Guide

MCP server

Use ScrapingBot from AI agents. ScrapingBot runs a Model Context Protocol server, so Claude, Cursor and other MCP clients can scrape pages, search Google and pull TikTok, Instagram and Amazon data as tools.

• Transport: Streamable HTTP. The server is stateless and answers each request with JSON; there is no SSE stream, so GET returns 405.
• Auth: your API key in an x-api-key header or as Authorization: Bearer.
• Billing: each tool call is a normal API request, with the same credits, concurrency slots and usage log.
• Safety: scraping tools refuse localhost, private and internal network addresses.
• Results: pages come back as markdown (or HTML with format: "html") and screenshots as an image. Every result stays under about 60,000 characters; long lists are trimmed with a note saying what was left out.

When calling the endpoint yourself, send Accept: application/json, text/event-stream; requests that don't accept both are rejected with 406.

Connect a client

Pick your client in the setup example and use a key from your dashboard in place of YOUR_API_KEY. New here? Create a free account for 100 credits. Setup for the Claude app, VS Code and more is on the MCP server page.

claude mcp add --transport http scrapingbot \
  https://scrapingbot.io/api/mcp \
  --header "x-api-key: YOUR_API_KEY"

export SCRAPINGBOT_API_KEY="YOUR_API_KEY"
codex mcp add scrapingbot --url https://scrapingbot.io/api/mcp \
  --bearer-token-env-var SCRAPINGBOT_API_KEY

{
  "mcpServers": {
    "scrapingbot": {
      "url": "https://scrapingbot.io/api/mcp",
      "headers": {
        "x-api-key": "YOUR_API_KEY"
      }
    }
  }
}

curl -X POST "https://scrapingbot.io/api/mcp" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'