Add a web scraping MCP server to Claude Code and Cursor

Coding agents are good at reasoning about data and bad at getting it: they cannot render a JavaScript page, get past a site that blocks data centers, or read TikTok. The Model Context Protocol (MCP) fixes that by letting the agent call outside tools. This guide connects ScrapingBot's MCP server to Claude Code, Cursor and Codex, and explains what each tool does and costs.

The server is hosted, so there is nothing to install: no package, no local browser and no proxies. Your agent sends tool calls to https://scrapingbot.io/api/mcp with your API key, and each call runs as a normal API request.

Claude Code

Run this once in your terminal, with your key in place of YOUR_API_KEY:

claude mcp add --transport http scrapingbot \
  https://scrapingbot.io/api/mcp \
  --header "x-api-key: YOUR_API_KEY"

Start a session and type /mcp to see the server and its tools; claude mcp list shows it from the shell. By default the server is added for the current project only. Add --scope user to make it available in every project.

Cursor

Add the server to .cursor/mcp.json in your project, or to ~/.cursor/mcp.json to use it everywhere, then check that it is switched on in Cursor's MCP settings:

{
  "mcpServers": {
    "scrapingbot": {
      "url": "https://scrapingbot.io/api/mcp",
      "headers": {
        "x-api-key": "YOUR_API_KEY"
      }
    }
  }
}

Other clients that support remote HTTP servers take the same URL and header, though their config files are shaped differently. The MCP server page has the exact setup for VS Code and the Claude app.

Codex

Codex reads the key from an environment variable and sends it as a bearer token, which the server also accepts:

export SCRAPINGBOT_API_KEY="YOUR_API_KEY"
codex mcp add scrapingbot --url https://scrapingbot.io/api/mcp \
  --bearer-token-env-var SCRAPINGBOT_API_KEY

Keep the key out of your repository

A key pasted into a project's config file ends up in git sooner or later. Both Claude Code and Cursor can read it from an environment variable instead. Set SCRAPINGBOT_API_KEY in your shell profile, then refer to it.

In Claude Code, --scope project writes the server to a .mcp.json file in the project root, meant to be shared through version control. That file expands ${VAR} in headers, so you can commit this:

{
  "mcpServers": {
    "scrapingbot": {
      "type": "http",
      "url": "https://scrapingbot.io/api/mcp",
      "headers": { "x-api-key": "${SCRAPINGBOT_API_KEY}" }
    }
  }
}

Cursor uses a slightly different syntax, ${env:NAME}:

"headers": { "x-api-key": "${env:SCRAPINGBOT_API_KEY}" }

One catch: writing --header "x-api-key: $SCRAPINGBOT_API_KEY" in the claude mcp add command does not keep the key out of config. Your shell replaces the variable before Claude Code sees it, so the key itself is saved.

Check the connection with curl

If a client will not connect, ask the server for its tool list directly. Listing tools does not call any API, so it costs nothing:

curl -X POST "https://scrapingbot.io/api/mcp" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

A working key returns a JSON-RPC result listing the 23 tools with their input schemas. The common failures each have a distinct answer:

The 23 tools

Each tool maps to an API in the docs and accepts the same parameters. Your agent reads the descriptions and picks one; listCapabilities exists for agents that want to discover everything first.

Any website

• scrapeWebsite: A page as markdown, or HTML with format: "html". Supports render_js, proxies, waits, cookies and screenshots, which come back as an image. (1 credit; 5 rendered, 10 premium, 75 stealth)
• extractStructuredData: Scrapes a page and returns the fields you describe, as JSON, using ai_query or ai_schema. (the page cost plus 5 credits)
• runBrowserScenario: Clicks, types, scrolls and waits in a real browser, then returns the page. (from 5 credits)
• getScrapeJob: Looks up an earlier scrape by its job_id. (free)
• pollJobUntilDone: Waits for a scrape job to complete, fail or time out. (free)

Google

• googleSearch: Web results by default, or images, videos, news, shopping, places or maps, for any country and language. (10 credits)
• googleReviews: Google Maps reviews for a place found with googleSearch. (10 credits)

Instagram

• instagramUser: A profile by username, user ID or URL. (5 credits)
• instagramSearch: Accounts, hashtags, places or posts. (5 credits)
• instagramMedia: Posts, reels, tagged posts, stories, one post, comments or comment replies. (5 credits)
• instagramFollowers: An account's followers or following, searchable by name. (5 credits)

TikTok

• tiktokVideo: A video's stats, caption, author, sound and download links, by URL. (1 credit)
• tiktokUser: A profile, or with include_posts its latest videos. (1 credit)
• tiktokSearch: Creators or videos by keyword. (1 credit)
• tiktokComments: Comments on a video, or the replies to one comment. (1 credit)
• tiktokFollowers: An account's followers or following. (1 credit)

Amazon

• amazonSearch: Products for a keyword, with price, rating and Prime. (10 credits)
• amazonProduct: Everything on a product page, by ASIN. (10 credits)
• amazonProducts: Up to 20 products in one call. (10 credits per product)
• amazonSuggestions: Amazon's search-box suggestions, for keyword research. (10 credits)
• amazonRankings: Best sellers or new releases for a department. (10 credits)

Everything else

• listCapabilities: Lists the tools, endpoints and guardrails. (free)
• providerRequest: Calls any TikTok, Instagram, Google or Amazon endpoint in the docs directly, for the few the other tools do not cover. (same credits as that API)

What to ask

You do not name tools; you describe the task, and the agent chooses. These are requests that fit the tools well, with the tool each one would most likely lead to:

• "Read https://quotes.toscrape.com/js/ and list the authors on the first page. The quotes are rendered with JavaScript."
  Likely tool: scrapeWebsite with render_js
• "Pull every product name, price and rating from this category page as JSON."
  Likely tool: extractStructuredData
• "Search Google for the latest release notes of our main dependency and summarize the breaking changes."
  Likely tool: googleSearch, then scrapeWebsite
• "What are people saying in the comments on this TikTok video? Group them by theme."
  Likely tool: tiktokComments
• "Which videos are using this TikTok sound?"
  Likely tool: providerRequest with the TikTok /music/posts endpoint
• "Compare these three Amazon products by price, rating and key features."
  Likely tool: amazonProducts
• "Write a fixture file for my parser from the real HTML of this page."
  Likely tool: scrapeWebsite with format: "html"

In a coding agent, the last kind of request is often the most useful: the agent fetches real pages or API responses, then writes the parser, test fixtures or types against what it actually saw instead of guessing.

Credits, limits and safety

• Billing is the API's billing. Each tool call is a normal API request, with the same credits, concurrency slots and usage log. Failed calls are refunded, and the agent cannot spend more than your balance.
• Results fit a context window. Pages come back as markdown, which is usually much smaller than the HTML. Every result stays under about 60,000 characters; long lists are trimmed with a note saying what was left out.
• Public addresses only. The scraping tools refuse localhost, private and internal network addresses, so a prompt cannot point them at your own network.
• Watch what the agent repeats. An agent that loops over a list can make many calls in one task. Cheap tools such as TikTok at 1 credit rarely matter; stealth scraping at 75 credits a page does, so say in the prompt when you want the cheapest option.

Where to go from here

• The MCP server page has setup for the Claude app, VS Code and other clients.
• The MCP reference lists each tool's inputs, and every tool links to the API it calls.
• After you sign up, the dashboard's MCP page has the setup with your key and a free connection check.

Common questions