How to Scrape Google Search Results Without Getting Blocked

To scrape Google search results reliably, stop parsing the results page and request the results as data. This guide shows why DIY Google scraping keeps breaking, how to get organic results, People also ask and related searches as JSON from a Google search API, and how to build a rank tracker in Python and Node.js that records positions per keyword and country over time.

Every example uses real responses from October 2026, and the limits section is blunt about what a SERP API does not give you, so you can decide quickly whether it fits.

Why scraping Google search results yourself keeps breaking

Most scrapers start like this:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.google.com/search", params={"q": "pour over coffee ratio"})
soup = BeautifulSoup(r.text, "html.parser")
print(r.status_code, len(r.text), len(soup.select("h3")))

# Our run in October 2026: 200 91978 0
# A successful status, about 90 KB of HTML, and not a single result title.

That output is the first lesson: the status code says success and the page has nothing in it. Since January 2025 Google has required JavaScript for Search, and a client that does not run it gets a page whose only job is to redirect to a "please enable JavaScript" check. Once you move to a headless browser, the next set of problems shows up:

• CAPTCHAs and rate limits. Repeated queries from one IP address, or from ranges with a poor reputation, get a CAPTCHA page or a 429. Datacenter proxies are flagged fastest; residential proxies last longer and cost more. The trade-offs are in our proxy types guide.
• Consent pages. Requests that look like they come from the EU can land on a cookie consent interstitial before any results.
• Layout churn. Google's markup uses generated class names and changes often. Selectors that worked last month return empty lists, without an error.
• Fewer results per request. In September 2025 Google stopped honoring the num=100 URL parameter, and told Search Engine Land it was not something it formally supports. Getting the top 100 now means ten requests instead of one.
• Silent degradation. The hardest failures are partial: a page with fewer blocks, a different locale, or a consent screen. You only notice if you validate the content, not the status code.

None of these is unsolvable, but together they turn a weekend script into an ongoing maintenance job. Seven web scraping challenges covers each one in more depth.

Your options: DIY, the official API, or a SERP API

If you are weighing SERP APIs against each other, our comparison of web scraping APIs covers the main options. The rest of this guide uses ScrapingBot's Google Search API.

How to get Google search results as JSON

Send a POST to /api/v1/google with "endpoint": "/search" and your query in params.q. Authenticate with the x-api-key header.

curl -X POST https://scrapingbot.io/api/v1/google \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "/search", "params": {"q": "pour over coffee ratio", "gl": "us", "hl": "en"}}'

Endpoint

POST /api/v1/google

Cost

10 credits per search

Results

Up to 10 per page

Timeout

15 seconds

The real response, trimmed:

{
  "success": true,
  "searchParameters": { "q": "pour over coffee ratio", "engine": "google", "gl": "us", "hl": "en", "type": "search" },
  "organic": [
    {
      "position": 1,
      "title": "What is your typical ratio? : r/pourover",
      "link": "https://www.reddit.com/r/pourover/comments/1ey0rx4/what_is_your_typical_ratio/",
      "snippet": "Just got into brewing at home and got the Kingrinder K6 and a Hario V60 for pour over…",
      "date": "2 years ago"
    },
    {
      "position": 2,
      "title": "Guide To Pour-Over Coffee",
      "link": "https://counterculturecoffee.com/blogs/counter-culture-coffee/guide-to-pour-over-coffee?srsltid=AU7gw4Wu…",
      "snippet": "We recommend somewhere between 1:14 and 1:20. That means that for every 1 gram of coffee…"
    },
    {
      "position": 5,
      "title": "Perfect Pour Over Coffee Recipe | Alton Brown",
      "link": "https://altonbrown.com/recipes/perfect-pour-over-coffee/",
      "snippet": "17 fluid ounces filtered water · 1/4 cup plus 1 tablespoon coffee, ground medium…",
      "rating": 4.4,
      "ratingCount": 65
    },
    …
  ],
  "peopleAlsoAsk": [
    { "question": "What is the best ratio for pour over coffee?" },
    { "question": "How many tablespoons of coffee for 16 oz of pour over?" },
    …
  ],
  "relatedSearches": [
    { "query": "Pour over coffee ratio calculator" },
    { "query": "Pour over coffee ratio grams" },
    …
  ],
  "duration": "1.00",
  "statusCode": 200,
  "creditsUsed": 10
}

What to notice in a real response:

• Fewer than 10 is normal. This page had 7 organic results; the second page had 9. Google fills the rest of the page with other blocks.
• Links carry tracking parameters. Several links end in ?srsltid=…, a value that changes between requests. Strip query strings before you compare URLs over time.
• Optional fields are optional. date appears in mixed formats ("2 years ago", "Aug 15, 2025"), and rating and ratingCount only appear on results such as recipes.
• Blocks can be missing. peopleAlsoAsk and relatedSearches are left out when Google does not show them.

The same request in Python and Node.js:

import requests

res = requests.post(
    "https://scrapingbot.io/api/v1/google",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"endpoint": "/search", "params": {"q": "pour over coffee ratio", "gl": "us", "hl": "en"}},
    timeout=30,
)
body = res.json()
if not body.get("success"):
    raise SystemExit(f"{res.status_code}: {body.get('error')}")

for r in body["organic"]:
    print(r["position"], r["title"], r["link"])
for q in body.get("peopleAlsoAsk", []):  # left out when Google shows none
    print("PAA:", q["question"])

const res = await fetch('https://scrapingbot.io/api/v1/google', {
  method: 'POST',
  headers: { 'x-api-key': 'YOUR_API_KEY', 'content-type': 'application/json' },
  body: JSON.stringify({ endpoint: '/search', params: { q: 'pour over coffee ratio', gl: 'us', hl: 'en' } }),
});
const body = await res.json();
if (!body.success) throw new Error(`${res.status}: ${body.error}`);

for (const r of body.organic) console.log(r.position, r.title, r.link);
for (const q of body.peopleAlsoAsk ?? []) console.log('PAA:', q.question); // left out when Google shows none

To translate the curl command into another language, paste it into the curl converter. Search operators such as site:, quotes and - work inside q as they do on Google.

Country, language and time filters

The same query returns different results in different markets, so always set the market explicitly:

// German results, as seen from Germany
{ "endpoint": "/search", "params": { "q": "kaffeevollautomat test", "gl": "de", "hl": "de" } }

// UK English results, second page
{ "endpoint": "/search", "params": { "q": "pour over coffee ratio", "gl": "gb", "hl": "en", "page": 2 } }

// Only pages from the past week, searched exactly as typed
{ "endpoint": "/search", "params": { "q": "coffee prices", "tbs": "qdr:w", "autocorrect": false } }

// Advanced operators work inside q
{ "endpoint": "/search", "params": { "q": "pour over ratio site:reddit.com -chemex" } }

Time filters are useful for news monitoring and for spotting fresh competitors. A past-week search for "coffee prices" returned results dated hours to days old:

{
  "searchParameters": { "q": "coffee prices", "engine": "google", "gl": "us", "hl": "en", "tbs": "qdr:w", "type": "search" },
  "organic": [
    { "position": 1, "title": "Dollar Strength Weighs on Coffee Prices", "link": "https://www.barchart.com/story/news/…", "date": "1 day ago", … },
    { "position": 2, "title": "Coffee C Futures Pricing", "link": "https://www.ice.com/products/15/coffee-c-futures/data", "date": "8 hours ago", … },
    { "position": 3, "title": "El Niño expected to drive up the cost of coffee and some …", "link": "https://www.facebook.com/…", … },
    …
  ],
  …
}

In our tests, gl visibly changed the results: the same query in the US and the UK shared some results but in a different order, and the UK page came back without People also ask or related searches.

Paging past the first 10 results

Each search returns one Google results page. We checked the edges:

• num tops out at 10. A request with "num": 20 returned the same 10 results as the default.
• Positions are counted within each page. Page 2 starts again at "position": 1.
• Pages can overlap. In one test the 10th result on page 1 came back as the 1st result on page 2.
• Pages can be short. 7 and 9 results per page are both normal.

So (page - 1) * 10 + position is only an approximation. For an accurate overall rank, walk the pages in order, reduce each link to host and path, skip anything already seen, and count. That is what the rank tracker below does. The start parameter (a result offset) is available as an alternative to page, but page is simpler.

Response field reference

A missing q returns 400 with Missing required parameter: q (search query) and costs nothing. Full details are in the Google Search API docs.

Using People also ask and related searches

The organic list tells you who ranks. The other two blocks tell you what people want to know, which makes them useful for content research:

• Find questions to answer. Run your core keywords, collect every peopleAlsoAsk question, de-duplicate them, and use them as H2s or FAQ entries.
• Expand a keyword list. relatedSearches gives Google's own suggestions, such as "pour over coffee ratio grams" and "pour over coffee ratio calculator" for our example query.
• Track intent over time. Store both blocks with each run; when the questions change, the intent behind the query is shifting.

Both blocks depend on the market and page. In our tests they appeared on page 1 for US English and were missing on the UK page and on page 2, so treat them as optional in your code.

Build a Google rank tracker in Python and Node.js

A rank tracker answers one question every day: where does my domain appear for each keyword, in each market? Here is the approach:

1. List keywords, markets and your domain. Decide which keywords to track, in which countries and languages (gl and hl), and which domain counts as yours.
2. Search page by page. Request page 1, and pages 2 and 3 only while your domain has not appeared, so you pay only for the pages you need.
3. Count unique results across pages. Reduce each link to host and path, skip repeats, and use the running count as the overall rank.
4. Store one row per keyword, market and day. Save the rank, URL and pages checked in SQLite. A run that fails midway records nothing rather than a false "not ranked".
5. Compare and schedule. Print the movement against the last stored rank, and run the script once a day from cron or a job runner.

The Python version uses requests and the standard library's sqlite3. Set SCRAPINGBOT_API_KEY in the environment or replace the placeholder, and change DOMAIN, KEYWORDS and MARKETS.

"""Google rank tracker: one row per keyword, market and day, in SQLite."""
import os
import sqlite3
import sys
import time
from datetime import date
from urllib.parse import urlsplit

import requests

API_URL = "https://scrapingbot.io/api/v1/google"
API_KEY = os.environ.get("SCRAPINGBOT_API_KEY", "YOUR_API_KEY")
DOMAIN = "example.com"  # the site you track
KEYWORDS = ["pour over coffee ratio", "chemex ratio"]
MARKETS = [("us", "en"), ("gb", "en")]  # (gl, hl) pairs
MAX_PAGES = 3  # look at up to ~30 results; each page is one 10-credit search


def search(q, gl, hl, page, attempts=4):
    params = {"q": q, "gl": gl, "hl": hl, "page": page}
    for attempt in range(attempts):
        try:
            r = requests.post(API_URL, headers={"x-api-key": API_KEY},
                              json={"endpoint": "/search", "params": params}, timeout=30)
            if r.status_code == 200 and r.json().get("success"):
                return r.json()
            if r.status_code in (400, 401, 402):
                sys.exit(f"{r.status_code}: {r.text[:200]}")  # fix the request or top up first
        except requests.RequestException:
            pass
        time.sleep(2 ** attempt)  # 429, 5xx and timeouts are refunded, so retry
    return None


def host(link):
    h = urlsplit(link).hostname or ""
    return h[4:] if h.startswith("www.") else h


def page_key(link):
    """Host + path, without tracking parameters such as srsltid."""
    parts = urlsplit(link)
    return host(link) + parts.path.rstrip("/")


def check_rank(q, gl, hl):
    """Return (rank, url, pages_checked). rank counts unique results across pages."""
    seen = []
    for page in range(1, MAX_PAGES + 1):
        res = search(q, gl, hl, page)
        if res is None:
            return None, None, page - 1  # could not finish; don't record a false "not ranked"
        organic = res.get("organic") or []
        for item in organic:
            key = page_key(item["link"])
            if key in seen:
                continue  # the same URL can close one page and open the next
            seen.append(key)
            h = host(item["link"])
            if h == DOMAIN or h.endswith("." + DOMAIN):
                return len(seen), item["link"], page
        if not organic:
            break  # no more results
    return 0, None, MAX_PAGES  # 0 = not in the results checked


def main():
    db = sqlite3.connect("ranks.db")
    db.execute("""CREATE TABLE IF NOT EXISTS ranks (
        day TEXT, keyword TEXT, gl TEXT, hl TEXT, rank INTEGER, url TEXT, pages INTEGER,
        PRIMARY KEY (day, keyword, gl, hl))""")
    today = date.today().isoformat()

    for keyword in KEYWORDS:
        for gl, hl in MARKETS:
            rank, url, pages = check_rank(keyword, gl, hl)
            if rank is None:
                print(f"{keyword} [{gl}]: incomplete, try again later")
                continue
            prev = db.execute(
                "SELECT rank FROM ranks WHERE keyword=? AND gl=? AND hl=? AND day<? "
                "ORDER BY day DESC LIMIT 1", (keyword, gl, hl, today)).fetchone()
            db.execute("INSERT OR REPLACE INTO ranks VALUES (?,?,?,?,?,?,?)",
                       (today, keyword, gl, hl, rank, url, pages))
            db.commit()
            now = f"#{rank}" if rank else f"not in top {pages * 10}"
            moved = ""
            if prev and prev[0] and rank:
                delta = prev[0] - rank
                moved = f" ({'up' if delta > 0 else 'down'} {abs(delta)})" if delta else " (no change)"
            print(f"{keyword} [{gl}-{hl}]: {now}{moved}")


if __name__ == "__main__":
    main()

The Node.js version has no dependencies. It needs Node.js 22.13 or later for node:sqlite, which still prints a harmless "experimental" warning on Node.js 24.

// Google rank tracker: one row per keyword, market and day, in SQLite.
// Node.js 22.13 or later (built-in fetch and node:sqlite).
import { DatabaseSync } from 'node:sqlite';

const API_URL = 'https://scrapingbot.io/api/v1/google';
const API_KEY = process.env.SCRAPINGBOT_API_KEY ?? 'YOUR_API_KEY';
const DOMAIN = 'example.com'; // the site you track
const KEYWORDS = ['pour over coffee ratio', 'chemex ratio'];
const MARKETS = [['us', 'en'], ['gb', 'en']]; // [gl, hl] pairs
const MAX_PAGES = 3; // look at up to ~30 results; each page is one 10-credit search

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function search(q, gl, hl, page, attempts = 4) {
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      const res = await fetch(API_URL, {
        method: 'POST',
        headers: { 'x-api-key': API_KEY, 'content-type': 'application/json' },
        body: JSON.stringify({ endpoint: '/search', params: { q, gl, hl, page } }),
        signal: AbortSignal.timeout(30_000),
      });
      const body = await res.json().catch(() => ({}));
      if (res.ok && body.success) return body;
      if ([400, 401, 402].includes(res.status)) throw new Error(`${res.status}: ${body.error}`);
    } catch (err) {
      if (/^40[012]:/.test(err.message)) throw err; // fix the request or top up first
    }
    await sleep(1000 * 2 ** attempt); // 429, 5xx and timeouts are refunded, so retry
  }
  return null;
}

const host = (link) => new URL(link).hostname.replace(/^www\./, '');
// Host + path, without tracking parameters such as srsltid.
const pageKey = (link) => host(link) + new URL(link).pathname.replace(/\/$/, '');

async function checkRank(q, gl, hl) {
  const seen = [];
  for (let page = 1; page <= MAX_PAGES; page++) {
    const res = await search(q, gl, hl, page);
    if (!res) return { rank: null, pages: page - 1 }; // incomplete: don't record "not ranked"
    const organic = res.organic ?? [];
    for (const item of organic) {
      const key = pageKey(item.link);
      if (seen.includes(key)) continue; // the same URL can close one page and open the next
      seen.push(key);
      const h = host(item.link);
      if (h === DOMAIN || h.endsWith('.' + DOMAIN)) return { rank: seen.length, url: item.link, pages: page };
    }
    if (!organic.length) break; // no more results
  }
  return { rank: 0, url: null, pages: MAX_PAGES }; // 0 = not in the results checked
}

const db = new DatabaseSync('ranks.db');
db.exec(`CREATE TABLE IF NOT EXISTS ranks (
  day TEXT, keyword TEXT, gl TEXT, hl TEXT, rank INTEGER, url TEXT, pages INTEGER,
  PRIMARY KEY (day, keyword, gl, hl))`);
const previous = db.prepare(
  'SELECT rank FROM ranks WHERE keyword = ? AND gl = ? AND hl = ? AND day < ? ORDER BY day DESC LIMIT 1');
const save = db.prepare('INSERT OR REPLACE INTO ranks VALUES (?, ?, ?, ?, ?, ?, ?)');
const today = new Date().toISOString().slice(0, 10);

for (const keyword of KEYWORDS) {
  for (const [gl, hl] of MARKETS) {
    const { rank, url = null, pages } = await checkRank(keyword, gl, hl);
    if (rank === null) {
      console.log(`${keyword} [${gl}]: incomplete, try again later`);
      continue;
    }
    const prev = previous.get(keyword, gl, hl, today);
    save.run(today, keyword, gl, hl, rank, url, pages);
    const now = rank ? `#${rank}` : `not in top ${pages * 10}`;
    let moved = '';
    if (prev?.rank && rank) {
      const delta = prev.rank - rank;
      moved = delta ? ` (${delta > 0 ? 'up' : 'down'} ${Math.abs(delta)})` : ' (no change)';
    }
    console.log(`${keyword} [${gl}-${hl}]: ${now}${moved}`);
  }
}

The output is one line per keyword and market. The second line below shows how a later run reports movement:

pour over coffee ratio [us-en]: #12
pour over coffee ratio [us-en]: #12 (up 3)

In our test data, the tracked domain sat 5th on page 2 behind a first page of 7 results, so its overall rank was 12, not the 15 that the "10 per page" formula gives.

Design notes

• Stop when found. If your domain ranks on page 1, a check costs one search. Only keywords where you rank lower pay for more pages.
• Matching uses the host. example.com also matches www.example.com and blog.example.com. Compare full paths instead if you track specific pages.
• "Not ranked" is only recorded after a complete check. If a search fails after retries, the run skips that keyword rather than storing a false drop.
• Retries are safe. 429, 5xx and timeouts are refunded, so retrying never pays twice.
• Raise MAX_PAGES deliberately. Tracking the top 100 means up to 10 searches, or 100 credits, per keyword and market.

Cost math for SERP tracking

Every search costs 10 credits. Here is how many searches a month each plan covers:

Worked examples, using 30-day months:

• 50 keywords, one market, daily, top 10: 1,500 searches, or 15,000 credits. Well inside Starter.
• 200 keywords, three markets, daily, up to the top 30: at most 600 × 3 × 30 = 54,000 searches (540,000 credits), which needs Startup. If every keyword is found on page 1, it drops to 18,000 searches, which fits Starter; real numbers land in between.
• 1,000 keywords, two markets, weekly, top 30: at most 2,000 × 3 × 4 = 24,000 searches, or 240,000 credits. Fits Starter.

Limits are on requests in flight, not per minute, so a tracker on Starter can run up to 10 searches at once. Going over returns an immediate, free 429; see concurrency and plans.

Limits to know before you build

• 10 results per page at most. Deeper results cost one search per page.
• Positions are per page. Compute overall rank yourself, as above.
• Web search returns organic results, People also ask and related searches. Images, videos, news, shopping, places, Maps and reviews are separate endpoints at the same price; ads are not included.
• Questions without answers. peopleAlsoAsk lists questions only.
• No reliable city targeting. location is not dependable; use gl and hl.
• Live results only. Every call is a fresh search. History is what you store.
• Web search only. Every request is a web search, whatever the query.

If those limits rule it out, you need a browser-based scraper and the maintenance that comes with it. If they do not, the API removes the hardest part of the job.

Next steps

1. Get a free key. 100 credits, no card, enough for 10 searches.
2. Run your real keywords in the playground in each market you care about, and check the results against a browser.
3. Run the rank tracker on five keywords for a few days before you scale it.

Building with an AI agent? The ScrapingBot MCP server includes a googleSearch tool, so Claude, Cursor and other MCP clients can search Google with your key. The same key also covers product data; the Amazon scraping guide builds a price monitor in the same style as the tracker above. For an overview of the Google API and more examples, see the Google Search API page.

Frequently asked questions

Is it legal to scrape Google search results?

Search results are public, and collecting them is widespread in SEO and market research, but Google's terms restrict automated access and its robots.txt disallows /search for crawlers. What applies to you depends on your jurisdiction and use, so ask a lawyer if it matters to your business. This is not legal advice.

Does Google have an official search API?

Google's Custom Search JSON API is closed to new customers, and existing customers have until January 1, 2027 to move to another solution, according to Google's documentation. For new projects that need Google web results as data, a SERP API is the practical option.

How do I get 100 results for a keyword?

Request pages 1 to 10. Google returns at most 10 organic results per page, and num above 10 still returns 10. Each page is a separate 10-credit search, and positions restart at 1 on every page, so count unique results across pages to get an overall rank.

Why does my Google scraper get CAPTCHAs or empty pages?

Google flags traffic that looks automated, from repeated requests on one IP address to clients that do not run JavaScript. You may get a CAPTCHA, a 429, or a 200 response with no results in it. A SERP API handles that on its side and returns results as JSON.

How much does a Google search via the API cost?

10 credits per successful search, and failed searches are refunded. On the Starter plan ($49.99 for 275,000 credits) that is 27,500 searches a month, about $1.82 per 1,000. New accounts get 100 free credits, enough for 10 searches.

Can I get results for a specific city?

Not reliably. The location parameter is passed through but city-level targeting is not dependable, so do not build local rank tracking on it. Use gl and hl to choose the country and language.

Does the API return ads, shopping, maps or image results?

Shopping, Maps, images, videos, news, places and reviews each have their own endpoint (/shopping, /maps, /images and so on), 10 credits a call. Web search returns organic results, People also ask and related searches. Ads are not included.

Does People also ask include the answers?

No. peopleAlsoAsk lists the questions only. The field is left out of the response when Google shows no such box for that query and page.