Web Crawling Explained: How Googlebot and AI Bots Decide If Your Site Gets Found

A search engine answers in a fraction of a second because it did most of the work long before you typed your query. This guide shows how crawlers discover your pages, which bots to welcome or block, and what it all means for SEO.

Bots called crawlers had already visited billions of pages and saved copies of them. This post walks through that process with diagrams. It covers what a crawler is, how it finds pages and decides what to fetch next, which bots you should welcome or block, how crawling affects SEO, and how the saved pages end up as the results you see.

1. What Is a Web Crawler?

A web crawler (also called a spider or bot) is a program that visits web pages automatically, downloads their content, collects the links on each page, and then follows those links to more pages. It does this continuously, spread across thousands of machines.

An analogy. Picture a postman who knocks on one house, copies what is written on the door, and writes down every other address mentioned there. He then visits each of those addresses in turn, and so on. The houses are pages and the addresses are links.

Why "spider"? Pages on the web are joined by hyperlinks, like threads in a spider's web. A crawler moves from page to page along those threads, exactly as a spider walks its web. In computer science terms the web is a directed graph, with pages as nodes and links as edges, and crawling is a graph traversal (Section 4 shows which kind).

A search engine works in four tiers, and crawling is the first. Without it there is nothing to index or rank. Figure 1 shows the whole pipeline.

1. Crawl: discover pages and download them.
2. Index: break pages into words and build a fast lookup.
3. Rank: score the matching pages and order them for the query.
4. Results: draw the page with title, URL and snippet, and learn from clicks.

2. Key Words Made Easy

Crawling comes with its own vocabulary. Here is a quick reference you can return to while reading.

Seed URL

The starting point or main door where the spider begins exploring.

URL Frontier

The spider's waiting line or to-do list of pages to visit next.

robots.txt

A "Keep Out!" sign on a door that tells spiders which rooms to avoid.

Sitemap

A helpful treasure map given by website owners listing all their good pages.

Crawl Budget

How much energy and time a spider is allowed to spend on one site.

Crawl Trap

An endless maze (like a calendar that goes on forever) that tricks spiders.

User-Agent

The badge or name tag worn by the robot, like Googlebot.

BFS

Visiting all neighboring rooms nearby before going down deep staircases.

Rendering

Loading interactive pages and code so the spider sees the final picture.

Bloom Filter

A super fast memory game to check: "Have I visited this page before?"

SimHash

A quick snapshot fingerprint to catch copycats and duplicate pages.

Canonical URL

The official "original copy" tag when multiple pages look identical.

DNS

The internet address book that turns website names into house numbers.

WAF

A security guard at the front gate blocking mean or naughty bots.

CDN

Helper toy boxes around the world holding copies so pages load fast everywhere.

PageRank

A popularity contest where links count as votes from friend pages.

BM25

A math game that scores how well special or rare words match your query.

3. What Is Search Indexing?

Crawling only collects pages. Indexing organizes them so they can be found in milliseconds. Searching the live web for every query would take months; searching an index takes a moment.

An analogy. The glossary at the back of a textbook says "photosynthesis: pages 12, 45, 88". You never read the whole book. A search index does this for every important word across billions of pages. It is called an inverted index (Section 11 shows one).

4. How Do Web Crawlers Work?

Every crawler runs the same basic loop, shown in Figure 2 as nine steps. The links found on each page go back into the queue, which is why the crawl keeps growing on its own.

1. Seed URLs start the crawl: trusted sites, sitemaps, and submitted URLs.
2. The URL frontier picks the next URL by priority (importance, freshness) and by politeness (one host at a time, with delays).
3. robots.txt check. The bot fetches and caches the file; blocked URLs are skipped.
4. DNS and fetch. The domain is resolved to an IP address and an HTTP GET is sent. The status code decides what happens next (see below).
5. Parse the initial HTML. Text, title, headings, canonical URL, structured data and every link in the raw response are extracted straight away.
6. Render JS (only if needed). If the page depends on JavaScript, it is queued for a headless Chromium renderer and the final DOM is parsed again for extra content and links. Pages that need no JavaScript skip this step; JavaScript pages are parsed twice. Rendering lags behind the first fetch, so server-side rendering is safer for key content.
7. Dedup. URLs are normalized (tracking parameters and fragments removed). A Bloom filter answers "seen this URL?" and fingerprints such as SimHash catch near-duplicate content.
8. Store. The clean page goes to the indexer.
9. Queue links. New URLs go back to the frontier with a priority score.

Which URL Comes Next? BFS, DFS and Best-First

Step 2 hides an important design choice. The crawl is a graph traversal, so the order in which URLs leave the frontier follows one of three strategies (Figure 3).

• BFS (breadth-first search) uses a queue (first in, first out). The crawler fetches every link on the seed page, then every link found on those pages, level by level. Pages close to the homepage are fetched first.
• DFS (depth-first search) uses a stack (last in, first out). It follows one chain of links as far as it goes, then backtracks. It can burn its whole budget inside one deep branch or a crawl trap.
• Best-first uses a priority queue. Each URL gets a score (link authority, freshness, change rate) and the highest score leaves first.

Research supports BFS as a sensible default. Najork and Wiener (2001) crawled 328 million pages breadth-first and found that high-PageRank pages arrive early, while average quality falls as the crawl continues. Cho, Garcia-Molina and Page (1998) found that ordering by partial PageRank did slightly better, with BFS close behind. Production crawlers therefore use BFS-like order with priorities and politeness rules on top, which is exactly what the two-level frontier in Step 2 does. The SEO side effect is practical: a page many clicks away from your homepage is reached late, so keep important pages shallow.

url = queue.popleft()    # BFS: take from the front, new links go to the back
url = stack.pop()        # DFS: take from the end, new links go on the same end
url = heappop(heap)[1]   # best-first: take the URL with the highest score

What the Status Code Tells the Crawler

Crawl Budget and Crawl Traps

Engines cannot crawl everything, so each site gets a limited crawl budget: a rate limit (how fast the bot may go without hurting your server) and demand (how much the engine wants your pages). Budget matters most for sites with hundreds of thousands of URLs. Crawling is never finished. A scheduler learns how often each page changes and revisits it accordingly (news homepages in minutes, static pages in weeks), often with a conditional request so an unchanged page costs almost nothing. At scale, each host is also assigned to one worker, which makes politeness easy to enforce.

One more limit worth knowing: when crawling for Search, Googlebot processes only the first 2 MB of an HTML or text file (and 64 MB of a PDF). Each CSS or JavaScript file is counted separately, and almost no real page reaches the limit.

A crawl trap is a structure that creates an almost endless supply of URLs. Figure 4 contrasts a healthy crawl with a trap. DFS is the strategy most easily caught by one.

5. List of Search Web Crawlers

These are the best-known search engine crawlers. The token is what you write after User-agent: in robots.txt.

6. AI Web Crawlers

AI companies run their own bots, and they visit for different reasons than search engines. The most useful way to understand them is by job, not by company. Figure 5 groups them into three.

Three practical points follow from the vendors' own documentation:

• Blocking training does not hide you from AI search. OpenAI states that GPTBot and OAI-SearchBot are independent settings. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
• User-triggered fetchers may not follow robots.txt. OpenAI notes robots.txt may not apply to ChatGPT-User because a person asked for the page, and Perplexity says the same of Perplexity-User. Anthropic says its bots honor robots.txt and support Crawl-delay. For policies that must hold, use a firewall (Section 10).
• AI crawlers generally do not run JavaScript. A 2024 study by Vercel and MERJ found that GPTBot and ClaudeBot downloaded script files but never executed them. Content that appears only after scripts run may be invisible to them. Behavior can change, so test your own pages.

7. Web Crawling vs Web Scraping

They use similar techniques but differ in goal, scope and behavior. A crawler is a city surveyor drawing a map of every street. A scraper is someone who walks into one shop and copies its price list for their own use.

Scraping is not always bad: price comparison, academic research and public-data journalism are legitimate. What matters is permission, volume, honesty and what is done with the data. Legal rules differ by country, so check the site's terms and local law. In practice one program can do both, so bot management judges behavior, not labels.

8. Should Crawler Bots Always Be Allowed?

No. The right policy is selective access based on value and risk.

Your Toolbox

Here is a starter robots.txt that follows the recipe blogger's policy:

# https://example.com/robots.txt
User-agent: *                # all bots
Disallow: /admin/
Disallow: /cart/
Disallow: /search            # internal search = crawl trap

User-agent: GPTBot           # block an AI training bot
Disallow: /

User-agent: OAI-SearchBot    # allow an AI search bot
Allow: /

Sitemap: https://example.com/sitemap.xml

Note that Google ignores the Crawl-delay line, so control Googlebot's speed through server health and Search Console instead.

9. How Do Crawlers Affect SEO?

Crawlers are the gatekeepers of SEO. Before any ranking factor matters, a page must be discovered, crawled, rendered and indexed. Figure 6 shows the four doors and what breaks each one.

10. Why Bot Management Must Take Crawling into Account

Crawlers are only one slice of automated traffic. Good bots (search crawlers, AI search bots, uptime monitors) sit next to bad bots (scrapers, credential stuffers, spam and DDoS bots). Managing crawling without managing bots means you either block helpful bots and lose traffic, or welcome harmful ones and lose money and security.

Figure 7 shows the decision flow for every visitor: verify who it is, check the value it brings, then check that it respects your limits.

How to Verify a Real Googlebot

1. Reverse DNS the visitor's IP. It should resolve to googlebot.com or google.com.
2. Forward DNS that hostname. It must return the same IP.
3. Or compare the IP with the operator's published ranges. Google and OpenAI publish JSON lists.

Why it matters: bot hits distort analytics; heavy crawling slows real users; scrapers copy pricing and content; and robots.txt only asks, while bot management enforces. Track the bot share of traffic, crawl requests per bot (from logs), error rates served to bots, and organic traffic after any rule change. One caution from Anthropic's documentation: blocking a crawler's IP addresses can stop it from even reading your robots.txt, so use robots.txt for opt-outs.

11. After the Crawl: Indexing, Ranking and the Results Page

The other three tiers turn the stored pages into the answer you see. Each deserves its own post, so this is a short overview.

Tier 2: Indexing and the Inverted Index

The indexer strips HTML, splits text into words (tokenize), lowercases and cleans them (normalize), and reduces forms such as "running" to "run" (stem). It then builds the inverted index: for each word, a posting list of the documents (and word positions) that contain it. Phrase search uses the positions. Posting lists are compressed (storing gaps between document IDs) and the whole index is sharded across thousands of machines and replicated.

Figure 8 shows a tiny inverted index built from three documents, and how a two-word query is answered by intersecting posting lists.

Tier 3: Query Processing and Ranking

1. Understand the query: fix typos, add synonyms, detect intent (informational, navigational, transactional, local).
2. Retrieve candidates: look up each word in every shard in parallel and merge the best matches.
3. Score relevance: BM25 gives more weight to rare, informative words, and repeating a word brings less and less extra credit, so keyword stuffing does not pay.
4. Add authority: PageRank treats a link as a vote, and votes from important pages weigh more.
5. Re-rank with machine learning: many signals (freshness, location, quality, page experience) and neural models judge meaning and intent.

BM25(page, query) = sum of IDF(word) x saturating_TF(word, length)
PageRank(p)       = (1 - d)/N + d x sum( PageRank(q) / outlinks(q) ),  d ~ 0.85

Tier 4: The Search Result Page (SERP)

Ranked document IDs become a page. For each result the engine fetches the stored title and text, builds a snippet that matches the query, adds rich features, and logs impressions and clicks that feed back into ranking. Figure 9 shows the funnel and a result's parts.

12. Conclusion

Search works as a pipeline: crawl to discover pages, index to organize them, rank to decide the order, and present the result. Crawling comes first because a page that is never fetched cannot be found. For site owners the takeaways are practical:

• Let verified crawlers reach the pages that matter.
• Keep crawlers out of traps such as endless filters, calendars and session URLs.
• Keep important pages close to the homepage.
• Set your AI bot policy by job: training, search or user-triggered.
• Back it with real bot management instead of relying on robots.txt alone.

Official References

• Google Search Central: Googlebot
• OpenAI: Overview of crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
• Anthropic: Does Anthropic crawl data from the web?
• Vercel and MERJ: The rise of the AI crawler (2024)
• RFC 9309: Robots Exclusion Protocol