Search Engines, Crawlers & robots.txt
Every dork from the last lesson (site:acme.com filetype:pdf) is really a filtered query over somebody else's index. Before you write another one, it's worth seeing how that index actually gets built, and meeting the handful of plain-text files that quietly describe a site's whole structure to any bot (or tester) that asks: robots.txt, sitemap.xml, and the newer llms.txt.
How a search engine builds its index
A crawler (also called a spider or bot) is a program that starts at a URL, downloads the page, follows every link it finds, and repeats, forever. Googlebot and Bingbot are the two best known. Whatever a crawler downloads gets processed and stored in the engine's index, a searchable copy of the web.
This is the missing link to dorking: site:acme.com filetype:pdf doesn't scan Acme's servers at all, it searches Google's stored copy of whatever Googlebot has crawled and indexed there. If Googlebot never found a page, no dork will surface it either, no matter how good the query is.
robots.txt, the crawler's rulebook (and a tester's cheat sheet)
Almost every site publishes a plain-text file at /robots.txt telling crawlers which parts of the site they'd prefer not be crawled:
User-agent: *
Allow: /public/
Sitemap: https://acme.com/sitemap.xml
- User-agent , which bot this block applies to (
*means all). - Disallow , a path the site asks that bot not to crawl.
- Allow , an exception carved out of a broader Disallow.
- Sitemap , often points straight at the sitemap file.
Here's the catch beginners miss: robots.txt is a request, not a lock. It only works because well-behaved crawlers choose to honour it. The paths it lists are still live, still reachable by anyone who requests them directly, curl included. Worse for the site owner (better for you): the file is itself a hint list, whoever wrote it just told you exactly which paths they consider sensitive enough to mention.
curl http://10.10.10.20/robots.txt
sitemap.xml, the official map
Where robots.txt says what to skip, sitemap.xml says the opposite, it's the site's own declared list of pages it wants indexed:
<urlset>
<url><loc>https://acme.com/</loc></url>
<url><loc>https://acme.com/products</loc></url>
<url><loc>https://acme.com/staging-checkout</loc></url>
</urlset>
Large sites often split this into a sitemap index, one sitemap file listing several other sitemap files. Either way, it's a free, complete, self-declared map of the site, sometimes including staging or draft pages someone forgot to remove.
curl http://10.10.10.20/sitemap.xml
llms.txt & AI crawlers, the newest wrinkle
The same idea is now repeating for AI assistants. llms.txt is an emerging convention, a plain Markdown file at /llms.txt where a site curates links to its most important pages and docs, written specifically so an LLM can read it directly instead of crawling the whole site. Where it exists, it's often an even more concise, owner-curated map than the sitemap.
Search engines aren't the only bots reading a site anymore. A new wave of AI crawlers scrape content to train or power LLMs, and sites are starting to name them explicitly in robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Worth recognising: GPTBot (OpenAI), ClaudeBot / anthropic-ai (Anthropic), CCBot (Common Crawl, which feeds many LLM training sets), Google-Extended (opts a site out of Gemini training, separately from normal Googlebot indexing), PerplexityBot, and Bytespider (ByteDance). Which of these a site blocks, or pointedly allows, is one more small signal about what they consider valuable or sensitive.
Putting it together, before you dork or brute-force
| File | What it tells you | Command |
|---|---|---|
/robots.txt | Paths the owner didn't want crawled or AI-scraped, often the interesting ones | curl http://target/robots.txt |
/sitemap.xml | The official, "intended" map of every page | curl http://target/sitemap.xml |
/llms.txt | Curated docs/links the site wants AI assistants to read | curl http://target/llms.txt |
The efficient order: robots.txt, then sitemap.xml, then llms.txt if it exists, then dorking to fill any gaps, and only then reach for the active brute-forcing in the next lesson. All three files cost one curl each, touch nothing sensitive, and routinely hand you the site's own map for free.
Try it (terminal)
curl http://10.10.10.20/robots.txt, read what the owner asked crawlers to skip.curl http://10.10.10.20/sitemap.xml, pull the official page map.curl http://10.10.10.20/llms.txt, check for a curated AI-facing map.site:acme.com filetype:pdf, the dork from last lesson, now with context for why it works.
Why this matters
Beginners jump straight to gobuster. Three free curl requests, robots.txt, sitemap.xml, llms.txt, often hand you the site's own map before you brute-force a single path, and they explain exactly why a Google dork finds what it finds.