Capture the FlagEasy
What robots.txt Hides
Lab for Search Engines, Crawlers & robots.txt
Scenario
robots.txt is a polite request to crawlers, and a gift to you. It lists exactly the paths the owner would rather you didn't visit. Read it, then visit one of them.
Objective
Let the site's own robots.txt point you at the path it doesn't want indexed.
lab ~ what robots.txt hidesnothing real runs
booting lab sandbox…
This sandbox is fully emulated in your browser, nothing real runs and nothing leaves your machine. Type help to see the tools available.