Deeper Web Recon: Crawling, Params & Screenshots

Directory brute-forcing (last lesson) finds paths from a wordlist. But the juiciest endpoints are rarely in a wordlist , they are buried in JavaScript, remembered by web archives, or hidden behind parameters no form shows. This lesson is the modern web-recon toolkit that surfaces them.

Crawl the app , katana

A crawler follows every link, form and script to map real endpoints. katana understands modern JavaScript apps (SPAs):

katana -u http://10.10.10.20            # crawl and print endpoints
katana -u http://10.10.10.20 -jc        # also parse linked JavaScript

Alternatives: hakrawler, gospider, Burp Suite's crawler.

Mine the archives , gau & waybackurls

The Wayback Machine and other archives remember URLs a site used years ago , old endpoints and parameters that are often still live and unpatched.

gau example.com | sort -u                # get all archived URLs, de-duplicated
waybackurls example.com                  # same idea, Wayback-only

Look for ?id=, ?file=, ?url=, /api/, .bak, admin paths , classic bug entry points.

Read the JavaScript , LinkFinder

Modern front-ends hard-code API paths (and sometimes tokens) into their JS bundles. LinkFinder extracts them:

linkfinder -i http://10.10.10.20/js/app.min.js -o cli

Also grep JS for secrets: API keys, tokens, internal hostnames. What the site calls in the background is often not linked anywhere visible.

Find hidden parameters , arjun

An endpoint may accept parameters that appear in no form or doc, ?debug=true, ?admin=1, ?user_id=. arjun brute-forces them, and hidden params are a top source of IDOR, SQLi and access-control bugs.

arjun -u http://10.10.10.20/api/v1/user

Fingerprint the target , whatweb & wafw00f

Know what you are attacking, and what defends it:

whatweb http://10.10.10.20     # tech stack: server, language, CMS, frameworks
wafw00f http://10.10.10.20     # is there a WAF (Cloudflare, Akamai) in front?
wpscan --url http://10.10.10.20   # if it's WordPress: version, plugins, users

A WAF changes your approach (slower, evasive); a known CMS points to CMS-specific tools.

Triage at scale , gowitness

With hundreds of live hosts, you can't visit each. gowitness (or aquatone/eyewitness) screenshots them all so you can eyeball the gallery for logins, dashboards and default pages.

gowitness scan --file live-hosts.txt

Putting it together

Crawl + archives + JS give you a big pile of URLs. Filter to live ones with httpx, pull parameters with arjun, then feed the interesting endpoints into your fuzzers and nuclei.

Try it (terminal)

1. katana -u http://10.10.10.20 , crawl the app.
2. gau example.com , historical URLs with old params.
3. arjun -u http://10.10.10.20/api/v1/user , hidden parameters.
4. wafw00f http://10.10.10.20 and whatweb http://10.10.10.20 , fingerprint.
5. gowitness scan , screenshot the live hosts.

Why this matters

The bug is usually on the endpoint nobody browses to, the archived one, the JavaScript-referenced API, the undocumented parameter. Deeper web recon is how you find those before anyone else does.