Web Crawler

A web crawler looks deceptively trivial. Download a page, pull out its links, download those, repeat. You could write the core loop in ten lines. The trap is that the naive loop fails on every axis that matters at scale: it has no throughput ceiling you can reason about, no manners (it will hammer a single host until you are firewalled or sued), and no memory (it re-downloads the same pages forever, burning bandwidth on content it already has). The interesting engineering is entirely in the parts the ten-line version skips. This guide builds the design the way you would in an interview, and it lines up with the SysGym web-crawler simulator so the two reinforce each other. The gym hands you seed URLs at 6,000 pages per second, then ramps offered load to 1.5x and finally 2x as extracted links feed back in (a deep crawl peaking at 12,000 pages/sec). While that happens it crashes a fetcher, stalls the page store, and slows the DNS resolver. You have to keep at least 97 percent of fetches succeeding, keep the busiest component under 90 percent utilization, and do it with a lean fleet. Everything below is in service of arriving at a design that clears those targets on purpose rather than by luck.