robots.txt

robots.txt is a plain text file at the root of your site that tells search-engine crawlers which parts they may or may not visit. It’s the front-door sign for bots — advisory, but respected by the major engines.

Why it matters

A single careless line can hide your whole site from Google — a frequent, painful cause of traffic vanishing after a redesign, when a “Disallow: /” gets left over from staging. Used well, it keeps bots out of low-value areas and points them to your sitemap.

Common misreading

Thinking robots.txt hides a page from search. Blocking crawling doesn’t guarantee a page stays out of the index — for that you use a noindex tag, and blocking the page can actually stop Google seeing that noindex.