robots.txt
Example: A neighborhood bakery discovers its new menu page is invisible on Google because a leftover "Disallow: /" line in robots.txt was telling every crawler to skip the whole site, and deleting that one line brings the pages back within days.
A tiny plain-text file at /robots.txt on your domain that tells search-engine crawlers which parts of your site they can and can't read. Usually a five-line file; sometimes the quiet reason a page isn't showing up in Google at all.
Why it matters
A broken or accidentally-restrictive robots.txt is one of the most common "why is my site invisible on Google" causes. Get yours wrong once and half your pages disappear from search overnight.
Frequently asked
What is robots.txt?
robots.txt is a tiny plain-text file at /robots.txt on your domain that tells search-engine crawlers which parts of your site they can and can't read. Usually a five-line file; sometimes the quiet reason a page isn't showing up in Google at all.
Why does robots.txt matter for a restaurant?
A broken or accidentally-restrictive robots.txt is one of the most common "why is my site invisible on Google" causes. Get yours wrong once and half your pages disappear from search overnight.
Browse all
149 terms.
Plain-English definitions for every term in your audit, organized by category.