Key takeaways
- robots.txt tells crawlers which URLs they may request; it lives at the root of your domain.
- It manages crawling, not indexing. A blocked page can still appear in Google if others link to it.
- To keep a page out of Google, use noindex or a password, not robots.txt.
- Most small business sites need a very short robots.txt, plus the sitemap line.
A broken robots.txt is one of the scariest things I find in a technical SEO audit, because it is invisible to visitors. The site looks fine; Google just cannot read it.
What is robots.txt?
robots.txt is a plain text file at the root of your domain, such as example.com/robots.txt, that tells search engine crawlers which URLs they may access. Google describes it as a way to manage crawler traffic and avoid overloading your site. Well-behaved crawlers read it before visiting other pages.
That description comes from Google's introduction to robots.txt.
Does robots.txt keep a page out of Google?
No. Google says plainly that robots.txt is not a mechanism for keeping a page out of Google. If a blocked URL is linked from other sites, Google can still index the address and show it, usually without a description. To keep a page out of results, use a noindex tag, a password or remove the page.
| Goal | Use |
|---|---|
| Stop crawlers from wasting time on filter or search URLs | robots.txt Disallow |
| Keep a page out of Google results | noindex meta tag or header |
| Keep private content private | Password or login |
| Remove a page that should not exist | Delete it and return 404 or 410 |
Google explains noindex separately. Important: if you block a page in robots.txt, Google cannot see its noindex tag. For step-by-step removal, read how to remove a page from Google.
What does a safe robots.txt look like?
For most business websites, a safe robots.txt allows everything except a few technical areas, and lists the sitemap. Blocking internal search results or endless filter combinations can make sense. Blocking CSS, JavaScript or whole folders of real pages usually does not, because Google needs them to render and understand the site.
User-agent: *
Disallow: /search
Sitemap: https://example.com/sitemap.xmlThe sitemap line helps crawlers find your XML sitemap. Platforms often generate this file for you: Shopify, for example, documents how to edit robots.txt.liquid.
What is the most dangerous robots.txt mistake?
A Disallow: / line left over from development. It tells every crawler to stay out of the entire site. It often happens when a staging site goes live with its settings copied. Check yourdomain.com/robots.txt after every launch or migration, and confirm with the URL Inspection tool in Search Console.
- Open yourdomain.com/robots.txt and read it line by line.
- Look for "Disallow: /" under User-agent: *.
- Make sure CSS and JavaScript folders are not blocked.
- Confirm the Sitemap line points to the right, live URL.
- Test a key page with Search Console's URL Inspection tool.
Can robots.txt control AI crawlers?
Partly. Many AI companies publish the names of their crawlers so site owners can allow or block them in robots.txt. Google, for example, documents a separate token for controlling use of content in some of its AI products. Blocking AI crawlers can also reduce your chances of being cited in AI answers, so decide on purpose.
Google lists its common crawlers, including that token. I explain the trade-off in how to show up in ChatGPT and AI Overviews, and robots.txt is one line of my technical SEO checklist. As an SEO consultant, I check it in every free diagnosis: book yours.
Frequently asked questions
Where do I put robots.txt?
At the root of the domain, so it opens at yourdomain.com/robots.txt.
Is robots.txt required?
No. Without one, crawlers assume they may visit everything.
Do all bots obey robots.txt?
Reputable crawlers do. Malicious bots can ignore it, so it is not a security tool.
Can I block one page with robots.txt?
You can block crawling, but use noindex if the goal is to keep it out of results.
Does every subdomain need its own robots.txt?
Yes. Each host reads its own file.
About the author
Juan José Giraldo is an SEO consultant in Medellín, Colombia, with 5+ years of experience. He works on technical and local SEO for 600+ US car dealerships at Reunion Marketing, an Inc. 5000 agency, and supervises the SEO of 18 US online stores with more than 3 million products at Prontomowers. Read more · LinkedIn.
Want to see these guides first? Choose JJ SEO Colombia as a preferred source on Google, and my articles are more likely to show up for you in Top Stories.
Follow on GoogleAdd as a preferred source