How to Fix Your robots.txt So You Don't Accidentally Block Google
One stray line in robots.txt can hide your entire site from search. Here is how to read the file, fix the dangerous mistakes, and keep crawlers focused.
Reviewed by Jacob Rhinehart for Strelva's co-founder; products and operations lead responsibilities.
Your robots.txt file tells search engines which paths they may crawl. It is a tiny text file with outsized power: a single wrong line can hide your entire website from Google. Most robots.txt disasters are accidents left over from a staging site, so the fix is usually about removing damage, not adding rules.
Read it first
Visit yoursite.com/robots.txt. A 404 here is fine; it simply means "crawl everything." If a file exists, scan for the dangerous patterns:
- Disallow: / blocks your whole site. This is the single most damaging line and almost always a mistake.
- Disallow rules on CSS, JS, or asset folders stop Google rendering your pages, which wrecks how it reads your layout and mobile-friendliness.
- A missing Sitemap line means crawlers have to discover your sitemap the hard way.
Step by step
- Remove any site-wide block. Delete Disallow: / unless you genuinely want to be invisible. On some platforms a "discourage search engines" checkbox quietly inserts this; uncheck it.
- Keep rendering resources open. Never block CSS, JavaScript, images, or fonts. Google needs them to see the page the way a visitor does.
- Block only low-value paths. Disallow admin, login, internal search results, and thank-you pages. These create noise and infinite URL variations.
- Reference your sitemap. Add Sitemap: https://yoursite.com/sitemap.xml as an absolute URL.
- Test before trusting it. Use the URL Inspection tool in Search Console on your homepage and a few key pages to confirm they are allowed.
Crawling versus indexing
An important nuance: robots.txt controls crawling, not indexing. A blocked page with inbound links can still appear in search with no description. To truly keep a page out of results, use a noindex tag on the page rather than a robots.txt block. When in doubt, do not block; Google is good at ignoring low-value pages on its own.
Frequently asked
What business owners ask next.
Should every site have a robots.txt file?
Not strictly. A missing file means 'crawl everything,' which is fine for many small sites. A file is useful once you want to block admin areas, internal search, or reference your sitemap.
I blocked a page in robots.txt but it still shows in Google. Why?
robots.txt stops crawling, not indexing. If other sites link to the page, Google may keep it listed without a snippet. Use a noindex tag on the page itself to remove it from results.
Is it safe to block my CSS and JavaScript folders?
No. Google renders your pages using those files. Block them and Google sees a broken, text-only version of your site, which can hurt rankings and mobile-friendliness checks.