Robots.txt Mistakes That Block Your Best Pages from Google

Traffic flatlined after launch. Search Console showed thousands of "Crawled - currently not indexed" pages. The team checked meta robots tags, then redirects, then page speed. Nobody opened /robots.txt until week three.

One line had Disallow: / left over from staging.

robots.txt is a small file with outsized impact. It does not guarantee indexing, but it can block crawling of paths you care about. Business sites lose visibility when staging rules go live, plugins append aggressive disallow blocks, or CMS defaults conflict with marketing URLs.

I work on technical SEO and crawl diagnostics with teams recovering from launch-week surprises. This guide covers the mistakes we see most often, how to verify fixes, and how robots rules interact with (but differ from) noindex.

What robots.txt Actually Does (And Does Not Do)

Does:

• Tells compliant crawlers which paths they may request

• Reduces crawl budget waste on admin, faceted search, or duplicate utility URLs

• Can block entire site (Disallow: /) or specific directories

Does not:

• Remove pages from Google's index by itself (blocked URLs may still appear as URL-only results if linked externally)

• Replace noindex when you want a page not indexed but crawlable for link discovery

• Apply to every bot equally (most major search engines respect it; some scrapers ignore it)

Think of robots.txt as a gate on crawling, not a fine-grained index control.

robots.txt vs noindex vs redirects

Tool	Use when
robots.txt Disallow	Block crawl of admin, internal search, print views, or infinite filter URLs
meta robots noindex	Page should not appear in search results but can be crawled (or use noindex + disallow together carefully)
301 redirect	URL permanently moved; pass users and signals to new location
404 / 410	URL retired; remove from index over time

Common confusion: blocking /blog/ in robots.txt while hoping those posts rank. Google cannot reliably index what it cannot crawl.

Staging pattern: many teams disallow all crawling on staging (Disallow: /). Production must replace that file entirely, not patch one line in a hurry.

Mistake 1: Staging Rules Left on Production

The classic launch incident:

User-agent: *
Disallow: /

Sometimes with an Allow: / added later that does not override intent clearly, or a CMS "discourage search engines" toggle that writes the same effect.

Verify on production:

curl -s https://yoursite.com/robots.txt

Open in a private browser. Confirm no sitewide disallow.

Search Console: URL Inspection on homepage and a money page. "Indexing allowed?" should be yes. If robots.txt blocks crawling, Inspection states it explicitly.

Mistake 2: Blocking CSS, JS, or Image Paths

Older SEO advice suggested blocking /wp-content/ or /assets/ to save crawl budget. Modern rendering needs CSS and JS for Google to understand pages.

Blocking asset paths can hurt rendering and mobile-friendly evaluation.

Safer approach: disallow admin and utility paths only. Allow assets needed for rendering unless you have a specific reason documented.

Mistake 3: Over-Broad Disallow on CMS or Plugin Paths

Plugins add rules like:

Disallow: /?*
Disallow: /search
Disallow: /tag/

Some are fine. Others accidentally block faceted URLs you want indexed, or entire query-string namespaces including harmless parameters.

Review each disallow against your URL strategy. If /search is site search results (thin duplicates), blocking is reasonable. If marketing uses tagged landing paths, verify they are not caught by a wildcard.

Mistake 4: Conflicting robots.txt and Sitemap

Your sitemap lists URLs under /services/ but robots.txt disallows /services/. Crawlers receive mixed instructions.

Fix alignment:

1. List sitemap URL at bottom of robots.txt (Sitemap: https://yoursite.com/sitemap.xml)

2. Ensure every major sitemap section is crawlable

3. Remove retired URLs from sitemap when you disallow or noindex them

After a website redesign and replatform, regenerate sitemap and robots together, not in separate tickets weeks apart.

Mistake 5: Multiple robots Files or CDN Overrides

Some setups serve different robots.txt per host (www vs apex), or a CDN page rule returns an old cached staging file.

Check:

• https://example.com/robots.txt

• https://www.example.com/robots.txt

• Response headers (cache TTL; staging files cached for days hurt)

Pick one canonical host and ensure robots.txt matches on both or redirects consistently.

Mistake 6: Using robots.txt Instead of noindex for Private Content

Teams disallow /client-portal/ hoping it stays private. URLs may still leak in external links and appear as unlinked search results.

Better for truly private content:

• Authentication required

• noindex on responses (and disallow if you also want to block crawl)

• Do not rely on "security through robots.txt"

Mistake 7: Wildcards Without Testing

Robots.txt supports limited wildcards (* and $ in Google-supported syntax). A pattern meant to block /*?sessionid= can block more than intended if written loosely.

Process:

• Document each rule with owner and reason

• Test with Google Search Console robots.txt tester (where available) or reputable validators

• Spot-check sample URLs from each section of the site

How to Verify After a Fix

1. curl production robots.txt from a neutral network

2. Search Console URL Inspection on 3 to 5 priority URLs

3. Crawl with a desktop SEO spider in "respect robots.txt" mode; note blocked money pages

4. Compare to staging file; diff should be intentional, not accidental copy-paste

5. Monitor Pages report for two to four weeks after fix; "Blocked by robots.txt" should drop on important paths

Do not request indexing on every URL daily. Fix the file once, confirm, then request indexing on top commercial pages if needed.

Safe Rollback Steps

If you discover sitewide block on production:

1. Replace robots.txt with corrected production version immediately

2. Purge CDN cache for /robots.txt if applicable

3. Re-verify with curl and URL Inspection

4. Submit sitemap in Search Console

5. Request indexing on homepage and top service URLs once

6. Post-mortem: add robots.txt diff to launch checklist for future releases

Speed matters. Every day with Disallow: / on production delays recovery after the fix.

Launch Checklist (robots.txt Section)

• Production robots.txt reviewed line by line, not copied from staging

• No sitewide disallow on production

• Money paths (/services/, /contact, key content) not disallowed

• Sitemap URL declared and sitemap paths allowed

• CSS/JS/assets not blocked without documented reason

• www and apex serve consistent rules

• CMS "discourage search engines" disabled on production

• URL Inspection passed on priority pages before announce

• Owner assigned for robots.txt changes (no silent plugin edits)

Key Takeaways

• robots.txt controls crawling, not full index removal.

• Staging disallow all on production is the most common high-impact mistake.

• Align robots.txt with sitemap and post-redesign URL plans.

• Do not block rendering assets without a clear reason.

• Verify with curl, URL Inspection, and crawl tools after every launch.

When did you last read your production robots.txt line by line: before launch, or after traffic looked wrong?

If robots rules, sitemap, and CMS launch settings sit with your build team, include web development and launch configuration in the same pre-go-live QA pass as redirects and analytics.