All posts
Robots Txt #robots-txt
Browse all Seodisias guides tagged robots txt. Practical, example-driven articles on SEO crawling, indexing, and AI search visibility.
2 posts
tutorials 12 min read
Why Pages Get Deindexed (and How to Diagnose It with a Crawl)
Hundreds of pages dropped from Google overnight and you have no idea why. Here is what 'deindexed' actually means, the eight causes that explain almost every case, and a 15 minute crawl-based triage to find yours.
#indexing#technical-seo#crawling
guides 11 min read
The Complete Guide to robots.txt: Rules, Examples, and AI Crawlers
robots.txt controls what search engines and AI crawlers can access. Learn the syntax, common patterns, and how to handle GPTBot, ClaudeBot, and PerplexityBot.
#robots-txt#ai-seo#geo