Why Do AI Crawlers Keep Hitting robots.txt Instead of My Content?
Over the past weeks I’ve been monitoring traffic from AI crawlers like OpenAI’s GPTBot, oai-searchbot and ClaudeBot. The data (see screenshots below) raises some interesting questions: Questions I Wanted to Answer Why does GPTBot visit robots.txt so many times, sometimes multiple times per day? Why does GPTBot prefer robots.txt over sitemap.xml? Why do I see…