What the Probe Saw
Our latest probe on openai.com reveals a concerning score of 20/100, placing it in the 'invisible' band. The primary issue detected was the inability to reach the homepage cleanly, resulting in an HTTP 403 error. This suggests either a server error or intentional blocking of automated requests, which affects AI crawlers similarly. While GPTBot, ClaudeBot, and PerplexityBot can access key pages, the absence of a /llms.txt file means these engines lack a curated map to the site's best content, forcing them to guess. Additionally, the probe couldn't detect any JSON-LD schema or assess the site's answer-first structure due to homepage inaccessibility.
Implications for Business Sites
For any business site, being largely invisible to AI crawlers can significantly hinder discoverability and user engagement. The absence of a /llms.txt file, as seen with openai.com, leaves AI engines guessing about the site's important content. This can lead to missed opportunities for traffic and visibility. Implementing a well-structured /llms.txt file, as detailed in our llms.txt guide , can help guide AI crawlers more effectively.
Moreover, ensuring that your homepage is accessible and not inadvertently blocking AI crawlers is crucial. Our AI crawler playbook provides insights into optimizing your site for better crawler access. Regularly monitoring your site's performance using tools like our free checker can help identify and resolve such issues promptly.
In summary, while openai.com faces challenges in visibility, other sites can learn from this by ensuring their domains are accessible and well-mapped for AI crawlers. For a deeper analysis, consider using our audit services.
Curious about your own domain's visibility? Run our checker to see how it scores.
No comments yet