Get the app
AI Research8 min reading time

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

Hacker News
Read full post
A study found that only 8.9% of websites block AI crawlers via robots.txt, yet 94.8% of audited sites never appear in AI assistant answers. Among assistants, Gemini cited audited businesses most frequently at 2.9%.
#ai-assistants#ai-crawlers#gemini#gptbot#oai-searchbot#robots-txt

More in AI Research

AI Research4 min read

An Anthropic researcher just quit, saying OpenAI and Anthropic are 'gambling with our lives'

Covered by 12 sources
AI Research3 min read

Worried Anthropic researchers warn that AI ‘could kill all humans’

Covered by 8 sources
AI Research4 min read

Suno trained its v6 AI music models with help from Warner and BMG

Covered by 5 sources