T. Rowe Price Expands Claude Across Investment Teams and Developers32m ago Val Bercovici, Chief AI Officer at WEKA – Interview Series39m ago Ben Jennings on the Bayeux tapestry – cartoon39m ago The latest Avatar: Seven Havens trailer gives us a deeper dive into the characters and story41m ago Agentic AI Is The New Attack Surface: How Can SAP Application Security Teams Repel It?42m ago Pony.ai Starts Fully Driverless Robotaxi Passenger Tests in Zagreb43m ago Euno raises $23m to give enterprise AI agents the context they lack45m ago Nvidia’s Huang Touts Cybersecurity as Next Big Market for AI48m ago Universal Music is launching an AI music platform with ElevenLabs55m ago Red Hat AI 3.5 tackles the GPU queue that can stall AI pilots55m ago Moonshot Secretly Routed User Requests Through Claude, Anthropic Says56m ago A Candid Abacus AI Review: The All-in-One AI Platform for Professionals & Enterprises56m ago Governments are turning to Claude to automate spying56m ago Anthropic Says It Blocked Possible Efforts to Build Biological Weapons57m ago

T. Rowe Price Expands Claude Across Investment Teams and Developers32m ago Val Bercovici, Chief AI Officer at WEKA – Interview Series39m ago Ben Jennings on the Bayeux tapestry – cartoon39m ago The latest Avatar: Seven Havens trailer gives us a deeper dive into the characters and story41m ago Agentic AI Is The New Attack Surface: How Can SAP Application Security Teams Repel It?42m ago Pony.ai Starts Fully Driverless Robotaxi Passenger Tests in Zagreb43m ago Euno raises $23m to give enterprise AI agents the context they lack45m ago Nvidia’s Huang Touts Cybersecurity as Next Big Market for AI48m ago Universal Music is launching an AI music platform with ElevenLabs55m ago Red Hat AI 3.5 tackles the GPU queue that can stall AI pilots55m ago Moonshot Secretly Routed User Requests Through Claude, Anthropic Says56m ago A Candid Abacus AI Review: The All-in-One AI Platform for Professionals & Enterprises56m ago Governments are turning to Claude to automate spying56m ago Anthropic Says It Blocked Possible Efforts to Build Biological Weapons57m ago

AI Research8 min reading time

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

Hacker NewsPublished Aug 2, 2026

A study found that only 8.9% of websites block AI crawlers via robots.txt, yet 94.8% of audited sites never appear in AI assistant answers. Among assistants, Gemini cited audited businesses most frequently at 2.9%.

#ai-assistants #ai-crawlers #gemini #gptbot #oai-searchbot #robots-txt

More in AI Research

AI Research4 min read

An Anthropic researcher just quit, saying OpenAI and Anthropic are 'gambling with our lives'

Covered by 12 sources1d ago

AI Research3 min read

Worried Anthropic researchers warn that AI ‘could kill all humans’

Covered by 8 sources1d ago

AI Research4 min read

Clay raises $115m at a $7.1bn valuation, more than double its 2025 price

The Next Web2h ago