AI is eating website traffic, websites are blocking AI – and reliable information is getting harder to find

Damien Geso / Getty Images For 30 years, the world wide web has run on a surprisingly profound social contract : most sites are free for search engines to access, but if you use their content you give credit by linking to the source.

Recently, that social contract has begun to collapse.

Artificial intelligence (AI) tools are crawling sites not to link to them , but to train models and generate answers (which may or may not be accurate).

When you search for something, ChatGPT’s response or Google’s AI Overviews may still include links to sources , but they’re a kind of optional extra to the main answer.

This has triggered a bad dynamic for website owners, the public, and even AI companies themselves: as websites lose traffic (and revenue), many are beginning to block AI scraping tools , meaning AI results depend more on low-quality websites (many of which are also generated by AI).

As a result, good information can be harder than ever to find.

How we got here In the early days of the world wide web, search engines and content creators came to an agreement about crawling (the practice of technologically examining a site to index it, so it can be served up in search results).

Content creators would provide access to their sites for free, and even allow search engines to reproduce small snippets of text.

In return, search engines provided links to the sites owned by content creators, who benefited from that web traffic.

If content creators didn’t like the deal, they could prevent search engines from crawling their site with instructions in a file called robots.txt .

But if AI tools no longer provide web traffic, it cuts content creators out of the economic loop.

There are also other costs associated with each visit to a website, so AI crawling can cost website providers money while not giving them any of the ad or other revenue that would come from human traffic.

AI crawlers also crawl more deeply and more intensely than traditional web crawlers, magnifying that cost.

This change in traffic patterns isn’t a small or hypothetical problem.