When News Publishers Declare War on the Wayback Machine: The Unintended Consequences of AI Anxiety

The Internet Archive has long been hailed as the “good guys” of the web—a nonprofit dedicated to preserving our digital heritage for future generations. But in an ironic twist, that very mission of open access has turned the Archive into a unexpected threat in the eyes of major news publishers. As AI companies ravenously scrape the web for training data, the Wayback Machine has become an unexpected backdoor that publishers never anticipated.

The Core Insight

What we’re witnessing is a perfect storm of technological unintended consequences. The Internet Archive was built to democratize information—to create a library of human knowledge accessible to everyone. But that same principle of openness now makes it a goldmine for AI companies seeking free training data.

When The Guardian examined their access logs, they discovered the Internet Archive was one of the most frequent crawlers of their content. The realization hit hard: every snapshot preserved in the Wayback Machine represents potential fuel for AI models. “A lot of these AI businesses are looking for readily available, structured databases of content,” explained Robert Hahn, The Guardian’s head of business affairs and licensing. “The Internet Archive’s API would have been an obvious place to plug their own machines into and suck out the IP.”

The New York Times has taken even more aggressive action, “hard blocking” the Internet Archive’s crawlers entirely. Their position is crystal clear: “The Wayback Machine provides unfettered access to Times content—including by AI companies—without authorization.”

Perhaps most significantly, Gannett—the largest newspaper conglomerate in the United States—added Internet Archive bots to their robots.txt files across all 209 of their publications in 2025. In September alone, they blocked 75 million AI bots, with approximately 70 million coming from OpenAI.

Why This Matters

This situation reveals a profound tension in our digital age: the clash between open information access and content protection in the AI era. The Internet Archive finds itself in an impossible position. Founder Brewster Kahle warned: “If publishers limit libraries, like the Internet Archive, then the public will have less access to the historical record.”

Computer scientist Michael Nelson from Old Dominion University captured this dilemma perfectly: “Common Crawl and Internet Archive are widely considered to be the ‘good guys’ and are used by ‘the bad guys’ like OpenAI. In everyone’s aversion to not be controlled by LLMs, I think the good guys are collateral damage.”

The evidence is clear that the Wayback Machine has been used to train AI models. An analysis of Google’s C4 dataset showed the Internet Archive was the 187th most present domain out of 15 million—significant enough to matter. And in May 2023, an AI company caused the Archive to go offline temporarily by sending tens of thousands of requests per second to extract text data.

Key Takeaways

• The Internet Archive is caught in the crossfire between publishers protecting their content and AI companies seeking training data
• Major publishers are taking action: The Guardian, The New York Times, and Gannett (209 outlets) have all restricted Internet Archive access
• The robots.txt mechanism is largely symbolic—AI companies aren’t legally obligated to comply, but it signals intent
• This represents a broader shift where “good actors” like Internet Archive and Common Crawl are penalized because of how their data gets misused
• The public loses: As archives become more restricted, historical web content becomes less accessible—a loss for researchers, journalists, and citizens alike
• The irony is stark: Publishers who once benefited from the Wayback Machine’s preservation of their content now see it as a liability

Looking Ahead

The Guardian hasn’t documented specific cases of their content being scraped via the Wayback Machine—they’re being proactive rather than reactive. This suggests the situation will only escalate as AI capabilities grow.

What’s clear is that we need new frameworks for digital preservation in the AI age. Simply blocking crawlers won’t stop AI companies (they can ignore robots.txt), but it will prevent legitimate historical research and digital preservation efforts.

The Internet Archive is working to implement rate-limiting and bulk download restrictions. But as Kahle himself acknowledged, the organization faces an impossible task: remain open as a library while preventing abuse by those who would use preserved content to build systems that might one day replace the very journalists whose work they’re archiving.

This is more than a technical problem—it’s a fundamental question about who controls access to humanity’s collective digital memory.

Based on analysis of “News publishers limit Internet Archive access due to AI scraping concerns” from Nieman Lab