Is the Internet Closing Its Gates to AI?

Question

For years, AI model creators treated the internet like an open buffet - a limitless source of data to train their models.

But now, that buffet is closing.

Publishers are blocking AI crawlers and demanding payment for access.

Why?

• Wikipedia found that most of its traffic spikes were from AI crawlers, not readers — driving up server costs while draining its donations and human traffic.

• Read the Docs got hit with a $5,000 bandwidth bill after an AI company downloaded 73 TB of data.

Normally, AI companies send crawlers to collect huge amounts of data from websites to train their models. But many of them do it without permission, which increases traffic and server costs for the websites.

So now, from The New York Times to Reuters, media houses are saying “No entry” to OpenAI and other bots.

Publisher’s solution - pay for what you use

The web has always been humanity’s collective brain, but if AI wants to keep learning from it, it needs to play fair.

What do you think - will it become difficult for smaller players to train their AI models now?