How Cloudflare is Tackling The Growing AI Bots Problem
Cloudflare has deployed a system enabling millions of websites to block AI bots from accessing its content without permission.
The technology targets AI firm bots, also known as crawlers, which are programs that explore the web to index and collect data for training AI systems.
Websites including Sky News, The Associated Press and Buzzfeed can now control access to their content through the system.
Cloudflare reports that its technology is already active on one million websites.
How the system is “a game-changer” for creative industries
The system applies by default to new users of Cloudflare services and sites that participated in an earlier effort to block crawlers.
The deployment addresses growing concerns from content creators about AI companies using their work without permission or payment. Writers, artists, musicians and actors have accused AI firms of training systems on their content without compensation.
Roger Lynch, CEO of Condé Nast, the publishing company whose titles include GQ, Vogue and The New Yorker, described the move as “a game-changer.
“This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.”
Publishers typically allow crawlers from search engines such as Google to access their sites because search companies direct visitors to their content in return.
However, Cloudflare argues that AI crawlers collect content including text, articles and images to generate answers without sending visitors to the original source, depriving content creators of revenue.
The BBC recently threatened legal action against Perplexity, demanding it stop using BBC content and pay compensation for material already used.
Perplexity accused the BBC of seeking to preserve what it called Google’s monopoly.
Matthew Prince, CEO of Cloudflare, says: “If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone.”
Cloudflare develops Pay Per Crawl system
Cloudflare is also developing a Pay Per Crawl system that would enable content creators to request payment from AI companies for utilising their content.
The system is an attempt to establish a new economic model for AI access to web content.
The company reports that AI crawlers generate more than 50 billion requests to the Cloudflare network daily, demonstrating the scale of AI bot activity across the internet.
Some AI crawlers are disregarding existing protocols for excluding bots, according to Cloudflare.
In response to persistent offenders, Cloudflare previously developed a system directing the worst miscreants to a labyrinth of web pages filled with AI-generated content.
The new system attempts to use technology to protect website content and give sites the option to charge AI firms for access.
Legislative battles are ongoing between governments, creators and AI firms over the extent to which creative industries should be protected from AI companies using their works to train systems without permission or payment.
In the UK, the government and artists including Sir Elton John have clashed over copyright protection measures.
Content creators, licensors and owners have pursued legal action on both sides of the Atlantic to prevent what they view as AI firms’ encroachment on creative rights.
Ed Newton-Rex warns of limited protection scope
Ed Newton-Rex, Founder of Fairly Trained, an organisation that certifies AI companies have trained their systems on properly licensed data, tells the BBC: “This is really only a sticking plaster when what’s required is major surgery.
“It will only offer protection for people on websites they control - it’s like having body armour that stops working when you leave your house.”
He argues: “The only real way to protect people’s content from theft by AI companies is through the law,” he says.
Baroness Beeban Kidron, a filmmaker campaigning for increased protection for creative industries, praised Cloudflare’s leadership in addressing the issue.
“Cloudflare sits at the heart of the digital world and it is exciting to see them take decisive action,” she tells the BBC.
“If we want a vibrant public sphere we need AI companies to contribute to the communities in which they operate, that means paying their fair share of tax, settling with those whose work they have stolen to build their products and as Cloudflare has just shown, using tech creatively to ensure equity between digital and human creators on an ongoing basis.”
