Cloudflare launches anti-AI bots tools to combat website scraping

A new free tool has been released by publicly listed cloud service company Cloudflare with the goal of preventing bots from gathering information from websites hosted on its network in order to create artificial intelligence (AI) models.

In a blog post announcing this update, the Cloudflare team also shared some data about how its clients are coping with the increase in bots that employ content scraping to create generative AI models.

Reports say all cloud service provider users, including those on free plans, have access to the new tool.

By modifying their site’s robots.txt file, which tells bots which pages on a website they may access, website owners can prevent the bots used for data scraping and model training, according to several AI providers including Google, OpenAI, and Apple.

“Customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company informed on its official blog.

READ ALSO:Microsoft coughs out over $14m to resolve claims of discriminating against workers

“We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection.”

The models consider, for example, the possibility that an AI bot is trying to hide from discovery by mimicking the appearance and behavior of a user of a web browser.

“When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we can fingerprint,” Cloudflare writes.

“Based on these signals, our models [are] able to appropriately flag traffic from evasive AI bots as bots.”

Join the conversation

Opinions

Support Ripples Nigeria, hold up solutions journalism

Balanced, fearless journalism driven by data comes at huge financial costs.

As a media platform, we hold leadership accountable and will not trade the right to press freedom and free speech for a piece of cake.

If you like what we do, and are ready to uphold solutions journalism, kindly donate to the Ripples Nigeria cause.

Your support would help to ensure that citizens and institutions continue to have free access to credible and reliable information for societal development.

Donate Now