Fed up with AI scraping your content? This open-source bot blocker can help - here's how

Anubis / Elyse Betters Picaro / ZDNET

Anyone who runs a website knows how annoying AI bots are these days. 

F5, the application delivery network company, found that more than half of all web visits come not from people but from data scrapers, including OpenAI, Anthropic, Google, and Perplexity AI bots. 

Also: AI bots scraping your data? This free tool gives those pesky crawlers the run-around

(Disclosure: Ziff Davis, ZDNET's parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)

Stopping AI bots in their tracks

People are sick and tired of wasting money on their sites only to have AI companies rip off everything of value. So, Xe Iaso, a technical educator and part-time bot fighter, wrote an open-source program, Anubis, to stop AI bots in their tracks.

Anubis isn't the only such program. Indeed, Iaso freely admits it's "basically the Cloudflare. Are you a bot? [aka Cloudflare Turnstile] page, but self-hostable." This enables you to run it on your own server without incurring any fees.

Also: How AI companies are secretly collecting training data from the web (and why it matters)

Anubis is designed to protect websites -- particularly those run by small organizations, open-source projects, and archives -- from the relentless onslaught of automated scrapers that threaten to overwhelm servers and increase hosting costs. The program is a web AI firewall utility. All incoming HTTP connections must successfully pass through it before reaching your actual website.

Tongue in cheek, Iaso describes Anubis as like the ancient Egyptian god, weighing the soul of your connection using one or more challenges to protect upstream resources from scraper bots. It does this by requiring visitors to solve a computational puzzle, which is trivial for PCs but expensive for bots operating at scale. 

The system checks whether visitors behave like real browsers, using JavaScript and cookies to verify authenticity. When a bot fails these challenges, it is blocked before reaching the website's core resources.

Anubis is an uncaptcha

Now, you may ask, "Isn't this just a CAPTCHA? And, aren't AI programs just as good at solving those as people are?" 

That's true -- they are. But as Iaso says, "Anubis is an uncaptcha. It uses features of your browser to automate a lot of the work that a CAPTCHA would, and right now, the main implementation is by having it run a bunch of cryptographic math with JavaScript to prove that you can run JavaScript in a way that can be validated on the server."

Also: Cloudflare just changed the internet, and it's bad news for the AI giants

She is well aware that many people are hesitant to run JavaScript due to security and privacy concerns. She's working on a non-JavaScript version of Anubis, but it's not here yet. It will be a while. On a Reddit thread, Iaso said she's "am working on a better one that doesn't rely on JS, but oh god, it is going to be a hell of a thing to implement."

Anubis is written in Go and licensed under the open-source MIT License. It's designed to be "as lightweight as possible to ensure that everyone can afford to protect the communities closest to them." On average, the program uses less than 128 MB of RAM on the server side.

Most of the workload is handled by visitors' PCs and smartphones. Still, the end-user processing load is so low that ordinary users won't notice. Indeed, since Anubis operates transparently, there are no CAPTCHAs to solve or images to click; most people won't even know that anything is happening. 

Also: This proxy provider I tested is the best for web scraping - and it's not IPRoyal or MarsProxies

The proof-of-work runs in the background, and only those with outdated browsers or JavaScript disabled may encounter issues. It's another story for bot farms -- their load quickly adds up.

In a blog, Iaso says:

At a high level, Anubis has a big old set of rules in your bot policy file. If clients match a rule, they are either passed through, blocked, or selected for secondary screening. By default, Anubis is meant to instantly work by stopping all the bleeding and letting administrators sleep without downtime alerts waking them up. This means that it's overly paranoid and aggressively challenges everything, similar to Cloudflare's "I'm under attack" mode.

My intent was that admins would start out with Anubis being quite paranoid and then slowly lessen the paranoia as they find better patterns and match out ways to do things. Users tend to use Anubis in its default configuration, but this default configuration interferes with RSS feed readers and other "good bots."

A nuclear response

The result is a tool that Iaso describes as a "bit of a nuclear response."

"This will result in your website being blocked from smaller scrapers and may inhibit 'good bots' like the Internet Archive. You can configure bot policy definitions to explicitly allowlist them, and we are working on a curated set of 'known good' bots to allow for a compromise between discoverability and uptime," Iaso says.

Also: Reddit sues Anthropic for scraping its users' content without consent

Many groups were ready for a nuclear response. Organizations such as GNOME, FFmpeg, and UNESCO have adopted Anubis to protect their online infrastructure. Since its release in January 2025, Anubis has been downloaded over 200,000 times and is credited with helping numerous organizations avoid outages and reduce the burden of unwanted AI scraping.

According to Duke University, a happy Anubis user, the school's library systems have successfully blocked about 90 percent of unwanted traffic and over 4 million unwanted HTTP requests per day, while improving service performance with minimal blockage for real users.

How to install and run Anubis

There are several ways to install and run Anubis. 

Typically, Anubis is meant to sit between your reverse proxy and your target service. Support is currently free. You can access it via its GitHub issue page or, for live chat, join Iaso's Patreon and ask in the Patreon Discord channel. There's also a commercial version of Anubis named BotStopper, which, at this point, just offers organizations more control over the program's branding.

Also: How global threat actors are weaponizing AI now, according to OpenAI

The battle between bot developers and defenders promises to be never-ending. Anubis's creators are updating the tool to counter new evasion tactics, such as headless browsers and advanced browser fingerprinting. The goal is to keep the internet accessible for humans while making it uneconomical for abusive bots to operate at scale. This is not easy. 

If you find the project useful, do support it. She can use all the help you can give.

Get the morning's top stories in your inbox each day with our Tech Today newsletter.