AI Bots Hammering Your Server? Block GPTBot & Co Safely

If your bandwidth graph in cPanel has been climbing for no obvious reason — no traffic spike in Google Analytics, no viral post, nothing — the culprit these days is usually an AI crawler. GPTBot, Bytespider, ClaudeBot and a growing list of others are hitting sites around the clock to scrape content for AI training and search, and a lot of them don't play nice. Here's how to spot them and shut them down without touching your real SEO traffic.

Symptom

A few patterns tend to show up together:

• Bandwidth or "Metrics > Raw Access Logs" in cPanel shows a big jump with no matching increase in real visitors
• Server load or CPU usage climbs at odd hours (2 AM local time is a classic one) with no cron jobs running
• AWStats or Webalizer lists unfamiliar user agents pulling hundreds or thousands of hits per day
• On a VPS, top or htop shows php-fpm or Apache workers pegged even though nobody's actively browsing the site
• Your hosting resource usage warning fires, or you get a 508 "Resource Limit Is Reached" error, with no clear cause

Cause

Since 2023, dozens of companies have launched bots that crawl the open web to train large language models or power AI-generated answers. Some of the more common ones:

The compliant ones will back off if you tell them to in robots.txt. A handful — Bytespider is the repeat offender we see in support tickets — crawl aggressively and don't always respect it, so robots.txt alone isn't a complete fix. That's the whole reason this turns into a support ticket: one bot ignoring your rules and re-crawling every page every few minutes.

Fix

1. Confirm it's actually AI bots

Before blocking anything, check who's really hitting you. On cPanel hosting, grab the raw log and count user agents:

cd ~/logs
zgrep -a "GPTBot\|Bytespider\|ClaudeBot\|CCBot\|PerplexityBot" yourdomain.com-*.gz | wc -l

On a VPS with root access, the domlog is usually at /usr/local/apache/domlogs/yourdomain.com (or Nginx's access log). Run:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

That gives you a ranked list of user agents by hit count — the AI bots will usually be near the top if they're the problem.

2. Block the compliant bots with robots.txt

This stops the well-behaved crawlers immediately. Add this to your robots.txt in the site's document root:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

None of this touches Googlebot, Bingbot, or any regular search crawler — those are separate user agents, so your search rankings are unaffected.

3. Hard-block the ones that ignore robots.txt

For shared cPanel hosting, add this to .htaccess in your document root (edit it via File Manager or SSH if you have it):

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Bytespider|CCBot|GPTBot|ClaudeBot|PerplexityBot|Amazonbot) [NC]
RewriteRule .* - [F,L]

That returns a 403 to any request carrying one of those user agent strings, robots.txt aside. If you're on Nginx (VPS or reverse proxy in front of Apache), do the equivalent in your server block:

if ($http_user_agent ~* (Bytespider|CCBot|GPTBot|ClaudeBot|PerplexityBot|Amazonbot)) {
    return 403;
}

Reload Nginx after saving: nginx -t && systemctl reload nginx.

4. If you're on Cloudflare, use the built-in toggle

If your domain is proxied through Cloudflare (orange cloud), Cloudflare ships a one-click "Block AI Bots" option under Security > Bots on paid plans, and a manual WAF rule works on the free plan too:

(http.user_agent contains "GPTBot") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "ClaudeBot")

Set the action to Block. This is the least maintenance-heavy option since Cloudflare updates its own bot-category list over time.

5. Rate-limit as a backstop (VPS)

Some bots rotate user agents or IPs to dodge a static blocklist. If load is still spiking after step 3, add a request-rate cap in Nginx:

limit_req_zone $binary_remote_addr zone=general:10m rate=5r/s;

server {
    location / {
        limit_req zone=general burst=10 nodelay;
    }
}

This caps any single IP at 5 requests/second regardless of what user agent it claims to be — useful when a bot is spoofing a browser string to get around your other rules.

Prevention

• Keep your blocklist current — new AI crawlers show up every few months, so revisit this every quarter
• If you use Cloudflare, turn on its managed AI bot category instead of a manual list; it updates automatically
• Watch AWStats or your log summary weekly rather than waiting for a resource-limit alert to tell you something's wrong
• On CSF/LFD-managed VPS, add a custom rule that temporarily bans an IP after a burst of hits matching known bot strings — this catches new bots you haven't explicitly listed yet
• Don't block indiscriminately with broad user-agent wildcards; a typo in your regex can accidentally catch Googlebot or Bingbot and tank your indexing

Frequently Asked Questions

Will blocking AI bots hurt my SEO or Google rankings?

No. GPTBot, ClaudeBot, Bytespider and the rest are entirely separate from Googlebot and Bingbot. Blocking them has zero effect on search indexing or rankings — you're only stopping content from being scraped for AI training or AI-generated answers.

Does adding rules to robots.txt actually stop the bots?

It stops the ones that respect it — GPTBot, ClaudeBot, Google-Extended, CCBot, and most of the major names do honor Disallow rules. A few, Bytespider being the most notorious, crawl anyway. For those you need the .htaccess/Nginx block in Step 3, not just robots.txt.

How do I find out which bots are actually hitting my site?

Check your raw access logs (cPanel: Metrics > Raw Access Logs, or the domlog file directly on a VPS) and count user agents as shown in Step 1. AWStats and Webalizer also break down traffic by user agent if you'd rather browse it visually.

I'm on shared cPanel hosting without root access — can I still block these bots?

Yes. The robots.txt and .htaccess methods in Steps 2 and 3 need no root access at all — just File Manager or FTP access to your document root. The Nginx and rate-limiting steps are for VPS/dedicated users with server config access.

Should I just block every unfamiliar bot user agent I see in my logs?

Not blindly. Some unfamiliar strings are legitimate services — uptime monitors, SEO audit tools you or a client use, payment gateway callbacks. Confirm what a user agent actually is before blocking it, and never use a wildcard broad enough to accidentally catch Googlebot or Bingbot.