Stop Scrapers & Bots Hammering Your VPS With Nginx Rate Limiting

If your VPS load average spikes every few hours for no obvious reason, and when you finally check the access log it's page after page of the same handful of IPs hitting every URL on your site in rapid succession, you don't have a traffic problem — you have a scraper or a badly-behaved bot hammering you faster than a real visitor ever would. Blocking it by IP is a losing game since the addresses rotate. What actually works is teaching Nginx to throttle request rates at the server level, so no single client can make more than a handful of requests per second no matter who they claim to be.

Symptom: One Client, Hundreds of Requests a Minute

A few things usually tip you off before you even open the logs:

• CPU or iowait spikes in short, regular bursts rather than a steady climb.
• PHP-FPM's process list (ps aux | grep php-fpm) is full, and most of the busy workers are serving the same handful of URLs.
• tail -f /var/log/nginx/access.log shows one IP (or a small IP block) requesting dozens of pages a second, often hitting search pages, category archives, or product pages that hit the database instead of the page cache.
• Your legit uptime monitor or real users start seeing slow page loads or 502/504 errors during these bursts, even though total daily traffic in your analytics looks normal.

Run a quick check to confirm it's concentrated on a few clients rather than genuinely distributed traffic:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

If the top handful of IPs each show thousands of hits against a log that only covers an hour or two, that's your scraper.

Cause: No Ceiling on Request Rate

By default, Nginx (and Apache) will happily serve as many requests as a client throws at it, as fast as it can. That's fine for a browser rendering one page at a time, but it's an open door for:

• Content scrapers pulling your entire product catalog or blog archive in minutes, often to feed a competitor's site or an AI training pipeline.
• Credential-stuffing and login brute force against wp-login.php or a custom login form — hundreds of password attempts a minute from a botnet.
• Misconfigured or aggressive crawlers that ignore Crawl-delay in robots.txt (which is only ever a polite suggestion, never enforced).
• Low-grade DDoS/stress traffic that isn't big enough to need a WAF or Cloudflare, but is more than enough to pin a single VPS's CPU.

None of this is stopped by a firewall rule unless you already know the IP, and IP-based blocking is a whack-a-mole game against rotating proxies. Rate limiting fixes the actual problem: however many IPs are involved, no single one gets to hammer your app faster than a human would.

Fix: Add limit_req_zone in Nginx

Nginx has this built in — no extra module needed. You define a shared-memory zone keyed by client IP, then apply a limit to specific locations (or the whole server block).

1. Define the Zone (in the http block)

Edit your main Nginx config, usually /etc/nginx/nginx.conf, and add this inside the http {} block — not inside a server {} block:

limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=login:10m rate=2r/m;

Here's what each part means:

2. Apply It in Your Server or Location Block

In your site's server {} block (usually under /etc/nginx/conf.d/ or your cPanel/WHM vhost include), apply the zone where it matters most:

server {
    # ... existing config ...

    location / {
        limit_req zone=general burst=20 nodelay;
        # ... existing proxy_pass / try_files ...
    }

    location = /wp-login.php {
        limit_req zone=login burst=3 nodelay;
        # ... existing config ...
    }
}

burst=20 lets a client briefly exceed the steady rate (useful for a real browser loading a page's CSS, JS, and images all at once) before the limit kicks in. nodelay serves the burst immediately instead of queuing requests with artificial delay — without it, legitimate visitors can feel like the site is lagging.

3. Log and Test Before You Tighten

Add a log level so you can see what's actually being throttled before you assume the numbers are right:

limit_req_log_level warn;
limit_req_status 429;

Reload Nginx and watch the error log for a bit:

nginx -t && systemctl reload nginx
tail -f /var/log/nginx/error.log | grep limiting

You'll see lines like limiting requests, excess: 15.500 by zone "general" when a client gets throttled. If real users start getting 429s during normal browsing, raise the rate and burst values rather than removing the limit entirely.

4. On a cPanel/WHM Server

WHM-managed servers regenerate Nginx-fronted or Apache configs from templates, so hand-edited vhost files can get overwritten on the next rebuild. If you're running Nginx as a reverse proxy in front of Apache (common on cPanel setups), put your limit_req_zone in a custom include file under /etc/nginx/conf.d/ that WHM won't touch, and reference the zone from your site's location block the same way. If you're not sure which layer is actually terminating traffic, run curl -I https://yourdomain.com and check the Server header to confirm Nginx is in the path at all before spending time here.

Prevention

• Set a stricter, separate zone for expensive endpoints — login forms, search, cart/checkout — where a burst of requests does the most database damage.
• Pair rate limiting with a page cache (LiteSpeed Cache, WP Super Cache, or Nginx fastcgi_cache) so the requests that do get through don't all hit PHP and MySQL directly.
• If you're already behind Cloudflare, enable its rate limiting or "I'm Under Attack" mode for spikes that are bigger than a single VPS can absorb — Nginx-level limits protect the origin, Cloudflare protects the edge.
• Review your top-IP report (the awk command above) monthly, even after adding limits, so you catch new patterns instead of assuming the problem is permanently solved.
• Combine this with fail2ban watching the Nginx error log for repeated limiting requests entries, so persistent offenders get temporarily firewalled instead of just slowed down.

Frequently Asked Questions

Will rate limiting block Googlebot or other search engine crawlers?

Not at a sane rate. Googlebot typically crawls well under 10 requests per second per site unless you've explicitly configured a higher crawl rate in Search Console. If you're worried, set a higher, separate zone matched by $http_user_agent for verified search bots, or simply monitor the error log after enabling limits to confirm you're not seeing legitimate crawlers throttled.

What's the difference between limit_req and limit_conn?

limit_req caps how many requests per second a client can make; limit_conn caps how many simultaneous open connections a client can hold. They solve different problems and are often used together — limit_conn is particularly useful against slow-loris-style attacks that hold connections open rather than firing rapid requests.

My site is behind Cloudflare — do I still need this on the origin Nginx?

Yes, for two reasons: Cloudflare's free-tier rate limiting is limited, and any traffic that bypasses Cloudflare (someone hitting your origin IP directly instead of your domain) skips its protection entirely. Origin-level limits are your last line of defense regardless of what's in front.

Why did I get 429 errors on my own site after adding this?

Your burst value is probably too low for how many assets a single page load fires off (CSS, JS, fonts, images can easily be 15-20 requests in under a second). Raise burst first before touching rate — burst absorbs legitimate page-load spikes without loosening the steady-state throttle that actually stops scrapers.

Can I apply different limits to different paths on the same site?

Yes — define one or more zones in the http block, then reference whichever zone fits each location block. A tight zone on /wp-login.php or /xmlrpc.php, a looser one on general pages, and no limit at all on static assets served straight from disk is a common, effective split.