Good Bots vs Bad Bots
Not all bots are created equal.
| Bots You Want | Bots You Don't Want |
|---|
| Googlebot — indexes your site for search | Scrapers — steal your content |
| Bingbot — same as Google, different engine | Credential stuffers — try leaked passwords |
| Ahrefs / Semrush — SEO analysis (debatable) | Vulnerability scanners — probe for weak points |
| Uptime monitors — Pingdom, Better Uptime | AI training crawlers — GPTBot, CCBot |
| Feed readers — RSS readers, Feedly | Price scrapers — competitor intelligence |
The goal: block bad bots while letting good ones through. A WAF with bot detection makes this automatic.
How SafeLine's Bot Protection Works
SafeLine uses three layers to identify bots:
Layer 1: Static analysis
- User-Agent pattern matching against known bots
- IP reputation database (known scanner IPs)
- Request header consistency checks
Layer 2: Behavioral analysis
- Request frequency and timing patterns
- Page navigation sequences
- Resource loading behavior (does it load CSS/JS/images?)
Layer 3: Active challenge
- JavaScript challenge (execute JS, prove you're a browser)
- CAPTCHA (image/text challenge)
- Cookie validation
Setting Up Bot Protection
In SafeLine Dashboard -> Bot Management:
1. Enable Bot Detection: ON
2. Action for suspicious bots: JavaScript Challenge
3. Action for confirmed bad bots: Block
4. Whitelist: Googlebot, Bingbot, Pingdom, Stripe webhooks
That's four settings and you're protected. The WAF handles the rest.
What It Caught on My Server
After enabling bot protection for 7 days:
| Bot Type | Detected | Action |
|---|
| Credential stuffing attempts | 2,847 | Blocked |
| Content scrapers | 1,203 | JS challenged (all failed) |
| Vulnerability scanners | 892 | Blocked |
| AI crawlers (GPTBot, CCBot) | 456 | JS challenged (all failed) |
| SEO crawlers (Ahrefs, Semrush) | 234 | Allowed (didn't trigger detection) |
| Googlebot | 3,456 | Whitelisted (never challenged) |
Zero legitimate users reported CAPTCHA fatigue. Google continued indexing normally.
Common Bot Patterns to Watch For
Pattern 1: The "I'm Googlebot" liar
User-Agent: Googlebot/2.1
IP: 45.33.32.156 (Linode VPS)
Real Googlebot IPs reverse-resolve to googlebot.com. SafeLine checks this automatically.
Pattern 2: The fast scraper
100 requests in 2 seconds
Every URL on the site, in alphabetical order
No CSS/JS/images loaded
Clear scraper pattern. Gets JS-challenged immediately.
Pattern 3: The credential list
POST /login 50 times in 10 minutes
Different email/password each time
All from the same IP
Stopped by rate limiting before bot detection even kicks in.
FAQ
Will bot protection break my analytics?
No. Google Analytics, Plausible, and similar tools use JavaScript execution (which a browser does but a simple scraper doesn't). Legitimate analytics are unaffected.
What about API clients (Postman, curl, SDKs)?
API endpoints with proper authentication should be added to the bot protection whitelist. Bots that authenticate with valid tokens are treated differently from anonymous scrapers.
Does it affect page load speed?
The JS challenge adds about 200ms on the first request for suspicious visitors. Legitimate users (real browsers, verified bots) see zero delay. The 200ms cost is paid entirely by bots.
Try SafeLine Community Edition — free, self-hosted, and takes 5 minutes to deploy:
bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en
Dashboard: https://<your-server-ip>:9443 | Docs
How much of your current traffic do you think is bots?