Web Scraping Without Getting Blocked: A Developer's Guide

Published July 24, 2026 · 7 min read

Getting blocked is the #1 frustration in web scraping. You write the code, it works in testing, you deploy it — and within hours, your IP is banned and you're staring at Cloudflare challenge pages instead of data.

Here's what actually causes blocks, and how to avoid them — from someone who's been on both sides of the cat-and-mouse game.

Why Sites Block Scrapers

Sites don't block scrapers because they hate developers. They block scrapers because:

• Server load: A scraper hitting 100 pages/second looks identical to a small DDoS attack
• Terms of service: Most sites prohibit automated access in their ToS
• Competitive data: Some industries (travel, ticketing, e-commerce) actively defend pricing data
• Content theft: Scrapers that republish content without attribution trigger DMCA alarms

Understanding why matters because it tells you how to scrape responsibly in ways that don't trigger defenses.

The Anti-Bot Detection Stack

Modern websites use a layered detection system. Here's what you're up against:

What Actually Works

1. Respect robots.txt (seriously)

The easiest way to get blocked is to ignore robots.txt. If a site explicitly disallows a path, scraping it will trigger alarms fast. Check it first:

curl -s https://target-site.com/robots.txt | grep Disallow

2. Use Real Browser Headers

The default python-requests/2.31.0 user agent is a giant "I'M A BOT" flag. Use a real Chrome user agent with a full header set:

const headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept-Encoding': 'gzip, deflate, br',
  'DNT': '1',
  'Connection': 'keep-alive',
  'Upgrade-Insecure-Requests': '1',
  'Sec-Fetch-Dest': 'document',
  'Sec-Fetch-Mode': 'navigate',
  'Sec-Fetch-Site': 'none',
  'Sec-Fetch-User': '?1',
  'Cache-Control': 'max-age=0'
};

3. Throttle Your Requests

The single biggest mistake: hitting a site too fast. A real human takes 3-30 seconds between page loads. A scraper requesting 10 pages/second looks like an attack.

// Good: random delays between requests
async function scrapeWithDelay(urls) {
  for (const url of urls) {
    await extractPage(url);
    // Random delay between 2-8 seconds
    const delay = 2000 + Math.random() * 6000;
    await new Promise(r => setTimeout(r, delay));
  }
}

4. Use a Real Browser for JavaScript-Heavy Sites

HTTP requests fail on SPAs, React apps, and any site that loads content with JavaScript. You'll get an empty <div id="root"> instead of actual data. This is where headless browsers come in — but running them yourself is a maintenance nightmare.

ExtractAPI runs every request through a real Chromium browser, so you get the rendered page without managing browser infrastructure:

curl -X POST https://extractapi.app/v1/extract \
  -H "Authorization: Bearer $EXTRACT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://react-ecommerce-site.com/products/123",
    "waitFor": ".product-details-loaded",
    "javascript": true
  }'

5. Rotate Patterns, Not Just IPs

Anti-bot systems profile behavior, not just IPs. If you request pages at exactly 3.0-second intervals, always visit pages in sequential order, and never load images or CSS — you'll get flagged even with rotating residential IPs.

Add randomness to:

• Request timing (not just delays, but actual jitter)
• Page visit order (don't always go 1 → 2 → 3)
• Session length (short sessions, long gaps between them)

When to Stop DIY-ing and Use an API

You should use a scraping API when:

• You're spending more time fighting anti-bot than writing business logic
• The target sites use Cloudflare, Akamai, or DataDome
• You need JavaScript rendering and can't maintain a browser pool
• Scale matters — running 10,000 extractions/day on your laptop won't work