Technical guide · reviewed September 2026
Responsible web scraping: reduce blocks without bypassing controls
Access limits are a signal, not a puzzle to defeat. A sustainable extractor identifies itself, follows the target’s published rules, requests only what it needs, and stops when the site says no. This approach also gives you cleaner failure data than trying to disguise an automated client.
Why a public page can still reject automation
Public visibility does not grant unlimited automated access. A site can enforce terms, robots guidance, authentication, consent, request quotas, network policy, or a challenge. Common responses include:
| Signal | What it can mean | Safe response |
|---|---|---|
| 403 | Access is forbidden or the request is not authorized | Stop and review permission; do not retry in a loop |
| 429 | Rate limit or quota was exceeded | Honor Retry-After when present and reduce demand |
| 5xx / timeout | Temporary target or network failure | Bounded retry with backoff, then surface an error |
| Challenge page | Interactive or automated-access control | Do not attempt to bypass it; seek an approved feed or permission |
Five practices that reduce avoidable load
1. Check permission and target guidance
Read the target’s terms and robots.txt before building a recurring job. Robots rules are a published crawler preference, not a substitute for authorization or a guarantee of access. If a page is disallowed or requires a login, use an official API, licensed feed, or an agreement instead.
2. Request less, less often
Cache unchanged pages, use conditional requests when the target supports them, schedule checks around business need, and cap concurrency per host. A small allowlist of exact product or article URLs is easier to govern than an open-ended crawler.
// Example: bounded work in your own application
for (const target of targets) {
await extractOne(target); // validate and log a sanitized result
await new Promise(resolve => setTimeout(resolve, 1500));
}3. Handle responses explicitly
Do not treat an HTML challenge page as the requested data. Record the status, a request ID, and a coarse error category. Keep raw target content and credentials out of logs and analytics.
if (response.status === 403) {
throw new Error('target_forbidden');
}
if (response.status === 429) {
const retryAfter = response.headers.get('retry-after');
throw new Error(`target_rate_limited:${retryAfter || 'unknown'}`);
}4. Use rendering only when the content needs it
Raw HTTP is usually the right tool for server-rendered pages and feeds. When a permitted public page fills its useful content after JavaScript runs, a browser-based request may help. ExtractAPI accepts javascript:true and optional selectors, but rendering does not bypass authentication, CAPTCHA, or a target’s access controls.
curl -X POST https://extractapi.app/v1/extract \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://demo.example/article","javascript":true}'5. Prefer a stable source when one exists
Ask for an official API, export, RSS feed, or partner endpoint when the data is important or recurring. A source designed for integration is easier to explain to your legal, security, and operations teams than a series of retries against a protected page.
What not to do
- Do not rotate identities, spoof headers, rotate proxies, or randomize behavior to evade a block.
- Do not solve or outsource CAPTCHAs, defeat paywalls, or automate a logged-in account without explicit permission.
- Do not infer that a browser-rendered response will succeed on every SPA or protected domain.
- Do not publish a target URL, extracted content, email address, API key, cookie, or authorization header in telemetry or error messages.
FAQ
Does robots.txt grant permission to scrape?
No. It communicates crawler preferences. Review terms, authorization, privacy obligations, and applicable law separately.
Should I retry a 403 or challenge page?
Not automatically. Stop, record a sanitized category, and look for an approved source or permission. A bounded retry for a transient 5xx or timeout is a different case.
Can a headless browser avoid being blocked?
It may render client-side content, but it does not guarantee access and should not be used to bypass controls. Test only targets your workflow is allowed to read.