First-request failures isolate the exit network
Split a failed run into two piles: failures after a few pages, and failures on the first page. A first-request 403 on a public URL has no session story behind it — no warmup, no cookie history, no rate pattern. The most likely variable is the exit network.
Test one route before scaling. Scaling obscures which variable failed and makes the bandwidth bill harder to explain.
ASN is IP-owner classification, not header inspection
ASN filtering is the target checking who owns the IP block, not what headers you sent. A datacenter IP maps back to a hosting network. WHOIS, BGP data, and IP intelligence services place that exit IP into a hosting, consumer ISP, mobile, or enterprise bucket — each AS is assigned a unique number recorded in the IANA ASN registry. Some targets check that bucket on the first request. Changing headers does not change the bucket.
| Route class | Typical ASN owner | Cost |
|---|---|---|
| Datacenter | Hosting or cloud AS | Check product page |
| Volume Residential | Broad consumer ISP pool | $0.89/GB |
| Premium Residential | Specific ISP, ASN, or carrier controls available | $5.00/GB |
Confirm the route is the failure before paying $5/GB
Keep a route_class column in the test sheet. The signal: same public URL, same method, same pace, same client — only the route changes. If datacenter fails on request one and residential loads content, mark it route-class sensitive. If failures only start after volume builds, stop blaming ASN and look at retry rate, pacing, and client behavior.
After the swap test, pull an ASN lookup on the exit IP. That confirms whether the block tracks the hosting classification or something else entirely.
Premium Residential ASN controls do not fix non-route problems
Premium Residential ASN controls narrow the public route. They do not make a private resource public. They do not fix rate limits, account checks, paywalls, robots-restricted paths, API contracts, or authorization walls. If datacenter, volume residential, and a premium filter all fail at the same point, the route was not the problem.
Usage logs show what the accepted-row counter hides
The accepted-row counter in a job result hides retries, redirects, blocked pages, failed pages, assets, and discarded responses. The Proxynade dashboard usage logs show host, outcome, latency, and byte totals, and export as CSV. Check those before moving anything to Premium. A wrong ASN guess at $5.00/GB burns money quietly and the logs are the only place it shows up.
Move only the failing slice, not the whole operation
One host failing does not mean every crawler task needs the expensive route. Split that host out, test the ISP or ASN filter on it specifically, and move only the slice that proves it requires Premium. Leave easy public pages on cheaper traffic.
ISP and ASN filtering FAQ
What does ASN filtering actually change? It controls which Autonomous System the exit IP belongs to. The target sees a consumer ISP AS rather than a hosting provider AS, which changes how its IP intelligence layer classifies the request.
Do I need Premium Residential for all targets? No. Datacenter works on targets that accept hosting networks. Volume Residential at $0.89/GB covers most broad residential use cases. Move to Premium at $5.00/GB only after ASN is confirmed as the failure point.
Does changing headers fix an ASN block? No. ASN is determined by the IP block owner, not by HTTP headers. WHOIS and BGP data classify the IP independently of what you send.
How do I confirm ASN caused the failure? Run the same request through datacenter and through residential exits on the same URL, same method, same pace. If datacenter fails on request one and residential loads content, the target is route-class sensitive.
Should I move the entire operation to Premium if one host fails? No. Split that host out, test the ISP or ASN filter on it, and move only the slice that proves it needs Premium. Leave other pages on cheaper traffic.
Route selection checklist
- Test one route before scaling.
- Run the same URL through datacenter and residential before drawing conclusions.
- Pull an ASN lookup on the exit IP after a swap test.
- Check usage logs for byte costs before moving to Premium.
- Move only the failing host slice to the more expensive route.
- If all route classes fail at the same point, look outside the route.