One warmed browser context costs 650–900 MB RSS
The run that set my current worker cap was 18,400 category and product URLs from a Portuguese retail site. It ran on a 64 GB Windows server with Playwright Chromium, a Portugal residential route through proxynade.net:2555, screenshots on failure, and trace capture still enabled from debugging. The question was whether twenty-four concurrent workers would fit.
One warmed context sat between 650 and 900 MB RSS. Twelve workers looked safe after five minutes, but that short test had not hit checkout retries, image blocks, or screenshot accumulation. At eighteen workers the job finished in just under six hours after two changes: recycle contexts every few hundred URL attempts, and keep screenshots only on the final failed try.
At twenty-four workers, RSS kept walking upward. Around the forty-minute mark the box was already over 50 GB used. Swap appeared, then one browser process died, and the proxy route looked guilty because the visible symptom was timeout noise. The memory graph told the less exciting story.
The cleanup mistake was specific
Traces were saved on every retry. A screenshot folder passed 11 GB. A few failed pages stayed open because the error handler logged first and closed the context second. After those three things were fixed, the same route handled the job without approaching swap.
The lesson is not "use less memory." It is: do not tune the machine and the proxy at the same time. Change one variable, keep the log, then decide whether the bottleneck was CPU, memory, network, or the exit route.
HTTP workers fail differently — queue bloat, not context leak
A requests or aiohttp scraper can stay small if responses are streamed and output is flushed immediately. It can also quietly waste RAM if it buffers full response bodies, keeps parser output in memory, and accumulates tens of thousands of failed URLs in a retry queue. More RAM hides that queue bug for a while. It does not fix it.
The proxy meter also gets mixed into the diagnosis. The scraper app may credit only accepted rows. The provider still meters blocked images, challenge pages, warmup traffic, and dead retries. The local counter can look calm while bytes and memory keep climbing. That usually points back to the script, not the route.
Size by measured worker peak, not averages
On a 64 GB machine, memory still has to cover the OS, queue process, logs, monitoring, the browser parent process, and normal spikes. If a browser worker peaks near 900 MB, dividing 64 GB by 900 MB gives a number that looks clean in a spreadsheet and falls apart during a longer run with retry storms.
| Workload type | Comfortable range | What kills the estimate |
|---|---|---|
| Bounded HTTP work | 48 GB – 96 GB | Unbounded retry queue, buffered response bodies |
| Browser jobs, short sessions | 64 GB – 128 GB | Trace capture on every retry, unclosed contexts |
| Browser jobs with screenshots + traces | 128 GB+ | Screenshot folder growth, long-running contexts |
Neither range matters without a measured worker peak. Run a small batch, record RSS under load, then multiply by the target concurrency and add OS headroom before picking hardware.
Add a config comment before raising the worker count
The job now has one comment at the top:
# cap=18; p95=6.4s; rss_peak=41gb; retry_cap=2; screenshots=final_fail_onlyThat is enough context for the next person who wants to raise the concurrency limit. Without it, they will repeat the same swap-then-blame-the-proxy diagnosis.
RAM sizing FAQ
How much RAM does one Playwright or Puppeteer browser worker use? A warmed Chromium context typically sits between 650 MB and 900 MB RSS. Use the high end for your per-worker estimate.
Why did my scraper OOM even though the per-worker numbers looked safe? Short tests miss retry storms, screenshot accumulation, and unclosed contexts from error handlers that log before they close. Measure under load, not at idle.
How much RAM do HTTP scrapers need compared to browser scrapers? HTTP scrapers stay small if responses are streamed and output is flushed immediately. Queue bloat and buffered response bodies cause growth, not concurrency itself.
Why does the provider meter more bytes than my app counted? The proxy meters every byte transferred: blocked images, challenge pages, warmup traffic, and retries. Your app may only credit accepted rows, so the two counters diverge when the error rate rises.
What RAM size fits most browser scraping jobs? Browser jobs with retry storms, screenshots, and traces are more comfortable in the 64 GB to 128 GB range. Bounded HTTP work often fits in 48 GB to 96 GB. Neither number matters without a measured worker peak.
Sizing checklist
- Measure idle worker RSS before setting a concurrency cap.
- Run a batch long enough to hit retries and screenshots.
- Stop adding workers before swap appears, not after.
- Change one variable at a time: machine, concurrency, or proxy route.
- Cap retry depth and close contexts in the finally block, not the catch block.