Deeman 9f010d8c0c perf(extract): parallel page fetching in tenants, drop EXTRACT_WORKERS env var ...

- playtomic_tenants.py: batch_size = len(proxy_urls) pages fired in parallel per
  batch; each page gets its own session + proxy; sorted(results) ensures
  deterministic done-detection; falls back to serial + THROTTLE_SECONDS when no
  proxies. Expected speedup: ~2.5 min → ~15 s with 10 proxies.
- .env.dev.sops, .env.prod.sops: remove EXTRACT_WORKERS (now derived from
  PROXY_URLS length)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-02-24 22:30:28 +01:00

8.9 KiB

Raw Blame History

View Raw