Files
padelnomics/extract
Deeman 6116445b56 perf(extract): auto-detect workers from proxies, skip throttle on success, crash-safe partial JSONL
- proxy.py: delete unused make_sticky_selector()
- utils.py: add load_partial_results() + flush_partial_batch() for crash-resumable extraction
- playtomic_availability.py:
  - drop MAX_WORKERS / EXTRACT_WORKERS — worker_count = len(proxy_urls) or 1
  - skip time.sleep(THROTTLE_SECONDS) on success when proxy_url is set; keep sleeps for 429/5xx
  - replace cursor-based resumption with .partial.jsonl sidecar (flush every 50 records)
  - _fetch_venues_parallel accepts on_result callback for incremental partial-file flushing
  - mirror auto-detect worker count in extract_recheck()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 22:21:05 +01:00
..