feat: extraction framework overhaul — extract_core shared package + SQLite state tracking
- Add extract/extract_core/ workspace package with three modules:
- state.py: SQLite run tracking (open_state_db, start_run, end_run, get_last_cursor)
- http.py: niquests session factory + etag normalization helpers
- files.py: landing_path, content_hash, write_bytes_atomic (atomic gzip writes)
- State lives at {LANDING_DIR}/.state.sqlite — no extra env var needed
- SQLite chosen over DuckDB: state tracking is OLTP (row inserts/updates), not analytical
- Refactor all 4 extractors (psdonline, cftc_cot, coffee_prices, ice_stocks):
- Replace inline boilerplate with extract_core helpers
- Add start_run/end_run tracking to every extraction entry point
- extract_cot_year returns int (bytes_written) instead of bool
- Update tests: assert result == 0 (not `is False`) for the return type change
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -123,7 +123,7 @@ def test_extract_cot_year_skips_existing_file(tmp_path, monkeypatch):
|
||||
|
||||
result = cot_execute.extract_cot_year(2024, mock_session)
|
||||
|
||||
assert result is False
|
||||
assert result == 0
|
||||
mock_session.get.assert_not_called() # No download should occur
|
||||
|
||||
|
||||
@@ -143,5 +143,5 @@ def test_extract_cot_year_returns_false_on_404(tmp_path, monkeypatch):
|
||||
|
||||
result = cot_execute.extract_cot_year(2006, mock_session)
|
||||
|
||||
assert result is False
|
||||
assert result == 0
|
||||
mock_session.get.assert_not_called()
|
||||
|
||||
Reference in New Issue
Block a user