feat(cot): add combined (futures+options) COT extractor and transform models
- extract/cftc_cot: refactor extract_cot_year() to accept url_template and
landing_subdir params; add _extract_cot() shared loop; add extract_cot_combined()
entry point using com_disagg_txt_{year}.zip → landing/cot_combined/
- pyproject.toml: add extract_cot_combined script entry point
- macros/__init__.py: add @cot_combined_glob() for cot_combined/**/*.csv.gzip
- fct_cot_positioning.sql: union cot_glob and cot_combined_glob in src CTE;
add report_type column (FutOnly_or_Combined) to cast_and_clean + deduplicated;
include FutOnly_or_Combined in hkey to avoid key collisions; add report_type to grain
- obt_cot_positioning.sql: add report_type = 'FutOnly' filter to preserve
existing serving behavior
- obt_cot_positioning_combined.sql: new serving model filtered to report_type =
'Combined'; identical analytics (COT index, net %, windows) on combined data
- pipelines.py: register extract_cot_combined; add to extract_all meta-pipeline
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,10 @@ PIPELINES = {
|
||||
"command": ["uv", "run", "--package", "cftc_cot", "extract_cot"],
|
||||
"timeout_seconds": 1800,
|
||||
},
|
||||
"extract_cot_combined": {
|
||||
"command": ["uv", "run", "--package", "cftc_cot", "extract_cot_combined"],
|
||||
"timeout_seconds": 1800,
|
||||
},
|
||||
"extract_prices": {
|
||||
"command": ["uv", "run", "--package", "coffee_prices", "extract_prices"],
|
||||
"timeout_seconds": 300,
|
||||
@@ -49,7 +53,7 @@ PIPELINES = {
|
||||
"timeout_seconds": 120,
|
||||
},
|
||||
"extract_all": {
|
||||
"command": ["meta", "extract", "extract_cot", "extract_prices", "extract_ice_all", "extract_weather"],
|
||||
"command": ["meta", "extract", "extract_cot", "extract_cot_combined", "extract_prices", "extract_ice_all", "extract_weather"],
|
||||
"timeout_seconds": 6600,
|
||||
},
|
||||
"transform": {
|
||||
@@ -68,7 +72,7 @@ PIPELINES = {
|
||||
|
||||
|
||||
META_PIPELINES: dict[str, list[str]] = {
|
||||
"extract_all": ["extract", "extract_cot", "extract_prices", "extract_ice_all", "extract_weather"],
|
||||
"extract_all": ["extract", "extract_cot", "extract_cot_combined", "extract_prices", "extract_ice_all", "extract_weather"],
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user