deemanone/beanflows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix COT pipeline: TRY_CAST nulls, dim_commodity leading zeros, correct CFTC codes

Browse Source 
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
  Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
  to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
  and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data

Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-20 23:22:46 +01:00
parent 0a83b2cb74
commit 2962bf5e3b
7 changed files with 126 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
transform/sqlmesh_materia/seeds/dim_commodity.csv
View File

@@ -1,2 +1,2 @@
usda_commodity_code;cftc_commodity_code;commodity_name;commodity_group
0711100;083731;Coffee, Green;Softs
"0711100";"083";"Coffee, Green";"Softs"
1 usda_commodity_code cftc_commodity_code commodity_name commodity_group
2 0711100 083731 083 Coffee, Green Softs