Fix COT pipeline: TRY_CAST nulls, dim_commodity leading zeros, correct CFTC codes
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs) - fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns - raw/cot_disaggregated: add columns() declaration for 33 varchar cols - dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models - seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code) to '083' (3-digit CFTC commodity code); add CSV quoting - test_cot_foundation.yaml: fix output key name, vars for time range, partial: true, and correct cftc_commodity_code to '083' - analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -13,7 +13,42 @@ MODEL (
|
||||
kind FULL,
|
||||
grain (cftc_commodity_code, report_date_as_yyyy_mm_dd, cftc_contract_market_code),
|
||||
start '2006-06-13',
|
||||
cron '@daily'
|
||||
cron '@daily',
|
||||
columns (
|
||||
market_and_exchange_names varchar,
|
||||
report_date_as_yyyy_mm_dd varchar,
|
||||
cftc_commodity_code varchar,
|
||||
cftc_contract_market_code varchar,
|
||||
contract_units varchar,
|
||||
open_interest_all varchar,
|
||||
prod_merc_positions_long_all varchar,
|
||||
prod_merc_positions_short_all varchar,
|
||||
swap_positions_long_all varchar,
|
||||
swap_positions_short_all varchar,
|
||||
swap_positions_spread_all varchar,
|
||||
m_money_positions_long_all varchar,
|
||||
m_money_positions_short_all varchar,
|
||||
m_money_positions_spread_all varchar,
|
||||
other_rept_positions_long_all varchar,
|
||||
other_rept_positions_short_all varchar,
|
||||
other_rept_positions_spread_all varchar,
|
||||
nonrept_positions_long_all varchar,
|
||||
nonrept_positions_short_all varchar,
|
||||
change_in_open_interest_all varchar,
|
||||
change_in_m_money_long_all varchar,
|
||||
change_in_m_money_short_all varchar,
|
||||
change_in_prod_merc_long_all varchar,
|
||||
change_in_prod_merc_short_all varchar,
|
||||
conc_gross_le_4_tdr_long_all varchar,
|
||||
conc_gross_le_4_tdr_short_all varchar,
|
||||
conc_gross_le_8_tdr_long_all varchar,
|
||||
conc_gross_le_8_tdr_short_all varchar,
|
||||
traders_tot_all varchar,
|
||||
traders_m_money_long_all varchar,
|
||||
traders_m_money_short_all varchar,
|
||||
traders_m_money_spread_all varchar,
|
||||
filename varchar
|
||||
)
|
||||
);
|
||||
|
||||
SELECT
|
||||
|
||||
Reference in New Issue
Block a user