Fix COT pipeline: TRY_CAST nulls, dim_commodity leading zeros, correct CFTC codes
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs) - fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns - raw/cot_disaggregated: add columns() declaration for 33 varchar cols - dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models - seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code) to '083' (3-digit CFTC commodity code); add CSV quoting - test_cot_foundation.yaml: fix output key name, vars for time range, partial: true, and correct cftc_commodity_code to '083' - analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,24 +1,23 @@
|
||||
-- Commodity dimension: conforms identifiers across source systems.
|
||||
--
|
||||
-- This is the ontology seed. Each row is a commodity tracked by BeanFlows.
|
||||
-- This is the ontology. Each row is a commodity tracked by BeanFlows.
|
||||
-- As new sources are added (ICO, futures prices, satellite), their
|
||||
-- commodity identifiers are added as columns here — not as separate tables.
|
||||
-- As new commodities are added (cocoa, sugar), rows are added here.
|
||||
--
|
||||
-- References:
|
||||
-- usda_commodity_code → raw.psd_alldata.commodity_code
|
||||
-- cftc_commodity_code → raw.cot_disaggregated.cftc_commodity_code
|
||||
-- usda_commodity_code → raw.psd_alldata.commodity_code (numeric string, e.g. '0711100')
|
||||
-- cftc_commodity_code → raw.cot_disaggregated.cftc_commodity_code (3-char, e.g. '083')
|
||||
--
|
||||
-- NOTE: Defined as FULL model (not SEED) to guarantee leading-zero preservation.
|
||||
-- Pandas CSV loading converts '083' → 83 even with varchar column declarations.
|
||||
|
||||
MODEL (
|
||||
name foundation.dim_commodity,
|
||||
kind SEED (
|
||||
path '$root/seeds/dim_commodity.csv',
|
||||
csv_settings (delimiter = ';')
|
||||
),
|
||||
columns (
|
||||
usda_commodity_code varchar,
|
||||
cftc_commodity_code varchar,
|
||||
commodity_name varchar,
|
||||
commodity_group varchar
|
||||
)
|
||||
kind FULL
|
||||
);
|
||||
|
||||
SELECT usda_commodity_code, cftc_commodity_code, commodity_name, commodity_group
|
||||
FROM (VALUES
|
||||
('0711100', '083', 'Coffee, Green', 'Softs')
|
||||
) AS t(usda_commodity_code, cftc_commodity_code, commodity_name, commodity_group)
|
||||
|
||||
Reference in New Issue
Block a user