67c048485b
Phase 1A — KC=F Coffee Futures Prices: - New extract/coffee_prices/ package (yfinance): downloads KC=F daily OHLCV, stores as gzip CSV with SHA256-based idempotency - SQLMesh models: raw/coffee_prices → foundation/fct_coffee_prices → serving/coffee_prices (with 20d/50d SMA, 52-week high/low, daily return %) - Dashboard: 4 metric cards + dual-line chart (close, 20d MA, 50d MA) - API: GET /commodities/<ticker>/prices Phase 1B — Data Methodology Page: - New /methodology route with full-page template (base.html) - 6 anchored sections: USDA PSD, CFTC COT, KC=F price, ICE warehouse stocks, data quality model, update schedule table - "Methodology" link added to marketing footer Phase 1C — Automated Pipeline: - supervisor.sh updated: runs extract_cot, extract_prices, extract_ice in sequence before transform - Webhook failure alerting via ALERT_WEBHOOK_URL env var (ntfy/Slack/Telegram) ICE Warehouse Stocks: - New extract/ice_stocks/ package (niquests): normalizes ICE Report Center CSV to canonical schema, hash-based idempotency, soft-fail on 404 with guidance - SQLMesh models: raw/ice_warehouse_stocks → foundation/fct_ice_warehouse_stocks → serving/ice_warehouse_stocks (30d avg, WoW change, 52w drawdown) - Dashboard: 4 metric cards + line chart (certified bags + 30d avg) - API: GET /commodities/<code>/stocks Foundation: - dim_commodity: added ticker (KC=F) and ice_stock_report_code (COFFEE-C) columns - macros/__init__.py: added prices_glob() and ice_stocks_glob() - pipelines.py: added extract_prices and extract_ice entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
24 lines
1.1 KiB
SQL
24 lines
1.1 KiB
SQL
-- Commodity dimension: conforms identifiers across source systems.
|
|
--
|
|
-- This is the ontology. Each row is a commodity tracked by BeanFlows.
|
|
-- As new sources are added (ICO, futures prices, satellite), their
|
|
-- commodity identifiers are added as columns here — not as separate tables.
|
|
-- As new commodities are added (cocoa, sugar), rows are added here.
|
|
--
|
|
-- References:
|
|
-- usda_commodity_code → raw.psd_alldata.commodity_code (numeric string, e.g. '0711100')
|
|
-- cftc_commodity_code → raw.cot_disaggregated.cftc_commodity_code (3-char, e.g. '083')
|
|
--
|
|
-- NOTE: Defined as FULL model (not SEED) to guarantee leading-zero preservation.
|
|
-- Pandas CSV loading converts '083' → 83 even with varchar column declarations.
|
|
|
|
MODEL (
|
|
name foundation.dim_commodity,
|
|
kind FULL
|
|
);
|
|
|
|
SELECT usda_commodity_code, cftc_commodity_code, ticker, ice_stock_report_code, commodity_name, commodity_group
|
|
FROM (VALUES
|
|
('0711100', '083', 'KC=F', 'COFFEE-C', 'Coffee, Green', 'Softs')
|
|
) AS t(usda_commodity_code, cftc_commodity_code, ticker, ice_stock_report_code, commodity_name, commodity_group)
|