Add Phase 1A-C + ICE warehouse stocks: prices, methodology, pipeline automation
Phase 1A — KC=F Coffee Futures Prices: - New extract/coffee_prices/ package (yfinance): downloads KC=F daily OHLCV, stores as gzip CSV with SHA256-based idempotency - SQLMesh models: raw/coffee_prices → foundation/fct_coffee_prices → serving/coffee_prices (with 20d/50d SMA, 52-week high/low, daily return %) - Dashboard: 4 metric cards + dual-line chart (close, 20d MA, 50d MA) - API: GET /commodities/<ticker>/prices Phase 1B — Data Methodology Page: - New /methodology route with full-page template (base.html) - 6 anchored sections: USDA PSD, CFTC COT, KC=F price, ICE warehouse stocks, data quality model, update schedule table - "Methodology" link added to marketing footer Phase 1C — Automated Pipeline: - supervisor.sh updated: runs extract_cot, extract_prices, extract_ice in sequence before transform - Webhook failure alerting via ALERT_WEBHOOK_URL env var (ntfy/Slack/Telegram) ICE Warehouse Stocks: - New extract/ice_stocks/ package (niquests): normalizes ICE Report Center CSV to canonical schema, hash-based idempotency, soft-fail on 404 with guidance - SQLMesh models: raw/ice_warehouse_stocks → foundation/fct_ice_warehouse_stocks → serving/ice_warehouse_stocks (30d avg, WoW change, 52w drawdown) - Dashboard: 4 metric cards + line chart (certified bags + 30d avg) - API: GET /commodities/<code>/stocks Foundation: - dim_commodity: added ticker (KC=F) and ice_stock_report_code (COFFEE-C) columns - macros/__init__.py: added prices_glob() and ice_stocks_glob() - pipelines.py: added extract_prices and extract_ice entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
46
transform/sqlmesh_materia/models/raw/coffee_prices.sql
Normal file
46
transform/sqlmesh_materia/models/raw/coffee_prices.sql
Normal file
@@ -0,0 +1,46 @@
|
||||
-- Raw KC=F Coffee C futures prices — technical ingestion layer.
|
||||
--
|
||||
-- Reads daily OHLCV gzip CSVs from the landing directory. All values are
|
||||
-- varchar; casting happens in foundation.fct_coffee_prices.
|
||||
--
|
||||
-- Source: Yahoo Finance via yfinance (KC=F ticker)
|
||||
-- Coverage: 1971-present (historical futures data)
|
||||
-- Frequency: daily (trading days only)
|
||||
|
||||
MODEL (
|
||||
name raw.coffee_prices,
|
||||
kind FULL,
|
||||
grain (Date),
|
||||
cron '@daily',
|
||||
columns (
|
||||
Date varchar,
|
||||
Open varchar,
|
||||
High varchar,
|
||||
Low varchar,
|
||||
Close varchar,
|
||||
Adj_Close varchar,
|
||||
Volume varchar,
|
||||
filename varchar
|
||||
)
|
||||
);
|
||||
|
||||
SELECT
|
||||
"Date" AS Date,
|
||||
"Open" AS Open,
|
||||
"High" AS High,
|
||||
"Low" AS Low,
|
||||
"Close" AS Close,
|
||||
"Adj Close" AS Adj_Close,
|
||||
"Volume" AS Volume,
|
||||
filename
|
||||
FROM read_csv(
|
||||
@prices_glob(),
|
||||
delim = ',',
|
||||
encoding = 'utf-8',
|
||||
compression = 'gzip',
|
||||
header = true,
|
||||
union_by_name = true,
|
||||
filename = true,
|
||||
all_varchar = true,
|
||||
ignore_errors = true
|
||||
)
|
||||
@@ -0,0 +1,37 @@
|
||||
-- Raw ICE certified warehouse stocks — technical ingestion layer.
|
||||
--
|
||||
-- Reads daily stock report gzip CSVs from the landing directory.
|
||||
-- All values are varchar; casting happens in foundation.fct_ice_warehouse_stocks.
|
||||
--
|
||||
-- Source: ICE Report Center (Coffee C certified warehouse stocks)
|
||||
-- Coverage: varies by download history
|
||||
-- Frequency: daily (ICE updates after market close)
|
||||
|
||||
MODEL (
|
||||
name raw.ice_warehouse_stocks,
|
||||
kind FULL,
|
||||
cron '@daily',
|
||||
columns (
|
||||
report_date varchar,
|
||||
total_certified_bags varchar,
|
||||
pending_grading_bags varchar,
|
||||
filename varchar
|
||||
)
|
||||
);
|
||||
|
||||
SELECT
|
||||
report_date,
|
||||
total_certified_bags,
|
||||
pending_grading_bags,
|
||||
filename
|
||||
FROM read_csv(
|
||||
@ice_stocks_glob(),
|
||||
delim = ',',
|
||||
encoding = 'utf-8',
|
||||
compression = 'gzip',
|
||||
header = true,
|
||||
union_by_name = true,
|
||||
filename = true,
|
||||
all_varchar = true,
|
||||
ignore_errors = true
|
||||
)
|
||||
Reference in New Issue
Block a user