diff --git a/CHANGELOG.md b/CHANGELOG.md index 882fc62..2defe28 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,46 @@ All notable changes to BeanFlows are documented here. ## [Unreleased] +### Added +- **ICE certified stock aging report** — Monthly age-bucket × port breakdown extracted via ICE API, stored as gzip CSV, modelled through raw→foundation→serving, exposed at `GET /api/v1/commodities//stocks/aging` +- **ICE historical warehouse stocks by port** — End-of-month data from Nov 1996 to present, downloaded from static ICE URL, full SQLMesh pipeline, exposed at `GET /api/v1/commodities//stocks/by-port` +- **ICE API-based URL discovery** — Replaces brittle `ICE_STOCKS_URL` env var; queries ICE Report Center JSON API, finds latest report by label, falls back automatically on 404 +- **XLS parsing** (`ice_stocks/xls_parse.py`) — Magic-byte format detection (`OLE2_MAGIC`) and xlrd-based row extraction; handles both CSV and XLS responses from ICE +- **`extract_ice_all`** CLI command — Runs all three ICE extractors (daily stocks, aging, by-port) in one command +- **`extract_all`** meta-pipeline — Sequences all four data source extractors (`extract`, `extract_cot`, `extract_prices`, `extract_ice_all`), stops on first failure +- **ICE aging dashboard section** — Metric cards (total bags, youngest %, oldest %, bucket count) + stacked horizontal bar chart (age buckets × ports) +- **ICE by-port dashboard section** — Metric cards (total, MoM change, 12m avg, 30yr history) + stacked area chart (30-year view by delivery port) +- **Origin Intelligence page** — Full redesign of country comparison: dark espresso selector panel + metric segment tabs + colored chips + Chart.js line chart + live-built rankings table with proportional bars +- **HATEOAS country comparison** — Clicking an origin immediately fetches an HTMX partial and swaps the canvas; URL updates via `history.pushState()`; no Apply button needed +- **Axis labels on all charts** — Y-axis titles on every overview and Origins chart; column unit header in YoY and rankings tables + +### Fixed +- **ICE XLS date parsing** — Handle `"Feb 20, 2026 1:35:39PM"` format (previously only `"1/30/2026"` worked); tries multiple `strptime` formats across single-token and three-token date candidates + +### Changed +- **Countries nav item renamed to "Origins"** — Matches "Origin Intelligence" page heading in both sidebar and mobile bottom nav +- **`dim_commodity`** — Added `ticker` and `ice_stock_report_code` columns (`KC=F`, `COFFEE-C`) +- **SQLMesh macros** — Added `ice_aging_glob()` and `ice_stocks_by_port_glob()` alongside existing `ice_stocks_glob()` + +## [0.2.0] - 2026-02-01 + +### Added +- **CFTC COT disaggregated futures** — Weekly trader positioning (managed money net, COT index 26w/52w, % of open interest); full SQLMesh pipeline; `GET /api/v1/commodities//positioning` +- **KC=F Coffee C futures prices** — Daily OHLCV from Yahoo Finance (1971–present), 20d/50d SMA, 52-week range; `GET /api/v1/commodities//prices` +- **ICE certified warehouse stocks (daily rolling)** — Certified Arabica bags, WoW change, 30d average, 52-week drawdown; `GET /api/v1/commodities//stocks` +- **COT positioning dashboard section** — Metric cards + dual-axis line chart (managed money net + COT index) +- **Price dashboard section** — Metric cards + three-line chart (close, 20d MA, 50d MA) +- **ICE daily stocks dashboard section** — Metric cards + area chart with 30d MA +- **`/methodology` page** — Full data source documentation: lineage, field definitions, update cadence for every source +- **Automated supervisor pipeline** — Systemd service runs all extractors daily; Discord/webhook alerting on failure +- **`extract_all` meta-pipeline precursor** — Individual pipeline entries for all four data sources + +### Fixed +- **COT pipeline nulls** — `TRY_CAST` instead of `CAST` in foundation model; leading zeros in CFTC commodity codes preserved +- **Concurrent DuckDB queries** — `cursor()` per thread prevents connection state corruption + +## [0.1.0] - 2025-12-01 + ### Added - **Coffee analytics dashboard** — Chart.js visualizations for global supply/demand time series, stock-to-use ratio trend, top producing countries bar chart, and YoY production change table - **Country comparison page** — Multi-select country picker with metric selector and overlay line chart at `/dashboard/countries` @@ -13,20 +53,16 @@ All notable changes to BeanFlows are documented here. - **Landing, features, and pricing pages** rewritten for coffee market intelligence positioning - **Health check** now verifies both SQLite and DuckDB connectivity - **Admin panel** shows commodity count and data year range from DuckDB -- **Docker** config updated with DuckDB volume mount - **Tests** — `test_dashboard.py` (8 tests), `test_api_commodities.py` (8 tests), analytics mock fixture in conftest ### Fixed -- **Pipeline time granularity** — Added `market_year` to GROUP BY in `cleaned.psdalldata__commodity_pivoted` and carried through to `serving.commodity_metrics`. Previously summed across all market years, making per-year metrics meaningless. +- **Pipeline time granularity** — Added `market_year` to GROUP BY in `cleaned.psdalldata__commodity_pivoted`; previously summed across all market years making per-year metrics meaningless ### Removed -- `items` table, FTS virtual table, and triggers from schema (boilerplate domain entity) +- `items` table, FTS virtual table, and triggers from schema (boilerplate) - Items CRUD from API routes -- Items count from dashboard stats ### Changed -- `PLAN_FEATURES` updated from generic (`basic`, `export`) to domain-specific (`dashboard`, `coffee_only`, `all_commodities`, `full_history`, `export`, `api`) +- `PLAN_FEATURES` updated from generic to domain-specific scopes - `PLAN_LIMITS` changed from `items`/`api_calls` to `commodities`/`history_years`/`api_calls` - API now requires Starter or Pro plan (free plan gets 403) -- Dashboard routes and API routes import analytics module (not individual functions) for testability -- Billing tests updated to match new feature and limit names