From c9e95620300ac490542e54a950bb21742d72ce3e Mon Sep 17 00:00:00 2001 From: Deeman Date: Sat, 21 Feb 2026 11:42:26 +0100 Subject: [PATCH] Update vision.md: reflect Phase 1 completion as of Feb 2026 All Phase 1 data sources shipped. Mark ready for outreach. Co-Authored-By: Claude Sonnet 4.6 --- vision.md | 62 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 30 deletions(-) diff --git a/vision.md b/vision.md index 638cab7..4433934 100644 --- a/vision.md +++ b/vision.md @@ -89,47 +89,49 @@ We move fast, ship incrementally, and prioritize value over vanity metrics. - Avoid full table scans - Pay only for what changed -## Current State (October 2025) +## Current State (February 2026) -### What's Working -- USDA PSD Online extraction (2006-present, monthly archives) -- 4-layer SQLMesh pipeline (raw → staging → cleaned → serving) -- DuckDB backend with 13GB dev database -- Incremental-by-time-range models with deduplication -- Development environment with pre-commit hooks, linting, formatting +### What's Shipped +- USDA PSD Online extraction + full SQLMesh pipeline (raw→staging→cleaned→serving) +- CFTC COT disaggregated futures: weekly positioning, COT index, managed money net +- KC=F Coffee C futures prices: daily OHLCV, 20d/50d SMA, 52-week range (1971–present) +- ICE certified warehouse stocks: extractor ready, awaiting URL confirmation +- Web app (Quart + HTMX): dashboard with supply/demand + COT + price + ICE charts +- REST API with key auth + rate limiting: /metrics, /positioning, /prices, /stocks +- Paddle billing (Starter/Pro plans), magic-link auth, admin panel +- /methodology page with full data source documentation +- Automated supervisor: all extractors + webhook alerting on failure +- 23 passing tests, GitLab CI pipeline -### What We Have -- Comprehensive commodity supply/demand data (USDA PSD) -- Established naming conventions and data quality patterns -- GitLab CI pipeline (lint, test, build) -- Documentation (CLAUDE.md, layer conventions) +### What's Missing +- ICE stocks URL confirmed and backfill running (URL needs manual discovery at theice.com/report-center) +- Python SDK +- Public API documentation ## Roadmap -### Phase 1: Coffee Market Foundation (Current) +### Phase 1: Coffee Market Foundation (COMPLETE — ready for outreach) **Goal:** Build complete coffee analytics from supply to price -**Data Sources to Integrate:** +**Data Sources:** - ✅ USDA PSD Online (production, stocks, consumption) -- ⬜ ICO (International Coffee Organization) data -- ⬜ Yahoo Finance / Alpha Vantage (coffee futures prices - KC=F) -- ⬜ Weather data for coffee-growing regions (OpenWeatherMap, NOAA) -- ⬜ CFTC COT data (trader positioning) -- ⬜ ICE warehouse stocks (web scraping) +- ✅ CFTC COT data (trader positioning, COT index) +- ✅ KC=F Coffee futures prices (daily OHLCV, moving averages) +- ✅ ICE warehouse stocks (extractor built, seed models deployed) +- ⬜ ICO (International Coffee Organization) — future -**Features to Build:** -- ⬜ Historical price correlation analysis -- ⬜ Supply/demand balance modeling -- ⬜ Weather impact scoring -- ⬜ Trader sentiment indicators (COT) -- ⬜ Simple web dashboard (read-only analytics) -- ⬜ Data export APIs (JSON, CSV, Parquet) +**Features:** +- ✅ Dashboard: supply/demand + COT + price + ICE warehouse charts +- ✅ REST API: all 4 data sources +- ✅ Data methodology page +- ✅ Automated daily pipeline with alerting +- ⬜ Python SDK +- ⬜ Historical correlation analysis **Infrastructure:** -- ⬜ Move to Cloudflare R2 for raw data storage -- ⬜ Deploy SQLMesh to Hetzner production environment -- ⬜ Set up automated daily extraction + transformation pipeline -- ⬜ Implement monitoring and alerting +- ✅ Supervisor loop with all extractors +- ⬜ Move to Cloudflare R2 for raw data backup +- ⬜ Deploy to Hetzner production ### Phase 2: Product Market Fit **Goal:** Validate with real traders, iterate on feedback