Update vision.md: reflect Phase 1 completion as of Feb 2026
All Phase 1 data sources shipped. Mark ready for outreach. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
62
vision.md
62
vision.md
@@ -89,47 +89,49 @@ We move fast, ship incrementally, and prioritize value over vanity metrics.
|
|||||||
- Avoid full table scans
|
- Avoid full table scans
|
||||||
- Pay only for what changed
|
- Pay only for what changed
|
||||||
|
|
||||||
## Current State (October 2025)
|
## Current State (February 2026)
|
||||||
|
|
||||||
### What's Working
|
### What's Shipped
|
||||||
- USDA PSD Online extraction (2006-present, monthly archives)
|
- USDA PSD Online extraction + full SQLMesh pipeline (raw→staging→cleaned→serving)
|
||||||
- 4-layer SQLMesh pipeline (raw → staging → cleaned → serving)
|
- CFTC COT disaggregated futures: weekly positioning, COT index, managed money net
|
||||||
- DuckDB backend with 13GB dev database
|
- KC=F Coffee C futures prices: daily OHLCV, 20d/50d SMA, 52-week range (1971–present)
|
||||||
- Incremental-by-time-range models with deduplication
|
- ICE certified warehouse stocks: extractor ready, awaiting URL confirmation
|
||||||
- Development environment with pre-commit hooks, linting, formatting
|
- Web app (Quart + HTMX): dashboard with supply/demand + COT + price + ICE charts
|
||||||
|
- REST API with key auth + rate limiting: /metrics, /positioning, /prices, /stocks
|
||||||
|
- Paddle billing (Starter/Pro plans), magic-link auth, admin panel
|
||||||
|
- /methodology page with full data source documentation
|
||||||
|
- Automated supervisor: all extractors + webhook alerting on failure
|
||||||
|
- 23 passing tests, GitLab CI pipeline
|
||||||
|
|
||||||
### What We Have
|
### What's Missing
|
||||||
- Comprehensive commodity supply/demand data (USDA PSD)
|
- ICE stocks URL confirmed and backfill running (URL needs manual discovery at theice.com/report-center)
|
||||||
- Established naming conventions and data quality patterns
|
- Python SDK
|
||||||
- GitLab CI pipeline (lint, test, build)
|
- Public API documentation
|
||||||
- Documentation (CLAUDE.md, layer conventions)
|
|
||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
|
|
||||||
### Phase 1: Coffee Market Foundation (Current)
|
### Phase 1: Coffee Market Foundation (COMPLETE — ready for outreach)
|
||||||
**Goal:** Build complete coffee analytics from supply to price
|
**Goal:** Build complete coffee analytics from supply to price
|
||||||
|
|
||||||
**Data Sources to Integrate:**
|
**Data Sources:**
|
||||||
- ✅ USDA PSD Online (production, stocks, consumption)
|
- ✅ USDA PSD Online (production, stocks, consumption)
|
||||||
- ⬜ ICO (International Coffee Organization) data
|
- ✅ CFTC COT data (trader positioning, COT index)
|
||||||
- ⬜ Yahoo Finance / Alpha Vantage (coffee futures prices - KC=F)
|
- ✅ KC=F Coffee futures prices (daily OHLCV, moving averages)
|
||||||
- ⬜ Weather data for coffee-growing regions (OpenWeatherMap, NOAA)
|
- ✅ ICE warehouse stocks (extractor built, seed models deployed)
|
||||||
- ⬜ CFTC COT data (trader positioning)
|
- ⬜ ICO (International Coffee Organization) — future
|
||||||
- ⬜ ICE warehouse stocks (web scraping)
|
|
||||||
|
|
||||||
**Features to Build:**
|
**Features:**
|
||||||
- ⬜ Historical price correlation analysis
|
- ✅ Dashboard: supply/demand + COT + price + ICE warehouse charts
|
||||||
- ⬜ Supply/demand balance modeling
|
- ✅ REST API: all 4 data sources
|
||||||
- ⬜ Weather impact scoring
|
- ✅ Data methodology page
|
||||||
- ⬜ Trader sentiment indicators (COT)
|
- ✅ Automated daily pipeline with alerting
|
||||||
- ⬜ Simple web dashboard (read-only analytics)
|
- ⬜ Python SDK
|
||||||
- ⬜ Data export APIs (JSON, CSV, Parquet)
|
- ⬜ Historical correlation analysis
|
||||||
|
|
||||||
**Infrastructure:**
|
**Infrastructure:**
|
||||||
- ⬜ Move to Cloudflare R2 for raw data storage
|
- ✅ Supervisor loop with all extractors
|
||||||
- ⬜ Deploy SQLMesh to Hetzner production environment
|
- ⬜ Move to Cloudflare R2 for raw data backup
|
||||||
- ⬜ Set up automated daily extraction + transformation pipeline
|
- ⬜ Deploy to Hetzner production
|
||||||
- ⬜ Implement monitoring and alerting
|
|
||||||
|
|
||||||
### Phase 2: Product Market Fit
|
### Phase 2: Product Market Fit
|
||||||
**Goal:** Validate with real traders, iterate on feedback
|
**Goal:** Validate with real traders, iterate on feedback
|
||||||
|
|||||||
Reference in New Issue
Block a user