deemanone/beanflows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update vision.md: reflect Phase 1 completion as of Feb 2026

Browse Source 
All Phase 1 data sources shipped. Mark ready for outreach.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-21 11:42:26 +01:00
parent 67c048485b
commit c9e9562030
1 changed files with 32 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
vision.md
View File

@@ -89,47 +89,49 @@ We move fast, ship incrementally, and prioritize value over vanity metrics.
- Avoid full table scans
- Pay only for what changed
## Current State (October 2025)
## Current State (February 2026)
### What's Working
- USDA PSD Online extraction (2006-present, monthly archives)
- 4-layer SQLMesh pipeline (raw → staging → cleaned → serving)
- DuckDB backend with 13GB dev database
- Incremental-by-time-range models with deduplication
- Development environment with pre-commit hooks, linting, formatting
### What's Shipped
- USDA PSD Online extraction + full SQLMesh pipeline (raw→staging→cleaned→serving)
- CFTC COT disaggregated futures: weekly positioning, COT index, managed money net
- KC=F Coffee C futures prices: daily OHLCV, 20d/50d SMA, 52-week range (1971–present)
- ICE certified warehouse stocks: extractor ready, awaiting URL confirmation
- Web app (Quart + HTMX): dashboard with supply/demand + COT + price + ICE charts
- REST API with key auth + rate limiting: /metrics, /positioning, /prices, /stocks
- Paddle billing (Starter/Pro plans), magic-link auth, admin panel
- /methodology page with full data source documentation
- Automated supervisor: all extractors + webhook alerting on failure
- 23 passing tests, GitLab CI pipeline
### What We Have
- Comprehensive commodity supply/demand data (USDA PSD)
- Established naming conventions and data quality patterns
- GitLab CI pipeline (lint, test, build)
- Documentation (CLAUDE.md, layer conventions)
### What's Missing
- ICE stocks URL confirmed and backfill running (URL needs manual discovery at theice.com/report-center)
- Python SDK
- Public API documentation
## Roadmap
### Phase 1: Coffee Market Foundation (Current)
### Phase 1: Coffee Market Foundation (COMPLETE — ready for outreach)
**Goal:** Build complete coffee analytics from supply to price
**Data Sources to Integrate:**
**Data Sources:**
- ✅ USDA PSD Online (production, stocks, consumption)
- ⬜ ICO (International Coffee Organization) data
- ⬜ Yahoo Finance / Alpha Vantage (coffee futures prices - KC=F)
- ⬜ Weather data for coffee-growing regions (OpenWeatherMap, NOAA)
- ⬜ CFTC COT data (trader positioning)
- ⬜ ICE warehouse stocks (web scraping)
- ✅ CFTC COT data (trader positioning, COT index)
- ✅ KC=F Coffee futures prices (daily OHLCV, moving averages)
- ✅ ICE warehouse stocks (extractor built, seed models deployed)
- ⬜ ICO (International Coffee Organization) — future
**Features to Build:**
- ⬜ Historical price correlation analysis
- ⬜ Supply/demand balance modeling
- ⬜ Weather impact scoring
- ⬜ Trader sentiment indicators (COT)
- ⬜ Simple web dashboard (read-only analytics)
- ⬜ Data export APIs (JSON, CSV, Parquet)
**Features:**
- ✅ Dashboard: supply/demand + COT + price + ICE warehouse charts
- ✅ REST API: all 4 data sources
- ✅ Data methodology page
- ✅ Automated daily pipeline with alerting
- ⬜ Python SDK
- ⬜ Historical correlation analysis
**Infrastructure:**
- ⬜ Move to Cloudflare R2 for raw data storage
- ⬜ Deploy SQLMesh to Hetzner production environment
- ⬜ Set up automated daily extraction + transformation pipeline
- ⬜ Implement monitoring and alerting
- ✅ Supervisor loop with all extractors
- ⬜ Move to Cloudflare R2 for raw data backup
- ⬜ Deploy to Hetzner production
### Phase 2: Product Market Fit
**Goal:** Validate with real traders, iterate on feedback