beanflows/vision.md

VISION.md

Mission

Build the fastest, most accurate, and most affordable commodity analytics platform for independent traders and small firms—without enterprise sales bullshit.

Product: BeanFlows.coffee

Tagline: Real-time commodity intelligence for traders who think for themselves.

Beachhead Market: Coffee commodities Long-term Vision: Expand to all major commodity markets (~35-40 global contracts)

Why We Exist

Platforms like Kpler dominate the commodity analytics space but are:

• Slow and complex
• Prohibitively expensive
• Designed for enterprise buyers with bloated sales processes
• Built on legacy infrastructure that prioritizes features over performance

We're building the anti-Kpler: better, faster, cheaper.

Who We Are

A two-person indie hacker startup:

• Data Engineer: Building the platform
• Commodity Trader: Domain expertise and product direction

We move fast, ship incrementally, and prioritize value over vanity metrics.

Technical Philosophy

Core Principles

1. Simplicity over complexity

   • Minimal dependencies
   • Clear, readable code
   • Avoid premature abstraction

2. Performance over features

   • DuckDB over Spark
   • Hetzner/Cloudflare over AWS
   • SQL/Python/C over heavyweight frameworks

3. Accuracy over speed-to-market

   • Data quality is non-negotiable
   • Rigorous validation at every layer
   • Build trust through reliability

4. Build over buy

   • We're not afraid to write code from scratch
   • Third-party tools must earn their place
   • Control our destiny, minimize vendor lock-in

Technology Stack

Languages:

• SQL (primary transformation language)
• Python (orchestration, extraction, APIs)
• C (performance-critical extensions)

Infrastructure:

• Storage: Cloudflare R2 (not S3)
• Compute: Hetzner bare metal (not AWS/GCP)
• Database: DuckDB (not Spark/Snowflake)
• Orchestration: SQLMesh + custom Python (not Airflow)

Development:

• Monorepo: uv workspace
• Package Manager: uv (not pip/poetry)
• Version Control: Git (GitLab)
• CI/CD: GitLab CI

Architectural Philosophy

Data-Oriented Design:

• No OOP spaghetti
• Data flows are explicit and traceable
• Functions transform data, not objects with hidden state

Layered Architecture:

• Raw → Staging → Cleaned → Serving
• Each layer has a single, clear purpose
• Immutable raw data, reproducible transformations

Incremental Everything:

• Models update incrementally by time ranges
• Avoid full table scans
• Pay only for what changed

Current State (February 2026)

What's Working

• USDA PSD Online extraction (2006-present, monthly archives)
• 4-layer SQLMesh pipeline (raw → staging → cleaned → serving)
• DuckDB backend (local dev + production lakehouse)
• Incremental-by-time-range models with deduplication
• Development environment with pre-commit hooks, linting, formatting
• Web app (BeanFlows.coffee) — Quart + HTMX, deployed via Docker
  • Magic-link auth + signup with waitlist flow
  • Coffee analytics dashboard: time series, top producers, stock-to-use trend, supply/demand balance, YoY change
  • Country comparison view
  • User settings + account management
  • API key management (create, revoke, prefix display)
  • Plan-based access control (free / starter / pro) with 5-year history cap on free tier
  • Billing via Paddle (subscriptions + webhooks)
  • Admin panel (users, waitlist, feedback, tasks)
  • REST API with Bearer token auth, rate limiting (1000 req/hr), CSV export
  • Feedback + waitlist capture
• GitLab CI pipeline (lint, test, build), regression tests for billing/auth/API

What We Have

• Comprehensive commodity supply/demand data (USDA PSD, 2006–present)
• Established naming conventions and data quality patterns
• Full product pipeline: data → DB → API → web dashboard
• Paddle billing integration (Starter + Pro tiers)
• Working waitlist to capture early interest

Roadmap

Phase 1: Coffee Market Foundation (In Progress → ~70% done)

Goal: Build complete coffee analytics from supply to price

Data Sources to Integrate:

• ✅ USDA PSD Online (production, stocks, consumption)
• ⬜ CFTC COT data (trader positioning — weekly, Coffee C futures code 083731)
• ⬜ Coffee futures prices — KC=F via Yahoo Finance / yfinance, or Databento for tick-level
• ⬜ ICO (International Coffee Organization) data — trade volumes, consumption stats
• ⬜ ICE certified warehouse stocks (daily CSV from ICE Report Center — free)
• ⬜ Weather data for growing regions — ECMWF/Open-Meteo (free), Brazil frost alerts

Features to Build:

• ✅ Web dashboard (supply/demand, stock-to-use trend, YoY, country comparison)
• ✅ REST API with key auth, plan-based access, rate limiting
• ✅ CSV export
• ⬜ CFTC COT integration → trader sentiment indicators
• ⬜ Historical price data → price/supply correlation analysis
• ⬜ Python SDK (pip install beanflows) — critical for the quant analyst beachhead
• ⬜ Data methodology documentation page — P0 for trust (see strategy doc)
• ⬜ Parquet export endpoint
• ⬜ Example Jupyter notebooks (show how to pipe data into common models)

Infrastructure:

• ⬜ Cloudflare R2 for raw data storage (rclone sync is partly planned)
• ⬜ Automated daily pipeline on Hetzner (SQLMesh prod + cron)
• ⬜ Pipeline monitoring + alerting (failure notifications)
• ⬜ Published SLA for data freshness

Phase 2: Product Market Fit

Goal: Validate with real traders, iterate on feedback

• ⬜ Beta access for small group of coffee traders
• ⬜ Usage analytics (what queries matter?)
• ⬜ Performance optimization based on real workloads
• ⬜ Pricing model experimentation ($X/month, pay-as-you-go?)

Phase 3: Expand Commodity Coverage

Goal: Prove architecture scales across commodities

Priority Markets:

1. Other softs (cocoa, sugar, cotton, OJ)
2. Grains (corn, wheat, soybeans)
3. Energy (crude oil, natural gas)
4. Metals (gold, silver, copper)

Reusable Patterns:

• Abstract extraction logic (API connectors, scrapers)
• Standardized staging layer for price/volume data
• Common serving models (time series, correlations, anomalies)

Phase 4: Advanced Analytics

Goal: Differentiation through unique insights

• ⬜ Satellite imagery integration (NASA, Planet) for crop monitoring
• ⬜ Custom yield forecasting models
• ⬜ Real-time alert system (price thresholds, supply shocks)
• ⬜ Historical backtesting framework for trading strategies
• ⬜ Sentiment analysis from news/reports (USDA GAIN, FAO)

Phase 5: Scale & Polish

Goal: Handle growth, maintain performance advantage

• ⬜ Multi-region deployment (low latency globally)
• ⬜ Advanced caching strategies
• ⬜ Self-service onboarding (no sales calls)
• ⬜ Public documentation and API reference
• ⬜ Community/forum for traders

Key Decisions & Trade-offs

Why DuckDB over Spark?

• Speed: In-process OLAP is faster for our workloads
• Simplicity: No cluster management, no JVM
• Cost: Runs on a single beefy server, not 100 nodes
• Developer experience: SQL-first, Python-friendly

Why SQLMesh over dbt/Airflow?

• Unified: Orchestration + transformation in one tool
• Performance: Built for incremental execution
• Virtual environments: Test changes without breaking prod
• Python-native: Extend with custom macros

Why Cloudflare R2 over S3?

• Cost: No egress fees (huge for data-heavy platform)
• Performance: Global edge network
• Simplicity: S3-compatible API, easy migration path

Why Hetzner over AWS?

• Cost: 10x cheaper for equivalent compute
• Performance: Bare metal = no noisy neighbors
• Simplicity: Less surface area, fewer services to manage

Why Monorepo?

• Atomic changes: Update extraction + transformation together
• Shared code: Reusable utilities across packages
• Simplified CI: One pipeline, consistent tooling

Anti-Goals

Things we explicitly do NOT want:

• ❌ Enterprise sales team
• ❌ Complex onboarding processes
• ❌ Vendor lock-in (AWS, Snowflake, etc.)
• ❌ OOP frameworks (Django ORM, SQLAlchemy magic)
• ❌ Microservices (until we need them, which is not now)
• ❌ Kubernetes (overkill for our scale)
• ❌ Feature bloat (every feature has a performance cost)

Success Metrics

Phase 1 (Foundation):

• All coffee data sources integrated
• Daily pipeline runs reliably (<5% failure rate)
• Query latency <500ms for common analytics

Phase 2 (PMF):

• 10+ paying beta users
• 90%+ data accuracy (validated against spot checks)
• Monthly churn <10%

Phase 3 (Expansion):

• 5+ commodity markets covered
• 100+ active users
• Break-even on infrastructure costs

Long-term (Scale):

• Cover all ~35-40 major commodity contracts
• 1000+ traders using the platform
• Recognized as the go-to alternative to Kpler for indie traders

Guiding Questions

When making decisions, ask:

1. Does this make us faster? (Performance)
2. Does this make us more accurate? (Data quality)
3. Does this make us simpler? (Maintainability)
4. Does this help traders make better decisions? (Value)
5. Can we afford to run this at scale? (Unit economics)

If the answer to any of these is "no," reconsider.

Current Priorities (Q1 2026)

Goal: Complete Phase 1 "whole product" and start beachhead outreach

Immediate (ship first):

1. CFTC COT data — extract weekly positioning data (CFTC code 083731), add to SQLMesh pipeline, expose via API. Completes the "USDA + CFTC" V1 promise from the strategy doc.
2. Coffee futures price (KC=F) — daily close via yfinance or Databento. Enables price/supply correlation in the dashboard. Core hook for trader interest.
3. Data methodology page — transparent docs for every field, every source, lineage. The #1 trust driver per the strategy doc. Required before outreach.
4. Python SDK (pip install beanflows) — one-line data access for quant analysts. The beachhead segment runs Python; this removes their biggest switching friction.

Then (before Series A of customers):

5. Automated daily pipeline on Hetzner — cron + SQLMesh prod, with failure alerting
6. Cloudflare R2 raw data backup + pipeline source
7. Example Jupyter notebooks — show before/after vs. manual WASDE workflow
8. ICE warehouse stocks — daily certified Arabica/Robusta inventory data (free from ICE Report Center)

Business (parallel, not blocking):

• Start direct outreach to 20–30 named analysts at mid-size commodity funds
• Weekly "BeanFlows Coffee Data Brief" newsletter (content marketing + credibility signal)
• Identify 1–2 early beta users willing to give feedback

---

Last Updated: February 2026 Next Review: End of Q1 2026