Commit Graph

30 Commits

Author SHA1 Message Date
Deeman
32e54f0381 countries: HATEOAS + HTMX — click origin to update chart instantly
Replace Apply button flow with immediate HTMX partial fetches:
- toggleCountry() does an optimistic UI update (row + badge) then
  calls htmx.ajax() targeting #cc-canvas with swap=innerHTML
- URL is pushed to history on every selection change (bookmarkable)
- HX-Request now returns countries_canvas.html fragment (chips +
  chart/empty + inline IIFE that re-syncs globals + re-inits Chart.js)
- Panel (dark) is never swapped; canvas fades during in-flight request
- PALETTE, buildRankings(), initChart() defined once on page load,
  called by both initial render and partial IIFE after each swap
- Apply button removed; Clear triggers fetchCanvas() with empty codes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 22:40:59 +01:00
Deeman
91a9fb83be redesign Countries page: commodity intelligence terminal aesthetic
Replace generic multi-select + plain card with a two-panel layout:
- Dark espresso selector panel (sticky, searchable, click-to-toggle)
  with country rows showing rank, name, production figure, checkbox
- Right canvas: metric segment tabs, selected-country chips (colored),
  Chart.js line chart with dark espresso tooltip, and a JS-built
  rankings table with proportional colored bars (latest year)
- Smooth fade-in animations, monospaced figures, copper accent palette

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 22:30:37 +01:00
Deeman
ff956b0138 ICE aging + by-port: serving models, API endpoints, dashboard integration
- serving/ice_aging_stocks.sql: pass-through from foundation, parses age
  bucket string to start/end days ints for correct sort order
- serving/ice_warehouse_stocks_by_port.sql: monthly by-port since 1996,
  adds MoM change, MoM %, 12-month rolling average
- analytics.py: get_ice_aging_latest(), get_ice_aging_trend(),
  get_ice_stocks_by_port_trend(), get_ice_stocks_by_port_latest()
- api/routes.py: GET /commodities/<code>/stocks/aging and
  GET /commodities/<code>/stocks/by-port with auth + rate limiting
- dashboard/routes.py: add 3 new queries to asyncio.gather(), pass to template
- index.html: aging stacked bar chart (age buckets × port) with 4 metric
  cards; by-port stacked area chart (30-year history) with 4 metric cards

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 21:52:35 +01:00
Deeman
3d3f375e01 Merge worktree-cot-integration: Phase 1 + scout MCP server
- Phase 1A-C: KC=F price extraction, SQLMesh models, dashboard charts, API endpoints
- ICE warehouse stocks: extraction package, SQLMesh models, dashboard + API
- Methodology page (/methodology) with all data sources documented
- Supervisor pipeline automation with webhook alerting
- Scout MCP server (tools/scout/) for browser recon via Pydoll
- msgspec added as workspace dependency for typed boundary structs
- vision.md updated to reflect Phase 1 completion (Feb 2026)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 15:57:49 +01:00
Deeman
67c048485b Add Phase 1A-C + ICE warehouse stocks: prices, methodology, pipeline automation
Phase 1A — KC=F Coffee Futures Prices:
- New extract/coffee_prices/ package (yfinance): downloads KC=F daily OHLCV,
  stores as gzip CSV with SHA256-based idempotency
- SQLMesh models: raw/coffee_prices → foundation/fct_coffee_prices →
  serving/coffee_prices (with 20d/50d SMA, 52-week high/low, daily return %)
- Dashboard: 4 metric cards + dual-line chart (close, 20d MA, 50d MA)
- API: GET /commodities/<ticker>/prices

Phase 1B — Data Methodology Page:
- New /methodology route with full-page template (base.html)
- 6 anchored sections: USDA PSD, CFTC COT, KC=F price, ICE warehouse stocks,
  data quality model, update schedule table
- "Methodology" link added to marketing footer

Phase 1C — Automated Pipeline:
- supervisor.sh updated: runs extract_cot, extract_prices, extract_ice in
  sequence before transform
- Webhook failure alerting via ALERT_WEBHOOK_URL env var (ntfy/Slack/Telegram)

ICE Warehouse Stocks:
- New extract/ice_stocks/ package (niquests): normalizes ICE Report Center CSV
  to canonical schema, hash-based idempotency, soft-fail on 404 with guidance
- SQLMesh models: raw/ice_warehouse_stocks → foundation/fct_ice_warehouse_stocks
  → serving/ice_warehouse_stocks (30d avg, WoW change, 52w drawdown)
- Dashboard: 4 metric cards + line chart (certified bags + 30d avg)
- API: GET /commodities/<code>/stocks

Foundation:
- dim_commodity: added ticker (KC=F) and ice_stock_report_code (COFFEE-C) columns
- macros/__init__.py: added prices_glob() and ice_stocks_glob()
- pipelines.py: added extract_prices and extract_ice entries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 11:41:43 +01:00
Deeman
1a39082514 Add sidenav layout for authenticated dashboard
- Create dashboard_base.html: standalone app shell with 56px sticky
  header (logo + user email + sign out), 220px left sidebar with
  Overview/Countries/Settings nav items (SVG icons, active state via
  request.path), and fixed mobile bottom tab bar (md:hidden)
- Add CSS component classes: .app-shell, .app-header, .app-sidebar,
  .sidebar-item, .app-content, .mobile-bottom-nav, .mobile-nav-item
- Extract feedback widget into _feedback_widget.html partial; include
  from both base.html and dashboard_base.html
- Switch index.html, countries.html, settings.html to extend
  dashboard_base.html; remove <main class="container-page"> wrappers
- Remove "Back to Dashboard" button from countries.html (sidebar
  provides persistent navigation)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 01:15:25 +01:00
Deeman
4dcf1e7e84 Fix dashboard error handling, settings billing route, update vision.md
- routes.py: return_exceptions=True on gather, log individual query failures
  with per-result defaults so one bad query doesn't blank the whole page
- settings.html: fix billing.portal → billing.manage (correct blueprint route)
- vision.md: update current state to February 2026, document shipped features

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 00:02:41 +01:00
Deeman
88e408b279 Make dashboard gather resilient to missing analytics tables
Use return_exceptions=True so a CatalogException from a single query
(e.g. table not yet populated in a fresh env) degrades gracefully
instead of crashing the whole dashboard render.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 23:33:39 +01:00
Deeman
2962bf5e3b Fix COT pipeline: TRY_CAST nulls, dim_commodity leading zeros, correct CFTC codes
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
  Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
  to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
  and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data

Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 23:28:10 +01:00
Deeman
0a83b2cb74 Add CFTC COT data integration with foundation data model layer
- New extraction package (cftc_cot): downloads yearly Disaggregated Futures ZIPs
  from CFTC, etag-based dedup, dynamic inner filename discovery, gzip normalization
- SQLMesh 3-layer architecture: raw (technical) → foundation (business model) → serving (mart)
- dim_commodity seed: conformed dimension mapping USDA ↔ CFTC codes — the commodity ontology
- fct_cot_positioning: typed, deduplicated weekly positioning facts for all commodities
- obt_cot_positioning: Coffee C mart with COT Index (26w/52w), WoW delta, OI ratios
- Analytics functions + REST API endpoints: /commodities/<code>/positioning[/latest]
- Dashboard widget: Managed Money net, COT Index card, dual-axis Chart.js chart
- 23 passing tests (10 unit + 2 SQLMesh model + existing regression suite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 23:28:10 +01:00
Deeman
d09ba91023 Remove password admin login, seed dev accounts, add regression tests
Admin flow:
- Remove /admin/login (password-based) and /admin/dev-login routes entirely
- admin_required now checks only the 'admin' role; redirects to auth.login
- auth/dev-login with an ADMIN_EMAILS address redirects directly to /admin/
- .env.example: replace ADMIN_PASSWORD with ADMIN_EMAILS=admin@beanflows.coffee

Dev seeding:
- Add dev_seed.py: idempotent upsert of 4 fixed accounts (admin, free,
  starter, pro) so every access tier is testable after dev_run.sh
- dev_run.sh: seed after migrations, show all 4 login shortcuts

Regression tests (37 passing):
- test_analytics.py: concurrent fetch_analytics calls return correct row
  counts (cursor thread-safety regression), column names are lowercase
- test_roles.py TestAdminAuthFlow: password login routes return 404,
  admin_required redirects to auth.login, dev-login grants admin role
  and redirects to admin panel when email is in ADMIN_EMAILS
- conftest.py: add mock_analytics fixture (fixes 7 pre-existing dashboard
  test errors); fix assertion text and lowercase metric param in tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 20:10:45 +01:00
Deeman
fef9f3d705 Fix concurrent DuckDB queries: use cursor() per thread
_conn.execute() is not thread-safe for concurrent calls from multiple
threads. asyncio.gather submits each analytics query to the thread pool
via asyncio.to_thread, causing race conditions that silently returned
empty result sets. _conn.cursor() creates an independent cursor that is
safe to use from separate threads simultaneously.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 19:04:09 +01:00
Deeman
d569ba0162 Fix metric column name casing: DuckDB returns lowercase, align everywhere
SQLMesh normalizes unquoted identifiers to lowercase in physical tables,
so commodity_metrics columns are e.g. 'production' not 'Production'.
Update ALLOWED_METRICS, all analytics.py SQL queries, dashboard routes,
and both dashboard templates (Jinja + JS chart references) to use
lowercase column names consistently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 17:14:52 +01:00
Deeman
d05e522c88 Add migration 0001: create feedback and waitlist tables
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 15:36:40 +01:00
Deeman
8e7af53ff6 Fix admin auth, impersonation session handling, and stale stripe column
- admin_required now accepts users with 'admin' role (via g.user) in
  addition to the password-based is_admin session flag, so both auth
  methods grant access
- impersonate stores the admin's user_id (not True) in admin_impersonating
  so stop-impersonating can restore the correct session
- stop_impersonating restores user_id from admin_impersonating instead of
  just popping it
- remove s.stripe_customer_id from get_user_by_id (Paddle project, no
  stripe_customer_id column in subscriptions)

Fixes 3 test_roles.py failures: test_admin_index_accessible_with_admin_role,
test_impersonate_stores_admin_id, test_stop_impersonating_restores_admin

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 02:33:10 +01:00
Deeman
48bea5c198 Add admin sidebar layout, feedback, waitlist, sitemap to BeanFlows web
- Admin sidebar layout (base_admin.html) with espresso/copper coffee theme,
  220px sidebar, responsive collapse, nav for Dashboard/Users/Tasks/Feedback/Waitlist
- Convert all admin templates to extend base_admin.html using Tailwind classes
- Feedback system: schema, public POST route (rate-limited), base.html widget
  with HTMX popover (coffee-themed), admin viewer with mark-read
- Waitlist mode: WAITLIST_MODE config, waitlist_gate decorator,
  capture_waitlist_email helper, auth route integration, confirmation pages,
  send_waitlist_confirmation worker task, admin table
- Sitemap.xml and robots.txt public routes
- Dashboard stats updated with waitlist_count, feedback_unread alongside
  existing commodity DuckDB analytics stats

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 02:27:26 +01:00
Deeman
e80e262e25 Fix NoneType error when user has no subscription
g.subscription is explicitly set to None in load_user, so
g.get("subscription", {}) returns None (key exists), not {}.
Use (g.get(...) or {}) to coalesce None to an empty dict.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 23:10:56 +01:00
Deeman
642b529c4d Fix column name mismatch after copier template update
The subscriptions table still had paddle_subscription_id but the new
code references provider_subscription_id.  Renamed the DB column and
updated all queries in billing/routes.py to match.

Also removed unused get_subscription import from dashboard/routes.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 23:09:12 +01:00
Deeman
94e8a5e6c3 Fix dev-login URLs in development script
- Update dev_run.sh with better dev-login URLs including appropriate email parameters
- Add user-login URL: auth/dev-login?email=trader@beanflows.coffee
- Add admin-login URL: auth/dev-login?email=admin@beanflows.coffee
- Keep admin-panel URL: admin/dev-login for direct admin session
- Add ADMIN_EMAILS config to .env.example for auto-granting admin role

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:55:57 +01:00
Deeman
18c6e0da4f Fix dashboard routes after copier update
- Remove import of get_user_with_subscription (function was removed)
- Use g.user and g.subscription from eager loading instead
- Fixes ImportError in dashboard routes

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:48:53 +01:00
Deeman
edd439245b Merge branch 'copier-update' 2026-02-19 22:46:41 +01:00
Deeman
32132974b2 Clean up web changes and add favicon
- Update uv.lock dependencies
- Remove web/CLAUDE.md (moved to root)
- Update base.html template
- Add favicon.svg

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:46:33 +01:00
Deeman
3f1cd8bd0c Update copier answers and docker-compose prod config
- Record v0.4.0 commit in .copier-answers.yml
- Apply flattened paths in docker-compose.prod.yml

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:35:55 +01:00
Deeman
4b7d4d5a74 Update from Copier template v0.4.0
- Accept RBAC system: user_roles table, role_required decorator, grant_role/revoke_role/ensure_admin_role functions
- Accept improved billing architecture: billing_customers table separation, provider-agnostic naming
- Accept enhanced user loading with subscription/roles eager loading in app.py
- Accept improved email templates with branded styling
- Accept new infrastructure: migration tracking, transaction logging, A/B testing
- Accept template improvements: Resend SDK, Tailwind build stage, UMAMI analytics config
- Keep beanflows-specific configs: BASE_URL 5001, coffee PLAN_FEATURES/PLAN_LIMITS
- Keep beanflows analytics integration and DuckDB health check
- Add new test files and utility scripts from template

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:22:13 +01:00
Deeman
1e8a173ae8 Merge branch 'frontend-upgrade'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:49:07 +01:00
Deeman
f722854c07 Rewrite frontend templates: Pico CSS → Tailwind + trader-focused copy
Replace all Pico CSS patterns (classless articles, role="button", inline
styles, var(--pico-*)) with Tailwind component classes. Add Fraunces
display font, mobile hamburger nav, brand chart colors, and new component
layer (hero, feature-card, metric-card, auth-card, pricing-card, etc.).

Rewrite marketing copy from generic SaaS boilerplate to coffee-trader
focused messaging. Rebrand pricing tiers to Explorer/Trader/Analyst.
Delete stale custom.css. No Python code changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:47:56 +01:00
Deeman
6dac8570ad Fix web/ startup errors and sync with boilerplate
- Load .env via python-dotenv in core.py
- Skip analytics DB open if file doesn't exist
- Guard dashboard analytics calls when DB not available
- Namespace admin templates under admin/ to avoid blueprint conflicts
- Add dev-login routes for user and admin (DEBUG only)
- Update .copier-answers.yml src_path to GitLab remote

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:37:44 +01:00
Deeman
d6d2aa8efe Merge remote-tracking branch 'origin/master'
# Conflicts:
#	infra/readme.md
2026-02-18 21:09:24 +01:00
Deeman
2748c606e9 Add BeanFlows MVP: coffee analytics dashboard, API, and web app
- Fix pipeline granularity: add market_year to cleaned/serving SQL models
- Add DuckDB data access layer with async query functions (analytics.py)
- Build Chart.js dashboard: supply/demand, STU ratio, top producers, YoY table
- Add country comparison page with multi-select picker
- Replace items CRUD with read-only commodity API (list, metrics, countries, CSV)
- Configure BeanFlows plan tiers (Free/Starter/Pro) with feature gating
- Rewrite public pages for coffee market intelligence positioning
- Remove boilerplate items schema, update health check for DuckDB
- Add test suite: 139 tests passing (dashboard, API, billing)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 16:11:50 +01:00
Hendrik Dreesmann
b702e6565a Update SQLMesh for R2 data access & Convert psd data to gzip 2025-11-02 00:26:01 +01:00