Commit Graph

22 Commits

Author SHA1 Message Date
Deeman
2962bf5e3b Fix COT pipeline: TRY_CAST nulls, dim_commodity leading zeros, correct CFTC codes
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
  Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
  to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
  and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data

Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 23:28:10 +01:00
Deeman
0a83b2cb74 Add CFTC COT data integration with foundation data model layer
- New extraction package (cftc_cot): downloads yearly Disaggregated Futures ZIPs
  from CFTC, etag-based dedup, dynamic inner filename discovery, gzip normalization
- SQLMesh 3-layer architecture: raw (technical) → foundation (business model) → serving (mart)
- dim_commodity seed: conformed dimension mapping USDA ↔ CFTC codes — the commodity ontology
- fct_cot_positioning: typed, deduplicated weekly positioning facts for all commodities
- obt_cot_positioning: Coffee C mart with COT Index (26w/52w), WoW delta, OI ratios
- Analytics functions + REST API endpoints: /commodities/<code>/positioning[/latest]
- Dashboard widget: Managed Money net, COT Index card, dual-axis Chart.js chart
- 23 passing tests (10 unit + 2 SQLMesh model + existing regression suite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 23:28:10 +01:00
Deeman
d09ba91023 Remove password admin login, seed dev accounts, add regression tests
Admin flow:
- Remove /admin/login (password-based) and /admin/dev-login routes entirely
- admin_required now checks only the 'admin' role; redirects to auth.login
- auth/dev-login with an ADMIN_EMAILS address redirects directly to /admin/
- .env.example: replace ADMIN_PASSWORD with ADMIN_EMAILS=admin@beanflows.coffee

Dev seeding:
- Add dev_seed.py: idempotent upsert of 4 fixed accounts (admin, free,
  starter, pro) so every access tier is testable after dev_run.sh
- dev_run.sh: seed after migrations, show all 4 login shortcuts

Regression tests (37 passing):
- test_analytics.py: concurrent fetch_analytics calls return correct row
  counts (cursor thread-safety regression), column names are lowercase
- test_roles.py TestAdminAuthFlow: password login routes return 404,
  admin_required redirects to auth.login, dev-login grants admin role
  and redirects to admin panel when email is in ADMIN_EMAILS
- conftest.py: add mock_analytics fixture (fixes 7 pre-existing dashboard
  test errors); fix assertion text and lowercase metric param in tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 20:10:45 +01:00
Deeman
fef9f3d705 Fix concurrent DuckDB queries: use cursor() per thread
_conn.execute() is not thread-safe for concurrent calls from multiple
threads. asyncio.gather submits each analytics query to the thread pool
via asyncio.to_thread, causing race conditions that silently returned
empty result sets. _conn.cursor() creates an independent cursor that is
safe to use from separate threads simultaneously.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 19:04:09 +01:00
Deeman
d569ba0162 Fix metric column name casing: DuckDB returns lowercase, align everywhere
SQLMesh normalizes unquoted identifiers to lowercase in physical tables,
so commodity_metrics columns are e.g. 'production' not 'Production'.
Update ALLOWED_METRICS, all analytics.py SQL queries, dashboard routes,
and both dashboard templates (Jinja + JS chart references) to use
lowercase column names consistently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 17:14:52 +01:00
Deeman
d05e522c88 Add migration 0001: create feedback and waitlist tables
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 15:36:40 +01:00
Deeman
8e7af53ff6 Fix admin auth, impersonation session handling, and stale stripe column
- admin_required now accepts users with 'admin' role (via g.user) in
  addition to the password-based is_admin session flag, so both auth
  methods grant access
- impersonate stores the admin's user_id (not True) in admin_impersonating
  so stop-impersonating can restore the correct session
- stop_impersonating restores user_id from admin_impersonating instead of
  just popping it
- remove s.stripe_customer_id from get_user_by_id (Paddle project, no
  stripe_customer_id column in subscriptions)

Fixes 3 test_roles.py failures: test_admin_index_accessible_with_admin_role,
test_impersonate_stores_admin_id, test_stop_impersonating_restores_admin

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 02:33:10 +01:00
Deeman
48bea5c198 Add admin sidebar layout, feedback, waitlist, sitemap to BeanFlows web
- Admin sidebar layout (base_admin.html) with espresso/copper coffee theme,
  220px sidebar, responsive collapse, nav for Dashboard/Users/Tasks/Feedback/Waitlist
- Convert all admin templates to extend base_admin.html using Tailwind classes
- Feedback system: schema, public POST route (rate-limited), base.html widget
  with HTMX popover (coffee-themed), admin viewer with mark-read
- Waitlist mode: WAITLIST_MODE config, waitlist_gate decorator,
  capture_waitlist_email helper, auth route integration, confirmation pages,
  send_waitlist_confirmation worker task, admin table
- Sitemap.xml and robots.txt public routes
- Dashboard stats updated with waitlist_count, feedback_unread alongside
  existing commodity DuckDB analytics stats

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 02:27:26 +01:00
Deeman
e80e262e25 Fix NoneType error when user has no subscription
g.subscription is explicitly set to None in load_user, so
g.get("subscription", {}) returns None (key exists), not {}.
Use (g.get(...) or {}) to coalesce None to an empty dict.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 23:10:56 +01:00
Deeman
642b529c4d Fix column name mismatch after copier template update
The subscriptions table still had paddle_subscription_id but the new
code references provider_subscription_id.  Renamed the DB column and
updated all queries in billing/routes.py to match.

Also removed unused get_subscription import from dashboard/routes.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 23:09:12 +01:00
Deeman
94e8a5e6c3 Fix dev-login URLs in development script
- Update dev_run.sh with better dev-login URLs including appropriate email parameters
- Add user-login URL: auth/dev-login?email=trader@beanflows.coffee
- Add admin-login URL: auth/dev-login?email=admin@beanflows.coffee
- Keep admin-panel URL: admin/dev-login for direct admin session
- Add ADMIN_EMAILS config to .env.example for auto-granting admin role

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:55:57 +01:00
Deeman
18c6e0da4f Fix dashboard routes after copier update
- Remove import of get_user_with_subscription (function was removed)
- Use g.user and g.subscription from eager loading instead
- Fixes ImportError in dashboard routes

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:48:53 +01:00
Deeman
edd439245b Merge branch 'copier-update' 2026-02-19 22:46:41 +01:00
Deeman
32132974b2 Clean up web changes and add favicon
- Update uv.lock dependencies
- Remove web/CLAUDE.md (moved to root)
- Update base.html template
- Add favicon.svg

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:46:33 +01:00
Deeman
3f1cd8bd0c Update copier answers and docker-compose prod config
- Record v0.4.0 commit in .copier-answers.yml
- Apply flattened paths in docker-compose.prod.yml

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:35:55 +01:00
Deeman
4b7d4d5a74 Update from Copier template v0.4.0
- Accept RBAC system: user_roles table, role_required decorator, grant_role/revoke_role/ensure_admin_role functions
- Accept improved billing architecture: billing_customers table separation, provider-agnostic naming
- Accept enhanced user loading with subscription/roles eager loading in app.py
- Accept improved email templates with branded styling
- Accept new infrastructure: migration tracking, transaction logging, A/B testing
- Accept template improvements: Resend SDK, Tailwind build stage, UMAMI analytics config
- Keep beanflows-specific configs: BASE_URL 5001, coffee PLAN_FEATURES/PLAN_LIMITS
- Keep beanflows analytics integration and DuckDB health check
- Add new test files and utility scripts from template

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-02-19 22:22:13 +01:00
Deeman
1e8a173ae8 Merge branch 'frontend-upgrade'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:49:07 +01:00
Deeman
f722854c07 Rewrite frontend templates: Pico CSS → Tailwind + trader-focused copy
Replace all Pico CSS patterns (classless articles, role="button", inline
styles, var(--pico-*)) with Tailwind component classes. Add Fraunces
display font, mobile hamburger nav, brand chart colors, and new component
layer (hero, feature-card, metric-card, auth-card, pricing-card, etc.).

Rewrite marketing copy from generic SaaS boilerplate to coffee-trader
focused messaging. Rebrand pricing tiers to Explorer/Trader/Analyst.
Delete stale custom.css. No Python code changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:47:56 +01:00
Deeman
6dac8570ad Fix web/ startup errors and sync with boilerplate
- Load .env via python-dotenv in core.py
- Skip analytics DB open if file doesn't exist
- Guard dashboard analytics calls when DB not available
- Namespace admin templates under admin/ to avoid blueprint conflicts
- Add dev-login routes for user and admin (DEBUG only)
- Update .copier-answers.yml src_path to GitLab remote

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:37:44 +01:00
Deeman
d6d2aa8efe Merge remote-tracking branch 'origin/master'
# Conflicts:
#	infra/readme.md
2026-02-18 21:09:24 +01:00
Deeman
2748c606e9 Add BeanFlows MVP: coffee analytics dashboard, API, and web app
- Fix pipeline granularity: add market_year to cleaned/serving SQL models
- Add DuckDB data access layer with async query functions (analytics.py)
- Build Chart.js dashboard: supply/demand, STU ratio, top producers, YoY table
- Add country comparison page with multi-select picker
- Replace items CRUD with read-only commodity API (list, metrics, countries, CSV)
- Configure BeanFlows plan tiers (Free/Starter/Pro) with feature gating
- Rewrite public pages for coffee market intelligence positioning
- Remove boilerplate items schema, update health check for DuckDB
- Add test suite: 139 tests passing (dashboard, API, billing)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 16:11:50 +01:00
Hendrik Dreesmann
b702e6565a Update SQLMesh for R2 data access & Convert psd data to gzip 2025-11-02 00:26:01 +01:00