Load .env from repo root first (created by `make secrets-decrypt-dev`),
falling back to web/.env for legacy setups. Also fixes import sort order
and removes unused httpx import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename env var to plural (CSV list) in CI yml to match the actual
config key. Add hendrik@beanflow.coffee and simon@beanflows.coffee
as hardcoded defaults so they get admin access without needing the
env var set explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dashboard/routes.py (4 places) and admin/routes.py still checked
analytics._conn is not None after _conn was removed in the two-file
refactor — causing AttributeError → 500 on every dashboard page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs fixed:
1. Cross-connection COPY: DuckDB doesn't support referencing another
connection's tables as src.serving.table. Replace with Arrow as
intermediate: src reads to Arrow, dst.register() + CREATE TABLE.
2. Catalog/schema name collision: naming the export file serving.duckdb
made DuckDB assign catalog name "serving" — same as the schema we
create inside it. Every serving.table query became ambiguous. Rename
to analytics.duckdb (catalog "analytics", schema "serving" = no clash).
SERVING_DUCKDB_PATH values updated: serving.duckdb → analytics.duckdb
in supervisor, service, bootstrap, dev_run.sh, .env.example, docker-compose.
3. Temp file: use _export.duckdb (not serving.duckdb.tmp) to avoid
the same catalog collision during the write phase.
Verified: 6 tables exported, serving.* queries work read-only.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On the first `./scripts/dev_run.sh` invocation (serving.duckdb absent),
automatically run extract → transform → export_serving from the repo root
so the dashboard is populated without any manual steps.
Subsequent runs skip the pipeline for a fast startup. Delete serving.duckdb
from the repo root to force a full pipeline re-run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The assert _db_path in fetch_analytics() would crash dashboard routes
locally when SERVING_DUCKDB_PATH is unset or serving.duckdb doesn't
exist yet. Change to graceful return [] so the app degrades cleanly.
Also add SERVING_DUCKDB_PATH=../serving.duckdb to local .env so the
web app will auto-connect once `materia pipeline run export_serving`
has been run for the first time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split the single lakehouse.duckdb into two files to eliminate the exclusive
write-lock conflict between SQLMesh (pipeline) and the Quart web app (reader):
lakehouse.duckdb — SQLMesh exclusive (all pipeline layers)
serving.duckdb — web app reads (serving tables only, atomically swapped)
Changes:
web/src/beanflows/analytics.py
- Replace persistent global _conn with per-thread connections (threading.local)
- Add _get_conn(): opens read_only=True on first call per thread, reopens
automatically on inode change (~1μs os.stat) to pick up atomic file swaps
- Switch env var from DUCKDB_PATH → SERVING_DUCKDB_PATH
- Add module docstring documenting architecture + DuckLake migration path
web/src/beanflows/app.py
- Startup check: use SERVING_DUCKDB_PATH
- Health check: use _db_path instead of _conn
src/materia/export_serving.py (new)
- Reads all serving.* tables from lakehouse.duckdb (read_only)
- Writes to serving_new.duckdb, then os.rename → serving.duckdb (atomic)
- ~50 lines; runs after each SQLMesh transform
src/materia/pipelines.py
- Add export_serving pipeline entry (uv run python -c ...)
infra/supervisor/supervisor.sh
- Add SERVING_DUCKDB_PATH env var comment
- Add export step: uv run materia pipeline run export_serving
infra/supervisor/materia-supervisor.service
- Add Environment=SERVING_DUCKDB_PATH=/data/materia/serving.duckdb
infra/bootstrap_supervisor.sh
- Add SERVING_DUCKDB_PATH to .env template
web/.env.example + web/docker-compose.yml
- Document both env vars; switch web service to SERVING_DUCKDB_PATH
web/src/beanflows/dashboard/templates/settings.html
- Minor settings page fix from prior session
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove 'Write' scope checkbox from API key creation form — BeanFlows
is a read-only data platform, write keys are meaningless to users.
Scope is now always 'read' via hidden input.
- Add try/except in billing.manage route so Paddle API failures (e.g.
no live credentials in dev) show a user-facing flash error instead
of a 500.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Apply button flow with immediate HTMX partial fetches:
- toggleCountry() does an optimistic UI update (row + badge) then
calls htmx.ajax() targeting #cc-canvas with swap=innerHTML
- URL is pushed to history on every selection change (bookmarkable)
- HX-Request now returns countries_canvas.html fragment (chips +
chart/empty + inline IIFE that re-syncs globals + re-inits Chart.js)
- Panel (dark) is never swapped; canvas fades during in-flight request
- PALETTE, buildRankings(), initChart() defined once on page load,
called by both initial render and partial IIFE after each swap
- Apply button removed; Clear triggers fetchCanvas() with empty codes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace generic multi-select + plain card with a two-panel layout:
- Dark espresso selector panel (sticky, searchable, click-to-toggle)
with country rows showing rank, name, production figure, checkbox
- Right canvas: metric segment tabs, selected-country chips (colored),
Chart.js line chart with dark espresso tooltip, and a JS-built
rankings table with proportional colored bars (latest year)
- Smooth fade-in animations, monospaced figures, copper accent palette
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- serving/ice_aging_stocks.sql: pass-through from foundation, parses age
bucket string to start/end days ints for correct sort order
- serving/ice_warehouse_stocks_by_port.sql: monthly by-port since 1996,
adds MoM change, MoM %, 12-month rolling average
- analytics.py: get_ice_aging_latest(), get_ice_aging_trend(),
get_ice_stocks_by_port_trend(), get_ice_stocks_by_port_latest()
- api/routes.py: GET /commodities/<code>/stocks/aging and
GET /commodities/<code>/stocks/by-port with auth + rate limiting
- dashboard/routes.py: add 3 new queries to asyncio.gather(), pass to template
- index.html: aging stacked bar chart (age buckets × port) with 4 metric
cards; by-port stacked area chart (30-year history) with 4 metric cards
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Create dashboard_base.html: standalone app shell with 56px sticky
header (logo + user email + sign out), 220px left sidebar with
Overview/Countries/Settings nav items (SVG icons, active state via
request.path), and fixed mobile bottom tab bar (md:hidden)
- Add CSS component classes: .app-shell, .app-header, .app-sidebar,
.sidebar-item, .app-content, .mobile-bottom-nav, .mobile-nav-item
- Extract feedback widget into _feedback_widget.html partial; include
from both base.html and dashboard_base.html
- Switch index.html, countries.html, settings.html to extend
dashboard_base.html; remove <main class="container-page"> wrappers
- Remove "Back to Dashboard" button from countries.html (sidebar
provides persistent navigation)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- routes.py: return_exceptions=True on gather, log individual query failures
with per-result defaults so one bad query doesn't blank the whole page
- settings.html: fix billing.portal → billing.manage (correct blueprint route)
- vision.md: update current state to February 2026, document shipped features
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use return_exceptions=True so a CatalogException from a single query
(e.g. table not yet populated in a fresh env) degrades gracefully
instead of crashing the whole dashboard render.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data
Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Admin flow:
- Remove /admin/login (password-based) and /admin/dev-login routes entirely
- admin_required now checks only the 'admin' role; redirects to auth.login
- auth/dev-login with an ADMIN_EMAILS address redirects directly to /admin/
- .env.example: replace ADMIN_PASSWORD with ADMIN_EMAILS=admin@beanflows.coffee
Dev seeding:
- Add dev_seed.py: idempotent upsert of 4 fixed accounts (admin, free,
starter, pro) so every access tier is testable after dev_run.sh
- dev_run.sh: seed after migrations, show all 4 login shortcuts
Regression tests (37 passing):
- test_analytics.py: concurrent fetch_analytics calls return correct row
counts (cursor thread-safety regression), column names are lowercase
- test_roles.py TestAdminAuthFlow: password login routes return 404,
admin_required redirects to auth.login, dev-login grants admin role
and redirects to admin panel when email is in ADMIN_EMAILS
- conftest.py: add mock_analytics fixture (fixes 7 pre-existing dashboard
test errors); fix assertion text and lowercase metric param in tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_conn.execute() is not thread-safe for concurrent calls from multiple
threads. asyncio.gather submits each analytics query to the thread pool
via asyncio.to_thread, causing race conditions that silently returned
empty result sets. _conn.cursor() creates an independent cursor that is
safe to use from separate threads simultaneously.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLMesh normalizes unquoted identifiers to lowercase in physical tables,
so commodity_metrics columns are e.g. 'production' not 'Production'.
Update ALLOWED_METRICS, all analytics.py SQL queries, dashboard routes,
and both dashboard templates (Jinja + JS chart references) to use
lowercase column names consistently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- admin_required now accepts users with 'admin' role (via g.user) in
addition to the password-based is_admin session flag, so both auth
methods grant access
- impersonate stores the admin's user_id (not True) in admin_impersonating
so stop-impersonating can restore the correct session
- stop_impersonating restores user_id from admin_impersonating instead of
just popping it
- remove s.stripe_customer_id from get_user_by_id (Paddle project, no
stripe_customer_id column in subscriptions)
Fixes 3 test_roles.py failures: test_admin_index_accessible_with_admin_role,
test_impersonate_stores_admin_id, test_stop_impersonating_restores_admin
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
g.subscription is explicitly set to None in load_user, so
g.get("subscription", {}) returns None (key exists), not {}.
Use (g.get(...) or {}) to coalesce None to an empty dict.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The subscriptions table still had paddle_subscription_id but the new
code references provider_subscription_id. Renamed the DB column and
updated all queries in billing/routes.py to match.
Also removed unused get_subscription import from dashboard/routes.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove import of get_user_with_subscription (function was removed)
- Use g.user and g.subscription from eager loading instead
- Fixes ImportError in dashboard routes
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
- Record v0.4.0 commit in .copier-answers.yml
- Apply flattened paths in docker-compose.prod.yml
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
- Load .env via python-dotenv in core.py
- Skip analytics DB open if file doesn't exist
- Guard dashboard analytics calls when DB not available
- Namespace admin templates under admin/ to avoid blueprint conflicts
- Add dev-login routes for user and admin (DEBUG only)
- Update .copier-answers.yml src_path to GitLab remote
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
