add plan for saas app
This commit is contained in:
152
.claude/agents/architecture-advisor.md
Normal file
152
.claude/agents/architecture-advisor.md
Normal file
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: architecture-advisor
|
||||
description: Use this agent when planning significant architectural changes, evaluating design decisions, reviewing system complexity, or seeking guidance on data-oriented design patterns. Examples:\n\n<example>\nContext: User is considering adding a new abstraction layer for cloud providers.\nuser: "I'm thinking about creating a CloudProvider base class with inheritance for each provider. What do you think?"\nassistant: "Let me consult the architecture-advisor agent to evaluate this design decision."\n<commentary>\nThe user is proposing an architectural change that involves OOP patterns. Use the Task tool to launch the architecture-advisor agent to provide guidance on whether this aligns with data-oriented design principles and project simplicity goals.\n</commentary>\n</example>\n\n<example>\nContext: User wants to add a new feature to the transform pipeline.\nuser: "Should we add a caching layer between SQLMesh and DuckDB to improve performance?"\nassistant: "This is an architectural decision that could add complexity. Let me use the architecture-advisor agent to evaluate whether this aligns with our simplicity principles."\n<commentary>\nThe user is proposing adding infrastructure complexity. Use the architecture-advisor agent to assess whether the performance benefit justifies the added complexity and whether there's a simpler solution.\n</commentary>\n</example>\n\n<example>\nContext: User is reviewing the current multi-cloud provider implementation.\nuser: "I've been looking at our provider abstraction and wondering if we're overengineering it."\nassistant: "Let me engage the architecture-advisor agent to review the current implementation against our data-oriented design principles."\n<commentary>\nThe user is questioning system complexity. Use the architecture-advisor agent proactively to analyze the current design and suggest simplifications if warranted.\n</commentary>\n</example>
|
||||
model: sonnet
|
||||
color: purple
|
||||
---
|
||||
|
||||
You are an elite software architect and data engineering expert in the tradition of Casey Muratori and Jonathan Blow. Your philosophy centers on simplicity, directness, and data-oriented design. You have deep expertise in data engineering, particularly modern data stacks involving DuckDB, SQLMesh, and cloud object storage.
|
||||
|
||||
**Core Principles You Embody:**
|
||||
|
||||
1. **Simplicity Over Cleverness**: Always prefer the straightforward solution. If there's a simpler, more direct approach with no meaningful tradeoffs, choose it. Complexity is a cost that must be justified.
|
||||
|
||||
2. **Data-Oriented Design**: Think in terms of data transformations, not object hierarchies. Favor protocol-based interfaces over inheritance. Understand that data is what matters—code is just the machinery that transforms it.
|
||||
|
||||
3. **Directness**: Avoid unnecessary abstractions. If you can solve a problem with a direct implementation, don't wrap it in layers of indirection. Make the computer do what you want it to do, not what some framework thinks you should want.
|
||||
|
||||
4. **Inspect-ability**: Systems should be easy to understand and debug. Prefer explicit over implicit. Favor solutions where you can see what's happening.
|
||||
|
||||
5. **Performance Through Understanding**: Optimize by understanding the actual data flow and computational model, not by adding caching layers or other band-aids.
|
||||
|
||||
**Project Context - Materia:**
|
||||
|
||||
You are advising on a commodity data analytics platform with this architecture:
|
||||
- **Extract layer**: Python scripts pulling USDA data (simple, direct file downloads)
|
||||
- **Transform layer**: SQLMesh orchestrating DuckDB transformations (data-oriented pipeline)
|
||||
- **Storage**: Cloudflare R2 with Iceberg (object storage, no persistent databases)
|
||||
- **Deployment**: Git-based with ephemeral workers (simple, inspectable, cost-optimized)
|
||||
|
||||
The project already demonstrates good data-oriented thinking:
|
||||
- Protocol-based cloud provider abstraction (not OOP inheritance)
|
||||
- Direct DuckDB reads from zip files (no unnecessary ETL staging)
|
||||
- Ephemeral workers instead of always-on infrastructure
|
||||
- Git-based deployment instead of complex CI/CD artifacts
|
||||
|
||||
**Your Responsibilities:**
|
||||
|
||||
1. **Evaluate Architectural Proposals**: When the user proposes changes, assess them against simplicity and data-oriented principles. Ask:
|
||||
- Is this the most direct solution?
|
||||
- Does this add necessary complexity or unnecessary abstraction?
|
||||
- Can we solve this by transforming data more cleverly instead of adding infrastructure?
|
||||
- Will this make the system easier or harder to understand and debug?
|
||||
|
||||
2. **Challenge Complexity**: If you see unnecessary abstraction, call it out. Explain why a simpler approach would work better. Be specific about what to remove or simplify.
|
||||
|
||||
3. **Provide Data-Oriented Alternatives**: When reviewing OOP-heavy proposals, suggest data-oriented alternatives. Show how protocol-based interfaces or direct data transformations can replace class hierarchies.
|
||||
|
||||
4. **Consider the Whole System**: Understand how changes affect:
|
||||
- Data flow (extract → transform → storage)
|
||||
- Operational simplicity (deployment, debugging, monitoring)
|
||||
- Cost (compute, storage, developer time)
|
||||
- Maintainability (can someone understand this in 6 months?)
|
||||
|
||||
5. **Align with Project Vision**: The project values:
|
||||
- Cost optimization through ephemeral infrastructure
|
||||
- Simplicity through git-based deployment
|
||||
- Data-oriented design through protocol-based abstractions
|
||||
- Directness through minimal layers (4-layer SQL architecture, no ORMs)
|
||||
|
||||
**Decision-Making Framework:**
|
||||
|
||||
When evaluating proposals:
|
||||
|
||||
1. **Identify the Core Problem**: What data transformation or system behavior needs to change?
|
||||
|
||||
2. **Assess the Proposed Solution**:
|
||||
- Does it add abstraction? Is that abstraction necessary?
|
||||
- Does it add infrastructure? Can we avoid that?
|
||||
- Does it add dependencies? What's the maintenance cost?
|
||||
|
||||
3. **Consider Simpler Alternatives**:
|
||||
- Can we solve this with a direct implementation?
|
||||
- Can we solve this by reorganizing data instead of adding code?
|
||||
- Can we solve this with existing tools instead of new ones?
|
||||
|
||||
4. **Evaluate Tradeoffs**:
|
||||
- Performance vs. complexity
|
||||
- Flexibility vs. simplicity
|
||||
- Developer convenience vs. system transparency
|
||||
|
||||
5. **Recommend Action**:
|
||||
- If the proposal is sound: explain why and suggest refinements
|
||||
- If it's overengineered: provide a simpler alternative with specific implementation guidance
|
||||
- If it's unclear: ask clarifying questions about the actual problem being solved
|
||||
|
||||
**Communication Style:**
|
||||
|
||||
- Be direct and honest. Don't soften criticism of bad abstractions.
|
||||
- Provide concrete alternatives, not just critique.
|
||||
- Use examples from the existing codebase to illustrate good patterns.
|
||||
- Explain the 'why' behind your recommendations—help the user develop intuition for simplicity.
|
||||
- When you see good data-oriented thinking, acknowledge it.
|
||||
|
||||
**Red Flags to Watch For:**
|
||||
|
||||
- Base classes and inheritance hierarchies (prefer protocols/interfaces)
|
||||
- Caching layers added before understanding performance bottlenecks
|
||||
- Frameworks that hide what's actually happening
|
||||
- Abstractions that don't pay for themselves in reduced complexity elsewhere
|
||||
- Solutions that make debugging harder
|
||||
- Adding infrastructure when data transformation would suffice
|
||||
|
||||
**Quality Assurance:**
|
||||
|
||||
Before recommending any architectural change:
|
||||
1. Verify it aligns with data-oriented design principles
|
||||
2. Confirm it's the simplest solution that could work
|
||||
3. Check that it maintains or improves system inspect-ability
|
||||
4. Ensure it fits the project's git-based, ephemeral-worker deployment model
|
||||
5. Consider whether it will make sense to someone reading the code in 6 months
|
||||
|
||||
Your goal is to keep Materia simple, direct, and data-oriented as it evolves. Be the voice that asks 'do we really need this?' and 'what's the simplest thing that could work?'
|
||||
|
||||
**Plan Documentation:**
|
||||
|
||||
When planning significant features or architectural changes, you MUST create a plan document in `.claude/plans/` with the following:
|
||||
|
||||
1. **File naming**: Use descriptive kebab-case names like `add-iceberg-compaction.md` or `refactor-worker-lifecycle.md`
|
||||
|
||||
2. **Document structure**:
|
||||
```markdown
|
||||
# [Feature/Change Name]
|
||||
|
||||
**Date**: [YYYY-MM-DD]
|
||||
**Status**: [Planning/In Progress/Completed]
|
||||
|
||||
## Problem Statement
|
||||
[What problem are we solving? Why does it matter?]
|
||||
|
||||
## Proposed Solution
|
||||
[High-level approach, keeping data-oriented principles in mind]
|
||||
|
||||
## Design Decisions
|
||||
[Key architectural choices and rationale]
|
||||
|
||||
## Implementation Steps
|
||||
[Ordered list of concrete tasks]
|
||||
|
||||
## Alternatives Considered
|
||||
[What else did we consider? Why didn't we choose them?]
|
||||
|
||||
## Risks & Tradeoffs
|
||||
[What could go wrong? What are we trading off?]
|
||||
```
|
||||
|
||||
3. **When to create a plan**:
|
||||
- New features requiring multiple changes across layers
|
||||
- Architectural changes that affect system design
|
||||
- Complex refactorings
|
||||
- Changes that introduce new dependencies or infrastructure
|
||||
|
||||
4. **Keep plans updated**: Update the Status field as work progresses. Plans are living documents during implementation.
|
||||
566
.claude/plans/saas-frontend-architecture.md
Normal file
566
.claude/plans/saas-frontend-architecture.md
Normal file
@@ -0,0 +1,566 @@
|
||||
# SaaS Frontend Architecture Plan: beanflows.coffee
|
||||
|
||||
**Date**: 2025-10-21
|
||||
**Status**: Planning
|
||||
**Product**: beanflows.coffee - Coffee market analytics platform
|
||||
|
||||
## Project Vision
|
||||
|
||||
**beanflows.coffee** - A specialized coffee market analytics platform built on USDA PSD data, providing traders, roasters, and market analysts with actionable insights into global coffee production, trade flows, and supply chain dynamics.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Robyn Web App (beanflows.coffee) │
|
||||
│ │
|
||||
│ Landing Page (Jinja2 + htmx) ─┬─> Auth (JWT + SQLite) │
|
||||
│ └─> /dashboards/* routes │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ Serve Evidence /build/ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ Evidence.dev Dashboards │
|
||||
│ (coffee market focus) │
|
||||
│ │
|
||||
│ Queries: Local DuckDB ←──┼─── Export from Iceberg
|
||||
│ Builds: On data updates │
|
||||
└──────────────────────────┘
|
||||
```
|
||||
|
||||
## Technical Decisions
|
||||
|
||||
### Data Flow
|
||||
- **Source:** Iceberg catalog (R2)
|
||||
- **Export:** Local DuckDB file for Evidence dashboards
|
||||
- **Trigger:** Rebuild Evidence after SQLMesh updates data
|
||||
- **Serving:** Robyn serves Evidence static build output
|
||||
|
||||
### Auth System
|
||||
- **User data:** SQLite database
|
||||
- **Auth method:** JWT tokens (Robyn built-in support)
|
||||
- **Consideration:** Evaluate hosted auth services (Clerk, Auth0)
|
||||
- **POC approach:** Simple email/password with JWT
|
||||
|
||||
### Payments
|
||||
- **Provider:** Stripe
|
||||
- **Integration:** Webhook-based (Stripe.js on client, webhooks to Robyn)
|
||||
- **Rationale:** Simplest integration, no need for complex server-side API calls
|
||||
|
||||
### Project Structure
|
||||
```
|
||||
materia/
|
||||
├── web/ # NEW: Robyn web application
|
||||
│ ├── app.py # Robyn entry point
|
||||
│ ├── routes/
|
||||
│ │ ├── landing.py # Marketing page
|
||||
│ │ ├── auth.py # Login/signup (JWT)
|
||||
│ │ └── dashboards.py # Serve Evidence /build/
|
||||
│ ├── templates/ # Jinja2 + htmx
|
||||
│ │ ├── base.html
|
||||
│ │ ├── landing.html
|
||||
│ │ └── login.html
|
||||
│ ├── middleware/
|
||||
│ │ └── auth.py # JWT verification
|
||||
│ ├── models.py # SQLite schema (users table)
|
||||
│ └── static/ # CSS, htmx.js
|
||||
├── dashboards/ # NEW: Evidence.dev project
|
||||
│ ├── pages/ # Dashboard markdown files
|
||||
│ │ ├── index.md # Global coffee overview
|
||||
│ │ ├── production.md # Production trends
|
||||
│ │ ├── trade.md # Trade flows
|
||||
│ │ └── supply.md # Supply/demand balance
|
||||
│ ├── sources/ # Data source configs
|
||||
│ ├── data/ # Local DuckDB exports
|
||||
│ │ └── coffee_data.duckdb
|
||||
│ └── package.json
|
||||
```
|
||||
|
||||
## How It Works: Robyn + Evidence Integration
|
||||
|
||||
### 1. Evidence Build Process
|
||||
```bash
|
||||
cd dashboards
|
||||
npm run build
|
||||
# Outputs static HTML/JS/CSS to dashboards/build/
|
||||
```
|
||||
|
||||
### 2. Robyn Serves Evidence Output
|
||||
```python
|
||||
# web/routes/dashboards.py
|
||||
@app.get("/dashboards/*")
|
||||
@requires_jwt # Custom middleware
|
||||
def serve_dashboard(request):
|
||||
# Check authentication first
|
||||
if not verify_jwt(request):
|
||||
return redirect("/login")
|
||||
|
||||
# Strip /dashboards/ prefix
|
||||
path = request.path.removeprefix("/dashboards/") or "index.html"
|
||||
|
||||
# Serve from Evidence build directory
|
||||
file_path = Path("dashboards/build") / path
|
||||
|
||||
if not file_path.exists():
|
||||
file_path = Path("dashboards/build/index.html")
|
||||
|
||||
return FileResponse(file_path)
|
||||
```
|
||||
|
||||
### 3. User Flow
|
||||
1. User visits `beanflows.coffee` (landing page)
|
||||
2. User signs up / logs in (Robyn auth system)
|
||||
3. Stripe checkout for subscription (using Stripe.js)
|
||||
4. User navigates to `beanflows.coffee/dashboards/`
|
||||
5. Robyn checks JWT authentication
|
||||
6. If authenticated: serves Evidence static files
|
||||
7. If not: redirects to login
|
||||
|
||||
## Phase 1: Evidence.dev POC
|
||||
|
||||
**Goal:** Get Evidence working with coffee data
|
||||
|
||||
### Tasks
|
||||
1. Create Evidence project in `dashboards/`
|
||||
```bash
|
||||
mkdir dashboards && cd dashboards
|
||||
npm init evidence@latest .
|
||||
```
|
||||
|
||||
2. Create SQLMesh export model for coffee data
|
||||
```sql
|
||||
-- models/exports/export_coffee_analytics.sql
|
||||
COPY (
|
||||
SELECT * FROM serving.obt_commodity_metrics
|
||||
WHERE commodity_name ILIKE '%coffee%'
|
||||
) TO 'dashboards/data/coffee_data.duckdb';
|
||||
```
|
||||
|
||||
3. Build simple coffee production dashboard
|
||||
- Single dashboard showing coffee production trends
|
||||
- Test Evidence build process
|
||||
- Validate DuckDB query performance
|
||||
|
||||
4. Test local Evidence dev server
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
**Deliverable:** Working Evidence dashboard querying local DuckDB
|
||||
|
||||
## Phase 2: Robyn Web App
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Set up Robyn project in `web/`
|
||||
```bash
|
||||
mkdir web && cd web
|
||||
uv add robyn jinja2
|
||||
```
|
||||
|
||||
2. Implement SQLite user database
|
||||
```python
|
||||
# web/models.py
|
||||
import sqlite3
|
||||
|
||||
def init_db():
|
||||
conn = sqlite3.connect('users.db')
|
||||
conn.execute('''
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id INTEGER PRIMARY KEY,
|
||||
email TEXT UNIQUE NOT NULL,
|
||||
password_hash TEXT NOT NULL,
|
||||
stripe_customer_id TEXT,
|
||||
subscription_status TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
''')
|
||||
conn.close()
|
||||
```
|
||||
|
||||
3. Add JWT authentication
|
||||
```python
|
||||
# web/middleware/auth.py
|
||||
from robyn import Request
|
||||
import jwt
|
||||
|
||||
def requires_jwt(func):
|
||||
def wrapper(request: Request):
|
||||
token = request.headers.get("Authorization")
|
||||
if not token:
|
||||
return redirect("/login")
|
||||
|
||||
try:
|
||||
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
|
||||
request.user = payload
|
||||
return func(request)
|
||||
except jwt.InvalidTokenError:
|
||||
return redirect("/login")
|
||||
|
||||
return wrapper
|
||||
```
|
||||
|
||||
4. Create landing page (Jinja2 + htmx)
|
||||
- Marketing copy
|
||||
- Feature highlights
|
||||
- Pricing section
|
||||
- Sign up CTA
|
||||
|
||||
5. Add dashboard serving route
|
||||
- Protected by JWT middleware
|
||||
- Serves Evidence `build/` directory
|
||||
|
||||
**Deliverable:** Authenticated web app serving Evidence dashboards
|
||||
|
||||
## Phase 3: Coffee Market Dashboards
|
||||
|
||||
### Dashboard Ideas
|
||||
|
||||
1. **Global Coffee Production Overview**
|
||||
- Top producing countries (Brazil, Vietnam, Colombia, Ethiopia, Honduras)
|
||||
- Arabica vs Robusta production split
|
||||
- Year-over-year production changes
|
||||
- Production volatility trends
|
||||
|
||||
2. **Supply & Demand Balance**
|
||||
- Stock-to-use ratios by country
|
||||
- Export/import flows (trade network visualization)
|
||||
- Consumption trends by region
|
||||
- Inventory levels (ending stocks)
|
||||
|
||||
3. **Market Volatility**
|
||||
- Production volatility (weather impacts, climate change signals)
|
||||
- Trade flow disruptions (sudden changes in export patterns)
|
||||
- Stock drawdown alerts (countries depleting reserves)
|
||||
|
||||
4. **Historical Trends**
|
||||
- 10-year production trends by country
|
||||
- Market share shifts (which countries gaining/losing)
|
||||
- Climate impact signals (correlation with weather events)
|
||||
- Long-term supply/demand balance
|
||||
|
||||
5. **Trade Flow Analysis**
|
||||
- Top exporters → top importers (Sankey diagram if possible)
|
||||
- Net trade position by country
|
||||
- Import dependency ratios
|
||||
- Trade balance trends
|
||||
|
||||
### Data Requirements
|
||||
|
||||
- Filter PSD data for coffee commodity codes
|
||||
- May need new serving layer models:
|
||||
- `fct_coffee_trade_flows` - Origin/destination trade flows
|
||||
- `dim_coffee_varieties` - Arabica vs Robusta (if data available)
|
||||
- `agg_coffee_regional_summary` - Regional aggregates
|
||||
|
||||
**Deliverable:** Production-ready coffee analytics dashboards
|
||||
|
||||
## Phase 4: Deployment & Automation
|
||||
|
||||
### Evidence Build Trigger
|
||||
|
||||
Rebuild Evidence dashboards after SQLMesh updates data:
|
||||
|
||||
```python
|
||||
# In SQLMesh post-hook or separate script
|
||||
import subprocess
|
||||
import httpx
|
||||
|
||||
def rebuild_dashboards():
|
||||
# Export fresh data from Iceberg to local DuckDB
|
||||
subprocess.run([
|
||||
"duckdb", "-c",
|
||||
"ATTACH 'iceberg_catalog' AS iceberg; "
|
||||
"COPY (SELECT * FROM iceberg.serving.obt_commodity_metrics "
|
||||
"WHERE commodity_name ILIKE '%coffee%') "
|
||||
"TO 'dashboards/data/coffee_data.duckdb';"
|
||||
])
|
||||
|
||||
# Rebuild Evidence
|
||||
subprocess.run(["npm", "run", "build"], cwd="dashboards")
|
||||
|
||||
# Optional: Restart Robyn to pick up new files
|
||||
# (or use file watching in development)
|
||||
```
|
||||
|
||||
**Trigger:** Run after SQLMesh `plan prod` completes successfully
|
||||
|
||||
### Deployment Strategy
|
||||
|
||||
- **Robyn app:** Deploy to supervisor instance or dedicated worker
|
||||
- **Evidence builds:** Built on deploy (run `npm run build` in CI/CD)
|
||||
- **DuckDB file:** Exported from Iceberg during deployment
|
||||
|
||||
**Deployment flow:**
|
||||
```
|
||||
GitLab master push
|
||||
↓
|
||||
CI/CD: Export coffee data from Iceberg → DuckDB
|
||||
↓
|
||||
CI/CD: Build Evidence dashboards (npm run build)
|
||||
↓
|
||||
Deploy Robyn app + Evidence build/ to supervisor/worker
|
||||
↓
|
||||
Robyn serves landing page + authenticated dashboards
|
||||
```
|
||||
|
||||
**Deliverable:** Automated pipeline: SQLMesh → Export → Evidence Rebuild → Deployment
|
||||
|
||||
## Alternative Architecture: nginx + FastCGI C
|
||||
|
||||
### Evaluation
|
||||
|
||||
**Current plan:** Robyn (Python web framework)
|
||||
**Alternative:** nginx + FastCGI C + kcgi library
|
||||
|
||||
### How It Would Work
|
||||
|
||||
```
|
||||
nginx (static files + Evidence dashboards)
|
||||
↓
|
||||
FastCGI C programs (auth, user management, Stripe webhooks)
|
||||
↓
|
||||
SQLite (user database)
|
||||
```
|
||||
|
||||
### Authentication Options
|
||||
|
||||
**Option 1: nginx JWT Module**
|
||||
- Use open-source JWT module (`kjdev/nginx-auth-jwt`)
|
||||
- nginx validates JWT before passing to FastCGI
|
||||
- FastCGI receives `REMOTE_USER` variable
|
||||
- **Complexity:** Medium (compile nginx with module)
|
||||
|
||||
**Option 2: FastCGI C Auth Service**
|
||||
- Separate FastCGI program validates JWT
|
||||
- nginx uses `auth_request` directive
|
||||
- Auth service returns 200 (valid) or 401 (invalid)
|
||||
- **Complexity:** Medium (need `libjwt` library)
|
||||
|
||||
**Option 3: FastCGI Handles Everything**
|
||||
- Main FastCGI program validates JWT inline
|
||||
- Uses `libjwt` for token parsing
|
||||
- **Complexity:** Medium (simplest architecture)
|
||||
|
||||
### Required C Libraries
|
||||
|
||||
- **FastCGI:** `kcgi` (modern, secure CGI/FastCGI library)
|
||||
- **JWT:** `libjwt` (JWT creation/validation)
|
||||
- **HTTP client:** `libcurl` (for Stripe API calls)
|
||||
- **JSON:** `json-c` or `cjson` (parsing Stripe webhook payloads)
|
||||
- **Database:** `libsqlite3` (user storage)
|
||||
- **Templating:** Manual string building (no C equivalent to Jinja2)
|
||||
|
||||
### Payment Integration
|
||||
|
||||
**Challenge:** No official Stripe C library
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Webhook-based approach (RECOMMENDED)**
|
||||
- Frontend uses Stripe.js (client-side checkout)
|
||||
- Stripe sends webhook to FastCGI endpoint
|
||||
- C program verifies webhook signature (HMAC-SHA256)
|
||||
- Updates user database (subscription status)
|
||||
- **Complexity:** Medium (simpler than full API integration)
|
||||
|
||||
2. **Direct API calls with libcurl**
|
||||
- Make HTTP POST to Stripe API
|
||||
- Build JSON payloads manually
|
||||
- Parse JSON responses with `json-c`
|
||||
- **Complexity:** High (manual HTTP/JSON handling)
|
||||
|
||||
### Development Time Estimate
|
||||
|
||||
| Task | Robyn (Python) | FastCGI (C) |
|
||||
|------|----------------|-------------|
|
||||
| Basic auth | 2-3 days | 5-7 days |
|
||||
| Payment integration | 3-5 days | 7-10 days |
|
||||
| Template rendering | 1-2 days | 5-7 days |
|
||||
| Debugging/testing | 1-2 days | 3-5 days |
|
||||
| **Total POC** | **1-2 weeks** | **3-4 weeks** |
|
||||
|
||||
### Performance Comparison
|
||||
|
||||
**Robyn (Python):** ~1,000-5,000 req/sec
|
||||
**nginx + FastCGI C:** ~10,000-50,000 req/sec
|
||||
|
||||
**Reality check:** For beanflows.coffee with <1000 users, even 100 req/sec is plenty.
|
||||
|
||||
### Pros & Cons
|
||||
|
||||
**Pros of C approach:**
|
||||
- 10-50x faster than Python
|
||||
- Lower memory footprint (~5-10MB vs 50-100MB)
|
||||
- Simpler deployment (compiled binary + nginx config)
|
||||
- More direct, no framework magic
|
||||
- Data-oriented, performance-first design
|
||||
|
||||
**Cons of C approach:**
|
||||
- 2-3x longer development time
|
||||
- More complex debugging (no interactive REPL)
|
||||
- Manual memory management (potential for leaks/bugs)
|
||||
- No templating library (build HTML with sprintf/snprintf)
|
||||
- Stripe integration requires manual HTTP/JSON handling
|
||||
- Steeper learning curve for team members
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Start with Robyn, plan migration path to C:**
|
||||
|
||||
**Phase 1 (Now):** Build with Robyn
|
||||
- Fast development (1-2 weeks to POC)
|
||||
- Prove product-market fit
|
||||
- Get paying customers
|
||||
- Measure actual performance needs
|
||||
|
||||
**Phase 2 (After launch):** Evaluate performance
|
||||
- Monitor Robyn performance under real load
|
||||
- If Robyn handles <1000 users easily → stay with it
|
||||
- If hitting bottlenecks → profile to find hot paths
|
||||
|
||||
**Phase 3 (Optional, if needed):** Incremental C migration
|
||||
- Rewrite hot paths only (e.g., auth service)
|
||||
- Keep Evidence dashboards static (nginx serves directly)
|
||||
- Hybrid architecture: nginx → C (auth) → Robyn (business logic)
|
||||
|
||||
### Hybrid Architecture (Best of Both Worlds)
|
||||
|
||||
```
|
||||
nginx
|
||||
↓
|
||||
├─> Static files (Evidence dashboards) [nginx serves directly]
|
||||
├─> Auth endpoints (/login, /signup) [FastCGI C - future optimization]
|
||||
└─> Business logic (/api/*, /webhooks) [Robyn - for flexibility]
|
||||
```
|
||||
|
||||
**When to migrate:**
|
||||
- When Robyn becomes measurable bottleneck (>80% CPU under normal load)
|
||||
- When response times exceed targets (>100ms p95)
|
||||
- When memory usage becomes concern (>500MB for simple app)
|
||||
|
||||
**Philosophy:** Measure first, optimize second. Data-oriented approach means we don't guess about performance, we measure and optimize only when needed.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Week 1:** Evidence POC + local DuckDB export
|
||||
- Create Evidence project
|
||||
- Export coffee data from Iceberg
|
||||
- Build simple production dashboard
|
||||
- Validate local dev workflow
|
||||
|
||||
2. **Week 2:** Robyn app + basic auth + Evidence embedding
|
||||
- Set up Robyn project
|
||||
- SQLite user database
|
||||
- JWT authentication
|
||||
- Landing page (Jinja2 + htmx)
|
||||
- Serve Evidence dashboards at `/dashboards/*`
|
||||
|
||||
3. **Week 3:** Coffee-specific dashboards + Stripe
|
||||
- Build 3-4 core coffee dashboards
|
||||
- Integrate Stripe checkout
|
||||
- Webhook handling for subscriptions
|
||||
- Basic user account page
|
||||
|
||||
4. **Week 4:** Automated rebuild pipeline + deployment
|
||||
- Automate Evidence rebuild after SQLMesh runs
|
||||
- CI/CD pipeline for deployment
|
||||
- Deploy to supervisor or dedicated worker
|
||||
- Monitoring and analytics
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Hosted auth:** Evaluate Clerk vs Auth0 vs roll-our-own
|
||||
- Clerk: $25/mo for 1000 MAU, nice DX
|
||||
- Auth0: Free tier 7500 MAU, more enterprise
|
||||
- Roll our own: $0, full control, more code
|
||||
- **Decision:** Start with roll-our-own JWT (simplest), migrate to hosted if auth becomes complex
|
||||
|
||||
2. **DuckDB sync:** How often to export from Iceberg?
|
||||
- Option A: Daily (after SQLMesh runs)
|
||||
- Option B: After every SQLMesh plan
|
||||
- **Decision:** Daily for now, automate after SQLMesh completion in production
|
||||
|
||||
3. **Evidence build time:** If builds are slow, need caching strategy
|
||||
- Monitor build times in Phase 1
|
||||
- If >60s, investigate Evidence cache options
|
||||
- May need incremental builds
|
||||
|
||||
4. **Multi-commodity future:** How to expand beyond coffee?
|
||||
- Code structure should be generic (parameterize commodity filter)
|
||||
- Could launch cocoa.flows, wheat.supply, etc.
|
||||
- Evidence supports parameterized pages (easy to expand)
|
||||
|
||||
5. **C migration decision point:** What metrics trigger rewrite?
|
||||
- CPU >80% sustained under normal load
|
||||
- Response times >100ms p95
|
||||
- Memory >500MB for simple app
|
||||
- User complaints about slowness
|
||||
|
||||
## Success Metrics
|
||||
|
||||
**Phase 1 (POC):**
|
||||
- Evidence site builds successfully
|
||||
- Coffee data loads from DuckDB (<2s)
|
||||
- One dashboard renders with real data
|
||||
- Local dev server runs without errors
|
||||
|
||||
**Phase 2 (MVP):**
|
||||
- Robyn app runs and serves Evidence dashboards
|
||||
- JWT auth works (login/signup flow)
|
||||
- Landing page loads <2s
|
||||
- Dashboard access restricted to authenticated users
|
||||
|
||||
**Phase 3 (Launch):**
|
||||
- Stripe integration works (test payment succeeds)
|
||||
- 3-4 coffee dashboards functional
|
||||
- Automated deployment pipeline working
|
||||
- Monitoring in place (uptime, errors, performance)
|
||||
|
||||
**Phase 4 (Growth):**
|
||||
- User signups (track conversion rate)
|
||||
- Active subscribers (MRR growth)
|
||||
- Dashboard usage (which insights most valuable)
|
||||
- Performance metrics (response times, error rates)
|
||||
|
||||
## Cost Analysis
|
||||
|
||||
**Current costs (data pipeline):**
|
||||
- Supervisor: €4.49/mo (Hetzner CPX11)
|
||||
- Workers: €0.01-0.05/day (ephemeral)
|
||||
- R2 Storage: ~€0.10/mo (Iceberg catalog)
|
||||
- **Total: ~€5/mo**
|
||||
|
||||
**Additional costs (SaaS frontend):**
|
||||
- Domain: €10/year (beanflows.coffee)
|
||||
- Robyn hosting: €0 (runs on supervisor or dedicated worker €4.49/mo)
|
||||
- Stripe fees: 2.9% + €0.30 per transaction
|
||||
- **Total: ~€5-10/mo base cost**
|
||||
|
||||
**Scaling costs:**
|
||||
- If need dedicated worker for Robyn: +€4.49/mo
|
||||
- If migrate to C: No additional cost (same infrastructure)
|
||||
- Stripe fees scale with revenue (good problem to have)
|
||||
|
||||
## Next Steps (When Ready)
|
||||
|
||||
1. Create `dashboards/` directory and initialize Evidence.dev
|
||||
2. Create SQLMesh export model for coffee data
|
||||
3. Build simple coffee production dashboard
|
||||
4. Set up Robyn project structure
|
||||
5. Implement basic JWT auth
|
||||
6. Integrate Evidence dashboards into Robyn
|
||||
|
||||
**Decision point:** After Phase 1 POC, re-evaluate C migration based on Evidence.dev capabilities and development experience.
|
||||
|
||||
## References
|
||||
|
||||
- Evidence.dev: https://docs.evidence.dev/
|
||||
- Robyn: https://github.com/sparckles/robyn
|
||||
- kcgi (C CGI library): https://kristaps.bsd.lv/kcgi/
|
||||
- libjwt: https://github.com/benmcollins/libjwt
|
||||
- nginx auth_request: https://nginx.org/en/docs/http/ngx_http_auth_request_module.html
|
||||
- Stripe webhooks: https://stripe.com/docs/webhooks
|
||||
Reference in New Issue
Block a user