Update SQLMesh for R2 data access & Convert psd data to gzip

2025-11-02 00:26:01 +01:00
parent fc27d5f887
commit b702e6565a
26 changed files with 3553 additions and 1786 deletions
--- a/MULTIAGENT_SYSTEM_README.md
+++ b/MULTIAGENT_SYSTEM_README.md
@@ -0,0 +1,537 @@
+# Multi-Agent System for Claude Code
+
+A lean, pragmatic multi-agent system for software and data engineering tasks. Designed for small teams (1-2 people) building data products.
+
+---
+
+## Philosophy
+
+**Simple, Direct, Procedural Code**
+- Functions over classes
+- Data-oriented design
+- Explicit over clever
+- Solve actual problems, not general cases
+
+Inspired by Casey Muratori and Jonathan Blow's approach to software development.
+
+---
+
+## System Structure
+
+```
+agent_system/
+├── README.md                      # This file
+├── coding_philosophy.md           # Core principles (reference for all agents)
+├── orchestrator.md                # Lead Engineer Agent
+├── code_analysis_agent.md         # Code exploration agent
+├── implementation_agent.md        # Code writing agent
+└── testing_agent.md               # Testing agent
+```
+
+---
+
+## Agents
+
+### 1. Orchestrator (Lead Engineer Agent)
+**File:** `orchestrator.md`
+
+**Role:** Coordinates all work, decides when to use workers
+
+**Responsibilities:**
+- Analyze task complexity
+- Decide: handle directly or spawn workers
+- Create worker specifications
+- Synthesize worker outputs
+- Make architectural decisions
+
+**Use:** This is your main agent for Claude Code
+
+### 2. Code Analysis Agent
+**File:** `code_analysis_agent.md`
+
+**Role:** Explore and understand code (read-only)
+
+**Responsibilities:**
+- Map code structure
+- Trace data flow
+- Identify patterns and issues
+- Answer specific questions about codebase
+
+**Use:** When you need to understand existing code before making changes
+
+### 3. Implementation Agent
+**File:** `implementation_agent.md`
+
+**Role:** Write simple, direct code
+
+**Responsibilities:**
+- Implement features
+- Modify existing code
+- Write SQLMesh models
+- Create Robyn routes
+- Build evidence.dev dashboards
+
+**Use:** For building and modifying code
+
+### 4. Testing Agent
+**File:** `testing_agent.md`
+
+**Role:** Verify code works correctly
+
+**Responsibilities:**
+- Write pytest tests
+- Create SQL test queries
+- Test data transformations
+- Validate edge cases
+
+**Use:** For creating test suites
+
+---
+
+## How It Works
+
+### Decision Tree
+
+```
+Task received
+    ↓
+Can I do this directly in <30 tool calls?
+    ↓
+YES → Handle directly (90% of tasks)
+NO  → ↓
+    ↓
+Is this truly parallelizable?
+    ↓
+YES → Spawn 2-3 workers (10% of tasks)
+NO  → Handle directly anyway
+```
+
+**Golden Rule:** Most tasks should be handled directly by the orchestrator. Only use multiple agents when parallelization provides clear benefit.
+
+### Example: Simple Task (Direct)
+
+```
+User: "Add an API endpoint to get user activity"
+
+Orchestrator: This is straightforward, <20 tool calls
+    ↓
+    Handles directly:
+    - Creates route in src/routes/activity.py
+    - Queries data lake
+    - Returns JSON
+    - Tests manually
+    - Done
+```
+
+**No workers needed.** Fast and simple.
+
+### Example: Complex Task (Multi-Agent)
+
+```
+User: "Migrate ETL pipeline to SQLMesh"
+
+Orchestrator: This is complex, will benefit from parallel work
+    ↓
+    Phase 1 - Analysis:
+    Spawns Code Analysis Agent
+    - Maps existing pipeline
+    - Identifies transformations
+    - Documents dependencies
+    → Writes to .agent_work/analysis/
+    ↓
+    Phase 2 - Implementation:
+    Spawns 2 Implementation Agents in parallel
+    - Agent A: Extract models
+    - Agent B: Transform models
+    → Both write to .agent_work/implementation/
+    ↓
+    Phase 3 - Testing:
+    Spawns Testing Agent
+    - Validates output correctness
+    → Writes to .agent_work/testing/
+    ↓
+    Orchestrator synthesizes:
+    - Reviews all outputs
+    - Resolves conflicts
+    - Creates migration plan
+    - Done
+```
+
+**Parallelization saves time** on truly complex work.
+
+---
+
+## Tech Stack
+
+### Data Engineering
+- **SQLMesh** - Data transformation framework (SQL models)
+- **DuckDB** - Analytics database (OLAP queries)
+- **Iceberg** - Data lake table format (on R2 storage)
+- **ELT** - Extract → Load → Transform (in warehouse)
+
+### SaaS Application
+- **Robyn** - Python web framework
+  - Hosts landing page, auth, payment
+  - Serves evidence.dev build at `/dashboard/`
+- **evidence.dev** - BI dashboards (SQL + Markdown → static site)
+
+### Architecture
+```
+User → Robyn
+        ├── / (landing, auth, payment)
+        ├── /api/* (API endpoints)
+        └── /dashboard/* (evidence.dev build)
+```
+
+---
+
+## Working Directory
+
+All agent work goes into `.agent_work/` with feature-specific subdirectories:
+
+```
+project_root/
+├── README.md                          # Architecture, setup, tech stack
+├── CLAUDE.md                          # Memory: decisions, patterns, conventions
+├── .agent_work/                       # Agent work (add to .gitignore)
+│   ├── feature-user-dashboard/        # Feature-specific directory
+│   │   ├── project_state.md           # Track this feature's progress
+│   │   ├── analysis/
+│   │   │   └── findings.md
+│   │   ├── implementation/
+│   │   │   ├── feature.py
+│   │   │   └── notes.md
+│   │   └── testing/
+│   │       ├── test_feature.py
+│   │       └── results.md
+│   └── feature-payment-integration/   # Another feature
+│       ├── project_state.md
+│       ├── analysis/
+│       ├── implementation/
+│       └── testing/
+├── models/                            # SQLMesh models
+├── src/                               # Application code
+└── tests/                             # Final test suite
+```
+
+**Workflow:**
+1. New feature → Create branch: `git checkout -b feature-name`
+2. Create `.agent_work/feature-name/` directory
+3. Track progress in `.agent_work/feature-name/project_state.md`
+4. Update global context in `README.md` and `CLAUDE.md` as needed
+
+**Global vs Feature Context:**
+- **README.md**: Current architecture, tech stack, how to run
+- **CLAUDE.md**: Memory file - decisions, patterns, conventions to follow
+- **project_state.md**: Feature-specific progress (in `.agent_work/feature-name/`)
+
+**Why `.agent_work/` instead of `/tmp/`:**
+- Persists across sessions
+- Easy to review agent work
+- Can reference with normal paths
+- Keep or discard as needed
+- Feature-scoped organization
+
+**Add to `.gitignore`:**
+```
+.agent_work/
+```
+
+---
+
+## Usage in Claude Code
+
+### Setting Up
+
+1. Copy agent system files to your project:
+   ```
+   mkdir -p .claude/agents/
+   cp agent_system/* .claude/agents/
+   ```
+
+2. Add to `.gitignore`:
+   ```
+   .agent_work/
+   ```
+
+3. Create `.agent_work/` directory:
+   ```
+   mkdir -p .agent_work/{analysis,implementation,testing}
+   ```
+
+### Using the Orchestrator
+
+In Claude Code, load the orchestrator:
+
+```
+@orchestrator.md
+
+[Your request here]
+```
+
+The orchestrator will:
+1. Analyze the task
+2. Decide if workers are needed
+3. Spawn workers if beneficial
+4. Handle directly if simple
+5. Synthesize results
+6. Deliver solution
+
+### When Workers Are Spawned
+
+The orchestrator automatically reads the appropriate agent file when spawning:
+
+```
+Orchestrator reads: code_analysis_agent.md
+    ↓
+Creates specific task spec
+    ↓
+Spawns Code Analysis Agent with:
+    - Agent instructions (from file)
+    - Task specification
+    - Output location
+    ↓
+Worker executes independently
+    ↓
+Writes to .agent_work/analysis/
+```
+
+---
+
+## Coding Philosophy
+
+All agents follow these principles (from `coding_philosophy.md`):
+
+### Core Principles
+1. **Functions over classes** - Use functions unless you truly need classes
+2. **Data is data** - Simple structures (dicts, lists), not objects hiding behavior
+3. **Explicit over implicit** - No magic, no hiding
+4. **Simple control flow** - Straightforward if/else, early returns
+5. **Build minimum that works** - Solve actual problem, not general case
+
+### What to Avoid
+❌ Classes wrapping single functions  
+❌ Inheritance hierarchies  
+❌ Framework magic  
+❌ Over-abstraction "for future flexibility"  
+❌ Configuration as code pyramids  
+
+### What to Do
+✅ Write simple, direct functions  
+✅ Make data transformations obvious  
+✅ Handle errors explicitly  
+✅ Keep business logic in SQL when possible  
+✅ Think about performance  
+
+---
+
+## Examples
+
+### Example 1: Build Dashboard
+
+**Request:** "Create dashboard showing user activity trends"
+
+**Orchestrator Decision:** Moderate complexity, 2 independent tasks
+
+**Execution:**
+1. Setup:
+   - Create branch: `git checkout -b feature-user-dashboard`
+   - Create `.agent_work/feature-user-dashboard/`
+   - Read `README.md` and `CLAUDE.md` for context
+
+2. Spawns Implementation Agent A
+   - Creates SQLMesh model (user_activity_daily.sql)
+   - Writes to `.agent_work/feature-user-dashboard/implementation-data/`
+
+3. Spawns Implementation Agent B (parallel)
+   - Creates evidence.dev dashboard
+   - Writes to `.agent_work/feature-user-dashboard/implementation-viz/`
+
+4. Orchestrator synthesizes
+   - Reviews both outputs
+   - Tests evidence build
+   - Deploys together
+   - Updates `.agent_work/feature-user-dashboard/project_state.md`
+
+**Result:** Working dashboard with data model
+
+### Example 2: Fix Bug
+
+**Request:** "This query is timing out, fix it"
+
+**Orchestrator Decision:** Simple, direct handling
+
+**Execution:**
+1. Setup:
+   - Create branch: `git checkout -b fix-query-timeout`
+   - Create `.agent_work/fix-query-timeout/`
+
+2. Orchestrator handles directly
+   - Runs EXPLAIN ANALYZE
+   - Identifies missing index
+   - Creates index
+   - Tests performance
+   - Documents in `.agent_work/fix-query-timeout/project_state.md`
+
+**Result:** Query now fast, documented
+
+### Example 3: Large Refactor
+
+**Request:** "Migrate 50 Python files from sync to async"
+
+**Orchestrator Decision:** Complex, phased approach
+
+**Execution:**
+1. Phase 1: Analysis
+   - Code Analysis Agent maps dependencies
+   - Identifies blocking calls
+   - Writes to `.agent_work/analysis/`
+
+2. Phase 2: Implementation (parallel)
+   - Implementation Agent A: Core modules (20 files)
+   - Implementation Agent B: API routes (15 files)
+   - Implementation Agent C: Utils (15 files)
+   - All write to `.agent_work/implementation/`
+
+3. Phase 3: Testing
+   - Testing Agent validates async behavior
+   - Writes to `.agent_work/testing/`
+
+4. Orchestrator synthesizes
+   - Resolves conflicts
+   - Integration testing
+   - Migration plan
+
+**Result:** Migrated codebase with tests
+
+---
+
+## Best Practices
+
+### For Orchestrator
+- Default to handling directly
+- Spawn workers only for truly parallel work
+- Give workers focused, non-overlapping tasks
+- Use extended thinking for planning
+- Document decisions in `project_state.md`
+
+### For Worker Specs
+**Good:**
+```
+AGENT: implementation
+OBJECTIVE: Create SQLMesh model for user_activity_daily
+SCOPE: Create models/user_activity_daily.sql
+CONSTRAINTS: DuckDB SQL, incremental by date, partition by event_date
+OUTPUT: .agent_work/implementation/models/
+BUDGET: 20 tool calls
+```
+
+**Bad:**
+```
+AGENT: implementation
+OBJECTIVE: Help with the data stuff
+```
+
+### For Long Tasks
+- Maintain `.agent_work/project_state.md`
+- Update after each major phase
+- Use compaction if approaching context limits
+- Load files just-in-time (not entire codebase)
+
+---
+
+## Context Management
+
+### Just-in-Time Loading
+
+Don't load entire codebases:
+```bash
+# Good: Survey, then target
+find models/ -name "*.sql" | head -10
+rg "SELECT.*FROM" models/
+cat models/specific_model.sql
+
+# Bad: Load everything
+cat models/*.sql
+```
+
+### Project State Tracking
+
+For long tasks (>50 turns), maintain state:
+
+```markdown
+## Project: [Name]
+## Phase: [Current]
+
+### Completed
+- [x] Task 1 - Agent - Outcome
+
+### Current
+- [ ] Task 2 - Agent - Status
+
+### Decisions
+1. Decision - Rationale
+
+### Next Steps
+1. Step 1
+2. Step 2
+```
+
+---
+
+## Troubleshooting
+
+### "Workers are duplicating work"
+**Cause:** Vague task boundaries  
+**Fix:** Be more specific, assign non-overlapping files
+
+### "Coordination overhead too high"
+**Cause:** Task not parallelizable  
+**Fix:** Handle directly, don't use workers
+
+### "Context window exceeded"
+**Cause:** Loading too much data  
+**Fix:** Use JIT loading, summarize outputs
+
+### "Workers stepping on each other"
+**Cause:** Overlapping responsibilities  
+**Fix:** Separate by file/module, clear boundaries
+
+---
+
+## Summary
+
+**System:**
+- 4 agents: Orchestrator + 3 workers
+- Orchestrator handles most tasks directly (90%)
+- Workers used for truly complex, parallelizable work (10%)
+
+**Philosophy:**
+- Simple, direct, procedural code
+- Data-oriented design
+- Functions over classes
+- Build minimum that works
+
+**Tech Stack:**
+- Data: SQLMesh, DuckDB, Iceberg, ELT
+- SaaS: Robyn, evidence.dev
+- Testing: pytest, DuckDB SQL tests
+
+**Working Directory:**
+- `.agent_work/` for all agent outputs
+- Add to `.gitignore`
+- Review, then move to final locations
+
+**Golden Rule:** When in doubt, go simpler. Most tasks don't need multiple agents.
+
+---
+
+## Getting Started
+
+1. Read `coding_philosophy.md` to understand principles
+2. Use `orchestrator.md` as your main agent in Claude Code
+3. Let orchestrator decide when to spawn workers
+4. Review outputs in `.agent_work/`
+5. Iterate based on results
+
+Start simple. Add complexity only when needed.