feat: copier update v0.9.0 — extraction docs, state tracking, architecture guides
Sync template from 29ac25b → v0.9.0 (29 template commits). Due to template's _subdirectory migration, new files were manually rendered rather than auto-merged by copier. New files: - .claude/CLAUDE.md + coding_philosophy.md (agent instructions) - extract utils.py: SQLite state tracking for extraction runs - extract/transform READMEs: architecture & pattern documentation - infra/supervisor: systemd service + orchestration script - Per-layer model READMEs (raw, staging, foundation, serving) Also fixes copier-answers.yml (adds 4 feature toggles, removes stale payment_provider key) and scopes CLAUDE.md gitignore to root only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
106
.claude/CLAUDE.md
Normal file
106
.claude/CLAUDE.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# CLAUDE.md — Padelnomics
|
||||
|
||||
This file tells Claude Code how to work in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Padelnomics is a SaaS application built with Quart (async Python), HTMX, and SQLite.
|
||||
It includes a full data pipeline:
|
||||
|
||||
```
|
||||
External APIs → extract → landing zone → SQLMesh transform → DuckDB → web app
|
||||
```
|
||||
|
||||
**Packages** (uv workspace):
|
||||
- `web/` — Quart + HTMX web application (auth, billing, dashboard)
|
||||
- `extract/padelnomics_extract/` — data extraction to local landing zone
|
||||
- `transform/sqlmesh_padelnomics/` — 4-layer SQL transformation (raw → staging → foundation → serving)
|
||||
- `src/padelnomics/` — CLI utilities, export_serving helper
|
||||
|
||||
## Skills: invoke these for domain tasks
|
||||
|
||||
### Working on extraction or transformation?
|
||||
|
||||
Use the **`data-engineer`** skill for:
|
||||
- Designing or reviewing SQLMesh model logic
|
||||
- Adding a new data source (extract + raw + staging models)
|
||||
- Performance tuning DuckDB queries
|
||||
- Data modeling decisions (dimensions, facts, aggregates)
|
||||
- Understanding the 4-layer architecture
|
||||
|
||||
```
|
||||
/data-engineer (or ask Claude to invoke it)
|
||||
```
|
||||
|
||||
### Working on the web app UI or frontend?
|
||||
|
||||
Use the **`frontend-design`** skill for UI components, templates, or dashboard layouts.
|
||||
|
||||
### Working on payments or subscriptions?
|
||||
|
||||
Use the **`paddle-integration`** skill for billing, webhooks, and subscription logic.
|
||||
|
||||
## Key commands
|
||||
|
||||
```bash
|
||||
# Install all dependencies
|
||||
uv sync --all-packages
|
||||
|
||||
# Lint & format
|
||||
ruff check .
|
||||
ruff format .
|
||||
|
||||
# Run tests
|
||||
uv run pytest tests/ -v
|
||||
|
||||
# Dev server
|
||||
./scripts/dev_run.sh
|
||||
|
||||
# Extract data
|
||||
LANDING_DIR=data/landing uv run extract
|
||||
|
||||
# SQLMesh plan + run (from repo root)
|
||||
uv run sqlmesh -p transform/sqlmesh_padelnomics plan
|
||||
uv run sqlmesh -p transform/sqlmesh_padelnomics plan prod
|
||||
|
||||
# Export serving tables (run after SQLMesh)
|
||||
DUCKDB_PATH=local.duckdb SERVING_DUCKDB_PATH=analytics.duckdb \
|
||||
uv run python -m padelnomics.export_serving
|
||||
```
|
||||
|
||||
## Architecture documentation
|
||||
|
||||
| Topic | File |
|
||||
|-------|------|
|
||||
| Extraction patterns, state tracking, adding new sources | `extract/padelnomics_extract/README.md` |
|
||||
| 4-layer SQLMesh architecture, materialization strategy | `transform/sqlmesh_padelnomics/README.md` |
|
||||
| Two-file DuckDB architecture (SQLMesh lock isolation) | `src/padelnomics/export_serving.py` docstring |
|
||||
|
||||
## Pipeline data flow
|
||||
|
||||
```
|
||||
data/landing/
|
||||
└── padelnomics/{year}/{etag}.csv.gz ← extraction output
|
||||
|
||||
local.duckdb ← SQLMesh exclusive (raw → staging → foundation → serving)
|
||||
|
||||
analytics.duckdb ← serving tables only, web app read-only
|
||||
└── serving.* ← atomically replaced by export_serving.py
|
||||
```
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `LANDING_DIR` | `data/landing` | Landing zone root (extraction writes here) |
|
||||
| `DUCKDB_PATH` | `local.duckdb` | SQLMesh pipeline DB (exclusive write) |
|
||||
| `SERVING_DUCKDB_PATH` | `analytics.duckdb` | Read-only DB for web app |
|
||||
|
||||
## Coding philosophy
|
||||
|
||||
- **Simple and procedural** — functions over classes, no "Manager" patterns
|
||||
- **Idempotent operations** — running twice produces the same result
|
||||
- **Explicit assertions** — assert preconditions at function boundaries
|
||||
- **Bounded operations** — set timeouts, page limits, buffer sizes
|
||||
|
||||
Read `coding_philosophy.md` (if present) for the full guide.
|
||||
Reference in New Issue
Block a user