Update SQLMesh for R2 data access & Convert psd data to gzip

This commit is contained in:
Hendrik Dreesmann
2025-11-02 00:26:01 +01:00
parent fc27d5f887
commit b702e6565a
26 changed files with 3553 additions and 1786 deletions

View File

@@ -8,9 +8,25 @@ Materia is a commodity data analytics platform built on a modern data engineerin
**Tech Stack:**
- Python 3.13 with `uv` package manager
- SQLMesh for SQL transformation and orchestration
- DuckDB as the analytical database
- Workspace structure with separate extract and transform packages
Build for simplicity and performance. Data oriented design.
Data Platform:
- duckdb
- cloudflare r2 iceberg catalog
- sqlmesh
- ephemeral workers on hetzner/oracle/ovh
- custom simple scheduler cli
Saas
- Quart framework - async flask api implementation
- No Js frameworks
- As simple as possible, as performant as possible
- DaisyUI
- htmx if interactivity needed (use sparingly)
- embedded dashboards with evidence.dev, static site BI generator
- Paddle as payment provider
Manager CLI
Materia cli is used to manage the whole project, like running pipelines, starting servers etc. and will be continuously extended.
## Environment Setup
@@ -374,3 +390,5 @@ All data is stored in Cloudflare R2 Data Catalog (Apache Iceberg) via REST API:
- NEVER hardcode secrets in plaintext
- Never add ssh keys to the git repo!
- If there is a simpler more direct solution and there is no other tradeoff, always choose the simpler solution
- Logo and wordmark of beanflows.coffee are stored in assets/
- Architecture change/feature plans of the architecture agent and their progress/status are stored in .claude/plans