Deeman
d2352c1876
Simplify SQLMesh to use single prod gateway with virtual environments
...
- Remove dev gateway (local DuckDB file no longer needed)
- Single prod gateway connects to R2 Iceberg catalog
- Use virtual environments for dev isolation (e.g., dev_<username>)
- Update CLAUDE.md with new workflow and environment strategy
- Create comprehensive transform/sqlmesh_materia/README.md
Benefits:
- Simpler configuration (one gateway instead of two)
- All environments use same R2 Iceberg catalog
- SQLMesh handles environment isolation automatically
- No need to maintain local 13GB materia_dev.db file
- before_all hooks only run for prod gateway (no conditional logic needed)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-13 21:47:04 +02:00
Deeman
6536724e00
Fix SQLMesh config: remove invalid init_script parameter
...
- Remove init_script from DuckDB connection config (not a valid parameter)
- Move R2 Iceberg catalog initialization to before_all hooks
- Hooks run before sqlmesh plan/run commands
- Uses SQLMesh @env_var() macro syntax for environment variables
Fixes CI/CD error: 'invalid duckdb connection config: invalid field init_script'
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-13 21:31:56 +02:00
Deeman
55bb84f0fa
implement cli/infra update cicd
2025-10-12 21:00:41 +02:00
Deeman
025dda16c6
update dedupe logic -> much faster now
2025-10-07 22:32:45 +02:00
Deeman
da89c2bf6e
update staging pipeline
2025-10-07 22:20:48 +02:00
Deeman
0a409acbea
update path
2025-09-10 18:56:32 +02:00
Deeman
85704a4bf1
Change layer naming
2025-09-10 18:46:18 +02:00
Deeman
f5f2dbc7a5
refactor
2025-08-25 20:50:25 +02:00
Simon Dmsn
5588be152b
Update 3 files
...
- /notebooks/03_Extraction.ipynb
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata_1_filter_silver_layer.sql
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata_2_filter_gold_layer.sql
2025-08-01 14:52:55 +00:00
Simon Dmsn
1c87488cc7
Update 4 files
...
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata.sql
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata_1_filter_silver_layer.sql
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata_2_filter_gold_layer.sql
- /transform/sqlmesh_materia/models/staging/stg_psd_alldata_0.sql
2025-08-01 14:45:34 +00:00
Simon Dmsn
82b27e7c55
Update 2 files
...
- /transform/sqlmesh_materia/seeds/commodity_exchange_codes.csv
- /transform/sqlmesh_materia/seeds/psd_codes_exchange_codes_merge.csv
2025-08-01 14:41:48 +00:00
Simon Dmsn
9d7cc4e1fb
Update file commodity_exchange_codes.csv
2025-08-01 14:26:19 +00:00
Simon Dmsn
4ad4386ccc
Update 2 files
...
- /transform/sqlmesh_materia/models/staging/Commodity Exchange Codes.xls
- /transform/sqlmesh_materia/seeds/commodity_exchange_codes.csv
2025-08-01 14:24:26 +00:00
Simon Dmsn
918b0071b1
Update file Commodity Exchange Codes.xls
2025-08-01 14:22:01 +00:00
Deeman
91f8968990
remove comment
2025-07-31 19:48:18 +02:00
Deeman
641f794d61
fix seeds; update models
2025-07-27 22:49:37 +02:00
Deeman
c0d8f60d1c
add reference data
2025-07-27 18:28:30 +02:00
Deeman
8b5d05b3c2
raw ingest model
2025-07-27 15:40:41 +02:00
Deeman
f5c73e32c5
testing sqlmesh
2025-07-27 00:18:14 +02:00
Deeman
9baa0d185c
testing sqlmesh
2025-07-27 00:18:03 +02:00
Deeman
f0de8a505b
update projects to packages
2025-07-26 22:32:47 +02:00
Deeman
0bbbd25b68
update projects to packages
2025-07-26 22:32:37 +02:00
Deeman
0a60bf8746
finish historical extraction
2025-07-13 23:20:55 +02:00
Deeman
b8ad73202c
finish historical extraction
2025-07-13 23:20:50 +02:00
Deeman
70bd8a52db
async is requesting stuff too fast
2025-07-13 18:08:25 +02:00
Deeman
8143c6ed8e
async is requesting stuff too fast
2025-07-13 18:08:19 +02:00
Deeman
c3c281fcd8
update structure
2025-07-08 22:41:59 +02:00
