c1d00dcdc4
Remove distributed R2/Iceberg/SSH pipeline architecture in favor of
local subprocess execution with NVMe storage. Landing data backed up
to R2 via rclone timer.
- Strip Iceberg catalog, httpfs, boto3, paramiko, prefect, pyarrow
- Pipelines run via subprocess.run() with bounded timeouts
- Extract writes to {LANDING_DIR}/psd/{year}/{month}/{etag}.csv.gzip
- SQLMesh reads LANDING_DIR variable, writes to DUCKDB_PATH
- Delete unused provider stubs (ovh, scaleway, oracle)
- Add rclone systemd timer for R2 backup every 6h
- Update supervisor to run pipelines with env vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
27 lines
567 B
Desktop File
27 lines
567 B
Desktop File
[Unit]
|
|
Description=Materia Supervisor - Pipeline Orchestration
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=root
|
|
WorkingDirectory=/opt/materia
|
|
ExecStart=/opt/materia/infra/supervisor/supervisor.sh
|
|
Restart=always
|
|
RestartSec=10
|
|
EnvironmentFile=/opt/materia/.env
|
|
Environment=LANDING_DIR=/data/materia/landing
|
|
Environment=DUCKDB_PATH=/data/materia/lakehouse.duckdb
|
|
|
|
# Resource limits
|
|
LimitNOFILE=65536
|
|
|
|
# Logging
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
SyslogIdentifier=materia-supervisor
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|