f207fb441d
Implements automated supervisor instance deployment that runs scheduled pipelines using a TigerBeetle-inspired continuous orchestration pattern. Infrastructure changes: - Update Pulumi to use existing R2 buckets (beanflows-artifacts, beanflows-data-prod) - Rename scheduler → supervisor, optimize to CCX11 (€4/mo) - Remove always-on worker (workers are now ephemeral only) - Add artifacts bucket resource for CLI/pipeline packages Supervisor architecture: - supervisor.sh: Continuous loop checking schedules every 15 minutes - Self-updating: Checks for new CLI versions hourly - Fixed schedules: Extract at 2 AM UTC, Transform at 3 AM UTC - systemd service for automatic restart on failure - Logs to systemd journal for observability CI/CD changes: - deploy:infra now runs on every master push (not just on changes) - New deploy:supervisor job: * Deploys supervisor.sh and systemd service * Installs latest materia CLI from R2 * Configures environment with Pulumi ESC secrets * Restarts supervisor service Future enhancements documented: - SQLMesh-aware scheduling (check models before running) - Model tags for worker sizing (heavy/distributed hints) - Multi-pipeline support, distributed execution - Cost optimization with multi-cloud spot pricing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
30 lines
562 B
Desktop File
30 lines
562 B
Desktop File
[Unit]
|
|
Description=Materia Supervisor - Pipeline Orchestration
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=root
|
|
WorkingDirectory=/opt/materia
|
|
Environment="PATH=/usr/local/bin:/usr/bin:/bin:/root/.pulumi/bin"
|
|
EnvironmentFile=/opt/materia/.env
|
|
|
|
# Restart policy
|
|
Restart=always
|
|
RestartSec=10
|
|
|
|
# Resource limits
|
|
LimitNOFILE=65536
|
|
|
|
# Logging
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
SyslogIdentifier=materia-supervisor
|
|
|
|
# Execute supervisor script
|
|
ExecStart=/opt/materia/supervisor.sh
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|