fix(pipeline): make availability chain incremental + fix supervisor
Convert the availability chain (stg_playtomic_availability → fct_availability_slot → fct_daily_availability) from FULL to INCREMENTAL_BY_TIME_RANGE so sqlmesh run processes only new daily intervals instead of re-reading all files. Supervisor changes: - run_transform(): plan prod --auto-apply → run prod (evaluates missing cron intervals, picks up new data) - git_pull_and_sync(): add plan prod --auto-apply before re-exec so model code changes are applied on deploy - supervisor.sh: same plan → run change Staging model uses a date-scoped glob (@start_ds) to read only the current interval's files. snapshot_date cast to DATE (was VARCHAR) as required by time_column. Clean up redundant TRY_CAST(snapshot_date AS DATE) in venue_pricing_benchmarks since it's already DATE from foundation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -33,10 +33,10 @@ do
|
||||
DUCKDB_PATH="${DUCKDB_PATH:-/data/padelnomics/lakehouse.duckdb}" \
|
||||
uv run --package padelnomics_extract extract
|
||||
|
||||
# Transform — plan detects new/changed models; run only executes existing plans.
|
||||
# Transform — run evaluates missing daily intervals for incremental models.
|
||||
LANDING_DIR="${LANDING_DIR:-/data/padelnomics/landing}" \
|
||||
DUCKDB_PATH="${DUCKDB_PATH:-/data/padelnomics/lakehouse.duckdb}" \
|
||||
uv run sqlmesh -p transform/sqlmesh_padelnomics plan prod --auto-apply
|
||||
uv run sqlmesh -p transform/sqlmesh_padelnomics run prod
|
||||
|
||||
# Export serving tables to analytics.duckdb (atomic swap).
|
||||
# The web app detects the inode change on next query — no restart needed.
|
||||
|
||||
Reference in New Issue
Block a user