fix(pipeline): stg_playtomic_availability glob reads all files, filters by date range
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s

The @start_ds in the glob pattern only matched files for the first day
of the batch, so incremental restates only loaded 1 day of data.
Changed to wildcard glob with explicit BETWEEN @start_ds AND @end_ds
filter on the date column.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-03-10 15:48:10 +01:00
parent 511a0ebac7
commit bd7fa1ae9a

View File

@@ -34,7 +34,7 @@ all_jsonl AS (
tenant_id,
slots AS slots_json
FROM read_json(
@LANDING_DIR || '/playtomic/*/*/availability_' || @start_ds || '*.jsonl.gz',
@LANDING_DIR || '/playtomic/*/*/availability_*.jsonl.gz',
format = 'newline_delimited',
columns = {
date: 'VARCHAR',
@@ -46,6 +46,7 @@ all_jsonl AS (
filename = true
)
WHERE tenant_id IS NOT NULL
AND CAST(date AS DATE) BETWEEN @start_ds AND @end_ds
),
raw_resources AS (
SELECT