fix(transform+content): unblock SQLMesh plan — three pipeline fixes
stg_playtomic_availability:
- Add maximum_object_size = 134217728 (128 MB) to both read_json calls;
daily files exceed the 16 MB default as venue count grows
- Add seed recheck file (1970-01-01_recheck_00.json.gz, gitignored with data/)
to avoid READ_JSON IOException when no recheck files exist
pseo_city_costs_de + pseo_city_pricing:
- Add QUALIFY ROW_NUMBER() OVER (PARTITION BY city_slug ...) = 1 to
deduplicate rows caused by dim_cities' loose LIKE join; reduces
pseo_city_costs_de from 2.6M → 222 rows (one per unique city)
content/__init__.py:
- DuckDB lowercases all column names at rest ("ratePeak" → "ratepeak"),
so calc_overrides dict comprehension never matched DEFAULTS keys.
Fix: build case-insensitive reverse map {k.lower(): k} and normalise
row keys before lookup. Applied in both generate_articles() and
preview_article().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -306,7 +306,14 @@ async def generate_articles(
|
||||
# Calculator content type: create scenario
|
||||
scenario_slug = None
|
||||
if config["content_type"] == "calculator":
|
||||
calc_overrides = {k: v for k, v in row.items() if k in DEFAULTS}
|
||||
# DuckDB lowercases all column names; build a case-insensitive
|
||||
# reverse map so "ratepeak" (stored) matches "ratePeak" (DEFAULTS).
|
||||
_defaults_ci = {k.lower(): k for k in DEFAULTS}
|
||||
calc_overrides = {
|
||||
_defaults_ci[k.lower()]: v
|
||||
for k, v in row.items()
|
||||
if k.lower() in _defaults_ci and v is not None
|
||||
}
|
||||
state = validate_state(calc_overrides)
|
||||
d = calc(state, lang=lang)
|
||||
|
||||
@@ -479,7 +486,14 @@ async def preview_article(
|
||||
|
||||
# Calculator: compute scenario in-memory
|
||||
if config["content_type"] == "calculator":
|
||||
calc_overrides = {k: v for k, v in row.items() if k in DEFAULTS}
|
||||
# DuckDB lowercases all column names; build a case-insensitive
|
||||
# reverse map so "ratepeak" (stored) matches "ratePeak" (DEFAULTS).
|
||||
_defaults_ci = {k.lower(): k for k in DEFAULTS}
|
||||
calc_overrides = {
|
||||
_defaults_ci[k.lower()]: v
|
||||
for k, v in row.items()
|
||||
if k.lower() in _defaults_ci and v is not None
|
||||
}
|
||||
state = validate_state(calc_overrides)
|
||||
calc(state, lang=lang) # validate state produces valid output
|
||||
ctx["scenario_slug"] = slug + "-" + str(row[natural_key])
|
||||
|
||||
Reference in New Issue
Block a user