refactor(transform): remove raw layer, read landing zone directly
- Delete 6 data raw models (coffee_prices, cot_disaggregated, ice_*, psd_data) — pure read_csv passthroughs with no added value - Move 3 PSD seed models raw/ → seeds/, rename schema raw.* → seeds.* - Update staging.psdalldata__commodity: read_csv(@psd_glob()) directly, join seeds.psd_* instead of raw.psd_* - Update 5 foundation models: inline read_csv() with src CTE, removing raw.* dependency (fct_coffee_prices, fct_cot_positioning, fct_ice_*) - Remove fixture-based SQLMesh test that depended on raw.cot_disaggregated (unit tests incompatible with inline read_csv; integration run covers this) - Update readme.md: 3-layer architecture (staging/foundation → serving) Landing files are immutable and content-addressed — the landing directory is the audit trail. A raw SQL layer duplicated file bytes into DuckDB with no added value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
-- Foundation fact: CFTC COT positioning, weekly grain, all commodities.
|
||||
--
|
||||
-- Casts raw varchar columns to proper types, cleans column names,
|
||||
-- computes net positions (long - short) per trader category, and
|
||||
-- deduplicates via hash key. Covers all commodities — filtering to
|
||||
-- Reads directly from the landing zone, casts varchar columns to proper types,
|
||||
-- cleans column names, computes net positions (long - short) per trader category,
|
||||
-- and deduplicates via hash key. Covers all commodities — filtering to
|
||||
-- a specific commodity happens in the serving layer.
|
||||
--
|
||||
-- Grain: one row per (cftc_commodity_code, report_date, cftc_contract_market_code)
|
||||
@@ -19,7 +19,19 @@ MODEL (
|
||||
cron '@daily'
|
||||
);
|
||||
|
||||
WITH cast_and_clean AS (
|
||||
WITH src AS (
|
||||
SELECT * FROM read_csv(
|
||||
@cot_glob(),
|
||||
compression = 'gzip',
|
||||
header = true,
|
||||
union_by_name = true,
|
||||
filename = true,
|
||||
all_varchar = true,
|
||||
max_line_size = 10000000
|
||||
)
|
||||
),
|
||||
|
||||
cast_and_clean AS (
|
||||
SELECT
|
||||
-- Identifiers
|
||||
trim(market_and_exchange_names) AS market_and_exchange_name,
|
||||
@@ -103,7 +115,7 @@ WITH cast_and_clean AS (
|
||||
prod_merc_positions_long_all,
|
||||
prod_merc_positions_short_all
|
||||
) AS hkey
|
||||
FROM raw.cot_disaggregated
|
||||
FROM src
|
||||
-- Reject rows with null commodity code or malformed date
|
||||
WHERE trim(cftc_commodity_code) IS NOT NULL
|
||||
AND len(trim(cftc_commodity_code)) > 0
|
||||
|
||||
Reference in New Issue
Block a user