ICE extraction overhaul: API discovery + aging report + historical backfill
- Replace brittle ICE_STOCKS_URL env var with API-based URL discovery via the private ICE Report Center JSON API (no auth required) - Add rolling CSV → XLS fallback in extract_ice_stocks() using find_latest_report() from ice_api.py - Add ice_api.py: fetch_report_listings(), find_latest_report() with pagination up to MAX_API_PAGES - Add xls_parse.py: detect_file_format() (magic bytes), xls_to_rows() using xlrd for OLE2/BIFF XLS files - Add extract_ice_aging(): monthly certified stock aging report by age bucket × port → ice_aging/ landing dir - Add extract_ice_historical(): 30-year EOM by-port stocks from static ICE URL → ice_stocks_by_port/ landing dir - Add xlrd>=2.0.1 (parse XLS), xlwt>=1.3.0 (dev, test fixtures) - Add SQLMesh raw + foundation models for both new datasets - Add ice_aging_glob(), ice_stocks_by_port_glob() macros - Add extract_ice_aging + extract_ice_historical pipeline entries - Add 12 unit tests (format detection, XLS roundtrip, API mock, CSV output) Seed files (data/landing/ice_aging/seed/ and ice_stocks_by_port/seed/) must be created locally — data/ is gitignored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
49
transform/sqlmesh_materia/models/raw/ice_aging_stocks.sql
Normal file
49
transform/sqlmesh_materia/models/raw/ice_aging_stocks.sql
Normal file
@@ -0,0 +1,49 @@
|
||||
-- Raw ICE certified stock aging report — technical ingestion layer.
|
||||
--
|
||||
-- Reads monthly aging report gzip CSVs from the landing directory.
|
||||
-- All values are varchar; casting happens in foundation.fct_ice_aging_stocks.
|
||||
--
|
||||
-- Source: ICE Report Center (Certified Stock Aging Report)
|
||||
-- Coverage: varies by download history
|
||||
-- Frequency: monthly (ICE updates after each delivery month)
|
||||
|
||||
MODEL (
|
||||
name raw.ice_aging_stocks,
|
||||
kind FULL,
|
||||
cron '@daily',
|
||||
columns (
|
||||
report_date varchar,
|
||||
age_bucket varchar,
|
||||
antwerp_bags varchar,
|
||||
hamburg_bremen_bags varchar,
|
||||
houston_bags varchar,
|
||||
miami_bags varchar,
|
||||
new_orleans_bags varchar,
|
||||
new_york_bags varchar,
|
||||
total_bags varchar,
|
||||
filename varchar
|
||||
)
|
||||
);
|
||||
|
||||
SELECT
|
||||
report_date,
|
||||
age_bucket,
|
||||
antwerp_bags,
|
||||
hamburg_bremen_bags,
|
||||
houston_bags,
|
||||
miami_bags,
|
||||
new_orleans_bags,
|
||||
new_york_bags,
|
||||
total_bags,
|
||||
filename
|
||||
FROM read_csv(
|
||||
@ice_aging_glob(),
|
||||
delim = ',',
|
||||
encoding = 'utf-8',
|
||||
compression = 'gzip',
|
||||
header = true,
|
||||
union_by_name = true,
|
||||
filename = true,
|
||||
all_varchar = true,
|
||||
ignore_errors = true
|
||||
)
|
||||
@@ -0,0 +1,51 @@
|
||||
-- Raw ICE historical end-of-month warehouse stocks by port — technical ingestion layer.
|
||||
--
|
||||
-- Reads historical by-port stock gzip CSVs from the landing directory.
|
||||
-- All values are varchar; casting happens in foundation.fct_ice_warehouse_stocks_by_port.
|
||||
--
|
||||
-- Source: ICE (EOM_KC_cert_stox_by_port_nov96-present.xls)
|
||||
-- Coverage: November 1996 to present
|
||||
-- Frequency: monthly (ICE updates the static file monthly)
|
||||
|
||||
MODEL (
|
||||
name raw.ice_warehouse_stocks_by_port,
|
||||
kind FULL,
|
||||
cron '@daily',
|
||||
columns (
|
||||
report_date varchar,
|
||||
new_york_bags varchar,
|
||||
new_orleans_bags varchar,
|
||||
houston_bags varchar,
|
||||
miami_bags varchar,
|
||||
antwerp_bags varchar,
|
||||
hamburg_bremen_bags varchar,
|
||||
barcelona_bags varchar,
|
||||
virginia_bags varchar,
|
||||
total_bags varchar,
|
||||
filename varchar
|
||||
)
|
||||
);
|
||||
|
||||
SELECT
|
||||
report_date,
|
||||
new_york_bags,
|
||||
new_orleans_bags,
|
||||
houston_bags,
|
||||
miami_bags,
|
||||
antwerp_bags,
|
||||
hamburg_bremen_bags,
|
||||
barcelona_bags,
|
||||
virginia_bags,
|
||||
total_bags,
|
||||
filename
|
||||
FROM read_csv(
|
||||
@ice_stocks_by_port_glob(),
|
||||
delim = ',',
|
||||
encoding = 'utf-8',
|
||||
compression = 'gzip',
|
||||
header = true,
|
||||
union_by_name = true,
|
||||
filename = true,
|
||||
all_varchar = true,
|
||||
ignore_errors = true
|
||||
)
|
||||
Reference in New Issue
Block a user