ff7301d6a8
- Replace brittle ICE_STOCKS_URL env var with API-based URL discovery via the private ICE Report Center JSON API (no auth required) - Add rolling CSV → XLS fallback in extract_ice_stocks() using find_latest_report() from ice_api.py - Add ice_api.py: fetch_report_listings(), find_latest_report() with pagination up to MAX_API_PAGES - Add xls_parse.py: detect_file_format() (magic bytes), xls_to_rows() using xlrd for OLE2/BIFF XLS files - Add extract_ice_aging(): monthly certified stock aging report by age bucket × port → ice_aging/ landing dir - Add extract_ice_historical(): 30-year EOM by-port stocks from static ICE URL → ice_stocks_by_port/ landing dir - Add xlrd>=2.0.1 (parse XLS), xlwt>=1.3.0 (dev, test fixtures) - Add SQLMesh raw + foundation models for both new datasets - Add ice_aging_glob(), ice_stocks_by_port_glob() macros - Add extract_ice_aging + extract_ice_historical pipeline entries - Add 12 unit tests (format detection, XLS roundtrip, API mock, CSV output) Seed files (data/landing/ice_aging/seed/ and ice_stocks_by_port/seed/) must be created locally — data/ is gitignored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
50 lines
1.2 KiB
SQL
50 lines
1.2 KiB
SQL
-- Raw ICE certified stock aging report — technical ingestion layer.
|
|
--
|
|
-- Reads monthly aging report gzip CSVs from the landing directory.
|
|
-- All values are varchar; casting happens in foundation.fct_ice_aging_stocks.
|
|
--
|
|
-- Source: ICE Report Center (Certified Stock Aging Report)
|
|
-- Coverage: varies by download history
|
|
-- Frequency: monthly (ICE updates after each delivery month)
|
|
|
|
MODEL (
|
|
name raw.ice_aging_stocks,
|
|
kind FULL,
|
|
cron '@daily',
|
|
columns (
|
|
report_date varchar,
|
|
age_bucket varchar,
|
|
antwerp_bags varchar,
|
|
hamburg_bremen_bags varchar,
|
|
houston_bags varchar,
|
|
miami_bags varchar,
|
|
new_orleans_bags varchar,
|
|
new_york_bags varchar,
|
|
total_bags varchar,
|
|
filename varchar
|
|
)
|
|
);
|
|
|
|
SELECT
|
|
report_date,
|
|
age_bucket,
|
|
antwerp_bags,
|
|
hamburg_bremen_bags,
|
|
houston_bags,
|
|
miami_bags,
|
|
new_orleans_bags,
|
|
new_york_bags,
|
|
total_bags,
|
|
filename
|
|
FROM read_csv(
|
|
@ice_aging_glob(),
|
|
delim = ',',
|
|
encoding = 'utf-8',
|
|
compression = 'gzip',
|
|
header = true,
|
|
union_by_name = true,
|
|
filename = true,
|
|
all_varchar = true,
|
|
ignore_errors = true
|
|
)
|