feat(extract): add OpenWeatherMap daily weather extractor
Adds extract/openweathermap package with daily weather extraction for 8
coffee-growing regions (Brazil, Vietnam, Colombia, Ethiopia, Honduras,
Guatemala, Indonesia). Feeds crop stress signal for commodity sentiment score.
Extractor:
- OWM One Call API 3.0 / Day Summary — one JSON.gz per (location, date)
- extract_weather: daily, fetches yesterday + today (16 calls max)
- extract_weather_backfill: fills 2020-01-01 to yesterday, capped at 500
calls/run with resume cursor '{location_id}:{date}' for crash safety
- Full idempotency via file existence check; state tracking via extract_core
SQLMesh:
- seeds.weather_locations (8 regions with lat/lon/variety)
- foundation.fct_weather_daily: INCREMENTAL_BY_TIME_RANGE, grain
(location_id, observation_date), dedup via hash key, crop stress flags:
is_frost (<2°C), is_heat_stress (>35°C), is_drought (<1mm), in_growing_season
Landing path: LANDING_DIR/weather/{location_id}/{year}/{date}.json.gz
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
16
uv.lock
generated
16
uv.lock
generated
@@ -14,6 +14,7 @@ members = [
|
||||
"extract-core",
|
||||
"ice-stocks",
|
||||
"materia",
|
||||
"openweathermap",
|
||||
"psdonline",
|
||||
"sqlmesh-materia",
|
||||
]
|
||||
@@ -1778,6 +1779,21 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/cf/df/d3f1ddf4bb4cb50ed9b1139cc7b1c54c34a1e7ce8fd1b9a37c0d1551a6bd/opentelemetry_api-1.39.1-py3-none-any.whl", hash = "sha256:2edd8463432a7f8443edce90972169b195e7d6a05500cd29e6d13898187c9950", size = 66356, upload-time = "2025-12-11T13:32:17.304Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openweathermap"
|
||||
version = "0.1.0"
|
||||
source = { editable = "extract/openweathermap" }
|
||||
dependencies = [
|
||||
{ name = "extract-core" },
|
||||
{ name = "niquests" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "extract-core", editable = "extract/extract_core" },
|
||||
{ name = "niquests", specifier = ">=3.14.1" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "orjson"
|
||||
version = "3.11.7"
|
||||
|
||||
Reference in New Issue
Block a user