docs: update inventory with ICE options research findings
- yfinance confirmed not viable (OPRA only, KC=F not covered) - CFTC COT combined report is the free immediate path (URL change only) - ICE Report Center settlement data viable with WebICE login automation - Barchart OnDemand has correct coverage but requires paid subscription - All OpenBB providers, Polygon.io, Nasdaq Data Link confirmed no KC=F coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -20,8 +20,9 @@ Purpose: Identify and track data sources feeding the BeanFlows DuckDB analytics
|
|||||||
| ICE Report Center — aging stocks | Warehouse / Inventory | ✅ Ingested | 4 | None | `extract_ice_stocks` → `fct_ice_aging_stocks` → `serving.ice_aging_stocks` |
|
| ICE Report Center — aging stocks | Warehouse / Inventory | ✅ Ingested | 4 | None | `extract_ice_stocks` → `fct_ice_aging_stocks` → `serving.ice_aging_stocks` |
|
||||||
| USDA PSD Online | Fundamentals (supply/demand) | ✅ Ingested | 5 | None | `extract_psd` → `stg_psdalldata__commodity` → `serving.commodity_metrics` |
|
| USDA PSD Online | Fundamentals (supply/demand) | ✅ Ingested | 5 | None | `extract_psd` → `stg_psdalldata__commodity` → `serving.commodity_metrics` |
|
||||||
| Open-Meteo ERA5 — weather | Crop weather | ✅ Ingested | 5 | None | `extract_openmeteo` → `fct_weather_daily` → `serving.weather_daily` |
|
| Open-Meteo ERA5 — weather | Crop weather | ✅ Ingested | 5 | None | `extract_openmeteo` → `fct_weather_daily` → `serving.weather_daily` |
|
||||||
| ICE Coffee C — options chain | Derivatives / Volatility | 🔲 Planned | 4 | None (yfinance) or paid | TBD |
|
| CFTC COT Combined (futures+options) | Positioning | 🔲 Planned | 4 | None (same ZIP pattern) | Extend `extract_cot` → `fct_cot_combined` variant |
|
||||||
| CFTC COT — options-and-futures combined | Positioning | 🔲 Planned | 3 | None (same ZIP) | `fct_cot_positioning` variant |
|
| ICE Report Center — options settlement | Derivatives / Volatility | 🔲 Planned | 3 | WebICE account (free) | `extract_ice_options` → `fct_ice_options_settlement` |
|
||||||
|
| Barchart OnDemand — options EOD | Derivatives / Volatility | ⏸ On hold | 3 | Paid subscription (contact sales) | `extract_barchart_options` → `fct_options_chain` |
|
||||||
| World Bank Commodity Prices (Pink Sheet) | Benchmark prices | 🔲 Planned | 3 | None | `extract_wb_prices` → `fct_wb_prices` |
|
| World Bank Commodity Prices (Pink Sheet) | Benchmark prices | 🔲 Planned | 3 | None | `extract_wb_prices` → `fct_wb_prices` |
|
||||||
| FAO Crop Calendar | Seasonality | 🔲 Planned | 3 | None (CSV) | Seed table |
|
| FAO Crop Calendar | Seasonality | 🔲 Planned | 3 | None (CSV) | Seed table |
|
||||||
| Freight / C4 route rates | Supply chain | 🔲 Planned | 2 | None (scrape) | `fct_freight_rates` |
|
| Freight / C4 route rates | Supply chain | 🔲 Planned | 2 | None (scrape) | `fct_freight_rates` |
|
||||||
@@ -194,12 +195,31 @@ Variables fetched:
|
|||||||
|
|
||||||
| Field | Value |
|
| Field | Value |
|
||||||
|-------|-------|
|
|-------|-------|
|
||||||
| Data Type | Per-strike open interest, volume, implied volatility for KC=F options |
|
| Data Type | Per-strike open interest, volume, implied volatility, greeks for KC=F options |
|
||||||
| Use Case | IV term structure, put/call skew, options positioning — leading indicator for futures moves |
|
| Use Case | IV term structure, put/call skew, options positioning — leading indicator for futures moves |
|
||||||
| Access Options | `yfinance` (free, limited history); barchart OnDemand API (paid); ICE Data Services (enterprise) |
|
|
||||||
| Priority | **High** |
|
| Priority | **High** |
|
||||||
|
|
||||||
See research note in `docs/ice-options-research.md` (to be added when research completes).
|
**Key constraint:** KC=F options trade on ICE Futures U.S. (MIC: IFUS), not on OPRA. This eliminates every equity-centric data provider from consideration (yfinance, Intrinio, Polygon.io, ORATS, OpenBB's six providers — all source from OPRA and have zero KC=F options data).
|
||||||
|
|
||||||
|
**Option A — CFTC COT Combined report (free, immediate):**
|
||||||
|
Change the existing extractor URL from `fut_disagg_txt_{year}.zip` to `com_disagg_txt_{year}.zip`. The combined report adds delta-weighted options exposure by trader category alongside the futures positions. Same ZIP format, same CSV schema, same 2006-present history. Does not give per-strike breakdown or implied volatility — it gives the aggregate "what does managed money's total directional exposure look like including their option book." Highest ROI change in the codebase.
|
||||||
|
|
||||||
|
**Option B — ICE Report Center settlement data (free, requires login automation):**
|
||||||
|
`https://www.theice.com/reports/end-of-day` publishes daily options settlement prices by strike and expiry, including the Black-76 implied volatility ICE uses for settlement. Free with a WebICE account (free to register). The site uses cookie-based authentication after T&C acceptance — automatable via an authenticated HTTP session. Gives EOD per-strike IV + OI + settlement price. No bulk historical archive on the free tier; CSV subscriptions are paid.
|
||||||
|
|
||||||
|
**Option C — Barchart OnDemand `getFuturesOptionsEOD` (paid, best self-service):**
|
||||||
|
`https://www.barchart.com/ondemand` provides per-strike agricultural futures options data (IV, OI, volume) via `getFuturesOptions` and `getFuturesOptionsEOD` REST endpoints. Correct instrument coverage; REST-based; integrates well with the existing extraction pattern. Paid commercial subscription required — no self-service pricing published, contact sales.
|
||||||
|
|
||||||
|
**Option D — Interactive Brokers TWS API (free with account, live only):**
|
||||||
|
`ibapi` Python package returns live KC options chain (bid/ask, IV, delta, gamma, theta, vega) for an active IB futures-enabled account. No historical bulk download — live/recent snapshots only. Free if account is active.
|
||||||
|
|
||||||
|
| Source | Per-Strike | IV | History | Cost |
|
||||||
|
|--------|-----------|-----|---------|------|
|
||||||
|
| CFTC COT Combined | No (aggregate) | No | 2006-present | Free |
|
||||||
|
| ICE Report Center (EOD settlement) | Yes | Settlement IV | Recent (web) | Free w/ login |
|
||||||
|
| Barchart OnDemand EOD | Yes | Yes | Deep (unspecified) | Paid |
|
||||||
|
| ICE Data Services | Yes | Yes (full surface) | Full archive | Institutional |
|
||||||
|
| yfinance / OpenBB / Polygon.io | None | — | — | N/A |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user