diff --git a/docs/data-sources-inventory.md b/docs/data-sources-inventory.md index 2ef7d98..9297b63 100644 --- a/docs/data-sources-inventory.md +++ b/docs/data-sources-inventory.md @@ -20,8 +20,9 @@ Purpose: Identify and track data sources feeding the BeanFlows DuckDB analytics | ICE Report Center — aging stocks | Warehouse / Inventory | ✅ Ingested | 4 | None | `extract_ice_stocks` → `fct_ice_aging_stocks` → `serving.ice_aging_stocks` | | USDA PSD Online | Fundamentals (supply/demand) | ✅ Ingested | 5 | None | `extract_psd` → `stg_psdalldata__commodity` → `serving.commodity_metrics` | | Open-Meteo ERA5 — weather | Crop weather | ✅ Ingested | 5 | None | `extract_openmeteo` → `fct_weather_daily` → `serving.weather_daily` | -| ICE Coffee C — options chain | Derivatives / Volatility | 🔲 Planned | 4 | None (yfinance) or paid | TBD | -| CFTC COT — options-and-futures combined | Positioning | 🔲 Planned | 3 | None (same ZIP) | `fct_cot_positioning` variant | +| CFTC COT Combined (futures+options) | Positioning | 🔲 Planned | 4 | None (same ZIP pattern) | Extend `extract_cot` → `fct_cot_combined` variant | +| ICE Report Center — options settlement | Derivatives / Volatility | 🔲 Planned | 3 | WebICE account (free) | `extract_ice_options` → `fct_ice_options_settlement` | +| Barchart OnDemand — options EOD | Derivatives / Volatility | ⏸ On hold | 3 | Paid subscription (contact sales) | `extract_barchart_options` → `fct_options_chain` | | World Bank Commodity Prices (Pink Sheet) | Benchmark prices | 🔲 Planned | 3 | None | `extract_wb_prices` → `fct_wb_prices` | | FAO Crop Calendar | Seasonality | 🔲 Planned | 3 | None (CSV) | Seed table | | Freight / C4 route rates | Supply chain | 🔲 Planned | 2 | None (scrape) | `fct_freight_rates` | @@ -194,12 +195,31 @@ Variables fetched: | Field | Value | |-------|-------| -| Data Type | Per-strike open interest, volume, implied volatility for KC=F options | +| Data Type | Per-strike open interest, volume, implied volatility, greeks for KC=F options | | Use Case | IV term structure, put/call skew, options positioning — leading indicator for futures moves | -| Access Options | `yfinance` (free, limited history); barchart OnDemand API (paid); ICE Data Services (enterprise) | | Priority | **High** | -See research note in `docs/ice-options-research.md` (to be added when research completes). +**Key constraint:** KC=F options trade on ICE Futures U.S. (MIC: IFUS), not on OPRA. This eliminates every equity-centric data provider from consideration (yfinance, Intrinio, Polygon.io, ORATS, OpenBB's six providers — all source from OPRA and have zero KC=F options data). + +**Option A — CFTC COT Combined report (free, immediate):** +Change the existing extractor URL from `fut_disagg_txt_{year}.zip` to `com_disagg_txt_{year}.zip`. The combined report adds delta-weighted options exposure by trader category alongside the futures positions. Same ZIP format, same CSV schema, same 2006-present history. Does not give per-strike breakdown or implied volatility — it gives the aggregate "what does managed money's total directional exposure look like including their option book." Highest ROI change in the codebase. + +**Option B — ICE Report Center settlement data (free, requires login automation):** +`https://www.theice.com/reports/end-of-day` publishes daily options settlement prices by strike and expiry, including the Black-76 implied volatility ICE uses for settlement. Free with a WebICE account (free to register). The site uses cookie-based authentication after T&C acceptance — automatable via an authenticated HTTP session. Gives EOD per-strike IV + OI + settlement price. No bulk historical archive on the free tier; CSV subscriptions are paid. + +**Option C — Barchart OnDemand `getFuturesOptionsEOD` (paid, best self-service):** +`https://www.barchart.com/ondemand` provides per-strike agricultural futures options data (IV, OI, volume) via `getFuturesOptions` and `getFuturesOptionsEOD` REST endpoints. Correct instrument coverage; REST-based; integrates well with the existing extraction pattern. Paid commercial subscription required — no self-service pricing published, contact sales. + +**Option D — Interactive Brokers TWS API (free with account, live only):** +`ibapi` Python package returns live KC options chain (bid/ask, IV, delta, gamma, theta, vega) for an active IB futures-enabled account. No historical bulk download — live/recent snapshots only. Free if account is active. + +| Source | Per-Strike | IV | History | Cost | +|--------|-----------|-----|---------|------| +| CFTC COT Combined | No (aggregate) | No | 2006-present | Free | +| ICE Report Center (EOD settlement) | Yes | Settlement IV | Recent (web) | Free w/ login | +| Barchart OnDemand EOD | Yes | Yes | Deep (unspecified) | Paid | +| ICE Data Services | Yes | Yes (full surface) | Full archive | Institutional | +| yfinance / OpenBB / Polygon.io | None | — | — | N/A | ---