docs: update inventory with ICE options research findings

- yfinance confirmed not viable (OPRA only, KC=F not covered)
- CFTC COT combined report is the free immediate path (URL change only)
- ICE Report Center settlement data viable with WebICE login automation
- Barchart OnDemand has correct coverage but requires paid subscription
- All OpenBB providers, Polygon.io, Nasdaq Data Link confirmed no KC=F coverage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-26 10:16:50 +01:00
parent 70415e23b8
commit b25b8780a7

View File

@@ -20,8 +20,9 @@ Purpose: Identify and track data sources feeding the BeanFlows DuckDB analytics
| ICE Report Center — aging stocks | Warehouse / Inventory | ✅ Ingested | 4 | None | `extract_ice_stocks``fct_ice_aging_stocks``serving.ice_aging_stocks` | | ICE Report Center — aging stocks | Warehouse / Inventory | ✅ Ingested | 4 | None | `extract_ice_stocks``fct_ice_aging_stocks``serving.ice_aging_stocks` |
| USDA PSD Online | Fundamentals (supply/demand) | ✅ Ingested | 5 | None | `extract_psd``stg_psdalldata__commodity``serving.commodity_metrics` | | USDA PSD Online | Fundamentals (supply/demand) | ✅ Ingested | 5 | None | `extract_psd``stg_psdalldata__commodity``serving.commodity_metrics` |
| Open-Meteo ERA5 — weather | Crop weather | ✅ Ingested | 5 | None | `extract_openmeteo``fct_weather_daily``serving.weather_daily` | | Open-Meteo ERA5 — weather | Crop weather | ✅ Ingested | 5 | None | `extract_openmeteo``fct_weather_daily``serving.weather_daily` |
| ICE Coffee C — options chain | Derivatives / Volatility | 🔲 Planned | 4 | None (yfinance) or paid | TBD | | CFTC COT Combined (futures+options) | Positioning | 🔲 Planned | 4 | None (same ZIP pattern) | Extend `extract_cot``fct_cot_combined` variant |
| CFTC COT — options-and-futures combined | Positioning | 🔲 Planned | 3 | None (same ZIP) | `fct_cot_positioning` variant | | ICE Report Center — options settlement | Derivatives / Volatility | 🔲 Planned | 3 | WebICE account (free) | `extract_ice_options``fct_ice_options_settlement` |
| Barchart OnDemand — options EOD | Derivatives / Volatility | ⏸ On hold | 3 | Paid subscription (contact sales) | `extract_barchart_options``fct_options_chain` |
| World Bank Commodity Prices (Pink Sheet) | Benchmark prices | 🔲 Planned | 3 | None | `extract_wb_prices``fct_wb_prices` | | World Bank Commodity Prices (Pink Sheet) | Benchmark prices | 🔲 Planned | 3 | None | `extract_wb_prices``fct_wb_prices` |
| FAO Crop Calendar | Seasonality | 🔲 Planned | 3 | None (CSV) | Seed table | | FAO Crop Calendar | Seasonality | 🔲 Planned | 3 | None (CSV) | Seed table |
| Freight / C4 route rates | Supply chain | 🔲 Planned | 2 | None (scrape) | `fct_freight_rates` | | Freight / C4 route rates | Supply chain | 🔲 Planned | 2 | None (scrape) | `fct_freight_rates` |
@@ -194,12 +195,31 @@ Variables fetched:
| Field | Value | | Field | Value |
|-------|-------| |-------|-------|
| Data Type | Per-strike open interest, volume, implied volatility for KC=F options | | Data Type | Per-strike open interest, volume, implied volatility, greeks for KC=F options |
| Use Case | IV term structure, put/call skew, options positioning — leading indicator for futures moves | | Use Case | IV term structure, put/call skew, options positioning — leading indicator for futures moves |
| Access Options | `yfinance` (free, limited history); barchart OnDemand API (paid); ICE Data Services (enterprise) |
| Priority | **High** | | Priority | **High** |
See research note in `docs/ice-options-research.md` (to be added when research completes). **Key constraint:** KC=F options trade on ICE Futures U.S. (MIC: IFUS), not on OPRA. This eliminates every equity-centric data provider from consideration (yfinance, Intrinio, Polygon.io, ORATS, OpenBB's six providers — all source from OPRA and have zero KC=F options data).
**Option A — CFTC COT Combined report (free, immediate):**
Change the existing extractor URL from `fut_disagg_txt_{year}.zip` to `com_disagg_txt_{year}.zip`. The combined report adds delta-weighted options exposure by trader category alongside the futures positions. Same ZIP format, same CSV schema, same 2006-present history. Does not give per-strike breakdown or implied volatility — it gives the aggregate "what does managed money's total directional exposure look like including their option book." Highest ROI change in the codebase.
**Option B — ICE Report Center settlement data (free, requires login automation):**
`https://www.theice.com/reports/end-of-day` publishes daily options settlement prices by strike and expiry, including the Black-76 implied volatility ICE uses for settlement. Free with a WebICE account (free to register). The site uses cookie-based authentication after T&C acceptance — automatable via an authenticated HTTP session. Gives EOD per-strike IV + OI + settlement price. No bulk historical archive on the free tier; CSV subscriptions are paid.
**Option C — Barchart OnDemand `getFuturesOptionsEOD` (paid, best self-service):**
`https://www.barchart.com/ondemand` provides per-strike agricultural futures options data (IV, OI, volume) via `getFuturesOptions` and `getFuturesOptionsEOD` REST endpoints. Correct instrument coverage; REST-based; integrates well with the existing extraction pattern. Paid commercial subscription required — no self-service pricing published, contact sales.
**Option D — Interactive Brokers TWS API (free with account, live only):**
`ibapi` Python package returns live KC options chain (bid/ask, IV, delta, gamma, theta, vega) for an active IB futures-enabled account. No historical bulk download — live/recent snapshots only. Free if account is active.
| Source | Per-Strike | IV | History | Cost |
|--------|-----------|-----|---------|------|
| CFTC COT Combined | No (aggregate) | No | 2006-present | Free |
| ICE Report Center (EOD settlement) | Yes | Settlement IV | Recent (web) | Free w/ login |
| Barchart OnDemand EOD | Yes | Yes | Deep (unspecified) | Paid |
| ICE Data Services | Yes | Yes (full surface) | Full archive | Institutional |
| yfinance / OpenBB / Polygon.io | None | — | — | N/A |
--- ---