beanflows

Files

Deeman d30ec9b66b Add R2 upload support with landing bucket path

## Changes

1. **Support ESC environment variable names**
   - Fallback to R2_ADMIN_ACCESS_KEY_ID if R2_ACCESS_KEY not set
   - Fallback to R2_ADMIN_SECRET_ACCESS_KEY if R2_SECRET_KEY not set
   - Allows script to work with Pulumi ESC (beanflows/prod) variables

2. **Use landing bucket path**
   - Changed R2 path from `psd/{etag}.zip` to `landing/psd/{etag}.zip`
   - All extracted data goes to landing bucket for consistent organization

3. **Updated Pulumi ESC environment**
   - Added R2_BUCKET=beanflows-data-prod
   - Fixed R2_ENDPOINT to remove bucket path (now just account URL)

## Testing

- ✅ R2 upload works: Uploaded to landing/psd/316039e2612edc1_0.zip
- ✅ R2 deduplication works: Skips upload if file exists
- ✅ Local mode still works without credentials

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-20 22:45:30 +02:00

src/psdonline

Add R2 upload support with landing bucket path

2025-10-20 22:45:30 +02:00

pyproject.toml

Refactor PSD extraction: simplify to latest-only + add R2 support

2025-10-20 22:02:15 +02:00

README.md

refactor

2025-08-25 20:50:25 +02:00

README.md

List of extracted source

https://apps.fas.usda.gov/psdonline/app/index.html#/app/downloads

TODO/Evaluate

Global Argricultural information service of USA, publishes reports on new developments globally as pdf files. Would need to do parsing/NLP to extract information but could be valuable source.

https://gain.fas.usda.gov/#/search