feat: landing zone backup to R2 via rclone + Litestream
Landing files (append-only JSON.gz) synced to R2 every 30 min via systemd timer + rclone. Extraction state DB (.state.sqlite) continuously replicated via Litestream (second DB entry). Auto-restore on container startup for both app.db and .state.sqlite. Reuses existing R2 bucket and credentials — no new env vars needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Landing zone backup to R2** — append-only landing files (`data/landing/*.json.gz`)
|
||||
synced to Cloudflare R2 every 30 minutes via systemd timer + rclone; extraction state
|
||||
DB (`.state.sqlite`) continuously replicated via Litestream (second DB entry in existing
|
||||
config); auto-restore on container startup for both `app.db` and `.state.sqlite`;
|
||||
`infra/restore_landing.sh` script for disaster recovery of landing files;
|
||||
`infra/landing-backup/` systemd service + timer units; rclone installation added to
|
||||
`infra/setup_server.sh`; reuses existing R2 bucket and credentials (no new env vars)
|
||||
|
||||
### Changed
|
||||
- **Extraction: one file per source** — replaced monolithic `execute.py` with per-source
|
||||
modules (`overpass.py`, `eurostat.py`, `playtomic_tenants.py`, `playtomic_availability.py`);
|
||||
|
||||
Reference in New Issue
Block a user