feat: landing zone backup to R2 via rclone + Litestream

Landing files (append-only JSON.gz) synced to R2 every 30 min via
systemd timer + rclone. Extraction state DB (.state.sqlite) continuously
replicated via Litestream (second DB entry). Auto-restore on container
startup for both app.db and .state.sqlite. Reuses existing R2 bucket
and credentials — no new env vars needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-23 14:06:16 +01:00
parent 5b6c4182f7
commit 76814dade7
8 changed files with 131 additions and 1 deletions

View File

@@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
### Added
- **Landing zone backup to R2** — append-only landing files (`data/landing/*.json.gz`)
synced to Cloudflare R2 every 30 minutes via systemd timer + rclone; extraction state
DB (`.state.sqlite`) continuously replicated via Litestream (second DB entry in existing
config); auto-restore on container startup for both `app.db` and `.state.sqlite`;
`infra/restore_landing.sh` script for disaster recovery of landing files;
`infra/landing-backup/` systemd service + timer units; rclone installation added to
`infra/setup_server.sh`; reuses existing R2 bucket and credentials (no new env vars)
### Changed
- **Extraction: one file per source** — replaced monolithic `execute.py` with per-source
modules (`overpass.py`, `eurostat.py`, `playtomic_tenants.py`, `playtomic_availability.py`);