feat: landing zone backup to R2 via rclone + Litestream
Landing files (append-only JSON.gz) synced to R2 every 30 min via systemd timer + rclone. Extraction state DB (.state.sqlite) continuously replicated via Litestream (second DB entry). Auto-restore on container startup for both app.db and .state.sqlite. Reuses existing R2 bucket and credentials — no new env vars needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
27
infra/restore_landing.sh
Executable file
27
infra/restore_landing.sh
Executable file
@@ -0,0 +1,27 @@
|
||||
#!/bin/sh
|
||||
# Restore landing zone files from R2.
|
||||
# The extraction state DB (.state.sqlite) is restored automatically by
|
||||
# the Litestream container on startup — this script handles the data files only.
|
||||
#
|
||||
# Requires: rclone, LITESTREAM_R2_* env vars (from /opt/padelnomics/.env)
|
||||
#
|
||||
# Usage:
|
||||
# source /opt/padelnomics/.env && bash infra/restore_landing.sh
|
||||
|
||||
set -eu
|
||||
|
||||
LANDING_DIR="${LANDING_DIR:-/data/padelnomics/landing}"
|
||||
BUCKET_PREFIX="${LITESTREAM_R2_BUCKET:?LITESTREAM_R2_BUCKET not set}/padelnomics/landing"
|
||||
|
||||
echo "==> Restoring landing zone from R2 to ${LANDING_DIR}..."
|
||||
|
||||
rclone sync ":s3:${BUCKET_PREFIX}" "$LANDING_DIR" \
|
||||
--s3-provider Cloudflare \
|
||||
--s3-access-key-id "${LITESTREAM_R2_ACCESS_KEY_ID:?not set}" \
|
||||
--s3-secret-access-key "${LITESTREAM_R2_SECRET_ACCESS_KEY:?not set}" \
|
||||
--s3-endpoint "https://${LITESTREAM_R2_ENDPOINT:?not set}" \
|
||||
--s3-no-check-bucket \
|
||||
--exclude ".state.sqlite*" \
|
||||
--progress
|
||||
|
||||
echo "==> Landing zone restored to ${LANDING_DIR}"
|
||||
Reference in New Issue
Block a user