fix: migration atomicity + deploy hardening + Litestream R2

Migration atomicity:
- Remove conn.commit() and executescript() from all up() functions (0000,
  0011, 0012, 0013, 0014, 0015); executescript() issued implicit COMMITs
  which broke the batch-rollback guarantee of the migration runner
- Rewrite 0000 with individual conn.execute() calls (was a single
  executescript block)

Deploy hardening:
- Add pre-migration DB backup step to deploy.sh: saves
  app.db.pre-deploy-<timestamp> in the volume before every migration
- On health-check failure: restore the backup, then stop + exit
- On success: clean up old backups (keep last 3)

Litestream:
- Enable R2 as primary replica in litestream.yml (env-var placeholders)
- Add local /app/data/backups as secondary replica
- docker-compose: add auto-restore on empty volume (sh entrypoint runs
  'litestream restore' before 'litestream replicate' if app.db missing)
- Add LITESTREAM_R2_* vars to .gitlab-ci.yml .env block and .env.example

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-20 10:28:59 +01:00
parent 363f93885d
commit c0c8607664
12 changed files with 315 additions and 171 deletions

View File

@@ -22,6 +22,13 @@ echo "==> Current: $CURRENT → Deploying: $TARGET"
echo "==> Building $TARGET..."
$COMPOSE --profile "$TARGET" build
# ── Backup DB before migration ────────────────────────────────
BACKUP_TAG="pre-deploy-$(date +%Y%m%d-%H%M%S)"
echo "==> Backing up database (${BACKUP_TAG})..."
$COMPOSE run --rm --entrypoint "" "${TARGET}-app" \
sh -c "cp /app/data/app.db /app/data/app.db.${BACKUP_TAG} 2>/dev/null || true"
# ── Migrate ─────────────────────────────────────────────────
echo "==> Running migrations..."
@@ -34,6 +41,13 @@ echo "==> Starting $TARGET (waiting for health check)..."
if ! $COMPOSE --profile "$TARGET" up -d --wait; then
echo "!!! Health check failed — rolling back"
$COMPOSE stop "${TARGET}-app" "${TARGET}-worker" "${TARGET}-scheduler"
LATEST=$($COMPOSE run --rm --entrypoint "" "${TARGET}-app" \
sh -c "ls -t /app/data/app.db.pre-deploy-* 2>/dev/null | head -1")
if [ -n "$LATEST" ]; then
echo "==> Restoring database from ${LATEST}..."
$COMPOSE run --rm --entrypoint "" "${TARGET}-app" \
sh -c "cp '${LATEST}' /app/data/app.db"
fi
exit 1
fi
@@ -63,6 +77,11 @@ NGINX
$COMPOSE up -d router
$COMPOSE exec router nginx -s reload
# ── Cleanup old pre-deploy backups (keep last 3) ─────────────
$COMPOSE run --rm --entrypoint "" "${TARGET}-app" \
sh -c "ls -t /app/data/app.db.pre-deploy-* 2>/dev/null | tail -n +4 | xargs rm -f" || true
# ── Stop old slot ───────────────────────────────────────────
if [ "$CURRENT" != "none" ]; then