[Feature] Full System Backup Orchestration, Snapshot Engine & Disaster Recovery #9

New issue

Open

opened 2026-09-08 11:31:48 +02:00 by JackPrince · 0 comments

JackPrince commented

2026-09-08 11:31:48 +02:00

Owner

Summary

Implement an end-to-end backup, snapshot orchestration, and disaster recovery pipeline directly within the backend. The system must consistently package database dumps and uploaded receipts/files, optionally encrypt archives at rest, sync them to configured storage targets (S3, Nextcloud, or local disk), and provide a streamlined CLI/UI restore mechanism for server migrations or disaster recovery.

Context & Compliance

GDPR Art. 32 (Security of Processing): Ensure ongoing confidentiality, integrity, availability, and the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident.
GoBD: Maintain complete, verifiable, and tamper-resistant audit trails and document archives.

Requirements

1. Backup Engine (Dump & Archive)

Database Dump: Generate transaction-safe database dumps.
Attachment Aggregation: Bundle the active upload directory (receipt images, PDFs, audit metadata).
Manifest Generation: Include a manifest.json with app version, timestamp, file hashes (SHA-256), and schema migration version.
At-Rest Encryption: Optional symmetric encryption for backup archives prior to transmission (e.g., AES-256 via OpenSSL or GPG) using an environment master key.
Target Dispatching: Stream/upload compressed archives (.tar.gz / .zip) to configured storage disks (S3-compatible, Nextcloud WebDAV, or local directory).

2. Scheduler & Retention Policies

Provide a dedicated CLI command (e.g., backup:run) for cronjobs and task runners.
Implement rotation/retention rules (e.g., keep the last 7 daily, 4 weekly, and 12 monthly archives) to prevent storage exhaustion.

3. Disaster Recovery & Migration (Restore Workflow)

CLI Restore Command: CLI command (e.g., backup:restore --file=... or --from-s3=...) that:
- Toggles maintenance mode.
- Validates archive integrity and schema compatibility using manifest.json.
- Re-imports database schema and records.
- Restores attachments to the configured disk location.
- Clears application caches and triggers post-migration sanity checks.
Playbook Documentation: Clear, step-by-step disaster recovery and migration guide ("Zero-to-Running") for operators.

Definition of Done (DoD)

End-to-end automated test verifies: generate test dataset -> run backup -> wipe database and file storage -> run restore -> confirm data integrity.
CLI commands return appropriate exit codes and structured log output.
Complete documentation added to docs/backup-recovery.md.

### Summary Implement an end-to-end backup, snapshot orchestration, and disaster recovery pipeline directly within the backend. The system must consistently package database dumps and uploaded receipts/files, optionally encrypt archives at rest, sync them to configured storage targets (S3, Nextcloud, or local disk), and provide a streamlined CLI/UI restore mechanism for server migrations or disaster recovery. ### Context & Compliance * **GDPR Art. 32 (Security of Processing):** Ensure ongoing confidentiality, integrity, availability, and the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident. * **GoBD:** Maintain complete, verifiable, and tamper-resistant audit trails and document archives. ### Requirements #### 1. Backup Engine (Dump & Archive) - [x] **Database Dump:** Generate transaction-safe database dumps. - [x] **Attachment Aggregation:** Bundle the active upload directory (receipt images, PDFs, audit metadata). - [ ] **Manifest Generation:** Include a `manifest.json` with app version, timestamp, file hashes (SHA-256), and schema migration version. - [ ] **At-Rest Encryption:** Optional symmetric encryption for backup archives prior to transmission (e.g., AES-256 via OpenSSL or GPG) using an environment master key. - [ ] **Target Dispatching:** Stream/upload compressed archives (`.tar.gz` / `.zip`) to configured storage disks (S3-compatible, Nextcloud WebDAV, or local directory). #### 2. Scheduler & Retention Policies - [ ] Provide a dedicated CLI command (e.g., `backup:run`) for cronjobs and task runners. - [ ] Implement rotation/retention rules (e.g., keep the last 7 daily, 4 weekly, and 12 monthly archives) to prevent storage exhaustion. #### 3. Disaster Recovery & Migration (Restore Workflow) - [ ] **CLI Restore Command:** CLI command (e.g., `backup:restore --file=...` or `--from-s3=...`) that: - Toggles maintenance mode. - Validates archive integrity and schema compatibility using `manifest.json`. - Re-imports database schema and records. - Restores attachments to the configured disk location. - Clears application caches and triggers post-migration sanity checks. - [ ] **Playbook Documentation:** Clear, step-by-step disaster recovery and migration guide ("Zero-to-Running") for operators. ### Definition of Done (DoD) - End-to-end automated test verifies: generate test dataset -> run backup -> wipe database and file storage -> run restore -> confirm data integrity. - CLI commands return appropriate exit codes and structured log output. - Complete documentation added to `docs/backup-recovery.md`.

JackPrince added the

Kind/Feature

Priority

Medium

labels

2026-09-08 11:31:56 +02:00

JackPrince self-assigned this

2026-09-08 11:31:58 +02:00

JackPrince added a new dependency

2026-09-08 11:32:25 +02:00

#8 [Feature] Storage Driver Abstraction (S3-Compatible & WebDAV / Nextcloud)

JackPrince referenced this issue