[Feature] Storage Driver Abstraction (S3-Compatible & WebDAV / Nextcloud) #8

New issue

Open

opened 2026-09-08 10:49:57 +02:00 by JackPrince · 0 comments

JackPrince commented

2026-09-08 10:49:57 +02:00

Owner

Summary

To guarantee data sovereignty, GDPR compliance, and European digital independence (mitigating dependency on US hyperscalers), the application requires an extensible storage abstraction layer. Beyond the local filesystem, the system must natively support S3-compatible open-source object stores (such as SeaweedFS, Garage, Ceph RGW, or self-hosted MinIO), European IaaS providers (e.g., Hetzner Object Storage/Storage Box, Wasabi EU), and WebDAV (Nextcloud / ownCloud).

Context & Compliance

GDPR / Data Residency: Eliminates unvetted third-country data transfers by granting operators complete control over where binary assets reside.
EU Cloud & Digital Sovereignty: Focuses on true open-source, on-premise alternatives (e.g., Garage, SeaweedFS, Ceph) to avoid vendor lock-in.
GoBD Requirements: Audit readiness, document immutability, and clean separation of receipt attachments from operational database records (laying the foundation for S3 Object Lock / WORM support).

Requirements

1. Unified Storage Abstraction Layer

Implement a unified filesystem contract for binary storage operations:
- put(path, contents, options)
- get(path) / readStream(path)
- delete(path)
- exists(path)
- temporaryUrl(path, expiration)

2. S3-Compatible Driver (Self-Hosted FOSS & Managed EU Providers)

Custom endpoint configuration (mandatory for self-hosted instances like SeaweedFS, Garage, Ceph, and Hetzner):
- S3_ENDPOINT (e.g., http://garage:3900 or https://fsn1.your-objectstorage.com)
- S3_KEY
- S3_SECRET
- S3_BUCKET
- S3_REGION (configurable with a sane default like eu-central-1 or garage)
- S3_USE_PATH_STYLE_ENDPOINT (boolean, defaults to true for self-hosted compatibility)
Ensure compatibility with S3 API subsets provided by modern FOSS engines (e.g., multipart upload handling, presigned URLs).

3. WebDAV / Nextcloud Driver

Native WebDAV integration (WEBDAV_BASE_URI, WEBDAV_USER, WEBDAV_PASSWORD / app token).
Safe auto-creation of recursive target directory structures (e.g., GoBDLogBook/receipts/YYYY/MM/).

4. Disk Configuration & Fallback

Multi-disk configuration (e.g., local, s3, nextcloud).
Configure target disk for document/receipt uploads via environment variable (STORAGE_DISK=s3).

Definition of Done (DoD)

CI pipeline validates file lifecycle (write, read, stream, delete) against an open-source S3 container (e.g., SeaweedFS or Garage) and a WebDAV test container via Docker Compose.
File operations execute transparently across all supported drivers without code changes in domain services.
Detailed configuration guide and sample variables added to .env.example and repository documentation.

### Summary To guarantee data sovereignty, GDPR compliance, and European digital independence (mitigating dependency on US hyperscalers), the application requires an extensible storage abstraction layer. Beyond the local filesystem, the system must natively support S3-compatible open-source object stores (such as SeaweedFS, Garage, Ceph RGW, or self-hosted MinIO), European IaaS providers (e.g., Hetzner Object Storage/Storage Box, Wasabi EU), and WebDAV (Nextcloud / ownCloud). ### Context & Compliance * **GDPR / Data Residency:** Eliminates unvetted third-country data transfers by granting operators complete control over where binary assets reside. * **EU Cloud & Digital Sovereignty:** Focuses on true open-source, on-premise alternatives (e.g., Garage, SeaweedFS, Ceph) to avoid vendor lock-in. * **GoBD Requirements:** Audit readiness, document immutability, and clean separation of receipt attachments from operational database records (laying the foundation for S3 Object Lock / WORM support). ### Requirements #### 1. Unified Storage Abstraction Layer - [ ] Implement a unified filesystem contract for binary storage operations: - `put(path, contents, options)` - `get(path)` / `readStream(path)` - `delete(path)` - `exists(path)` - `temporaryUrl(path, expiration)` #### 2. S3-Compatible Driver (Self-Hosted FOSS & Managed EU Providers) - [x] Custom endpoint configuration (mandatory for self-hosted instances like SeaweedFS, Garage, Ceph, and Hetzner): - `S3_ENDPOINT` (e.g., `http://garage:3900` or `https://fsn1.your-objectstorage.com`) - `S3_KEY` - `S3_SECRET` - `S3_BUCKET` - `S3_REGION` (configurable with a sane default like `eu-central-1` or `garage`) - `S3_USE_PATH_STYLE_ENDPOINT` (boolean, defaults to `true` for self-hosted compatibility) - [x] Ensure compatibility with S3 API subsets provided by modern FOSS engines (e.g., multipart upload handling, presigned URLs). #### 3. WebDAV / Nextcloud Driver - [x] Native WebDAV integration (`WEBDAV_BASE_URI`, `WEBDAV_USER`, `WEBDAV_PASSWORD` / app token). - [ ] Safe auto-creation of recursive target directory structures (e.g., `GoBDLogBook/receipts/YYYY/MM/`). #### 4. Disk Configuration & Fallback - [x] Multi-disk configuration (e.g., `local`, `s3`, `nextcloud`). - [ ] Configure target disk for document/receipt uploads via environment variable (`STORAGE_DISK=s3`). ### Definition of Done (DoD) - CI pipeline validates file lifecycle (write, read, stream, delete) against an open-source S3 container (e.g., SeaweedFS or Garage) and a WebDAV test container via Docker Compose. - File operations execute transparently across all supported drivers without code changes in domain services. - Detailed configuration guide and sample variables added to `.env.example` and repository documentation.

JackPrince added the

Priority

Medium

Kind/Feature

labels

2026-09-08 10:49:57 +02:00

JackPrince self-assigned this

2026-09-08 10:49:57 +02:00

JackPrince added a new dependency

2026-09-08 11:32:25 +02:00

#9 [Feature] Full System Backup Orchestration, Snapshot Engine & Disaster Recovery

JackPrince referenced this issue