pgBackRest: Multi-Destination PostgreSQL Backups in CloudNativePG
How I replaced Barman Cloud Plugin with pgBackRest to get true dual-destination full backups to both Backblaze B2 and Cloudflare R2, then migrated my entire PostgreSQL infrastructure to PostgreSQL 18.
posted
2025·12·19
read
14 min
cat
database
words
2,793
on this page
“The backup that only exists in one place doesn’t exist at all.”
The Problem: Barman’s Dirty Secret
I had what I thought was a solid PostgreSQL backup strategy for my Immich database. CloudNativePG with Barman Cloud Plugin, two ObjectStores configured—one for Backblaze B2, one for Cloudflare R2. Daily ScheduledBackups for each destination. Belt and suspenders.
Then I checked the actual buckets.
B2: Full backups, WAL files, everything present.
R2: Empty. Nothing. Not a single file.
Both ScheduledBackups were writing to B2. The R2 ObjectStore was configured, referenced in the ScheduledBackup—and completely ignored.
Turns out, Barman Cloud Plugin has a limitation I hadn’t spotted. The barmanObjectName parameter in ScheduledBackup? It’s ignored. The plugin only uses whatever’s configured in the cluster’s plugin configuration. Both my scheduled backups were hitting the same destination.
This isn’t theoretical concern. If Backblaze goes down, I can’t copy backups to R2 because they’re only in B2. My 3-2-1 backup strategy was actually a 2-1-1.
The Alternative: pgBackRest
After researching alternatives, I found three options for PostgreSQL backups with CloudNativePG plugins:
Barman (what I was using) - Single destination limitation
WAL-G - Similar architecture, no multi-repo support
pgBackRest - Native multi-repository support
pgBackRest from Dalibo looked promising. It’s designed from the ground up for multi-repository backups—WAL archiving goes to ALL configured repositories simultaneously. The catch: it’s experimental, has 16 GitHub stars, and the documentation is thin.
I gave it a shot anyway.
Deploying the pgBackRest Plugin
Unlike Barman which has a Helm chart, the Dalibo pgBackRest plugin requires manual deployment. I created a new directory structure:
1
2
3
4
5
6
7
kubernetes/apps/database/cloudnative-pg/pgbackrest/
├── kustomization.yaml
├── crd.yaml # Repository CRD
├── rbac.yaml # ServiceAccount, ClusterRole, RoleBinding
├── certificate.yaml # Self-signed TLS for plugin communication
├── deployment.yaml # The controller
└── service.yaml # Exposes the controller to CNPG
stanza creation failed: can't parse pgbackrest JSON: invalid character 'P'
After way too much debugging, I found the issue. I’d named the service pgbackrest. Kubernetes automatically creates environment variables for services: PGBACKREST_SERVICE_HOST, PGBACKREST_PORT, etc.
pgBackRest interprets anyPGBACKREST_* environment variable as configuration. The sidecar was trying to parse PGBACKREST_PORT_9090_TCP_ADDR as a pgBackRest option and choking on the JSON output.
The fix: rename the service to cnpg-pgbackrest. No more environment variable conflicts.
Leader Lease Conflict
After fixing the service name, the controller was stuck:
1
attempting to acquire leader lease database/822e3f5c.cnpg.io...
Both Barman and pgBackRest plugins use the same leader election lease name. I had to disable Barman first:
Note the R2 bucket and endpoint use Flux variable substitution (${VARIABLE_NAME}) instead of hardcoded values. These get populated from an ExternalSecret that pulls from 1Password, with the Flux Kustomization configured to substitute variables from that secret. This keeps sensitive values like Cloudflare account IDs out of git history.
ERROR: [056]: unable to find primary cluster - cannot proceed
HINT: are all available clusters in recovery?
CNPG defaults to running backups on replicas to reduce load on the primary. But pgBackRest can’t run backups from replicas without SSH access to the primary—which doesn’t exist in Kubernetes.
The fix is simple: tell CNPG to run backups on the primary:
1
2
3
4
5
6
7
8
9
10
11
12
apiVersion:postgresql.cnpg.io/v1kind:ScheduledBackupmetadata:name:immich18-daily-b2spec:schedule:"0 3 * * *"target:primary # This is the keymethod:plugincluster:name:immich18pluginConfiguration:name:pgbackrest.dalibo.com
The Multi-Repo Full Backup Discovery
With the target fixed, backups started working. WAL archiving was going to both repositories—I could see files appearing in both B2 and R2. But when I checked the full backups:
1
2
3
4
5
6
7
# B2aws s3 ls s3://<your-bucket>/immich18/ --profile backblaze-b2 --recursive | wc -l
1285# R2aws s3 ls s3://<your-bucket>/immich18/ --profile cloudflare-r2 --region auto --recursive | wc -l
11
WAL archives were in both. Full backup was only in B2.
This is actually intentional behavior in pgBackRest. From the documentation:
WAL archiving: Pushes to ALL configured repositories simultaneously
Full backups: Only runs against ONE repository (defaults to repo1)
The reasoning makes sense—full backups are large and expensive. Doing them twice doubles storage costs. WAL goes everywhere for redundancy.
But for disaster recovery, I needed full backups in both locations.
The selectedRepository Parameter
After digging through the plugin source code, I found the solution. The plugin accepts a selectedRepository parameter:
1
2
3
4
selectedRepo,ok:=request.Parameters["selectedRepository"]if!ok{selectedRepo="1"// use first repo by default
}
Note: the parameter is selectedRepository, not repo. The code defaults to repository 1 if not specified.
Documentation is sparse (I had to read source code)
For a homelab where I’m willing to debug issues, it’s the right choice. For production, I’d wait for the plugin to mature or implement external replication (rclone sync between buckets).
Part 2: The Full Migration
With pgBackRest working for Immich, I decided to go all-in and migrate my entire PostgreSQL infrastructure:
Rename immich18 to postgres18-immich - Clearer naming convention
Create postgres18-cluster - New cluster to replace postgres17 (which used Barman)
Migrate all 46 databases from postgres17 to postgres18-cluster
Shared Credentials
Rather than managing separate S3 credentials for each cluster, I consolidated to shared pgBackRest credentials:
apiVersion:external-secrets.io/v1beta1kind:ExternalSecretmetadata:name:cnpg-secretnamespace:databasespec:secretStoreRef:kind:ClusterSecretStorename:onepassword-connecttarget:name:cnpg-secretdata:# pgBackRest B2 - shared across all clusters- secretKey:b2-access-key-idremoteRef:key:backblazeproperty:BACKBLAZE_PGBACKREST_ACCESS_KEY- secretKey:b2-secret-access-keyremoteRef:key:backblazeproperty:BACKBLAZE_PGBACKREST_SECRET_ACCESS_KEY# pgBackRest R2 - shared across all clusters- secretKey:r2-access-key-idremoteRef:key:cloudflareproperty:CLOUDFLARE_PGBACKREST_ACCESS_KEY- secretKey:r2-secret-access-keyremoteRef:key:cloudflareproperty:CLOUDFLARE_PGBACKREST_SECRET_ACCESS_KEY
Each cluster gets its own S3 bucket but shares the same credentials, simplifying secret management.
Creating postgres18-cluster
The new cluster uses standard CloudNativePG PostgreSQL 18 (no VectorChord needed - that’s only for Immich):
kubectl exec -n database postgres17-1 -c postgres -- psql -U postgres -c \
"SELECT datname, usename, client_addr, state FROM pg_stat_activity \
WHERE datname IS NOT NULL ORDER BY datname;"