Backup Longhorn volumes to S3

Introduction

Need a backup solution for your Kubernetes Longhorn volumes? S3 storage is a widely used and reliable choice. This step-by-step guide covers how to set it up. Remember that Longhorn's built-in replication protects against hardware failures, but it does not replace a true backup strategy.

Prerequisites

1. Kubernetes basic knowledge.
2. An active AWS account with the AWS CLI configured.
3. AWS Access and Secret keys with permissions to access the target S3 bucket.
4. A running Kubernetes (K8s) cluster with the Sealed Secrets controller installed.
5. Longhorn deployed via Helm.

Step-by-step

1. Create a S3 bucket.

   aws s3api create-bucket --bucket k8s-longhorn-s3-backups
   {
       "Location": "/k8s-longhorn-s3-backups",
       "BucketArn": "arn:aws:s3:::k8s-longhorn-s3-backups"
   }
   

2. Create a SealedSecret containing your AWS Access and Secret keys to grant Longhorn access to your S3 backup target.

   export AWS_ACCESS_KEY_ID='toBeDefined'
   export AWS_SECRET_ACCESS_KEY='toBeDefined'
   kubectl create secret generic s3-secret --dry-run=client \
       --namespace=longhorn-system \
       --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
       --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
       -o yaml \
       | kubeseal --format=yaml
   

3. Configure the default backup target (helm values) in Longhorn to point to your AWS S3 bucket, ensuring the system knows where to store and retrieve your backups.

   apiVersion: helm.toolkit.fluxcd.io/v2
   kind: HelmRelease
   metadata:
     name: longhorn
     namespace: longhorn-system
   spec:
     values:
       defaultBackupStore:
         backupTarget: s3://k8s-longhorn-s3-backups@us-east-1/
         backupTargetCredentialSecret: s3-secret
   

4. Create a recurring backup job with a daily schedule. You can control which volumes are included by using the daily-backup group label.

   apiVersion: longhorn.io/v1beta2
   kind: RecurringJob
   metadata:
     name: daily-backup
     namespace: longhorn-system
   spec:
     concurrency: 1
     cron: 0 0 * * *
     groups:
       - daily-backup
     labels: {}
     name: daily-backup
     parameters: {}
     retain: 7
     task: backup    
   

5. Navigate to the Longhorn UI, select the volumes you wish to include in the backup, and assign them to the daily-backup group within the Recurring Jobs Schedule section.

   

Conclusion

Congratulations! You have successfully implemented a backup solution for your volumes, which are now securely stored in AWS. Note that you can also utilize any S3-compatible alternative, such as Scaleway Object Storage.