Deploy MinIO in Kubernetes with Tenant Separation

Introduction

If you're looking for a self-hosted, S3-compatible object storage solution for a Kubernetes project, MinIO is one of the best options available due to its simplicity, ease of use, and tenant separation. Here's how I installed the MinIO Operator and deployed a MinIO tenant for my NextCloud homelab.

Prerequisites

A Kubernetes cluster running version 1.32 with Helm.
Sealed Secrets installed.
FluxCD (optional).
Cloudflare Tunnel (optional).

Step-by-step Guide

Install the MinIO Operator

The MinIO Operator manages all MinIO tenants, so I first installed it using Helm:

helm repo add minio-operator https://operator.min.io
helm repo update
helm upgrade --install operator minio-operator/operator \
  --namespace minio-operator \
  --create-namespace \
  --version '^7.0.0'

This is the FluxCD HelmRelease I used:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: minio-operator
  namespace: minio-operator
spec:
  interval: 24h
  url: https://operator.min.io
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: operator
  namespace: minio-operator
spec:
  interval: 24h
  chart:
    spec:
      chart: operator
      version: '^7.0.0'
      sourceRef:
        kind: HelmRepository
        name: minio-operator
        namespace: minio-operator
      interval: 24h

Deploy MinIO tenant

I created a nextcloud namespace for the MinIO tenant:

kubectl create namespace nextcloud

Next, I created sealed secrets for the MinIO console user:

kubectl -n nextcloud create secret generic minio-env-configuration \
    --from-literal=config.env="$(cat <<EOF
export MINIO_ROOT_USER='nextcloud'
export MINIO_ROOT_PASSWORD=$(openssl rand -base64 32)
EOF
)" --dry-run=client -o yaml | kubeseal --format yaml > minio-env-configuration.sealed-secret.yaml

And the access and secret keys for the NextCloud application:

export MINIO_ACCESS_KEY=$(openssl rand -base64 24 | tr -d '/+=' | cut -c1-20)
export MINIO_SECRET_KEY=$(openssl rand -base64 32 | tr -d '/+=' | cut -c1-40)
kubectl -n nextcloud create secret generic minio-user --from-literal=CONSOLE_ACCESS_KEY=$MINIO_ACCESS_KEY --from-literal=CONSOLE_SECRET_KEY=$MINIO_SECRET_KEY --dry-run=client -o yaml | kubeseal --format yaml > minio-user.sealed-secret.yaml

Finally, I deployed the MinIO tenant:

helm upgrade --install minio minio-operator/tenant \
  --namespace nextcloud \
  --version '^7.0.0' \
  --set tenant.name=nextcloud \
  --set tenant.configuration.name=minio-env-configuration \
  --set tenant.configSecret.name=minio-env-configuration \
  --set tenant.configSecret.accessKey=false \
  --set tenant.configSecret.secretKey=false \
  --set tenant.configSecret.existingSecret=true \
  --set tenant.users[0].name=minio-user \
  --set tenant.env[0].name=MINIO_STORAGE_CLASS_STANDARD \
  --set tenant.env[0].value=EC:2 \
  --set tenant.pools[0].servers=6 \
  --set tenant.pools[0].name=pool-0 \
  --set tenant.pools[0].volumesPerServer=1 \
  --set tenant.pools[0].size=500Gi \
  --set tenant.pools[0].storageClassName=longhorn-single \
  --set tenant.buckets[0].name=nextcloud \
  --set tenant.certificate.requestAutoCert=false

This is the FluxCD HelmRelease I used:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: minio
  namespace: nextcloud
spec:
  interval: 24h
  chart:
    spec:
      chart: tenant # https://artifacthub.io/packages/helm/minio-operator/tenant
      version: '^7.0.0'
      sourceRef:
        kind: HelmRepository
        name: minio-operator
        namespace: minio-operator
      interval: 24h
  values:
    tenant:
      name: nextcloud
      configuration:
        name: minio-env-configuration
      configSecret:
        name: minio-env-configuration
        accessKey: false
        secretKey: false
        existingSecret: true
      users:
        - name: minio-user
      env:
        - name: MINIO_STORAGE_CLASS_STANDARD
          value: EC:2
      pools:
        - servers: 6
          name: pool-0
          volumesPerServer: 1
          size: 500Gi
          storageClassName: longhorn-single
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: v1.min.io/tenant
                        operator: In
                        values:
                          - cloud
                      - key: v1.min.io/pool
                        operator: In
                        values:
                          - pool-0
                  topologyKey: kubernetes.io/hostname
      buckets:
        - name: nextcloud
      certificate:
        requestAutoCert: false
        # externalCertSecret:
        #   - name: nextcloud-tls
        #     type: cert-manager.io/v1

Check the deployment status:

kubectl -n nextcloud get pods
NAME                         READY   STATUS    RESTARTS      AGE
nextcloud-pool-0-0           2/2     Running   0             10h
nextcloud-pool-0-1           2/2     Running   0             10h
nextcloud-pool-0-2           2/2     Running   0             10h
nextcloud-pool-0-3           2/2     Running   0             10h
nextcloud-pool-0-4           2/2     Running   0             10h
nextcloud-pool-0-5           2/2     Running   0             10h

Expose the MinIO Console using Cloudflare Tunnel.

This is an optional step where I used Cloudflare Tunnel to securely expose the MinIO console via a public endpoint:
```
ingress:
  - hostname: minio.harrytang.com
     service: http://nextcloud-console.nextcloud:9090
```
Accessing the UI requires the credentials created earlier:

Conclusion

With MinIO, I easily deployed a MinIO tenant for my NextCloud project. The best part is that for other projects, I can simply create a new namespace and deploy a separate tenant, ensuring enhanced security thanks to the separation feature.

Deploy MinIO in Kubernetes with Tenant Separation

Introduction

Prerequisites

Step-by-step Guide

Conclusion

References

Comments