Encrypting Kubernetes Secrets at Rest

Introduction

Everybody constantly criticises storing passwords in plain text; however, if you are running vanilla K8s clusters, your confidential data, such as Secrets, are unfortunately not encrypted and stored as base64 encoded, which can be easily decoded.

In this tutorial, I will show you how to encrypt your K8s sensitive data at rest using the secretbox options. It is recommended to use kms v2 for production.

Prerequisites

• A running K8s cluster.
• Administrator/root access to the control plane nodes.

Step-by-step Guide

1. SSH to one of the control plane nodes and create the encryption configuration enc.yaml file:

   sudo mkdir /etc/kubernetes/enc
   sudo touch /etc/kubernetes/enc/enc.yaml
   

2. Make sure that only the user who runs the kube-apiserver can read the configuration file:

   sudo chmod 600 /etc/kubernetes/enc/enc.yaml
   

3. Generate a new encryption key:

   head -c 32 /dev/urandom | base64
   

4. Update the encryption configuration file:

   ---
   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
         - secrets
       providers:
         - secretbox:
             keys:
               - name: key1
                 secret: <BASE 64 ENCODED SECRET>
         - identity: {}
   

   Then, securely copy this file to other control plane nodes, if any.

5. Update the kube-apiserver (all control plane nodes) to use the new encryption configuration file: Edit the manifest /etc/kubernetes/manifests/kube-apiserver.yaml to:

   ---
   #
   # This is a fragment of a manifest for a static Pod.
   # Check whether this is correct for your cluster and for your API server.
   #
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.30.40:443
     creationTimestamp: null
     labels:
       app.kubernetes.io/component: kube-apiserver
       tier: control-plane
     name: kube-apiserver
     namespace: kube-system
   spec:
     containers:
     - command:
       - kube-apiserver
       ...
       - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml  # add this line
       volumeMounts:
       ...
       - name: enc                           # add this line
         mountPath: /etc/kubernetes/enc      # add this line
         readOnly: true                      # add this line
       ...
     volumes:
     ...
     - name: enc                             # add this line
       hostPath:                             # add this line
         path: /etc/kubernetes/enc           # add this line
         type: DirectoryOrCreate             # add this line
     ...    
   

Testing

From now on, only newly created secrets will be encrypted. Let's create one to test:

1. Create a test secret:

   kubectl -n default create secret generic testsecret  --from-literal=mykey=mydata
   

2. Exec into the ETCD pod to view the newly created secret:

   kubectl exec -i -t -n kube-system etcd-$(hostname) -c etcd -- sh -c "sh"
   
   ETCDCTL_API=3 etcdctl \
      --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
      --cert=/etc/kubernetes/pki/etcd/server.crt \
      --key=/etc/kubernetes/pki/etcd/server.key  \
      get /registry/secrets/default/testsecret
   

3. The output is similar to this (abbreviated):

   /registry/secrets/default/testsecret
   ;????U?:?sd?y~???2???)ey1:?W?wu5,?J???^?v??i?|
   ???                   ???2???|    ???2???
   

   As you can see, the data is encrypted.

4. Delete the secret:

   kubectl -n default delete secret testsecret
   

Encrypt Secrets

The simplest way to encrypt all current secrets is to delete and recreate them. This command will do the trick:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Conclusion

Walla! You now have a more secure K8s cluster with all confidential data encrypted at rest.

References

• Encrypting Confidential Data at Rest.

Comments