Encrypting Kubernetes Secrets at Rest

Introduction

Everybody constantly criticises storing passwords in plain text; however, if you are running vanilla K8s clusters, your confidential data, such as Secrets, are unfortunately not encrypted and stored as base64 encoded, which can be easily decoded.

In this tutorial, I will show you how to encrypt your K8s sensitive data at rest using the secretbox options. It is recommended to use kms v2 for production.

Prerequisites

A running K8s cluster.
Administrator/root access to the control plane nodes.

Step-by-step Guide

SSH to one of the control plane nodes and create the encryption configuration enc.yaml file:
```
sudo mkdir /etc/kubernetes/enc
sudo touch /etc/kubernetes/enc/enc.yaml
```
Make sure that only the user who runs the kube-apiserver can read the configuration file:
```
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
```
Generate a new encryption key:
```
head -c 32 /dev/urandom | base64
```

Update the encryption configuration file:

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

Then, securely copy this file to other control plane nodes, if any.

Update the kube-apiserver (all control plane nodes) to use the new encryption configuration file: Edit the manifest /etc/kubernetes/manifests/kube-apiserver.yaml to:

---
#
# This is a fragment of a manifest for a static Pod.
# Check whether this is correct for your cluster and for your API server.
#
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.30.40:443
  creationTimestamp: null
  labels:
    app.kubernetes.io/component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml  # add this line
    volumeMounts:
    ...
    - name: enc                           # add this line
      mountPath: /etc/kubernetes/enc      # add this line
      readOnly: true                      # add this line
    ...
  volumes:
  ...
  - name: enc                             # add this line
    hostPath:                             # add this line
      path: /etc/kubernetes/enc           # add this line
      type: DirectoryOrCreate             # add this line
  ...

Testing

From now on, only newly created secrets will be encrypted. Let's create one to test:

Create a test secret:

kubectl -n default create secret generic testsecret  --from-literal=mykey=mydata

Exec into the ETCD pod to view the newly created secret:

kubectl exec -i -t -n kube-system etcd-$(hostname) -c etcd -- sh -c "sh"

ETCDCTL_API=3 etcdctl \
   --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
   --cert=/etc/kubernetes/pki/etcd/server.crt \
   --key=/etc/kubernetes/pki/etcd/server.key  \
   get /registry/secrets/default/testsecret

The output is similar to this (abbreviated):

/registry/secrets/default/testsecret
;????U?:?sd?y~???2???)ey1:?W?wu5,?J???^?v??i?|
???                   ???2???|    ???2???

As you can see, the data is encrypted.

Delete the secret:

kubectl -n default delete secret testsecret

Encrypt Secrets

The simplest way to encrypt all current secrets is to delete and recreate them. This command will do the trick:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Conclusion

Walla! You now have a more secure K8s cluster with all confidential data encrypted at rest.

References

Encrypting Confidential Data at Rest.