Encrypting Kubernetes Secrets at Rest

Introduction

Everybody constantly criticises storing passwords in plain text; however, if you are running vanilla K8s clusters, your confidential data, such as Secrets, are unfortunately not encrypted and stored as base64 encoded in etcd, which can be easily decoded.

In this tutorial, I will show you how to encrypt your K8s sensitive data at rest using the secretbox options. It is recommended to use kms v2 for production.

Prerequisites

A running K8s cluster.
Administrator/root access to the control plane nodes.

Step-by-step Guide

SSH to one of the control plane nodes and create the encryption configuration enc.yaml file:
```
sudo mkdir /etc/kubernetes/enc
sudo touch /etc/kubernetes/enc/enc.yaml
```
Make sure that only the user who runs the kube-apiserver can read the configuration file:
```
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
```
Generate a new encryption key:
```
head -c 32 /dev/urandom | base64
```

Update the encryption configuration file:

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

Then, securely copy this file to other control plane nodes, if any.

Update the kube-apiserver (all control plane nodes) to use the new encryption configuration file: Edit the manifest /etc/kubernetes/manifests/kube-apiserver.yaml to:

---
#
# This is a fragment of a manifest for a static Pod.
# Check whether this is correct for your cluster and for your API server.
#
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.30.40:443
  creationTimestamp: null
  labels:
    app.kubernetes.io/component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml  # add this line
    volumeMounts:
    ...
    - name: enc                           # add this line
      mountPath: /etc/kubernetes/enc      # add this line
      readOnly: true                      # add this line
    ...
  volumes:
  ...
  - name: enc                             # add this line
    hostPath:                             # add this line
      path: /etc/kubernetes/enc           # add this line
      type: DirectoryOrCreate             # add this line
  ...

Testing

From now on, only newly created secrets will be encrypted. Let's create one to test:

Create a test secret:

kubectl -n default create secret generic testsecret  --from-literal=mykey=mydata

Exec into the ETCD pod to view the newly created secret:

kubectl exec -i -t -n kube-system etcd-$(hostname) -c etcd -- sh -c "sh"

ETCDCTL_API=3 etcdctl \
   --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
   --cert=/etc/kubernetes/pki/etcd/server.crt \
   --key=/etc/kubernetes/pki/etcd/server.key  \
   get /registry/secrets/default/testsecret

The output is similar to this (abbreviated):

/registry/secrets/default/testsecret
;????U?:?sd?y~???2???)ey1:?W?wu5,?J???^?v??i?|
???                   ???2???|    ???2???

As you can see, the data is encrypted.

Delete the secret:

kubectl -n default delete secret testsecret

Encrypt Secrets

The simplest way to encrypt all current secrets is to delete and recreate them. This command will do the trick:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Conclusion

Walla! You now have a more secure K8s cluster with all confidential data encrypted at rest.

References

Encrypting Confidential Data at Rest.