Performing Kubernetes etcd Defragmentation
Introduction
Etcd defragmentation is the process of reorganizing the etcd database to reclaim unused disk space. Over time, as keys are added, modified, and deleted, the k8s etcd database can become fragmented. This fragmentation leads to inefficient use of disk space and can impact performance.
The main goal of defragmentation is to optimize the database by removing the fragmented and unused data, thereby freeing up disk space and improving the overall efficiency and performance of the etcd instance.
Prerequisites
- Access to the k8s cluster.
- Etcd v3.5.
Step-by-step Guide
If you have an observability stack monitoring your Kubernetes (k8s) cluster, you might encounter this warning:
etcd cluster "kube-etcd": database size in use on instance 10.1.0.210:2381 is 49.88% of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.
-
Exec into the etcd pod:
kubectl exec -i -t -n kube-system etcd-cp1 -c etcd -- sh -c "sh"
-
Run the
defrag
command:export ETCDCTL_API=3 export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl defrag
-
Verify and make sure etcd is healthy
etcdctl endpoint health
Repeat these steps for the other etcd pods if you are running an etcd cluster in HA (High Availability) mode.
Conclusion
Etcd defragmentation is crucial for maintaining optimal performance and efficient disk space usage. Regularly defragmenting your etcd cluster reclaims unused space and ensures the health of your Kubernetes infrastructure.