Harry Tang

Performing Kubernetes etcd Defragmentation

Etcd Cluster
Etcd Cluster

Introduction

Etcd defragmentation is the process of reorganizing the etcd database to reclaim unused disk space. Over time, as keys are added, modified, and deleted, the k8s etcd database can become fragmented. This fragmentation leads to inefficient use of disk space and can impact performance.

The main goal of defragmentation is to optimize the database by removing the fragmented and unused data, thereby freeing up disk space and improving the overall efficiency and performance of the etcd instance.

Prerequisites

  • Access to the k8s cluster.
  • Etcd v3.5.

Step-by-step Guide

If you have an observability stack monitoring your Kubernetes (k8s) cluster, you might encounter this warning:

etcd cluster "kube-etcd": database size in use on instance 10.1.0.210:2381 is 49.88% of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.

  1. Exec into the etcd pod:

    kubectl exec -i -t -n kube-system etcd-cp1 -c etcd -- sh -c "sh"

  2. Run the defrag command:

    export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl defrag

  3. Verify and make sure etcd is healthy

    etcdctl endpoint health

Repeat these steps for the other etcd pods if you are running an etcd cluster in HA (High Availability) mode.

Conclusion

Etcd defragmentation is crucial for maintaining optimal performance and efficient disk space usage. Regularly defragmenting your etcd cluster reclaims unused space and ensures the health of your Kubernetes infrastructure.

Comments