How I Upgrade My Kubernetes Cluster
Introduction
I am currently managing my Kubernetes cluster for my hobby projects. Here, I show you how I upgraded my cluster to a newer version.
Prerequisites
- SSH access to all k8s cluster nodes.
- Basic knowledge of Kubernetes.
Step-by-step Guide
My k8s cluster was initialized using Kubeadm and has three control plane nodes and five worker nodes. I also use CRI-O as the container runtime. Here are the steps I did to upgrade my cluster.
-
SSH to one of the three control plane nodes and list all nodes with their status:
kubectl get nodes NAME STATUS ROLES AGE VERSION cp1 Ready control-plane 224d v1.29.6 cp2 Ready control-plane 145d v1.29.6 cp3 Ready control-plane 229d v1.29.6 node1 Ready <none> 230d v1.29.6 node2 Ready <none> 226d v1.29.6 node3 Ready <none> 224d v1.29.6 node4 Ready <none> 168d v1.29.6 node5 Ready <none> 216d v1.29.6
-
Get the latest version of Kubeadm and CRI-O:
sudo apt-get update K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1) CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1) echo $K8S_VERSION $CRI_O_VERSION 1.29.7-1.1 1.29.7-1.1
Or, you can also specify your preferred versions:
KUBERNETES_VERSION=v1.30 CRIO_VERSION=v1.30 curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/cri-o.list sudo apt-get update K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1) CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1) echo $K8S_VERSION $CRI_O_VERSION
-
Unhold the
kubeadm
, and begin the upgrade and re-hold:sudo apt-mark unhold kubeadm sudo apt-get install -y kubeadm=$K8S_VERSION sudo apt-mark hold kubeadm
-
Plan and upgrade the control plane:
sudo kubeadm upgrade plan sudo kubeadm upgrade apply UPGRADING_VERSION [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.29.7" [upgrade/versions] Cluster version: v1.29.6 [upgrade/versions] kubeadm version: v1.29.7 [upgrade] Are you sure you want to proceed? [y/N]: y
-
Next, upgrade kubelte, kubectl & cri-o:
kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager' sudo apt-mark unhold kubelet kubectl cri-o sudo apt-get install -y kubelet=$K8S_VERSION sudo apt-get install -y kubectl=$K8S_VERSION sudo apt-get install -y cri-o=$CRI_O_VERSION sudo apt-mark hold kubelet kubectl cri-o kubectl uncordon $(hostname) sudo systemctl restart kubelet
-
Repeat steps 2 through 3 to upgrade the remaining two control plane nodes and all worker nodes, then run:
sudo kubeadm upgrade node kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager' sudo apt-mark unhold kubelet kubectl cri-o sudo apt-get install -y kubelet=$K8S_VERSION kubectl=$K8S_VERSION cri-o=$CRI_O_VERSION sudo apt-mark hold kubelet kubectl cri-o kubectl uncordon $(hostname) sudo systemctl restart kubelet
-
Finally, check the status of the k8s cluster:
kubectl get nodes NAME STATUS ROLES AGE VERSION cp1 Ready control-plane 224d v1.29.7 cp2 Ready control-plane 145d v1.29.7 cp3 Ready control-plane 229d v1.29.7 node1 Ready <none> 230d v1.29.7 node2 Ready <none> 226d v1.29.7 node3 Ready <none> 224d v1.29.7 node4 Ready <none> 168d v1.29.7 node5 Ready <none> 216d v1.29.7
-
Bonus - Is it a good time to remove unused images:
sudo crictl rmi --prune
Note: If you encounter some problems related to update CRI-O with overwrite error, run this command:
sudo dpkg -i --force-overwrite /var/cache/apt/archives/cri-o_$CRI_O_VERSION_arm64.deb
Conclusion
Upgrading the K8s cluster regularly is essential to keeping it healthy. However, I always try to use the second-latest version to ensure stability and avoid potential bugs that might be present in the newest release.