Harry Tang

How I Upgrade My Kubernetes Cluster

Upgrade Kubernetes
Upgrade Kubernetes

Introduction

I am currently managing my Kubernetes cluster for my hobby projects. Here, I show you how I upgraded my cluster to a newer version.

Prerequisites

  • SSH access to all k8s cluster nodes.
  • Basic knowledge of Kubernetes.

Step-by-step Guide

My k8s cluster was initialized using Kubeadm and has three control plane nodes and five worker nodes. I also use CRI-O as the container runtime. Here are the steps I did to upgrade my cluster.

  1. SSH to one of the three control plane nodes and list all nodes with their status:

    kubectl get nodes
NAME    STATUS   ROLES           AGE    VERSION
cp1     Ready    control-plane   224d   v1.29.6
cp2     Ready    control-plane   145d   v1.29.6
cp3     Ready    control-plane   229d   v1.29.6
node1   Ready    <none>          230d   v1.29.6
node2   Ready    <none>          226d   v1.29.6
node3   Ready    <none>          224d   v1.29.6
node4   Ready    <none>          168d   v1.29.6
node5   Ready    <none>          216d   v1.29.6

  2. Get the latest version of Kubeadm and CRI-O:

    sudo apt-get update
K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1)
CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1)
echo $K8S_VERSION $CRI_O_VERSION
1.29.7-1.1 1.29.7-1.1

    Or, you can also specify your preferred versions:

    KUBERNETES_VERSION=v1.30
CRIO_VERSION=v1.30
curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list    
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/cri-o.list 
sudo apt-get update
K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1)
CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1)
echo $K8S_VERSION $CRI_O_VERSION

  3. Unhold the kubeadm, and begin the upgrade and re-hold:

    sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=$K8S_VERSION
sudo apt-mark hold kubeadm

  4. Plan and upgrade the control plane:

    sudo kubeadm upgrade plan
sudo kubeadm upgrade apply UPGRADING_VERSION

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.29.7"
[upgrade/versions] Cluster version: v1.29.6
[upgrade/versions] kubeadm version: v1.29.7
[upgrade] Are you sure you want to proceed? [y/N]: y

  5. Next, upgrade kubelte, kubectl & cri-o:

    kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager'
sudo apt-mark unhold kubelet kubectl cri-o
sudo apt-get install -y kubelet=$K8S_VERSION
sudo apt-get install -y kubectl=$K8S_VERSION
sudo apt-get install -y cri-o=$CRI_O_VERSION
sudo apt-mark hold kubelet kubectl cri-o
kubectl uncordon $(hostname)
sudo systemctl restart kubelet

  6. Repeat steps 2 through 3 to upgrade the remaining two control plane nodes and all worker nodes, then run:

    sudo kubeadm upgrade node
kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager'
sudo apt-mark unhold kubelet kubectl cri-o
sudo apt-get install -y kubelet=$K8S_VERSION kubectl=$K8S_VERSION cri-o=$CRI_O_VERSION
sudo apt-mark hold kubelet kubectl cri-o
kubectl uncordon $(hostname)
sudo systemctl restart kubelet

  7. Finally, check the status of the k8s cluster:

    kubectl get nodes
NAME    STATUS   ROLES           AGE    VERSION
cp1     Ready    control-plane   224d   v1.29.7
cp2     Ready    control-plane   145d   v1.29.7
cp3     Ready    control-plane   229d   v1.29.7
node1   Ready    <none>          230d   v1.29.7
node2   Ready    <none>          226d   v1.29.7
node3   Ready    <none>          224d   v1.29.7
node4   Ready    <none>          168d   v1.29.7
node5   Ready    <none>          216d   v1.29.7

  8. Bonus - Is it a good time to remove unused images:

    sudo crictl rmi --prune

Note: If you encounter some problems related to update CRI-O with overwrite error, run this command:

sudo dpkg -i --force-overwrite /var/cache/apt/archives/cri-o_$CRI_O_VERSION_arm64.deb

Conclusion

Upgrading the K8s cluster regularly is essential to keeping it healthy. However, I always try to use the second-latest version to ensure stability and avoid potential bugs that might be present in the newest release.

References

Comments