How I Upgrade My Kubernetes Cluster

Upgrade Kubernetes
Upgrade Kubernetes

Introduction

I am currently managing my Kubernetes cluster for my hobby projects. Here, I show you how I upgraded my cluster to a newer version.

Prerequisites

  • SSH access to all k8s cluster nodes.
  • Basic knowledge of Kubernetes.

Step-by-step Guide

My k8s cluster was initialized using Kubeadm and has three control plane nodes and five worker nodes. I also use CRI-O as the container runtime. Here are the steps I did to upgrade my cluster.

  1. SSH to one of the three control plane nodes and list all nodes with their status:

    kubectl get nodes
    NAME    STATUS   ROLES           AGE    VERSION
    cp1     Ready    control-plane   224d   v1.29.6
    cp2     Ready    control-plane   145d   v1.29.6
    cp3     Ready    control-plane   229d   v1.29.6
    node1   Ready    <none>          230d   v1.29.6
    node2   Ready    <none>          226d   v1.29.6
    node3   Ready    <none>          224d   v1.29.6
    node4   Ready    <none>          168d   v1.29.6
    node5   Ready    <none>          216d   v1.29.6
    
  2. Get the latest version of Kubeadm and CRI-O:

    sudo apt-get update
    K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1)
    CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1)
    echo $K8S_VERSION $CRI_O_VERSION
    1.29.7-1.1 1.29.7-1.1
    

    Or, you can also specify your preferred versions:

    KUBERNETES_VERSION=v1.30
    CRIO_VERSION=v1.30
    curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list    
    curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
    echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/cri-o.list 
    sudo apt-get update
    K8S_VERSION=$(apt-cache madison kubeadm | awk '{print $3}' | head -1)
    CRI_O_VERSION=$(apt-cache madison cri-o | awk '{print $3}' | head -1)
    echo $K8S_VERSION $CRI_O_VERSION
    
  3. Unhold the kubeadm, and begin the upgrade and re-hold:

    sudo apt-mark unhold kubeadm
    sudo apt-get install -y kubeadm=$K8S_VERSION
    sudo apt-mark hold kubeadm
    
  4. Plan and upgrade the control plane:

    sudo kubeadm upgrade plan
    sudo kubeadm upgrade apply UPGRADING_VERSION
    
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [preflight] Running pre-flight checks.
    [upgrade] Running cluster health checks
    [upgrade/version] You have chosen to change the cluster version to "v1.29.7"
    [upgrade/versions] Cluster version: v1.29.6
    [upgrade/versions] kubeadm version: v1.29.7
    [upgrade] Are you sure you want to proceed? [y/N]: y
    
  5. Next, upgrade kubelte, kubectl & cri-o:

    kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager'
    sudo apt-mark unhold kubelet kubectl cri-o
    sudo apt-get install -y kubelet=$K8S_VERSION
    sudo apt-get install -y kubectl=$K8S_VERSION
    sudo apt-get install -y cri-o=$CRI_O_VERSION
    sudo apt-mark hold kubelet kubectl cri-o
    kubectl uncordon $(hostname)
    sudo systemctl restart kubelet     
    
  6. Repeat steps 2 through 3 to upgrade the remaining two control plane nodes and all worker nodes, then run:

    sudo kubeadm upgrade node
    kubectl drain $(hostname) --delete-emptydir-data --ignore-daemonsets --pod-selector='app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager'
    sudo apt-mark unhold kubelet kubectl cri-o
    sudo apt-get install -y kubelet=$K8S_VERSION kubectl=$K8S_VERSION cri-o=$CRI_O_VERSION
    sudo apt-mark hold kubelet kubectl cri-o
    kubectl uncordon $(hostname)
    sudo systemctl restart kubelet
    
  7. Finally, check the status of the k8s cluster:

    kubectl get nodes
    NAME    STATUS   ROLES           AGE    VERSION
    cp1     Ready    control-plane   224d   v1.29.7
    cp2     Ready    control-plane   145d   v1.29.7
    cp3     Ready    control-plane   229d   v1.29.7
    node1   Ready    <none>          230d   v1.29.7
    node2   Ready    <none>          226d   v1.29.7
    node3   Ready    <none>          224d   v1.29.7
    node4   Ready    <none>          168d   v1.29.7
    node5   Ready    <none>          216d   v1.29.7
    
  8. Bonus - Is it a good time to remove unused images:

    sudo crictl rmi --prune
    

Note: If you encounter some problems related to update CRI-O with overwrite error, run this command:

sudo dpkg -i --force-overwrite /var/cache/apt/archives/cri-o_$CRI_O_VERSION_arm64.deb

Conclusion

Upgrading the K8s cluster regularly is essential to keeping it healthy. However, I always try to use the second-latest version to ensure stability and avoid potential bugs that might be present in the newest release.

References

Comments