Bootstrap Kubernetes clusters with Ansible and Flux

Bootstrap clusters with Ansible and Flux

Introduction

This part covers the day-zero bootstrap flow for Kubernetes cluster management using the repo pattern from part 1. The bootstrap starts with fresh Ubuntu nodes, prepares containerd and Kubernetes packages with Ansible, initializes kubeadm, installs Cilium, restores secrets when needed, and bootstraps Flux to the correct cluster path.

Run this first on staging when validating the process. Use the same flow for production clusters once the inventory, DNS, and secrets are ready.

Prerequisites

SSH access to all nodes as ubuntu.
Correct inventory in ansible/<cluster>-inventory.ini.
A DNS name for the API server, such as k8s.example.com.
An edited init-config.yaml on the init controller.
An edited cilium.yaml on the init controller.
A working SSH deploy key for your GitOps repository.
A Flux signing keyring if you sign GitOps commits.
A Sealed Secrets private key if restoring an existing cluster.

Pick the inventory

Use one inventory per cluster:

bash

export CLUSTER=staging
export INVENTORY=ansible/staging-inventory.ini

Create ansible/staging-inventory.ini with your real node IPs:

ini

[initController]
192.0.2.10

[joinControllers]
192.0.2.11
192.0.2.12

[controllers]
192.0.2.10
192.0.2.11
192.0.2.12

[workers]
192.0.2.20
192.0.2.21
192.0.2.22

[nodes]
192.0.2.10
192.0.2.11
192.0.2.12
192.0.2.20
192.0.2.21
192.0.2.22

[all:vars]
ansible_user=ubuntu

For your first production cluster:

bash

export CLUSTER=production-a
export INVENTORY=ansible/production-a-inventory.ini

For another production cluster:

bash

export CLUSTER=production-b
export INVENTORY=ansible/production-b-inventory.ini

Check that Ansible can reach the nodes:

bash

ansible nodes -m ping -i $INVENTORY

Prepare node packages

The ansible/setup.yaml playbook prepares every node for Kubernetes cluster management. It installs base packages, resets iptables, loads kernel modules, enables IPv4 and IPv6 forwarding, installs containerd, runc, CNI plugins, kubelet, kubeadm, and kubectl.

Run it against the selected inventory:

bash

ansible-playbook ansible/setup.yaml -i $INVENTORY

The default variables are tuned for arm64 nodes:

yaml

containerd_version: '2.2.0'
containerd_arch: 'arm64'
runc_version: '1.4.0'
runc_arch: 'arm64'
cni_version: '1.8.0'
cni_arch: 'arm64'
k8s_version: 'v1.34'

Create ansible/setup.yaml:

yaml

---
- name: Setup nodes for containerd and Kubernetes
  hosts: nodes
  become: true
  vars:
    containerd_version: '2.2.0'
    containerd_arch: 'arm64'
    runc_version: '1.4.0'
    runc_arch: 'arm64'
    cni_version: '1.8.0'
    cni_arch: 'arm64'
    k8s_version: 'v1.34'

  tasks:
    - name: Update apt cache and upgrade packages
      ansible.builtin.apt:
        update_cache: true
        upgrade: dist
        cache_valid_time: 3600

    - name: Install required packages
      ansible.builtin.apt:
        name:
          - apt-transport-https
          - bash-completion
          - ca-certificates
          - curl
          - cryptsetup
          - gpg
          - iputils-ping
          - nfs-common
          - socat
          - vim
        state: present

    - name: Create modules-load file for Kubernetes
      ansible.builtin.copy:
        dest: /etc/modules-load.d/k8s.conf
        mode: '0644'
        content: |
          br_netfilter
          overlay
          dm_crypt
          nfs

    - name: Load kernel modules
      ansible.builtin.shell: |
        for m in br_netfilter overlay dm_crypt nfs; do
          /sbin/modprobe $m || true
        done
      changed_when: false

    - name: Set sysctl values for Kubernetes
      ansible.builtin.copy:
        dest: /etc/sysctl.d/k8s.conf
        mode: '0644'
        content: |
          net.ipv4.ip_forward=1
          net.ipv6.conf.all.forwarding=1

    - name: Reload sysctl settings
      ansible.builtin.command: sysctl --system

    - name: Download containerd
      ansible.builtin.get_url:
        url: 'https://github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ containerd_arch }}.tar.gz'
        dest: '/tmp/containerd.tar.gz'
        mode: '0644'

    - name: Extract containerd
      ansible.builtin.unarchive:
        src: /tmp/containerd.tar.gz
        dest: /usr/local
        remote_src: true

    - name: Download runc
      ansible.builtin.get_url:
        url: 'https://github.com/opencontainers/runc/releases/download/v{{ runc_version }}/runc.{{ runc_arch }}'
        dest: '/tmp/runc.{{ runc_arch }}'
        mode: '0755'

    - name: Install runc
      ansible.builtin.command: install -m 755 /tmp/runc.{{ runc_arch }} /usr/local/sbin/runc
      args:
        creates: /usr/local/sbin/runc

    - name: Create CNI directory
      ansible.builtin.file:
        path: /opt/cni/bin
        state: directory
        mode: '0755'

    - name: Download CNI plugins
      ansible.builtin.get_url:
        url: 'https://github.com/containernetworking/plugins/releases/download/v{{ cni_version }}/cni-plugins-linux-{{ cni_arch }}-v{{ cni_version }}.tgz'
        dest: /tmp/cni-plugins.tgz
        mode: '0644'

    - name: Extract CNI plugins
      ansible.builtin.unarchive:
        src: /tmp/cni-plugins.tgz
        dest: /opt/cni/bin
        remote_src: true

    - name: Download containerd systemd unit
      ansible.builtin.get_url:
        url: https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
        dest: /etc/systemd/system/containerd.service
        mode: '0644'

    - name: Enable containerd
      ansible.builtin.systemd:
        name: containerd
        enabled: true
        state: started
        daemon_reload: true

    - name: Add Kubernetes apt key
      ansible.builtin.shell: |
        set -e
        mkdir -p /etc/apt/keyrings
        curl -fsSL https://pkgs.k8s.io/core:/stable:/{{ k8s_version }}/deb/Release.key \
          | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
      args:
        creates: /etc/apt/keyrings/kubernetes-apt-keyring.gpg

    - name: Add Kubernetes apt repository
      ansible.builtin.copy:
        dest: /etc/apt/sources.list.d/kubernetes.list
        mode: '0644'
        content: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{ k8s_version }}/deb/ /\n"

    - name: Install Kubernetes packages
      ansible.builtin.apt:
        update_cache: true
        name:
          - kubelet
          - kubeadm
          - kubectl
        state: present

    - name: Hold Kubernetes packages
      ansible.builtin.command: apt-mark hold kubelet kubeadm kubectl

    - name: Enable kubelet
      ansible.builtin.systemd:
        name: kubelet
        enabled: true
        state: started

Override them only when the cluster hardware or Kubernetes minor version changes:

bash

ansible-playbook ansible/setup.yaml -i $INVENTORY \
  -e containerd_arch=amd64 \
  -e runc_arch=amd64 \
  -e cni_arch=amd64

Prepare control-plane tools

The ansible/setup-cp.yaml playbook installs helper tools on control-plane nodes:

bash

ansible-playbook ansible/setup-cp.yaml -i $INVENTORY

It installs or configures:

kubectl bash completion.
k alias.
Helm CLI.
Flux CLI.
kubeseal CLI.

These tools are needed for Kubernetes cluster management tasks such as Cilium install, Flux bootstrap, and Sealed Secrets generation.

Create ansible/setup-cp.yaml:

yaml

---
- name: Install control-plane helper tools
  hosts: controllers
  become: true
  vars:
    bashrc_path: '/home/{{ ansible_user }}/.bashrc'
  tasks:
    - name: Add kubectl completion
      ansible.builtin.lineinfile:
        path: '{{ bashrc_path }}'
        line: 'source <(kubectl completion bash)'
        create: true
      become_user: '{{ ansible_user }}'

    - name: Add kubectl alias
      ansible.builtin.lineinfile:
        path: '{{ bashrc_path }}'
        line: 'alias k=kubectl'
        create: true
      become_user: '{{ ansible_user }}'

    - name: Install Helm
      ansible.builtin.shell: curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
      args:
        creates: /usr/local/bin/helm

    - name: Install Flux
      ansible.builtin.shell: curl -s https://fluxcd.io/install.sh | bash
      args:
        creates: /usr/local/bin/flux

    - name: Install kubeseal
      ansible.builtin.shell: |
        set -e
        version="0.28.0"
        arch="arm64"
        curl -fsSL -o /tmp/kubeseal.tar.gz \
          "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${version}/kubeseal-${version}-linux-${arch}.tar.gz"
        tar -xzf /tmp/kubeseal.tar.gz -C /tmp kubeseal
        install -m 0755 /tmp/kubeseal /usr/local/bin/kubeseal
      args:
        creates: /usr/local/bin/kubeseal

Create encryption config

The kubeadm config mounts /etc/kubernetes/enc/enc.yaml into the API server and enables encryption at rest for Kubernetes Secrets.

Create the encryption config on all controller nodes before kubeadm init:

bash

ansible-playbook ansible/encrypt-k8s.yaml -i $INVENTORY \
  -e enc_secret_b64=$(head -c 32 /dev/urandom | base64)

Store the generated secret securely. If the file already exists, the playbook preserves it.

Create ansible/encrypt-k8s.yaml:

yaml

---
- name: Configure Kubernetes encryption at rest
  hosts: controllers
  become: true
  gather_facts: false
  vars:
    enc_dir: /etc/kubernetes/enc
    enc_file: /etc/kubernetes/enc/enc.yaml
  tasks:
    - name: Ensure encryption directory exists
      ansible.builtin.file:
        path: '{{ enc_dir }}'
        state: directory
        owner: root
        group: root
        mode: '0755'

    - name: Check for existing encryption config
      ansible.builtin.stat:
        path: '{{ enc_file }}'
      register: enc_stat

    - name: Require enc_secret_b64 for new config
      ansible.builtin.fail:
        msg: "Pass -e enc_secret_b64=$(head -c 32 /dev/urandom | base64)"
      when: not enc_stat.stat.exists and (enc_secret_b64 is not defined or enc_secret_b64 | trim == '')

    - name: Create encryption config
      ansible.builtin.copy:
        dest: '{{ enc_file }}'
        owner: root
        group: root
        mode: '0600'
        content: |
          ---
          apiVersion: apiserver.config.k8s.io/v1
          kind: EncryptionConfiguration
          resources:
            - resources:
                - secrets
              providers:
                - secretbox:
                    keys:
                      - name: key1
                        secret: "{{ enc_secret_b64 }}"
                - identity: {}
      when: not enc_stat.stat.exists

Edit kubeadm config

Start from:

txt

kubeadm/init-config.yaml

Create kubeadm/init-config.yaml, then copy it to the init controller as init-config.yaml:

yaml

---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
bootstrapTokens:
  - token: xxx.yyyyyy
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
    groups:
      - system:bootstrappers:kubeadm:default-node-token
localAPIEndpoint:
  advertiseAddress: YOUR_CONTROL_PLANE_NODE_IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  kubeletExtraArgs:
    - name: node-ip
      value: YOUR_CONTROL_PLANE_NODE_IP
  name: YOUR_CONTROL_PLANE_NODE_NAME
  taints:
    - key: node-role.kubernetes.io/control-plane
      effect: NoSchedule
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  certSANs:
    - k8s.host.com
  extraArgs:
    - name: encryption-provider-config
      value: /etc/kubernetes/enc/enc.yaml
  extraVolumes:
    - name: enc
      hostPath: /etc/kubernetes/enc
      mountPath: /etc/kubernetes/enc
      pathType: DirectoryOrCreate
      readOnly: true
controlPlaneEndpoint: k8s.host.com:6443
controllerManager:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      - name: listen-metrics-urls
        value: http://0.0.0.0:2381
kubernetesVersion: v1.34.2
networking:
  dnsDomain: k8s.host.com
  podSubnet: 172.16.0.0/16,fd00::/104
  serviceSubnet: 172.31.0.0/16,fd00:1::/112
scheduler:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

Keep kube-proxy disabled during init because this pattern uses Cilium kube-proxy replacement.

The kubeadm playbook runs:

bash

kubeadm init --config=init-config.yaml --upload-certs --skip-phases=addon/kube-proxy

Initialize the cluster

Set the API server host. This must match the DNS name or IP that join nodes can reach:

bash

export API_SERVER_HOST=k8s.example.com

Run the init playbook:

bash

ansible-playbook ansible/init-k8s.yaml -i $INVENTORY \
  -e api_server_host=$API_SERVER_HOST

Create ansible/init-k8s.yaml:

yaml

---
- name: Require API server host
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Validate api_server_host
      ansible.builtin.assert:
        that:
          - api_server_host is defined
          - api_server_host != ''
        fail_msg: "Pass -e api_server_host=k8s.example.com"

- name: Initialize first control-plane node
  hosts: initController
  become: true
  gather_facts: true
  tasks:
    - name: Run kubeadm init
      ansible.builtin.shell: |
        set -o pipefail
        kubeadm init --config=init-config.yaml --upload-certs --skip-phases=addon/kube-proxy 2>&1 \
          | tee /tmp/kubeadm_init.out
      args:
        executable: /bin/bash
      register: kubeadm_init
      failed_when: kubeadm_init.rc != 0 and 'already initialized' not in kubeadm_init.stdout

    - name: Create kubeconfig directory
      ansible.builtin.file:
        path: /home/ubuntu/.kube
        state: directory
        owner: ubuntu
        group: ubuntu
        mode: '0700'

    - name: Copy admin kubeconfig
      ansible.builtin.copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/ubuntu/.kube/config
        remote_src: true
        owner: ubuntu
        group: ubuntu
        mode: '0600'

    - name: Extract worker join command
      ansible.builtin.shell: |
        kubeadm token create --print-join-command
      register: worker_join_raw
      changed_when: false

    - name: Upload certs for control-plane joins
      ansible.builtin.shell: |
        kubeadm init phase upload-certs --upload-certs | tail -n1
      register: certificate_key_raw
      changed_when: false

    - name: Save join commands
      ansible.builtin.set_fact:
        worker_join_cmd: "{{ worker_join_raw.stdout }}"
        control_join_cmd: "{{ worker_join_raw.stdout }} --control-plane --certificate-key {{ certificate_key_raw.stdout }}"

- name: Join remaining control-plane nodes
  hosts: joinControllers
  become: true
  tasks:
    - name: Skip if already joined
      ansible.builtin.stat:
        path: /etc/kubernetes/kubelet.conf
      register: kubelet_conf

    - name: Join control-plane node
      ansible.builtin.shell: "{{ hostvars[groups['initController'][0]].control_join_cmd }}"
      when: not kubelet_conf.stat.exists

- name: Join worker nodes
  hosts: workers
  become: true
  tasks:
    - name: Skip if already joined
      ansible.builtin.stat:
        path: /etc/kubernetes/kubelet.conf
      register: kubelet_conf

    - name: Join worker node
      ansible.builtin.shell: "{{ hostvars[groups['initController'][0]].worker_join_cmd }}"
      when: not kubelet_conf.stat.exists

The playbook:

Runs kubeadm init on initController.
Copies /etc/kubernetes/admin.conf to /home/ubuntu/.kube/config.
Extracts the join token, discovery hash, and certificate key.
Joins joinControllers as control-plane nodes.
Joins workers as worker nodes.
Restarts kubelet after join.

Verify nodes:

bash

kubectl get nodes -o wide

At this point nodes may be NotReady until Cilium is installed.

Install Cilium

Copy and edit:

txt

kubeadm/cilium.yaml

Create kubeadm/cilium.yaml:

yaml

k8sServiceHost: k8s.example.com
ipv6:
  enabled: true
bpf:
  masquerade: true
externalIPs:
  enabled: true
gatewayAPI:
  enabled: false
  hostNetwork:
    enabled: true
  gatewayClass:
    create: "true"
hostPort:
  enabled: true
hubble:
  peerService:
    clusterDomain: k8s.example.com
  relay:
    enabled: true
  ui:
    enabled: true
ingressController:
  default: true
  enabled: true
  enforceHttps: false
  loadbalancerMode: shared
  service:
    insecureNodePort: 32080
    secureNodePort: 32443
    type: NodePort
ipam:
  operator:
    clusterPoolIPv4PodCIDRList:
      - 172.16.0.0/16
k8sServicePort: 6443
kubeProxyReplacement: true
nodePort:
  enabled: true
socketLB:
  enabled: true
  hostNamespaceOnly: false

Install Cilium:

bash

helm repo add cilium https://helm.cilium.io/
helm repo update
helm upgrade --install cilium cilium/cilium \
  --version 1.18.2 \
  --namespace kube-system \
  -f cilium.yaml

Wait for Cilium:

bash

kubectl -n kube-system rollout status ds/cilium --timeout=10m
kubectl -n kube-system rollout status deployment/cilium-operator --timeout=10m
kubectl get nodes

Restore Sealed Secrets key

If rebuilding a cluster that already has sealed secrets in Git, restore the Sealed Secrets private key before the controller tries to decrypt cluster secrets:

bash

kubectl apply -f main.key

For a new cluster, let the Sealed Secrets controller create a new key, then back it up immediately:

bash

kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > main.key

Keep the key outside Git.

Bootstrap Flux

Prepare the Flux SSH key and GPG keyring on a controller node:

bash

export CLUSTER_DOMAIN=k8s.example.com
export CLUSTER_PATH=clusters/staging

For production, change CLUSTER_PATH:

bash

export CLUSTER_PATH=clusters/production-a

or:

bash

export CLUSTER_PATH=clusters/production-b

Bootstrap Flux:

bash

flux bootstrap git \
  --cluster-domain=$CLUSTER_DOMAIN \
  --components source-controller,kustomize-controller,helm-controller,notification-controller \
  --components-extra image-reflector-controller,image-automation-controller,source-watcher \
  --url=ssh://git@github.com/YOUR_ORG/k8s \
  --branch=main \
  --private-key-file=./id_ed25519 \
  --path=$CLUSTER_PATH \
  --gpg-key-id=29127FBA5514724E \
  --gpg-key-ring=./keyring.gpg \
  --author-email=gitops@example.com \
  --author-name=mrgitops

This creates the Flux controllers and points the root Kustomization at the selected cluster path.

Watch first reconciliation

Check the source and Kustomizations:

bash

flux get sources git -A
flux get kustomizations -A

Then watch pods:

bash

kubectl get pods -A -w

If a phase fails because a sealed secret is missing, add the required sealed secret, commit, push, and reconcile again.

Control-plane nodes as workers

For small clusters like staging, you may want control-plane nodes to run workloads:

bash

kubectl taint nodes --all node-role.kubernetes.io/control-plane- || true

Reset a failed bootstrap

Use reset only when you intentionally want to destroy the cluster state on nodes:

bash

ansible-playbook ansible/reset-k8s.yaml -i $INVENTORY

The reset playbook runs kubeadm reset, removes CNI config, removes /var/lib/longhorn/, resets iptables, and prints the firewall state.

Do not run this against production nodes unless you are rebuilding the cluster.

Create ansible/reset-k8s.yaml only for rebuild scenarios:

yaml

---
- name: Reset Kubernetes nodes
  hosts: nodes
  become: true
  gather_facts: false
  tasks:
    - name: Reset kubeadm
      ansible.builtin.command: kubeadm reset --force
      ignore_errors: true

    - name: Remove CNI configuration
      ansible.builtin.file:
        path: /etc/cni/net.d
        state: absent

    - name: Remove Longhorn data directory
      ansible.builtin.file:
        path: /var/lib/longhorn
        state: absent

    - name: Flush iptables rules
      ansible.builtin.command: iptables -F

    - name: Delete custom iptables chains
      ansible.builtin.command: iptables -X

    - name: Reset default iptables policies
      ansible.builtin.shell: |
        iptables -P INPUT ACCEPT
        iptables -P FORWARD ACCEPT
        iptables -P OUTPUT ACCEPT

Bootstrap verification

A healthy bootstrap should pass these checks:

bash

kubectl get nodes -o wide
kubectl -n kube-system get pods
kubectl -n flux-system get pods
flux check
flux get kustomizations -A
flux get helmreleases -A

For Kubernetes cluster management, do not move on to application work until all infrastructure phases are either ready or waiting only on known secrets that you are about to create.

What comes next

Part 3 explains the Flux and Kustomize configuration model, including the infrastructure phases and cluster overlays.

References

Part 1 - Overview
Part 2 - Bootstrap Kubernetes clusters

If you found this useful, you can buy me a coffee! Thanks for the support!