Configure Kubernetes clusters with Flux phases part 3

Introduction

After part 2, Flux is running and points at clusters/staging, but the four infrastructure phases are still empty. This part fills them: the base manifests every cluster shares, the overlays that hold each cluster's differences, and the commands to reconcile, suspend, and validate a phase.

By the end, staging runs storage, networking, observability, operators, and certificate issuers from Git. Kubernetes cluster management stays repeatable because every cluster reads the same base, and only its overlay changes. The cluster names staging, production-a, and production-b come from part 1; replace them with your own.

Root cluster path

Flux starts at one cluster path:

txt

clusters/staging
clusters/production-a
clusters/production-b

The generated Flux sync file points at that path:

yaml

spec:
  interval: 10m0s
  path: ./clusters/staging
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Anything imported from that cluster directory becomes part of the cluster's desired state.

ArtifactGenerator

clusters/staging/artifacts.yaml from part 1 splits the monorepo into smaller Flux artifacts. The staging cluster gets these:

txt

infrastructure  from infrastructure/**
apps            from apps/base/** and apps/staging/**
databases       from databases/base/** and databases/staging/**
tenants         from tenants/staging/**

Each artifact contains only the paths one layer needs, so the apps Kustomization never sees database or tenant manifests from other clusters.

When you push a commit, the generator copies the new revision into the artifacts, and every Kustomization that reads a changed artifact reconciles on its own. You do not need a separate trigger per layer.

Infrastructure Kustomizations

clusters/staging/infrastructure.yaml, also from part 1, defines one Flux Kustomization per phase. All four read the infrastructure artifact and wait for the previous phase. Only the path changes:

txt

infra-phase0  path ./phase0/staging
infra-phase1  path ./phase1/staging  depends on infra-phase0
infra-phase2  path ./phase2/staging  depends on infra-phase1
infra-phase3  path ./phase3/staging  depends on infra-phase2

Production clusters use the same file with ./phase0/production-a, ./phase0/production-b, and so on.

Phase 0 configuration

Phase 0 installs the platform foundation:

txt

Longhorn
Cilium
kubelet-csr-approver
Sealed Secrets
Reloader

The base directory is:

txt

infrastructure/phase0/base

Create infrastructure/phase0/base/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sealed-secrets.yaml
  - longhorn.yaml
  - csr-approver.yaml
  - reloader.yaml
  - cilium.yaml
  # Volume Snapshot CRDs
  - https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
  - https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
  - https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
  # Gateway API CRDs for Cilium 1.18
  - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml

The Gateway API CRDs belong in phase 0 because Cilium needs them before it can serve a Gateway, and phase 3 creates the first one. Cilium 1.18 expects the v1.3.0 CRDs.

Create infrastructure/phase0/base/longhorn.yaml:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: longhorn-system
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: longhorn
  namespace: longhorn-system
spec:
  interval: 24h
  url: https://charts.longhorn.io
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: longhorn
  namespace: longhorn-system
spec:
  interval: 24h
  chart:
    spec:
      chart: longhorn
      version: '~1.11.0'
      sourceRef:
        kind: HelmRepository
        name: longhorn
        namespace: longhorn-system
      interval: 24h
  values:
    persistence:
      defaultDataLocality: best-effort
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-rwx
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: '3'
  staleReplicaTimeout: '2880'
  fsType: ext4
  nfsOptions: vers=4.2,noresvport,softerr,timeo=600,retrans=5
  dataLocality: best-effort

Create infrastructure/phase0/base/cilium.yaml:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: cilium
  namespace: kube-system
spec:
  interval: 24h
  url: https://helm.cilium.io/
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: cilium
      version: '1.18.2'
      sourceRef:
        kind: HelmRepository
        name: cilium
        namespace: kube-system
      interval: 24h
  values:
    ipv6:
      enabled: true
    bpf:
      masquerade: true
    externalIPs:
      enabled: true
    gatewayAPI:
      enabled: true
      hostNetwork:
        enabled: true
      gatewayClass:
        create: "true"
    hostPort:
      enabled: true
    hubble:
      relay:
        enabled: true
      ui:
        enabled: true
    ingressController:
      default: true
      enabled: true
      enforceHttps: false
      loadbalancerMode: shared
      service:
        insecureNodePort: 32080
        secureNodePort: 32443
        type: NodePort
    ipam:
      operator:
        clusterPoolIPv4PodCIDRList:
          - 172.16.0.0/16
    k8sServicePort: 6443
    kubeProxyReplacement: true
    nodePort:
      enabled: true
    socketLB:
      enabled: true
      hostNamespaceOnly: false

Flux takes over the cilium Helm release that part 2 installed and upgrades it with only these values. Keep them in step with kubeadm/cilium.yaml: a key you leave out here falls back to the chart default on the next reconcile.

There is one intended difference. kubeadm/cilium.yaml sets gatewayAPI.enabled: false because the Gateway API CRDs do not exist during bootstrap. Phase 0 installs the CRDs, so the HelmRelease turns the feature on. The API server host and Hubble domain differ per cluster, so they live in the overlay below.

Create infrastructure/phase0/base/sealed-secrets.yaml:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: sealed-secrets
  namespace: kube-system
spec:
  interval: 24h
  url: https://bitnami.github.io/sealed-secrets
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: sealed-secrets
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: sealed-secrets
      version: '~2.18.0'
      sourceRef:
        kind: HelmRepository
        name: sealed-secrets
        namespace: kube-system
      interval: 24h
  values:
    fullnameOverride: sealed-secrets-controller
    keyrenewperiod: 2160h # 90 days

fullnameOverride keeps the controller name that kubeseal looks for by default. The controller adds a new sealing key every 90 days, so back up the keys again after each renewal.

Create infrastructure/phase0/base/csr-approver.yaml. Part 2 set serverTLSBootstrap: true, so each kubelet requests a serving certificate, and this controller approves those requests. metrics-server in phase 2 depends on these certificates:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: kubelet-csr-approver
  namespace: kube-system
spec:
  interval: 24h
  url: https://postfinance.github.io/kubelet-csr-approver
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kubelet-csr-approver
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: kubelet-csr-approver
      version: '~1.2.0'
      sourceRef:
        kind: HelmRepository
        name: kubelet-csr-approver
        namespace: kube-system
      interval: 24h

Create infrastructure/phase0/base/reloader.yaml. Reloader rolls annotated workloads when a ConfigMap or Secret they use changes, which matters once sealed secrets start rotating:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: stakater
  namespace: kube-system
spec:
  interval: 24h
  url: https://stakater.github.io/stakater-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: reloader
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: reloader
      version: '~2.2.0'
      sourceRef:
        kind: HelmRepository
        name: stakater
        namespace: kube-system
      interval: 24h

Create infrastructure/phase0/staging/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - path: patch.cilium.yaml
    target:
      kind: HelmRelease
      name: cilium
      namespace: kube-system
  - path: patch.longhorn.yaml
    target:
      kind: HelmRelease
      name: longhorn
      namespace: longhorn-system

Create infrastructure/phase0/staging/patch.cilium.yaml with the same API host you used in kubeadm/cilium.yaml:

yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  values:
    k8sServiceHost: k8s.example.com
    hubble:
      peerService:
        clusterDomain: k8s.example.com

To test a new Longhorn version, patch it in staging first. After testing, promote the version to infrastructure/phase0/base/longhorn.yaml or to the production overlays. Create infrastructure/phase0/staging/patch.longhorn.yaml:

yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: longhorn
  namespace: longhorn-system
spec:
  chart:
    spec:
      version: '~1.12.0'

Phase 1 configuration

Phase 1 installs GitOps alerts, private access, tunnels, recurring jobs, and observability:

txt

infrastructure/phase1/base/gitops.yaml
infrastructure/phase1/base/tunnel.yaml
infrastructure/phase1/base/longhorn.recurring-jobs.yaml
infrastructure/phase1/base/tailscale.yaml
infrastructure/phase1/base/observability.yaml

Cluster patches live in:

txt

infrastructure/phase1/staging
infrastructure/phase1/production-a
infrastructure/phase1/production-b

Use the patches for cluster-specific values such as:

Tailscale hostname and tags.
Cloudflare tunnel name.
Prometheus external URL.
Alertmanager external URL.
Loki values.
kube-prometheus-stack values.

Most of this phase reads secrets: the Slack webhook, Grafana admin login, Tailscale OAuth client, and tunnel credentials. Part 4 creates them as sealed secrets in the cluster overlay. Until they exist, infra-phase1 stays not ready.

Create infrastructure/phase1/base/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gitops.yaml
  - longhorn.recurring-jobs.yaml
  - observability.yaml
  - tailscale.yaml
  - tunnel.yaml

Create a minimal infrastructure/phase1/base/gitops.yaml:

yaml

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: slack
  namespace: flux-system
spec:
  summary: 'Kubernetes Cluster Notification'
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: HelmRepository
      name: '*'
    - kind: Kustomization
      name: '*'
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  secretRef:
    name: slack-url

The * wildcard only matches objects in the Alert's own namespace, flux-system. That covers the Git source and every phase Kustomization. HelmReleases live in their component namespaces, so to hear about a failed chart upgrade, add one entry per namespace:

yaml

    - kind: HelmRelease
      name: '*'
      namespace: longhorn-system
    - kind: HelmRelease
      name: '*'
      namespace: kube-system

Create infrastructure/phase1/base/longhorn.recurring-jobs.yaml. Both jobs apply to every volume in the default group:

yaml

---
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: trim-filesystem
  namespace: longhorn-system
spec:
  concurrency: 1
  cron: 0 */12 * * *
  name: trim-filesystem
  task: filesystem-trim
  groups:
    - default
---
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: snapshot-cleanup
  namespace: longhorn-system
spec:
  concurrency: 1
  cron: 12 0 * * *
  groups:
    - default
  name: snapshot-cleanup
  task: snapshot-cleanup

Create a minimal infrastructure/phase1/base/tailscale.yaml. The hostname and tags are placeholders that every cluster overrides:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: tailscale
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: tailscale
  namespace: tailscale
spec:
  interval: 24h
  url: https://pkgs.tailscale.com/helmcharts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: tailscale-operator
  namespace: tailscale
spec:
  interval: 24h
  chart:
    spec:
      chart: tailscale-operator
      version: '~1.94.0'
      sourceRef:
        kind: HelmRepository
        name: tailscale
        namespace: tailscale
  values:
    operatorConfig:
      hostname: 'cluster-k8s-tailscale-operator' # patched per cluster
      defaultTags:
        - 'tag:cluster-k8s-operator' # patched per cluster
    proxyConfig:
      defaultTags: 'tag:cluster-k8s' # patched per cluster
    apiServerProxyConfig:
      mode: 'true'

In auth mode ('true'), the operator proxies tailnet users to the Kubernetes API under their Tailscale identity, plus any Kubernetes groups your tailnet grants assign, so RBAC can bind them. Part 5 shows the RoleBinding.

Create a minimal infrastructure/phase1/base/tunnel.yaml if you use Cloudflare Tunnel. The tunnel name is also a placeholder:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: community-charts
  namespace: kube-system
spec:
  interval: 24h
  url: https://community-charts.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cloudflared
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: cloudflared
      version: '~2.2.1'
      sourceRef:
        kind: HelmRepository
        name: community-charts
        namespace: kube-system
  values:
    replica:
      allNodes: false
      count: 2
    tunnelSecrets:
      existingConfigJsonFileSecret:
        name: tunnel-credentials
      existingPemFileSecret:
        name: tunnel-pem
    tunnelConfig:
      name: cluster-k8s # patched per cluster

Create a minimal infrastructure/phase1/base/observability.yaml:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: observability
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: observability
spec:
  interval: 1h
  url: https://prometheus-community.github.io/helm-charts
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: grafana
  namespace: observability
spec:
  interval: 1h
  url: https://grafana.github.io/helm-charts
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: grafana-community
  namespace: observability
spec:
  interval: 24h
  url: https://grafana-community.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kube-prom-stack
  namespace: observability
spec:
  interval: 24h
  chart:
    spec:
      chart: kube-prometheus-stack
      version: '~77.14.0'
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: observability
  values:
    crds:
      upgradeJob:
        enabled: true
        forceConflicts: true
    prometheus:
      prometheusSpec:
        replicas: 1
        retention: 7d
    grafana:
      enabled: true
      admin:
        existingSecret: grafana-admin
        userKey: ADMIN_USER
        passwordKey: ADMIN_PASSWORD
    alertmanager:
      config:
        route:
          receiver: 'slack'
          routes:
            - receiver: 'null'
              matchers:
                - alertname =~ "InfoInhibitor|Watchdog"
        receivers:
          - name: 'null'
          - name: 'slack'
            slack_configs:
              - api_url_file: '/etc/alertmanager/secrets/slack-url/SLACK_API_URL'
                channel: '#alerts'
                send_resolved: true
      alertmanagerSpec:
        secrets:
          - slack-url

alertmanagerSpec.secrets mounts the slack-url secret into the Alertmanager pods, and api_url_file reads the webhook from that file, so the URL never appears in the HelmRelease. The null route drops Watchdog, which fires all the time by design, and InfoInhibitor, which only exists to silence info-level alerts.

The two Grafana repositories serve the Loki chart (grafana-community) and the Alloy chart (grafana). Their values are long enough for their own guide: Deploy Kubernetes observability with Loki and Alloy has them as plain Helm values that drop into HelmRelease values.

The staging overlay adds its sealed secrets and replaces the placeholders. Create infrastructure/phase1/staging/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
  - sealed-secrets.yaml
patches:
  - path: patch.tailscale.yaml
    target:
      kind: HelmRelease
      name: tailscale-operator
      namespace: tailscale
  - path: patch.tunnel.yaml
    target:
      kind: HelmRelease
      name: cloudflared
      namespace: kube-system
  - path: patch.provider.yaml
    target:
      kind: Provider
      name: slack

Create infrastructure/phase1/staging/patch.tailscale.yaml:

yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: tailscale-operator
  namespace: tailscale
spec:
  values:
    operatorConfig:
      hostname: 'staging-k8s-tailscale-operator'
      defaultTags:
        - 'tag:staging-k8s-operator'
    proxyConfig:
      defaultTags: 'tag:staging-k8s'

Create infrastructure/phase1/staging/patch.tunnel.yaml. Each ingress rule maps a public hostname to a cluster service, and the last rule catches everything else:

yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cloudflared
  namespace: kube-system
spec:
  values:
    tunnelConfig:
      name: staging-k8s
    ingress:
      - hostname: grafana.example.com
        service: http://kube-prom-stack-grafana.observability:80
      - hostname: storage.example.com
        service: http://longhorn-frontend.longhorn-system:80
      - service: http_status:404

Create infrastructure/phase1/staging/patch.provider.yaml to send this cluster's Flux alerts to its own Slack channel:

yaml

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
spec:
  channel: staging-k8s

Phase 2 configuration

Phase 2 installs operators, metrics, dashboards, alerts, and maintenance components:

txt

cert-manager
trust-manager
mariadb-operator
CloudNativePG
metrics-server
kured
Falco
Longhorn ServiceMonitor and alerts
cert-manager dashboard and alerts
Galera dashboard
CSR approver ServiceMonitor

kured reads kured-env and Falco reads slack-webhook. Part 4 seals both into infrastructure/phase2/staging/sealed-secrets.yaml.

Create infrastructure/phase2/base/kustomization.yaml with the components you want to run. This starter keeps the operators and leaves dashboards and PrometheusRule files for later. The cert-manager ones are in Cert-manager Grafana dashboard and alert rules setup:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - mariadb-operator.yaml
  - metrics-server.yaml
  - kured.yaml
  - falco.yaml
  - cloudnative-pg.yaml

Create infrastructure/phase2/staging/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
  - sealed-secrets.yaml

When you want to test a new chart version for cert-manager, CloudNativePG, Falco, kured, the MariaDB operator, or metrics-server, add a version patch here first. Part 6 walks through one.

Create infrastructure/phase2/base/cert-manager.yaml:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: jetstack
  namespace: cert-manager
spec:
  interval: 24h
  url: https://charts.jetstack.io
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  interval: 24h
  chart:
    spec:
      chart: cert-manager
      version: '~1.19.0'
      sourceRef:
        kind: HelmRepository
        name: jetstack
        namespace: cert-manager
  values:
    crds:
      enabled: true
    prometheus:
      servicemonitor:
        enabled: true
        labels:
          release: kube-prom-stack
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: trust-manager
  namespace: cert-manager
spec:
  interval: 24h
  chart:
    spec:
      chart: trust-manager
      version: '~0.19.0'
      sourceRef:
        kind: HelmRepository
        name: jetstack
        namespace: cert-manager
  values:
    crds:
      enabled: true

Create infrastructure/phase2/base/mariadb-operator.yaml. The operator's CRDs ship as a separate chart, so the file has two releases that must stay on the same version:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: mariadb-operator
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: mariadb-operator
  namespace: mariadb-operator
spec:
  interval: 24h
  url: https://helm.mariadb.com/mariadb-operator
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mariadb-operator-crds
  namespace: mariadb-operator
spec:
  interval: 24h
  chart:
    spec:
      chart: mariadb-operator-crds
      version: '~25.10.0' # keep in step with mariadb-operator
      sourceRef:
        kind: HelmRepository
        name: mariadb-operator
        namespace: mariadb-operator
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mariadb-operator
  namespace: mariadb-operator
spec:
  interval: 24h
  chart:
    spec:
      chart: mariadb-operator
      version: '~25.10.0' # keep in step with mariadb-operator-crds
      sourceRef:
        kind: HelmRepository
        name: mariadb-operator
        namespace: mariadb-operator
  values:
    metrics:
      enabled: true
      serviceMonitor:
        additionalLabels:
          release: kube-prom-stack

Create infrastructure/phase2/base/cloudnative-pg.yaml:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: cnpg-system
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: cnpg
  namespace: cnpg-system
spec:
  interval: 24h
  url: https://cloudnative-pg.github.io/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cnpg
  namespace: cnpg-system
spec:
  interval: 24h
  chart:
    spec:
      chart: cloudnative-pg
      version: '~0.27.1'
      sourceRef:
        kind: HelmRepository
        name: cnpg
        namespace: cnpg-system

Create infrastructure/phase2/base/metrics-server.yaml:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  interval: 24h
  url: https://kubernetes-sigs.github.io/metrics-server/
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: metrics-server
      version: '~3.13.0'
      sourceRef:
        kind: HelmRepository
        name: metrics-server
        namespace: kube-system

Create infrastructure/phase2/base/kured.yaml:

yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: kubereboot
  namespace: kube-system
spec:
  interval: 24h
  url: https://kubereboot.github.io/charts/
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kured
  namespace: kube-system
spec:
  interval: 24h
  chart:
    spec:
      chart: kured
      version: '~5.11.0'
      sourceRef:
        kind: HelmRepository
        name: kubereboot
        namespace: kube-system
  values:
    metrics:
      create: true
      namespace: observability
      labels:
        release: kube-prom-stack
    extraEnvVars:
      - name: KURED_NOTIFY_URL
        valueFrom:
          secretKeyRef:
            name: kured-env
            key: KURED_NOTIFY_URL
      - name: KURED_DRAIN_POD_SELECTOR
        value: app!=csi-attacher,app!=csi-provisioner,longhorn.io/component!=instance-manager

Create infrastructure/phase2/base/falco.yaml:

yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: falco
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: falcosecurity
  namespace: falco
spec:
  interval: 24h
  url: https://falcosecurity.github.io/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: falco
  namespace: falco
spec:
  interval: 24h
  chart:
    spec:
      chart: falco
      version: '~6.3.0'
      sourceRef:
        kind: HelmRepository
        name: falcosecurity
        namespace: falco
  values:
    falcosidekick:
      enabled: true
      config:
        extraEnv:
          - name: SLACK_WEBHOOKURL
            valueFrom:
              secretKeyRef:
                name: slack-webhook
                key: url
    metrics:
      enabled: true
    serviceMonitor:
      create: true
      labels:
        release: kube-prom-stack

Phase 3 configuration

Phase 3 installs gateway and issuer resources:

txt

infrastructure/phase3/base/namespaces.yaml
infrastructure/phase3/base/letsencrypt.cluster-issuer.yaml
infrastructure/phase3/base/zerossl.cluster-issuer.yaml
infrastructure/phase3/base/cilium.operator-restart.yaml

Cluster overlays patch issuer credentials and add cluster-specific gateways:

txt

infrastructure/phase3/staging/hello.gw.yaml
infrastructure/phase3/production-a/app.gw.yaml

Create infrastructure/phase3/base/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cilium.operator-restart.yaml
  - namespaces.yaml
  - letsencrypt.cluster-issuer.yaml
  - zerossl.cluster-issuer.yaml

Create infrastructure/phase3/base/namespaces.yaml for the shared Gateways:

yaml

apiVersion: v1
kind: Namespace
metadata:
  name: gateway

Both issuers solve DNS-01 challenges through Cloudflare. They read the cloudflare-api-token secret, and ZeroSSL also reads zero-ssl-eabsecret. Part 4 creates both in infrastructure/phase3/staging/sealed-secrets.yaml.

Create infrastructure/phase3/base/letsencrypt.cluster-issuer.yaml:

yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-cluster-issuer-account-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Create infrastructure/phase3/base/zerossl.cluster-issuer.yaml if you also want ZeroSSL:

yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: zerossl
spec:
  acme:
    email: admin@example.com
    server: https://acme.zerossl.com/v2/DV90
    privateKeySecretRef:
      name: zerossl-cluster-issuer-account-key
    externalAccountBinding:
      keyID: YOUR_ZEROSSL_EAB_KEY_ID
      keySecretRef:
        name: zero-ssl-eabsecret
        key: secret
      keyAlgorithm: HS256
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

The Gateway API CRDs and the Cilium HelmRelease arrive in the same phase, so the Cilium operator can start before it sees the CRDs. When that happens, the cilium GatewayClass stays stuck until the operator restarts. Phase 3 runs that restart once as a Job, after every earlier phase is ready.

Create infrastructure/phase3/base/cilium.operator-restart.yaml:

yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cilium-operator-restart
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cilium-operator-restart
  namespace: kube-system
rules:
  - apiGroups:
      - apps
    resources:
      - deployments
    verbs:
      - get
      - patch
      - update
  - apiGroups:
      - apps
    resources:
      - deployments/status
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cilium-operator-restart
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: cilium-operator-restart
    namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cilium-operator-restart
---
apiVersion: batch/v1
kind: Job
metadata:
  name: cilium-operator-restart-once
  namespace: kube-system
spec:
  template:
    spec:
      restartPolicy: OnFailure
      serviceAccountName: cilium-operator-restart
      containers:
        - name: kubectl
          image: bitnami/kubectl
          command:
            - /bin/sh
            - -c
            - |
              kubectl -n kube-system rollout restart deployment/cilium-operator
              kubectl -n kube-system rollout status deployment/cilium-operator --timeout=10m

Create infrastructure/phase3/staging/kustomization.yaml:

yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
  - sealed-secrets.yaml
  - hello.gw.yaml

Create infrastructure/phase3/staging/hello.gw.yaml. It is a test Gateway that proves cert-manager, the issuer, and Cilium work together:

yaml

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hello
  namespace: gateway
spec:
  secretName: hello-tls
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - hello.example.com
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: hello
  namespace: gateway
spec:
  gatewayClassName: cilium
  listeners:
    - name: http
      hostname: hello.example.com
      port: 8080
      protocol: HTTP
    - name: https
      hostname: hello.example.com
      port: 8443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: hello-tls

With gatewayAPI.hostNetwork.enabled, each listener port is opened on the nodes themselves. Cilium's Envoy has no Linux capabilities by default, so it cannot bind ports below 1024 unless you set envoy.securityContext.capabilities.keepCapNetBindService=true. Unprivileged ports such as 8080 and 8443 avoid that.

HelmRelease management

Every component above follows the same shape: a HelmRepository that points at the chart source, and a HelmRelease that picks the chart, version, and values. The version line decides how much Flux may change on its own.

A tilde range lets Flux take patch releases as they appear:

yaml

version: '~1.11.0' # any 1.11.x

An exact version changes only when you commit a new one:

yaml

version: '1.18.2'

Pin networking components like Cilium exactly. A new Cilium version restarts the agent on every node, and you want that to happen when you choose, after a test in staging. Use tilde ranges where a patch release is low risk and you want fixes without a commit.

Cluster-specific values

Patch values that are truly cluster-specific:

Public hostnames.
API server hostnames.
Storage sizes.
Tailscale tags.
Cloudflare tunnel names.
Alerting endpoints.
Certificate issuer details.
App ingress hostnames.

Avoid copying full base manifests into overlays. Patch only the fields that differ.

Reconcile a phase

After editing a phase:

bash

git add infrastructure/phase0/staging/patch.longhorn.yaml
git commit -m "Test Longhorn version on staging"
git push

Then pull the new revision instead of waiting for the next interval:

bash

flux reconcile source git flux-system -n flux-system

The ArtifactGenerator copies the commit into the infrastructure artifact, and infra-phase0 reconciles because its artifact changed. To force a phase again without a new commit:

bash

flux reconcile kustomization infra-phase0 -n flux-system

Do not add --with-source here. The phases read an ExternalArtifact, and the Flux CLI can only reconcile Git, OCI, and Bucket sources that way, so the command fails.

Check status:

bash

flux get kustomizations -n flux-system
flux get helmreleases -A
kubectl get pods -A

Suspend a phase

During a risky maintenance task, suspend a Kustomization:

bash

flux suspend kustomization infra-phase0 -n flux-system

Resume when ready:

bash

flux resume kustomization infra-phase0 -n flux-system

Use suspension sparingly. A suspended phase stops drift correction for that phase.

Validate before pushing

Render every overlay locally before you push:

bash

for phase in phase0 phase1 phase2 phase3; do
  kubectl kustomize infrastructure/$phase/staging > /dev/null \
    && echo "$phase ok" || echo "$phase FAILED"
done

kubectl kustomize exits with an error when a file is missing or a patch matches no resource. Flux reports the same error on the phase, but only after the push, and the phase stays failed until you fix it.

Promote from staging to production

Use this flow:

Make the change in the staging overlay.
Reconcile staging.
Verify workloads and alerts.
If the change is safe for all clusters, move it into the base.
If only one production cluster needs it, add or update that production overlay.
Reconcile one production cluster at a time.

staging runs the same phases, artifacts, and commands as production, so a change that passes there has already gone through the real workflow.

What comes next

Part 4 creates the sealed secrets these phases are waiting for, then adds databases, apps, and tenants on top of the infrastructure.

References

If you found this useful, you can buy me a coffee! Thanks for the support!