Monaf Horany

Menu

All runbooks

Runbooks

A Kubernetes cluster on bare-metal VMs, with kubeadm

From a fresh Ubuntu VM to a working single control-plane cluster: static networking, containerd, the kernel modules nobody mentions until it fails, kubeadm init, Flannel, Helm and the dashboard.

kubeadmcontainerdFlannelHelmUbuntu10 steps · 45–90 min

Addresses in this guide are placeholders in 10.0.0.0/24 and the domain is example.dev. Everything else is verbatim from the working notes.

  1. 01

    Pin the node's address

    A control plane that changes IP is a control plane you rebuild. Give every node a static address before anything else — kubeadm bakes the advertise address into the cluster's certificates.

    sudo nano /etc/netplan/50-cloud-init.yaml
    bash
    network:
  version: 2
  ethernets:
    enp0s3:
      dhcp4: no
      addresses: [10.0.0.79/24]
      routes:
        - to: default
          via: 10.0.0.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]
    bash
    sudo netplan apply
  2. 02

    Disable swap, permanently

    The kubelet refuses to start with swap enabled. swapoff -a handles the running system; the fstab edit is what stops it coming back on the next reboot.

    bash
    sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

sudo apt update && sudo apt upgrade
  3. 03

    Install containerd via the Docker repository

    Remove anything the distro shipped first — mixed container runtimes are a long afternoon.

    bash
    for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
  sudo apt-get remove $pkg
done
    Add Docker's official GPG key and repository
    bash
    sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  4. 04

    Set SystemdCgroup — the one that bites

    containerd's default config uses the cgroupfs driver while the kubelet uses systemd. They disagree silently: the cluster comes up, then pods restart under memory pressure for reasons that make no sense. Generate the default config and flip this one value.

    bash
    containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo nano /etc/containerd/config.toml
    Find SystemdCgroup and set it to true
    toml
    SystemdCgroup = true
    bash
    sudo systemctl restart containerd
sudo systemctl enable containerd
sudo systemctl status containerd
  5. 05

    Load the kernel modules and sysctls the CNI needs

    Without br_netfilter and IP forwarding, pod-to-pod traffic across nodes simply does not route — and nothing tells you why. Both are declared as files so they survive a reboot.

    bash
    sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system
  6. 06

    Install kubelet, kubeadm and kubectl

    bash
    sudo apt-get install -y apt-transport-https ca-certificates curl gpg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet
  7. 07

    Initialise the control plane

    The pod CIDR has to match what the CNI expects — 10.244.0.0/16 is Flannel's default, and changing one without the other produces a cluster where nothing ever becomes Ready.

    bash
    sudo kubeadm init \
  --apiserver-advertise-address=10.0.0.55 \
  --pod-network-cidr=10.244.0.0/16 \
  --kubernetes-version=v1.33.3
    Take ownership of the kubeconfig
    bash
    mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  8. 08

    Install the CNI, then Helm

    Nodes stay NotReady until a network plugin is installed. That is expected, not a fault.

    bash
    kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
  9. 09

    Reach the cluster from your workstation

    Run kubectl from the desktop, not the server. Copy admin.conf across and install the client binary.

    Windows
    bash
    curl.exe -LO "https://dl.k8s.io/release/v1.33.0/bin/windows/amd64/kubectl.exe"
curl.exe -LO "https://dl.k8s.io/v1.33.0/bin/windows/amd64/kubectl.exe.sha256"
kubectl version --client
  10. 10

    Expose the dashboard and mint a token

    The dashboard ships as a ClusterIP service, which is unreachable from outside the cluster. Switching it to NodePort is the quickest way in on a lab cluster.

    bash
    kubectl -n kubernetes-dashboard edit svc kubernetes-dashboard-kong-proxy
# change  type: ClusterIP  ->  type: NodePort

kubectl -n kubernetes-dashboard get svc kubernetes-dashboard-kong-proxy
# then browse to https://<node-ip>:<nodeport>
    admin-user.yaml
    yaml
    apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: admin-user
    namespace: kubernetes-dashboard
    bash
    kubectl apply -f admin-user.yaml
kubectl -n kubernetes-dashboard create token admin-user