Setting up a Kubernetes cluster today is straightforward. With tools like kubeadm or managed platforms such as Amazon EKS and Google Kubernetes Engine, you can get a working cluster in just a few minutes. While these approaches make it easy to get a cluster running, they abstract away the most important part — how Kubernetes actually works under the hood.

In this guide, we take a different approach.

Instead of relying on automation, we bootstrap a Kubernetes cluster completely from scratch using binaries. Every component is configured manually — from generating TLS certificates and kubeconfig files to setting up the API server, controller manager, scheduler, kubelet, and container runtime. This process exposes what managed services hide: how each component interacts, how authentication and authorization work, and how the control plane communicates with worker nodes.

If you've used Kubernetes before but want to truly understand what's happening behind tools like kubeadm or managed platforms, this guide will give you that depth.

Architecture Overview — What Are We Actually Building?

Before we start downloading binaries and writing config files, it's important to understand what a Kubernetes cluster actually looks like when you strip away the abstraction layers.

A Kubernetes cluster is made up of two types of nodes:

Control Plane

The control plane is the brain of the cluster. It's responsible for making decisions — which pod goes where, what the desired state of the system is, and whether the actual state matches it. It does not run your application containers. It runs the following components:

etcd — A distributed key-value store that holds the entire cluster state. It stores metadata and configuration for resources like pods, services, secrets, and configmaps. It acts as the single source of truth. If etcd becomes unavailable, the control plane cannot operate correctly.
kube-apiserver — The front door to the cluster. Every interaction — whether it's kubectl, the kubelet, or the scheduler — goes through the API server. It validates requests, authenticates clients using TLS certificates, enforces RBAC, and persists everything to etcd. No other component talks to etcd directly.
kube-controller-manager — Runs a set of control loops that continuously compare the desired state (what you declared in YAML) against the actual state (what's running on the nodes). If something drifts — a pod crashes, a node goes down — the controller manager takes corrective action automatically.
kube-scheduler — Watches for newly created pods that don't have a node assigned yet. It evaluates resource availability, affinity rules, and taints/tolerations, then assigns the pod to the best-fit node by writing the nodeName field. It does not start the container — it only decides where it should run.

Worker Nodes

Worker nodes are the machines that actually run your application containers. Each worker runs the following:

kubelet — The agent that runs on every worker node. It watches the API server for pods assigned to its node, then instructs the container runtime to pull images and start containers. It also reports the node's health and pod status back to the API server.
kube-proxy — Maintains network rules (iptables or IPVS) on each node so that traffic to a Kubernetes Service gets routed to the correct backend pod, regardless of which node that pod is running on.
Container Runtime (e.g., containerd, CRI-O) — The component that actually pulls container images and manages the lifecycle of containers. The kubelet communicates with the runtime using the Container Runtime Interface (CRI).

Phase 1: Infrastructure Provisioning & Prerequisites

Before installing any Kubernetes components, we need to provision the underlying compute infrastructure.

For this setup, we use Vagrant with VirtualBox to spin up two virtual machines locally — one acting as the control plane and the other as a worker node.

Here is the Vagrantfile used to define the environment:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"

  # Controller Node (Control Plane)
  config.vm.define "controller-0" do |c|
    c.vm.hostname = "controller-0"
    c.vm.network "private_network", ip: "10.240.0.10"
    c.vm.provider "virtualbox" do |vb|
      vb.memory = 2048
      vb.cpus = 2
      vb.name = "kthw-controller-0"
    end
  end

  # Worker Node 0
  config.vm.define "worker-0" do |w|
    w.vm.hostname = "worker-0"
    w.vm.network "private_network", ip: "10.240.0.20"
    w.vm.provider "virtualbox" do |vb|
      vb.memory = 2048
      vb.cpus = 2
      vb.name = "kthw-worker-0"
    end
  end
end

To prevent IP conflicts, Kubernetes uses three separate networks. We define these boundaries upfront:

The Node Network: 10.240.0.0/24
The Pod Network: 10.200.0.0/16
The Service Network: 10.32.0.0/24

Running vagrant up provisions two clean Ubuntu 22.04 (Jammy) machines, each with 2 vCPUs and 2 GB of RAM.

OS-Level Prerequisites

On All Nodes (controller-0 and worker-0)

# Disable swap (required by kubelet)
sudo swapoff -a

# Make swap disable persistent
sudo sed -i '/ swap / s/^/#/' /etc/fstab

Why this is required:

The kubelet expects swap to be disabled because Kubernetes relies on accurate memory accounting for scheduling decisions. If memory is swapped to disk, it becomes difficult for the scheduler to enforce resource limits reliably.

Downloading Kubernetes Binaries

On the Controller Node (controller-0)

K8S_VER="v1.32.0"
ETCD_VER="v3.5.16"

# Kubernetes control plane binaries
for BIN in kube-apiserver kube-controller-manager kube-scheduler kubectl; do
  curl -LO "https://dl.k8s.io/release/\({K8S_VER}/bin/linux/amd64/\){BIN}"
  chmod +x "${BIN}"
  sudo mv "${BIN}" /usr/local/bin/
done

# etcd and etcdctl
curl -LO "https://github.com/etcd-io/etcd/releases/download/\({ETCD_VER}/etcd-\){ETCD_VER}-linux-amd64.tar.gz"
tar -xzf etcd-${ETCD_VER}-linux-amd64.tar.gz
sudo mv etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/

On the Worker Node (worker-0)

K8S_VER="v1.32.0"
CONTAINERD_VER="1.7.22"
RUNC_VER="v1.1.12"
CNI_VER="v1.4.0"
CRICTL_VER="v1.30.0"

# Kubernetes worker binaries
for BIN in kubelet kube-proxy kubectl; do
  curl -LO "https://dl.k8s.io/release/\({K8S_VER}/bin/linux/amd64/\){BIN}"
  chmod +x ${BIN}
  sudo mv ${BIN} /usr/local/bin/
done

# containerd (container runtime)
curl -LO "https://github.com/containerd/containerd/releases/download/v\({CONTAINERD_VER}/containerd-\){CONTAINERD_VER}-linux-amd64.tar.gz"
sudo tar -xzf containerd-${CONTAINERD_VER}-linux-amd64.tar.gz -C /usr/local

# runc (low-level container runtime that actually creates Linux containers)
curl -LO "https://github.com/opencontainers/runc/releases/download/${RUNC_VER}/runc.amd64"
chmod +x runc.amd64
sudo mv runc.amd64 /usr/local/bin/runc

# crictl (CLI tool for debugging CRI-compatible container runtimes)
curl -LO "https://github.com/kubernetes-sigs/cri-tools/releases/download/\({CRICTL_VER}/crictl-\){CRICTL_VER}-linux-amd64.tar.gz"
sudo tar -xzf crictl-${CRICTL_VER}-linux-amd64.tar.gz -C /usr/local/bin/

# CNI plugins (networking)
sudo mkdir -p /opt/cni/bin
curl -LO "https://github.com/containernetworking/plugins/releases/download/\({CNI_VER}/cni-plugins-linux-amd64-\){CNI_VER}.tgz"
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VER}.tgz -C /opt/cni/bin/

Kernel Modules & Networking (Worker Node Only)

# Load required kernel modules and persist them
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
overlay
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Configure networking parameters
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply changes
sudo sysctl --system

Why this is required:

overlay — Required by containerd for managing container image layers.
br_netfilter — Allows iptables to inspect bridged traffic used in pod networking.
net.ipv4.ip_forward = 1 — Enables packet forwarding for pod-to-pod communication.
net.bridge.bridge-nf-call-iptables = 1 — Ensures bridged traffic is processed by iptables for service routing.

Phase 2: Public Key Infrastructure (PKI) — Certificates and Authentication

Prerequisite: Before diving into this section, you should be familiar with public key cryptography (public/private keys), how a Certificate Authority (CA) works, and the basics of TLS. This guide focuses on how these concepts are applied within Kubernetes.

Why Does Kubernetes Need Certificates?

When building a cluster, each component runs as an independent binary. There is no built-in trust between components — the API server cannot assume that a request from a kubelet is legitimate, and the kubelet cannot verify that it is communicating with the correct API server.

Kubernetes establishes trust using mutual TLS (mTLS). Every component is assigned its own X.509 certificate, signed by a common Certificate Authority. When two components communicate, they both present their certificates and verify that they are signed by the trusted CA.

Beyond encryption, certificates in Kubernetes also define identity. The API server extracts the Common Name (CN) and Organization (O) fields from the certificate and maps them directly to Kubernetes users and groups. These identities are then used by RBAC to determine what actions are permitted.

In other words, a certificate in Kubernetes is not just a security mechanism — it is also an identity and authorization mechanism.

Setting Up the Certificate Authority

Before generating certificates for individual components, we need to establish a root of trust.

The Certificate Authority (CA) is responsible for signing and validating all certificates in the cluster. Any component that trusts this CA will trust certificates signed by it.

We use cfssl (Cloudflare's PKI toolkit) to generate and manage these certificates.

# Install cfssl and cfssljson
CFSSL_VER="1.6.5"
curl -LO "https://github.com/cloudflare/cfssl/releases/download/v\({CFSSL_VER}/cfssl_\){CFSSL_VER}_linux_amd64"
curl -LO "https://github.com/cloudflare/cfssl/releases/download/v\({CFSSL_VER}/cfssljson_\){CFSSL_VER}_linux_amd64"
chmod +x cfssl_* cfssljson_*
sudo mv cfssl_${CFSSL_VER}_linux_amd64 /usr/local/bin/cfssl
sudo mv cfssljson_${CFSSL_VER}_linux_amd64 /usr/local/bin/cfssljson

# Create the CA configuration (defines how certificates will be signed)
cat > ca-config.json <<EOF
{
  "signing": {
    "default": { "expiry": "8760h" },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

# Create the CA certificate signing request (CSR)
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "Kubernetes", "OU": "CA" }]
}
EOF

# Generate the CA certificate and private key
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

Why this is required:

The CA (Certificate Authority) is the root of trust for the entire cluster.
Every component (API server, kubelet, etcd, clients) will trust certificates signed by this CA.
The ca-config.json defines how certificates will be issued (validity, usage, etc.).
The CSR (ca-csr.json) defines the identity of the CA itself.

This process generates the following files:

ca.pem — The public certificate. Distributed to all components so they can verify certificates signed by this CA.
ca-key.pem — The private key. Must be kept secure, as it is used to sign all other certificates.
ca.csr — The certificate signing request. Not required after certificate generation.

Generating Certificates for Each Component

With the Certificate Authority in place, we now generate certificates for each Kubernetes component.

Each component is assigned a unique identity using a certificate. These identities are used by the API server for both authentication and authorization.

1. Admin Client Certificate

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "system:masters", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin

Used by kubectl to interact with the cluster. The system:masters group grants full administrative privileges.

2. Kube Controller Manager Certificate

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "system:kube-controller-manager", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

Authenticates the controller manager to the API server. The identity is mapped to a built-in ClusterRole that allows it to manage cluster resources.

3. Kube Scheduler Certificate

cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "system:kube-scheduler", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

Authenticates the scheduler to the API server. Grants permission to read pods and assign them to nodes.

4. Kube Proxy Certificate

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "system:node-proxier", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

Authenticates kube-proxy to the API server. Grants access to Service and Endpoint resources used to configure networking rules.

5. Kubelet Certificate (Worker Node)

WORKER_IP="10.240.0.20"

cat > worker-0-csr.json <<EOF
{
  "CN": "system:node:worker-0",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "system:nodes", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  -hostname=worker-0,${WORKER_IP} \
  worker-0-csr.json | cfssljson -bare worker-0

The -hostname flag ensures the certificate is valid for the node's hostname and IP, allowing the API server to trust this node.

6. Service Account Key Pair

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "Kubernetes", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

Used by the API server and controller manager to sign and verify ServiceAccount tokens.

7. Kubernetes API Server Certificate

CONTROLLER_IP="10.240.0.10"

KUBERNETES_HOSTNAMES=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,localhost

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "O": "Kubernetes", "OU": "Kubernetes The Hard Way" }]
}
EOF

cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=kubernetes \
  -hostname=10.32.0.1,\({CONTROLLER_IP},127.0.0.1,\){KUBERNETES_HOSTNAMES} \
  kubernetes-csr.json | cfssljson -bare kubernetes

This certificate includes all IP addresses and DNS names used to access the API server. If any required SAN is missing, TLS verification will fail for clients.

Output: kubernetes.pem, kubernetes-key.pem

Generating Kubeconfig Files

Certificates prove identity, but components also need to know where to connect and which credentials to use. This is what kubeconfig files provide.

A kubeconfig is a YAML file that contains:

Cluster → API server endpoint + CA certificate
Credentials → client certificate and private key
Context → binds cluster and credentials together

Each component gets its own kubeconfig file.

First, install kubectl (the CLI tool for talking to Kubernetes):

KUBECTL_VER="v1.32.0"
curl -LO "https://dl.k8s.io/release/${KUBECTL_VER}/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

1. Kubelet Kubeconfig (worker-0)

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://10.240.0.10:6443 \
  --kubeconfig=worker-0.kubeconfig

kubectl config set-credentials system:node:worker-0 \
  --client-certificate=worker-0.pem \
  --client-key=worker-0-key.pem \
  --embed-certs=true \
  --kubeconfig=worker-0.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:node:worker-0 \
  --kubeconfig=worker-0.kubeconfig

kubectl config use-context default --kubeconfig=worker-0.kubeconfig

The --server points to the controller node because the kubelet must reach the API server over the network.

2. Kube Proxy Kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://10.240.0.10:6443 \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

3. Kube Controller Manager Kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context default \
  --kubeconfig=kube-controller-manager.kubeconfig

Uses 127.0.0.1 because it runs on the same machine as the API server.

4. Kube Scheduler Kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=kube-scheduler.pem \
  --client-key=kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

5. Admin Kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
  --client-certificate=admin.pem \
  --client-key=admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=admin \
  --kubeconfig=admin.kubeconfig

kubectl config use-context default --kubeconfig=admin.kubeconfig

Distributing Kubeconfig Files and Certificates

Once all kubeconfig files are generated, we transfer everything required by the worker node in a single step.

scp worker-0.kubeconfig kube-proxy.kubeconfig \
    worker-0.pem worker-0-key.pem ca.pem \
    worker-0:~/

Why this matters:

The worker node only receives the minimum required files:

ca.pem — used to verify the API server's identity
worker-0.pem and worker-0-key.pem — used by the kubelet to authenticate itself
kubeconfig files — define how components connect to the API server

With the infrastructure provisioned, binaries installed, certificates generated, and kubeconfig files distributed, we now have everything needed to bring the cluster to life.

Phase 3: Bootstrapping the Control Plane

At this point, all required configurations — including certificates, kubeconfig files, and system prerequisites — are in place. We can now proceed with installing and bootstrapping the Kubernetes components.

1. etcd

First, create the required directories and place the TLS certificates in the correct location:

# Create directories for etcd config and data storage
sudo mkdir -p /etc/etcd /var/lib/etcd
sudo chmod 700 /var/lib/etcd

# Copy the TLS certificates etcd needs to its config directory
sudo cp ca.pem /etc/etcd/
sudo cp kubernetes.pem /etc/etcd/etcd-server.pem
sudo cp kubernetes-key.pem /etc/etcd/etcd-server-key.pem

# Set variables for etcd configuration
INTERNAL_IP="10.240.0.10"   # Controller's private IP
ETCD_NAME=$(hostname -s)

etcd Systemd Service

Create the systemd unit file at /etc/systemd/system/etcd.service:

[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/etcd/etcd-server.pem \
  --key-file=/etc/etcd/etcd-server-key.pem \
  --peer-cert-file=/etc/etcd/etcd-server.pem \
  --peer-key-file=/etc/etcd/etcd-server-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://${INTERNAL_IP}:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster controller-0=https://10.240.0.10:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Note: Replace \({ETCD_NAME} and \){INTERNAL_IP} with their respective values before saving the file.

Key Flags Explained

Flag	Purpose
`--name`	Name of this etcd member
`--cert-file` / `--key-file`	TLS certificate and key for client connections
`--peer-cert-file` / `--peer-key-file`	TLS certificate and key for etcd-to-etcd communication
`--trusted-ca-file`	CA used to verify client certificates
`--peer-trusted-ca-file`	CA used to verify peer certificates
`--client-cert-auth`	Require clients to present a valid certificate
`--peer-client-cert-auth`	Require peers to present a valid certificate
`--initial-advertise-peer-urls`	Tell other etcd nodes how to reach this member
`--listen-peer-urls`	Listen for peer traffic on port 2380
`--listen-client-urls`	Listen for client traffic on port 2379
`--advertise-client-urls`	Tell clients where to connect
`--initial-cluster-token`	Unique token to prevent joining the wrong cluster
`--initial-cluster`	List of all etcd members in the cluster
`--initial-cluster-state`	`new` for a fresh cluster (vs `existing` for joining)
`--data-dir`	Where etcd stores its data on disk

After configuring the systemd unit, start and enable the etcd service:

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

Verify etcd is running and healthy:

sudo ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/etcd-server.pem \
  --key=/etc/etcd/etcd-server-key.pem

You should see controller-0 listed as a member. If this works, etcd is healthy.

2. kube-apiserver

Once etcd is up, we configure the kube-apiserver.

Encrypting Secrets at Rest

Kubernetes can encrypt secrets at rest in etcd. We generate an encryption key and create a config that tells the API server to use it.

# Generate a random 32-byte encryption key and encode it as base64
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

# Create the encryption config YAML file
cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

Copying Certificates to the Config Directory

# Create the directory where K8s stores its config/certs on the controller
sudo mkdir -p /var/lib/kubernetes/

# Copy all certificates + encryption config the API server needs
sudo cp ca.pem ca-key.pem kubernetes.pem kubernetes-key.pem \
  service-account.pem service-account-key.pem \
  encryption-config.yaml /var/lib/kubernetes/

kube-apiserver Systemd Service

Create this file at /etc/systemd/system/kube-apiserver.service:

[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=${INTERNAL_IP} \
  --allow-privileged=true \
  --apiserver-count=1 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --etcd-servers=https://127.0.0.1:2379 \
  --event-ttl=1h \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --runtime-config='api/all=true' \
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \
  --service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \
  --service-account-issuer=https://${INTERNAL_IP}:6443 \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Key Flags Explained

Flag	Why It's Important
`--authorization-mode=Node,RBAC`	Without this, there's no access control. `Node` restricts kubelets to only their own resources. `RBAC` controls everything else.
`--client-ca-file=ca.pem`	This is what enables mTLS. The API server uses this CA to verify every client certificate. Remove this and anyone can connect.
`--etcd-servers=https://127.0.0.1:2379`	Points to etcd. If this is wrong, the API server has no database and nothing works.
`--service-cluster-ip-range=10.32.0.0/24`	Defines the virtual IP pool for Services. Must match what kube-proxy and controller-manager are configured with.
`--tls-cert-file` / `--tls-private-key-file`	The API server's own identity. Every client verifies this certificate before trusting the connection.
`--encryption-provider-config`	Without this, Secrets are stored as plain base64 in etcd. Anyone with etcd access can read them.
`--service-account-signing-key-file`	Signs the JWT tokens that pods use to authenticate to the API server. Without this, ServiceAccounts don't work.

3. kube-controller-manager

# Copy the controller-manager's kubeconfig
sudo cp kube-controller-manager.kubeconfig /var/lib/kubernetes/

kube-controller-manager Systemd Service

Create this file at /etc/systemd/system/kube-controller-manager.service:

[Unit]
Description=Kubernetes Controller Manager

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=0.0.0.0 \
  --cluster-cidr=10.200.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --leader-elect=true \
  --root-ca-file=/var/lib/kubernetes/ca.pem \
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Key Flags Explained

Flag	Purpose
`--cluster-cidr=10.200.0.0/16`	The pod IP range. Must match what the CNI plugin is configured with.
`--cluster-signing-cert-file` / `--cluster-signing-key-file`	The CA used to sign certificate signing requests (e.g., kubelet cert rotation).
`--service-cluster-ip-range`	Must match the API server's value.

4. kube-scheduler

# Copy the scheduler's kubeconfig to the K8s config directory
sudo cp kube-scheduler.kubeconfig /var/lib/kubernetes/

Scheduler Configuration File

# Create a scheduler configuration file
sudo mkdir -p /etc/kubernetes/config

cat > /etc/kubernetes/config/kube-scheduler.yaml <<EOF
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF

kube-scheduler Systemd Service

Create this file at /etc/systemd/system/kube-scheduler.service:

[Unit]
Description=Kubernetes Scheduler

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/config/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Starting the Control Plane

# Reload systemd, enable, and start all control plane components
sudo systemctl daemon-reload
sudo systemctl enable kube-apiserver kube-controller-manager kube-scheduler
sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler

# Wait ~10 seconds for everything to initialize, then verify
kubectl cluster-info --kubeconfig admin.kubeconfig

You should see: Kubernetes control plane is running at https://127.0.0.1:6443

RBAC for Kubelet Authorization

The API server needs to call back into kubelets for operations like kubectl logs, kubectl exec, and kubectl port-forward. By default, it doesn't have permission to do this. We create an RBAC rule to grant it.

cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups: [""]
    resources: ["nodes/proxy", "nodes/stats", "nodes/log", "nodes/spec", "nodes/metrics"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

Bootstrapping the Worker Node — kubelet, kube-proxy, Networking & DNS

With the control plane running and healthy, we now bootstrap the worker node. This is where your application containers will actually run. The worker needs three services: containerd (container runtime), kubelet (node agent), and kube-proxy (network proxy).

Configure kubelet

Placing Certificates and Kubeconfig

HOSTNAME=$(hostname -s)   # Get the short hostname (e.g., "worker-0")

# Create directories kubelet needs
sudo mkdir -p /var/lib/kubelet /var/lib/kubernetes /var/run/kubernetes

# Copy this worker's cert, key, and kubeconfig into place
# These were generated in Phase 1 (certs) and Phase 2 (kubeconfigs)
sudo cp \({HOSTNAME}.pem \){HOSTNAME}-key.pem /var/lib/kubelet/
sudo cp ${HOSTNAME}.kubeconfig /var/lib/kubelet/kubeconfig
sudo cp ca.pem /var/lib/kubernetes/

Kubelet Configuration File

Create this file at /var/lib/kubelet/kubelet-config.yaml:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
cgroupDriver: systemd
containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/worker-0.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/worker-0-key.pem"
registerNode: true

⚠️ Replace worker-0 in tlsCertFile and tlsPrivateKeyFile with the actual hostname of your worker node.

Key Config Fields Explained

Field	Purpose
`authentication.anonymous.enabled: false`	Don't allow unauthenticated requests to kubelet
`authentication.webhook.enabled: true`	Use the API server to verify if requests are authorized
`authentication.x509.clientCAFile`	CA cert to verify client certificates
`authorization.mode: Webhook`	Ask the API server "is this user allowed to do this?"
`clusterDNS: ["10.32.0.10"]`	IP address of the CoreDNS service (configured later in this guide)
`cgroupDriver: systemd`	Must match containerd's cgroup driver
`containerRuntimeEndpoint`	How kubelet talks to containerd via CRI
`resolvConf`	DNS config file for pods
`runtimeRequestTimeout`	Timeout for container runtime operations
`registerNode: true`	Automatically register this node with the API server

kubelet Systemd Service

Create this file at /etc/systemd/system/kubelet.service:

[Unit]
Description=Kubernetes Kubelet
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Configure kube-proxy

Placing Kubeconfig

sudo mkdir -p /var/lib/kube-proxy

# Copy the kube-proxy kubeconfig (generated in Phase 2)
sudo cp kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig

kube-proxy Configuration File

Create this file at /var/lib/kube-proxy/kube-proxy-config.yaml:

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"

kube-proxy Systemd Service

Create this file at /etc/systemd/system/kube-proxy.service:

[Unit]
Description=Kubernetes Kube Proxy

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Start Worker Services

# Same systemd pattern: reload → enable → start
# This time we start 3 services at once: containerd, kubelet, and kube-proxy
sudo systemctl daemon-reload
sudo systemctl enable containerd kubelet kube-proxy
sudo systemctl start containerd kubelet kube-proxy

Verify the Worker Node

Run this from the controller node (not the worker):

# kubectl get nodes shows all worker nodes that have registered with the API server
kubectl get nodes --kubeconfig admin.kubeconfig

You should see worker-0 listed. At this point, the node will likely show NotReady — that's expected because we haven't installed a CNI networking plugin yet.

Pod Networking (CNI)

Without a CNI plugin, nodes will show NotReady and pods cannot communicate across nodes. You need to choose one networking solution — we'll use Flannel for its simplicity.

Matching the Pod Subnet

Your controller manager is configured to give pods IPs in the 10.200.0.0/16 range. Flannel defaults to 10.244.0.0/16. We need to match it.

Installing and Configuring Flannel

Download the Flannel manifest:

curl -LO https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Replace Flannel's default network with your 10.200.0.0/16 network:

sed -i 's/10.244.0.0\/16/10.200.0.0\/16/g' kube-flannel.yml

Fixing the Network Interface for Vagrant

Vagrant typically assigns two network interfaces to all VMs. Flannel defaults to using the first interface, which is usually the NAT interface — not the one we want. To fix this, add the --iface flag to point Flannel at the correct host-only interface:

sed -i '/- --kube-subnet-mgr/a \        - --iface=enp0s8' kube-flannel.yml

Apply Flannel

kubectl apply -f kube-flannel.yml --kubeconfig admin.kubeconfig

After a few seconds, verify the node is now Ready:

kubectl get nodes --kubeconfig admin.kubeconfig

DNS — CoreDNS Deployment

CoreDNS provides DNS-based service discovery inside the cluster. Without it, pods cannot resolve service names like my-service.default.svc.cluster.local.

Install CoreDNS

# Download the official CoreDNS deployment tool
git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes

Generate and Apply the Configuration

This script takes the deployment template, injects the cluster DNS IP (10.32.0.10) and domain (cluster.local), and pipes it directly into Kubernetes:

./deploy.sh -i 10.32.0.10 | kubectl apply -f -

Verify CoreDNS

kubectl get pods -n kube-system --kubeconfig admin.kubeconfig

You should see CoreDNS pods running. Once they're in Running state, DNS resolution will work inside the cluster.

What Actually Happens When You Run `kubectl run nginx`?

We've set up every component by hand. Now let's trace exactly what happens internally when you deploy a pod — step by step.

Step 1: kubectl → API Server

When you run:

kubectl run nginx --image=nginx --kubeconfig admin.kubeconfig

kubectl reads the kubeconfig file, extracts the API server endpoint (https://10.240.0.10:6443), and sends an HTTP POST request to create a Pod resource.

The request includes the admin client certificate (admin.pem) for authentication. This is the certificate you generated with CN=admin and O=system:masters.

Step 2: API Server — Authentication

The kube-apiserver receives the request and performs TLS client authentication:

It checks the client certificate against the CA (--client-ca-file=ca.pem) — the same CA you created in Phase 2.
It extracts the identity: CN → admin (username), O → system:masters (group).

If the certificate is invalid or not signed by the trusted CA, the request is rejected here.

Step 3: API Server — Authorization (RBAC)

Next, the API server checks: "Is the user admin in group system:masters allowed to create pods?"

Because you configured --authorization-mode=Node,RBAC, it evaluates RBAC rules. The system:masters group is bound to the cluster-admin ClusterRole by default, which grants full access. The request is authorized.

Step 4: API Server — Admission Controllers

The request passes through the admission controllers you enabled:

--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,
                           ServiceAccount,DefaultStorageClass,ResourceQuota

NamespaceLifecycle — Verifies the default namespace exists.
ServiceAccount — Automatically attaches the default service account and mounts its token as a volume.
LimitRanger / ResourceQuota — Checks if any limits apply (none in our setup).

If any admission controller rejects the request, the pod is not created.

Step 5: API Server → etcd

The API server serializes the Pod object and writes it to etcd (--etcd-servers=https://127.0.0.1:2379). At this point:

The pod exists in the cluster's desired state.
The pod has no nodeName assigned — it's Pending.
No container is running anywhere yet.

This is the same etcd you configured with TLS certs and started as a systemd service.

Step 6: Scheduler — Assigns a Node

The kube-scheduler is continuously watching the API server for pods with no nodeName. It detects the new nginx pod and begins its scheduling cycle:

Filtering — Eliminates nodes that can't run the pod (insufficient CPU/memory, taints, etc.).
Scoring — Ranks remaining nodes based on resource availability, affinity, and spread.
Binding — Picks the best node (worker-0) and writes the nodeName field back to the API server.

The scheduler uses its kubeconfig (kube-scheduler.kubeconfig) to authenticate to the API server — the same kubeconfig you generated with the system:kube-scheduler certificate.

After this step, the pod is still Pending — but now it has a node assigned.

Step 7: Kubelet — Detects the Pod

The kubelet on worker-0 is continuously watching the API server for pods assigned to its node. It detects the nginx pod and begins execution:

It reads the pod spec from the API server.
It calls the container runtime (containerd) via CRI to pull the nginx image.

The kubelet authenticates to the API server using the worker-0.kubeconfig with the system:node:worker-0 certificate — the same node-specific certificate you generated with the -hostname=worker-0,10.240.0.20 flag.

Step 8: Containerd — Pulls the Image and Creates the Container

Containerd receives the request from the kubelet through the Container Runtime Interface (CRI) via the Unix socket you configured:

containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"

It performs the following:

Pulls the image — Downloads nginx:latest from Docker Hub.
Creates the container — Uses runc (the low-level OCI runtime you installed) to create a Linux container with the appropriate namespaces, cgroups, and filesystem layers.
Starts the container — The nginx process begins running inside the container.

Command Palette

Architecture Overview — What Are We Actually Building?

Control Plane

Worker Nodes

Phase 1: Infrastructure Provisioning & Prerequisites

OS-Level Prerequisites

On All Nodes (controller-0 and worker-0)

Downloading Kubernetes Binaries

On the Controller Node (controller-0)

On the Worker Node (worker-0)

Kernel Modules & Networking (Worker Node Only)

Phase 2: Public Key Infrastructure (PKI) — Certificates and Authentication

Why Does Kubernetes Need Certificates?

Setting Up the Certificate Authority

Generating Certificates for Each Component

1. Admin Client Certificate

2. Kube Controller Manager Certificate

3. Kube Scheduler Certificate

4. Kube Proxy Certificate

5. Kubelet Certificate (Worker Node)

6. Service Account Key Pair

7. Kubernetes API Server Certificate

Generating Kubeconfig Files

1. Kubelet Kubeconfig (worker-0)

2. Kube Proxy Kubeconfig

3. Kube Controller Manager Kubeconfig

4. Kube Scheduler Kubeconfig

5. Admin Kubeconfig

Distributing Kubeconfig Files and Certificates

Phase 3: Bootstrapping the Control Plane

1. etcd

etcd Systemd Service

Key Flags Explained

2. kube-apiserver

Encrypting Secrets at Rest

Copying Certificates to the Config Directory

kube-apiserver Systemd Service

Key Flags Explained

3. kube-controller-manager

kube-controller-manager Systemd Service

Key Flags Explained

4. kube-scheduler

Scheduler Configuration File

kube-scheduler Systemd Service

Starting the Control Plane

RBAC for Kubelet Authorization

Bootstrapping the Worker Node — kubelet, kube-proxy, Networking & DNS

Configure kubelet

Placing Certificates and Kubeconfig

Kubelet Configuration File

Key Config Fields Explained

kubelet Systemd Service

Configure kube-proxy

Placing Kubeconfig

kube-proxy Configuration File

kube-proxy Systemd Service

Start Worker Services

Verify the Worker Node

Pod Networking (CNI)

Matching the Pod Subnet

Installing and Configuring Flannel

Fixing the Network Interface for Vagrant

Apply Flannel

DNS — CoreDNS Deployment

Install CoreDNS

Generate and Apply the Configuration

Verify CoreDNS

What Actually Happens When You Run kubectl run nginx?

Step 1: kubectl → API Server

Step 2: API Server — Authentication

Step 3: API Server — Authorization (RBAC)

Step 4: API Server — Admission Controllers

Step 5: API Server → etcd

Step 6: Scheduler — Assigns a Node

Step 7: Kubelet — Detects the Pod

Step 8: Containerd — Pulls the Image and Creates the Container

Comments

What Actually Happens When You Run `kubectl run nginx`?