Prerequisites⌗

CPU: 2 Core
RAM: >= 2Gb
OS: Ubuntu Server 22.04 LTS
Kubeadm v1.25.2
Kubelet v1.25.2
Kubectl v1.25.2
Containerd.io
Proxy (Optional, not needed if your network environment can access Google services normally)

It is recommended not to clone virtual machines to deploy other nodes, otherwise some unpredictable errors may occur due to conflicts with Host, MAC, and UUID!

Environment Information⌗

My virtual machines are all running in the 10.0.8.0/24 network segment, and I’m running Surge Proxy on my macOS. The specific IP information is as follows:

S01: 10.0.8.81 (cluster-master)
S02: 10.0.8.82 (cluster-node)
S03: 10.0.8.83 (cluster-node)
macOS: 10.0.8.18

All Google sources used below will use macOS as a proxy.

Installing ZSH⌗

This is optional. The main reason for using ZSH is to use the zsh-autosuggestions plugin, which can greatly improve efficiency when managing Kubernetes.

sudo apt install zsh
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

After completing the above commands, edit the ~/.zshrc file to add the zsh-autosuggestions plugin:

plugins=( 
    # other plugins...
    zsh-autosuggestions
)

Apply ZSH settings:

source ~/.zshrc

Disable Swap⌗

sudo swapoff -a
sudo rm /swap.img

Edit the /etc/fstab file, delete the line containing swap.img, if it doesn’t exist, ignore it:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/df8f75a8-980b-4055-bafa-5bdef04872b9 / ext4 defaults 0 1
/swap.img	none	swap	sw	0	0

Configure Network Forwarding⌗

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Set required sysctl parameters, these persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl parameters without reboot
sudo sysctl --system

Install Containerd⌗

sudo apt-get remove docker docker-engine docker.io containerd runc

# Update APT package index and install packages
sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up APT repository address
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install containerd.io

# Install docker engine and compose plugin (optional)
sudo apt-get install docker-ce docker-ce-cli docker-compose-plugin

Configure Containerd⌗

sudo rm -rf /etc/containerd/conf.toml
containerd config default | sudo tee /etc/containerd/config.toml

Edit the /etc/containerd/config.toml configuration file, set runc to use systemd cgroup driver:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Change the value of SystemdCgroup from the default false to true, then restart Containerd.

sudo systemctl daemon-reload
sudo systemctl restart containerd

Containerd Proxy⌗

This part is optional. If your network environment can access Google services normally, it is not needed. Also note that if you configure a proxy, you need to cancel the proxy configuration after pulling the images, otherwise it will cause inability to access the Kubernetes Service API normally!

sudo mkdir -p /etc/systemd/system/containerd.service.d

cat <<EOF | sudo tee /etc/systemd/system/containerd.service.d/proxy.conf
[Service]
Environment="HTTP_PROXY=http://PROXY_IP:8234"
Environment="HTTPS_PROXY=http://PROXY_IP:8234"
Environment="NO_PROXY="10.96.0.1,localhost,127.0.0.1,::1"
EOF

Setting up a proxy for Containerd is mainly used when pulling Google images, but when starting Calico-related Pods, they will not start due to the proxy settings. When getting Pod information, you will see the following content:

plugin type="calico" failed (add): error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": Service Unavailable

Because a proxy was configured for Containerd, the containers started cannot access the Kubernetes Service IP normally. This problem troubled me for a long time. I looked up various materials and online issues on Google, but still couldn’t solve it. Finally, I disabled the Containerd proxy after starting the Kubernetes cluster, then restarted Containerd and it worked normally.

sudo mv /etc/systemd/system/containerd.service.d/proxy.conf /etc/systemd/system/containerd.service.d/proxy.conf.back
sudo systemctl daemon-reload
sudo systemctl restart containerd

Configure crictl⌗

cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 5
debug: false
EOF

After configuration, you can use crictl to manage Pods and containers!

Set sudo to Allow Environment Variables⌗

Edit /etc/sudoers and uncomment the line Defaults:%sudo env_keep += "http_proxy https_proxy ftp_proxy all_proxy no_proxy".

Install Kube CLIs⌗

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

# Download Google Cloud public signing key
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

# Add Kubernetes APT source
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install kubelet, kubeadm and kubectl, and pin their version:
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Set Domain Name Resolution⌗

Add the names cluster-master and cluster-endpoint to /etc/hosts, with the corresponding address being the current server’s IP.

Summary⌗

At this point, the preparation environment for Kubernetes is ready, and you can now initialize the cluster through kubeadm. Although I spent two days troubleshooting proxy issues, I’m glad it was finally resolved.

Ah, network problems really make many people take unnecessary detours!

I hope this is helpful, Happy hacking…

Preparing Kubernetes Cluster Environment