Setting up a K3S cluster

I wanted to set up a Kubernetes cluster on my networking lab that consists of three RPis and a PC, and run some actual workload. I used K3S, a lightweight Kubernetes distribution, because it seemed to be the one to go to for running Kubernetes on RPis, ex. Jeff Geerling's video on installing K3S. For image registry, I tried Harbor open source registry v2.4.1. The cluster ingress is fronted by a reverse proxy.

Diagram

Network diagram of the Kubernetes cluster

Setup K3S cluster

The K3S single-server setup is the simplest way to setup a cluster. The amount of simplication it does is phenomenal. It packages up all the functionalities of etcd3, control plane, networking, DNS, and ingress (Traefik) in one binary and I did not have to interact with any of the components to have a working cluster.

At first, I tried to run a K3S server node on RPi 3 but it did not work well, which I assume is because of the memory constraint. Running K3s agent nodes on RPi 3 instead was not an issue.

On K3S server node

sudo k3s server

On K3S agent nodes

sudo k3s agent --server https://${SERVER_HOST}:6443 --token "${NODE_TOKEN}"

Install Harbor

Screenshot of Harbor UI

Harbor is an open source registry that has many additional features like access control, vuln scanning, and others that are overkill for my use.

Since I am running the Harbor instance in the same node as the K3S server, I configure it to listen to port 8443 instead of 443.

Set up certs for image registry

I used smallstep Certificate Manager to provision an internal cert for image registry.

step ca certificate server --san gauss.lab.jaeyoung.se:8443 server.crt server.key --not-after 31d

Install the root CA in each node and configure K3S nodes to use the root CA to pull images.

sudo cp root_ca.crt /usr/local/share/ca-certificates/root_ca.crt
sudo update-ca-certificates

mkdir -p /etc/rancher/k3s
cat <<EOF > /etc/rancher/k3s/registries.yaml
configs:
  "gauss.lab.jaeyoung.se:8443":
    tls:
      ca_file: /usr/local/share/ca-certificates/root_ca.crt
EOF

Ref. https://rancher.com/docs/k3s/latest/en/installation/private-registry/

Run the Harbor installer

Make changes to harbor.yml to configure the HTTP port and the storage directory, then Harbor can be installed. It installs as several Docker containers.

One thing I find really neat is that, since Harbor v2 supports manifest lists, if you build images for both linux/amd64 and linux/arm64, the images will work on both platforms transparently. I use ko to build and push multi-platform images in one command:

KO_DOCKER_REPO=gauss.lab.jaeyoung.se:8443/library ko publish --platform=linux/amd64,linux/arm64 -P .

The cluster is up and running

Yay.