Building my homelab on $90 of Raspberry Pis

Four Pi 4Bs, a PoE switch, and one weekend. What it taught me about k8s that I could not have learned in EKS.

The hardware

4× Raspberry Pi 4B (4GB RAM) — purchased used
1× TP-Link TL-SG1005P PoE switch
4× PoE HATs for the Pis
1× USB-C power brick for the switch
A small wooden rack shelf I built from a cutting board

Total cost: ~$90 USD at time of purchase.

Why not just use EKS

I’d been using managed Kubernetes for a year. I understood how to USE Kubernetes. I had no idea how it actually worked.

The homelab forced me to understand:

What etcd actually stores and why it’s separate
How the scheduler actually picks nodes (it’s just an API call to the API server)
Why coreDNS needs resources but the control plane doesn’t want to share a node with it
What happens when a node goes down and the control plane can’t reach it

All of these things are abstracted away in EKS. The abstraction is wonderful when you’re shipping. It’s terrible when you’re learning.

k3s, not kubeadm

I chose k3s over kubeadm for one reason: it works on ARM. k3s also uses SQLite instead of etcd by default, which I later swapped out for external etcd to understand the HA story.

The k3s install:

curl -sfL https://get.k3s.io | sh -
# on agent nodes:
curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

That’s the whole install.

What broke and what I learned from it

Everything broke, in sequence:

SD card corruption on node 3 — I learned about PodDisruptionBudgets
OOM on the control plane node — I learned about resource requests vs limits
DNS resolution failing after a reboot — I learned about coreDNS config and the k8s DNS spec
Flux reconciliation loop thrashing — I learned about retry backoff and the importance of suspend

None of these things would have broken in EKS. EKS is designed to not break. That’s its entire value proposition.