Intro: A Tale of 2 Servers, UDP Ports, and K3s

The Unintended Cluster Origin #

It all started with an incident that was somewhat… ironically amusing.

I had a web project that needed deploying urgently. Instead of renting a VPS online as usual, I texted a senior friend in the industry to ask whether he had a spare server going. He threw me the details on the spot, enthusiastic, no hesitation.

But life is not a dream. During infrastructure setup, I found this server had UDP blocked by the ISP in both directions. Unfortunately my project’s architecture required that protocol.

Quick note for anyone not deep in networking: there are two common ways to send data across a network. TCP does a handshake, retransmits lost packets, and guarantees everything arrives complete and in order — at the cost of speed. UDP just fires, no handshake, and a lost packet stays lost — in exchange for speed, which is why most modern VPN solutions run over UDP. And the reason ISPs block UDP by default is equally simple: because there is no handshake, an attacker can spoof the source address and use your server to amplify a DDoS aimed at someone else.

Unable to fix it from the OS side, my friend opened a ticket with the provider’s technical support to request the port be opened, and we waited.

The project could not wait. I bit the bullet, spent my own money, and bought a brand new server from another provider to stay on schedule.

The next morning support reported the ticket resolved. But they could only open UDP for domestic IP ranges; international traffic stayed blocked entirely.

What shocked me more was learning the truth at that point: the server with the blocked port was not an old spare he had lying around. It was a brand new machine he had quietly bought out of his own pocket to rent to me. And he wanted no extra fee at all, he simply meant to help me hit the deadline.

Brotherly enthusiasm plus the bad luck of hitting a network policy straight out of the box made for an ironic situation. Naturally I could not refuse that kindness, so I accepted.

And just like that, from a guy sweating over where to host one web app, I suddenly had 2 servers.

Letting one carry all the work while the beefier box sits idle is an unacceptable waste. A plan popped into my head immediately: “Why not connect these two machines into a cluster, so they share the load and back each other up?”

That was the origin of this series.

I turned to Kubernetes — the gold standard for distributed systems. And on day one of deployment it threw cold water in my face: far too heavy. The reason is the part called the control plane: the cluster’s brain, the set of components that remember the desired state and continuously reconcile the system toward it. It runs permanently, and on a mid-sized VPS it consumes nearly all the RAM and CPU, leaving very little for the application that was supposed to run there.

The solution came named K3s — a Kubernetes distribution with the fat trimmed out: components meant for large-scale cloud environments removed, the rest packed into a single binary, while keeping the core intact and staying compatible with normal Kubernetes tooling.

This is the journey from the world of Docker into the world of Kubernetes.

[!NOTE]

For anyone curious about the actual architecture I built to solve that awkward network problem:

• With UDP blocked internationally, the internal VPN connection kept dropping and fell back to running over TCP — much slower for this kind of traffic. So I chose Tailscale as the core network layer: it builds a private network connecting machines directly wherever they sit, and handles punching through NAT and firewalls itself.
• To get around the block and cut latency, I self-hosted a DERP server right on the K3s cluster. DERP is Tailscale’s relay: when two machines cannot connect directly, traffic detours through it — and self-hosting means that detour goes through my own machine instead of a server halfway around the world.
• Once everything was working, I connected my home PC directly into the cluster to carry AI workloads. Borrowing the GPU at home saves a large amount against renting one.

Yes, a hybrid cloud cluster combining VPS instances and hardware at home. It was also the first time I really got my hands into Kubernetes for a personal project. So this series is not only an architecture log, it is a practical K3s guide from scratch, for anyone who accidentally-on-purpose ends up with 2 servers.