Cloud & DevOps

Running Your Own Kubernetes: What Managed Clusters Were Quietly Doing for You

We moved a multi-market platform off GKE Autopilot onto self-managed k3s. The cluster came up in ten minutes. The interesting part was the inventory of things Autopilot had been doing that we suddenly owned.

August 21, 2026 · 9 min read · Cartolinks Engineering

Photo by Wolfgang Weiser on Pexels

Standing up a self-managed Kubernetes cluster in 2026 is genuinely easy. One config file, one command, and about ten minutes later you have a working cluster with a CNI, a cloud controller, a CSI driver, and an autoscaler. That part is a solved problem.

The hard part is everything the managed control plane was doing on your behalf that you never had to name. This is that inventory, written down after moving roughly thirty microservices across several markets from GKE Autopilot onto k3s.

The setup, briefly

k3s (Kubernetes in a single binary), provisioned by a config-driven CLI that also creates the servers, the private network, and the firewall, then installs Cilium as the CNI, the cloud controller manager, the CSI driver for block storage, and the cluster autoscaler. On top of that we install ingress-nginx, cert-manager, and a pull-based deployment controller.

That prod figure covers the cluster plus the database, messaging, and monitoring VMs that sit alongside it. Worth being precise about, because a cluster-only number is misleading — self-hosting means the things that used to be managed services are now line items too.

Everything Autopilot was doing for us

• The control plane and etcd. Someone now owns upgrades, backups, and the decision about how many control-plane nodes to run. Ours is currently one, which is honest but not HA — and the reason is mundane: the account's server quota. Two-node etcd is *worse* than one (it can't form a quorum majority), so the choice is one or three, and three wasn't available yet.
• Node autoscaling. Available as an addon, but currently disabled in prod for the same quota reason. Autopilot's node auto-provisioning was invisible; here it's a feature you enable, budget for, and can be blocked from using.
• Registry pull authentication. Workload Identity handled this transparently. Now it's an explicit image-pull secret created in every namespace by the bootstrap script — seven namespaces in prod, and a missed one means ImagePullBackOff.
• Push-based CI deploys. Our cluster API is IP-allowlisted, so a hosted CI runner physically cannot reach it. Rather than open a hole in the firewall for CI, deploys became pull-based: CI pushes an image, an in-cluster controller notices and rolls the deployment. No CI credentials and no inbound API access — arguably a security upgrade, but one we were pushed into.
• TLS certificates. cert-manager and Let's Encrypt replace managed certs, and bring three new ways to fail that we'll come back to.
• Logs and metrics. The managed logging agent is now a self-hosted Prometheus, Loki, and Grafana stack that someone has to size, store, and keep alive.

The economics quietly invert

The single most useful line in our architecture doc is about cluster count: on a managed platform a cluster is nearly free; on self-managed it's yours to babysit. Each additional cluster means control-plane nodes, its own upgrade cycle, its own etcd, its own monitoring.

So the default flipped. On GKE, spinning up a cluster for isolation was a reasonable reflex. Now the rule is: default to a namespace, and escalate to a separate cluster only if one of three things is true — it needs hard blast-radius isolation, it has a genuinely independent lifecycle, or it has a very different scaling profile (spiky batch versus steady web).

The same inversion hits load balancers. On GKE, giving a service type: LoadBalancer was a habit. Here every one of those is a billed load balancer plus a public IP, so everything that can go behind a single shared ingress does — including, as it turned out, gRPC.

The bill made us measure things we'd never measured

This was the genuine surprise. Because nodes are finite and visibly billed, we did a seven-day study of actual resource consumption across production before setting requests on the new cluster. The long tail of services averaged under 5m CPU and peaked below 50m, with memory peaking under 110Mi.

They had been requesting 200m CPU and 800Mi of memory. That is roughly an 8× over-request on CPU and 6× on memory, carried for as long as anyone could remember, because on a platform that silently provisions nodes to fit your requests, nothing ever forces the question.

One BI workload went from an 8 GB heap to 4 GB with no measured change in behaviour. Self-hosting didn't make our workloads more efficient — it made the inefficiency legible.

Five things that actually bit us

1. Anything you layer onto a Helm release your provisioner owns is silently reversible. Our CNI is managed by the cluster provisioning tool. Re-running that tool reverts the CNI configuration we added on top — with no error and no warning. The fix is an idempotent re-apply script run after every cluster create or upgrade, using --reuse-values so you merge rather than clobber the provisioner's own settings.
2. k3s doesn't expose control-plane component metrics the way a kubeadm cluster does. The default scrape jobs for the controller manager, scheduler, proxy, and etcd will sit permanently red until you disable them. Four red targets on day one is alarming and entirely expected.
3. Liveness probes will kill healthy pods if you enable them globally. Some of our services return 500 on their health endpoint when a downstream dependency is down — which is a reasonable thing for a health check to say, and a catastrophic thing for a liveness probe to act on. Probes go on per service, after each health path is validated. Meanwhile the opposite bug also bit us: a frontend that rewrites its bundle for two to three minutes before it starts listening counted as Ready the whole time, so rollouts briefly served nothing.
4. Default chart values assume a big node. Our log store's default caches request about 9 GB of RAM *each*. They will never schedule on a modest node, and the install just hangs. Read the resource requests in any chart before installing it.
5. Mutable image tags plus a force-pull policy will serve you a stale image. A node had cached a layer for a tag that kept moving. The fix was immutable, monotonically increasing tags per build, so the pod spec changes every deploy and a cached layer can never be silently reused.

Would we do it again?

For this workload, yes — but the honest version of the answer has a boundary in it. Self-hosting the application tier turned out to be very manageable. The genuinely risky part was giving up the *managed database*, which is a separate decision that happens to travel with the migration and deserves to be made on its own terms.

If you're weighing this: the cluster is the easy part. Budget your effort for the perimeter — certificates, egress addresses, image pulls, backups, and the observability stack — because that's where the managed platform was doing the most invisible work.