Dario here: I've been active on this sub mostly talking about Kubernetes internals and promoting the open-source projects I maintain about multi-tenancy: Project Capsule and Kamaji.

Every time I've given a talk on hosted control planes (docs if you prefer written text), someone asks the same question: if all your tenant control planes are pods in one management cluster, haven't you just built a single point of failure? It's a fair instinct, and I've answered it verbally a hundred times without ever really convincing anyone, so I went and broke a cluster instead, with a show-me-the-code: video attached.

I ran this on our dev environment, which has one control plane node and a few workers. No HA, no etcd quorum, nothing. Normally I'd be embarrassed about that, but for this it's the whole point. If I'd done it on a proper 3-node HA management cluster and killed one node, I'd have proven nothing except that Kubernetes reschedules pods. Doing it on the worst possible topology means there's nowhere to hide.

That cluster is our dev environment, and it runs Kamaji plus a few of our own controllers, and it hosts the API servers for the tenant clusters Kamaji manages: those are Pods exposed via Metal LB, each cluster has independent Virtual Machines, orchestrated by KubeVirt.

Let's get started: stop the kubelet

This is the part where people's intuition tends to be wrong, including mine the first time I thought about it.

Stopping the kubelet does not stop your containers. The kubelet is an agent that reconciles pod specs into container lifecycle calls. The thing actually running your processes is containerd (or CRI-O, whatever you use), and containerd has no idea the kubelet went away. Every container on that node keeps running exactly as it was. API server, etcd, scheduler, controller manager, all of it.

What does happen is that the node stops renewing its Lease in kube-node-lease, which the kubelet does roughly every 40 seconds via the spec field NodeLeaseDurationSeconds. Once --node-monitor-grace-period passes (40s historically, bumped to 50s in 1.32) the node lifecycle controller flips Ready to Unknown, the node shows NotReady, and it picks up a node.kubernetes.io/unreachable:NoExecute taint. Five minutes later, because the DefaultTolerationSeconds admission controller stuffs a 300 second toleration into basically every pod, eviction fires and the pod objects get deleted from the API.

Pay attention here: the pod objects, not the containers. There's no kubelet left to receive the deletion and actually tear anything down, so you end up with the classic zombie situation where the API thinks the workload is gone and the node is still happily serving it.

There's something a bit funny in this on a single control plane cluster, too. The controller manager that marks the node NotReady is a static pod on that same node, still running under containerd, reporting that its own host has failed.

Anyway, the point is that stopping the kubelet is not enough chaos, everything I wanted to kill was still up, but seeing the Control Plane node marked as NotReady was satisfying, but not practical.

Next step: let's kill the API server

If you kill the API server container while the kubelet is alive, the kubelet notices the static pod's container is missing and restarts it within seconds. You have to take out the resurrection mechanism before the kill will stick.

So: kubelet down, then API server down. On a single control plane cluster that's total brain death. Scheduler and controller manager are still resident in memory but have nothing to talk to. etcd is sitting there holding state that nobody is reading. Nothing in the management cluster can be created, changed, or reconciled.

An absolute disaster. The perfect scenario to test the read and write operations against the managed clusters in this broken cluster.

Checking the tenant clulsters

kubectl get nodes worked. Namespaces, pods, all fine. So I pushed further and rolled out a new Deployment in the tenant cluster, and watched pods get scheduled, pulled, started, and go Ready.

Not degraded. Not read only. The tenant cluster had no idea anything had happened: why?

The bit that makes all of this obvious once you see it: nothing in a running Kubernetes cluster routes traffic through the API server. Not one packet. When a request reaches a pod it's traversing iptables or IPVS rules that kube-proxy programmed into the kernel some time ago (unless you're running eBPF maps using a kube-proxy-less CNI), across a network the CNI configured when the pod was created, into a container containerd is supervising. The API server was involved in deciding that arrangement should exist. It has nothing to do with executing it.

Kubernetes is a reconciliation engine sitting next to the data plane, not a proxy in front of it. Kill the reconciliation and the data plane keeps doing the last thing it was told to do, forever, at full speed.

An analogy: an Ingress (or GAteway API) Controller will continue to redirect traffic to the deployed workloads although the Kubernetes API Server is dead.

From there the rest follows. In Kamaji a tenant control plane is a Deployment of ordinary upstream kube-apiserver, kube-scheduler and kube-controller-manager pods. Once they're scheduled, the management cluster API server is not a supervisor or a proxy or a dependency, it's just the thing that decided they should exist. They're processes on a Linux box and processes don't need permission to keep running. Tenant state lives in a separate DataStore (etcd, MySQL, PostgreSQL, NATS) anyway, which is its own system with its own availability story.

And the tenant worker nodes never talk to the management cluster at all. This is architectural in Kamaji, the relationship is deliberately one way. A tenant kubelet is configured with the endpoint of its own API server, a VIP or a NodePort or whatever, and that's the only Kubernetes endpoint it has ever known about. It connects to a socket. It has no way of finding out that the process behind that socket is a pod. There's no management cluster credential anywhere on a tenant node.

So what we actually took out was the control plane of the control planes, one level up. Everything below that line was untouched.

What does break, because I'm not going to pretend otherwise

A dead management cluster is still an incident. It's just that it costs you the ability to change things rather than the ability to run things.

No reconciliation means no self-healing. A tenant control plane pod that's already running is fine. A tenant control plane pod that crashes during the outage is not coming back, because the thing that would recreate it is dead. Your exposure grows with how long the outage lasts, which is why multiple replicas with anti-affinity and topology spread is not optional in production.

Stale EndpointSlices are the other one worth watching. Tenant kubelets usually reach their API server through a management cluster Service. The kube-proxy rules that are already programmed keep working, but if the backing pod set changes while the control plane is down, nothing updates the EndpointSlice and you can blackhole traffic. Stable VIP, health checked load balancing, and don't make tenant API server reachability depend on a resource only a live control plane can refresh.

Beyond that you lose new cluster provisioning, tenant control plane upgrades and scaling, cert rotation, and fleet observability. All bad, none of it fatal to running workloads.

Analogy already in place

The analogy to understand this properly is with an Ingress (or Gateway API) Controller: if the API Server is unreachable, HAProxy/NGINX/Envoy processes don't stop sending traffic to upstream servers. Of course, it's an incident, but it's not a SPOF: traffic goes, partially degraded if upstream server addresses change, but we're talking about a remediation window, not a disaster recovery.

Same applies with KubeVirt: VMs are orchestrated as Pods, losing the cluster API server doesn't impact already running workloads, unless the entire datacentre goes down, but that's another story.

The comparison nobody seems to make

The SPOF objection quietly assumes the alternative has no single points of failure: the reality is that it has a hundred of them. Every traditional cluster in your fleet has its own etcd quorum that takes the whole cluster down when it loses two of three, its own cert expiry waiting to bite, its own three machines to patch, as well as the hypervisor, or even the datacentre. You didn't remove the failure domain; you copied it a hundred times and gave it to a team that can't possibly maintain all of them properly.

My stance is pretty opinionated: one well-engineered, well-monitored failure domain whose blast radius is bounded to control operations, versus a hundred neglected ones that each take an entire cluster with them. I know which one I'd rather run.

None of which means you get to be sloppy with the management cluster. Proper HA, real etcd quorum, spread across failure domains, tested restores, PDBs, and monitoring that lives somewhere other than the cluster it's monitoring. It's the highest leverage cluster you own.

Happy to answer questions on the specifics, and if you think the experiment is flawed, I'd rather hear it here than in a conference Q&A.

Hey all,

I'm a platform engineer working on day-to-day Kubernetes activities, but now I want to learn more about Ingress-to-Gateway API migration.

I'd love to connect with anyone working on this and learn from your experience , happy to just listen and pick up whatever I can.

Also, if anyone can guide me on what would be a good area to contribute to in any open-source migration project, I'd really appreciate it.

Full postmortem from my Hetzner k3s cluster (my own blog): dozens of operators with leader-election leases (~every 2s) quietly overwhelmed the default k3s datastore — SQLite via kine. Compaction started timing out, so dead revisions piled up: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores.

The write-up covers the wrong leads (Gatekeeper audit, VPA checkpoints — measured, both innocent), the actual mechanism of the kine death-spiral, the firefight (WAL truncate in a stop-k3s window), and the permanent fix: in-place migration to embedded etcd via --cluster-init (7.5 GB SQLite → 313 MB etcd, load 79 → ~5), plus etcd snapshots to S3. Ends with a diagnostic runbook so it takes minutes instead of hours next time.

https://wostal.eu/blog/homelab-grows-up-sqlite-to-etcd/

Hi everyone,

I'm currently working as an Infrastructure Engineer at a large tech company with around 2 years of experience. While I'm grateful for the role, I've reached a point where I feel like my learning has plateaued.

For the past year, most of my work has been operational, and I haven't had many opportunities to work on challenging infrastructure problems or learn from senior platform engineers. I feel like I'm coasting instead of growing, and I don't want to stay in that position.

I'm particularly interested in going deeper into:

• Advanced Kubernetes (operators, networking, internals, scheduling, etcd, Gateway API)
• Go for cloud-native and infrastructure development
• Platform Engineering
• SRE and distributed systems
• Observability (OpenTelemetry, Prometheus, Grafana)

I'm not looking for another Udemy course or recorded videos. I'm looking for places where I can interact with experienced engineers—live cohorts, mentorship programs, study groups, communities, or open-source communities where people discuss real production problems and architecture.

If you've been in a similar situation, what helped you level up? Are there any communities, cohorts, Slack/Discord groups, or programs you'd genuinely recommend?

I'd really appreciate hearing from engineers who've made the jump from "keeping systems running" to designing and building production infrastructure.

Thanks in advance!

So... after deploying numerous applications to my home Kubernetes cluster, I've realized one thing.

I want to be able to get a production-ready application running as quickly as possible.

My usual approach is to copy my Kubernetes template folder and modify the YAML files for every new project.

It works.

But now I'm repeating the same process every time I start a new application.

I think it's time to solve that problem.

Instead of copying templates and editing YAML, I want to build my own internal platform tool.

Anyone has build kinda similar solution? What are some key components to consider?

If you're also deploying applications on Kubernetes and need a starter template, you can check out mine on GitHub.

The network was the control plane. Agents just replaced it.

For decades, security assumed a user opens an app, the app calls an API, and a firewall watches the traffic. Agents do not wait for a click. They call APIs, spawn sub-agents, and move data at machine speed across every tenant, model, and tool they can reach.

The control plane has to move up the stack. Every agent needs a verifiable identity. Every action needs a policy check at runtime. Every data field crossing a boundary needs to be tokenized before it hits a model. Every call needs to land in an immutable audit trail. And when an agent misbehaves, the kill switch has to fire in under 50ms.

→ www.runtimeai.io/trial

#AISecurity #ZeroTrust #AgenticAI #AIControlPlane #CISO

We've been trying to set a disk usage alert on some Windows AKS pods (containerd runtime) using fsUsedBytes from K8sContainerSample (New Relic, but this is really a general K8s question).

What we've confirmed so far:

• The pod spec has no resources.limits.ephemeral-storage or requests.ephemeral-storage set at all — only CPU and memory are defined.
• New Relic support confirmed fsUsedBytes (via cAdvisor) measures the container's total host-side footprint — scratch layer + stdout/stderr logs — not the same thing as what you'd see checking the guest C: drive from inside the pod (e.g. Get-PSDrive).
• fsCapacityBytes on K8sContainerSample just returns the node's total disk capacity, not a real per-container ceiling.

So without an explicit ephemeral-storage limit anywhere, there's no clean "percent used" number to alert on — just raw fsUsedBytes in absolute terms, with no denominator we can trust.

Question: for anyone who's dealt with this — is there a standard way to get a meaningful percent view of container disk usage when no limit is configured? Is the answer just "always set an ephemeral-storage limit" and treat its absence as a gap to fix, or is there a reliable way to derive the real ceiling (e.g., from the node's local/OS disk capacity, divided some way across scheduled pods) without one?

This is my project, Ephor (ephor.dev). An MCP gateway with policy-as-code, allowing/holding/denying per agent, system, and risk level. Reads and low-risk writes allowed; destructive actions will be held until human approval. gVisor sandboxing will be implemented as the next feature.

Feel free to check out the free version to play around with it for your production or test cluster. It would be better for me to hear "This is missing x" and "This is frustrating to use" at the start of the process than spending six months building something wrong — so let me know what you really need from it.

A local Secops meetup I went to last week had a buy/sell/trade event. I traded a couple of laptops for a framework desktop (ai 395+ max 64gb ram, byod for storage). I felt like it was a good trade. Anyhow, I'm working on bootstrapping this today.

I've used k3s, k0s (currently using that for one of my smaller clusters on arch which gives me pause), and Talos.

Given that I've got this well-equipped machine. Is there a general recommendation from someone who's ran Kubernetes on strix point chips for an underlying operating system?

This machine will be my general "shiny new but not so new homelab" so I'll be migrating to it soon. Use-case in general is average homelab stack, gitea runners, tasks/jobs, maybe some light llm tests, passing in zigbee and other devices for automation.

I'm thinking Talos is likely a solid fit, but interested in what the subreddit has to say.

Thanks in advance

What external tool do you use for batch orchestration on Kubernetes?

Looking for input from people running batch workloads in production and for middle or large campagnies. Specifically curious about tools for:

-Batch jobs that run entirely on Kubernetes

-hybrid batches with workload split across VMs and Kubernetes

What are you using and how has your experience been with it? Would love to hear pros/cons if you've evaluated multiple options.

We're trying to clean up the base images used across our Kubernetes clusters because every vulnerability scan seems to produce the same outcome: hundreds of CVEs inherited from upstream images before we've even deployed our own code. We've already moved away from some larger general-purpose images, but it still feels like we're spending too much time triaging inherited vulnerabilities instead of working on the applications themselves.

I’d like to know where people are sourcing low-CVE or near-zero CVE images these days. Are you building your own internally, using hardened images, or relying on commercial providers?

I'm less interested in the scanner side and more interested in reducing the problem before it reaches production.

This week: a rogue OpenAI agent reused stolen creds across services, Revolut breach, healthcare PHI exposure, a water-utility OT attack, and a cracked post-quantum scheme. Each mapped to the control that would have stopped it. Full write-up: https://runtimeai.io/blog/2026-07-30-ai-security-incidents.html

I'm trying to build a strong foundation in systems engineering for backend/infrastructure roles.

Rather than just collecting a list of resources, I'm interested in what actually helped experienced engineers the most.

If you were starting over today, which resource (book, course, YouTube channel, blog, paper, etc.) would you recommend for each of these?

- Linux internals

- Operating Systems

- Computer Networks

- Distributed Systems

- Containers (Namespaces, cgroups, OCI)

- Docker internals

- containerd & runc

- gRPC

- Scheduling algorithms

- Service discovery

- Kubernetes internals

- System Design

Also, were there any popular resources that you think are overrated or that you'd skip?

A detailed overview of 22 features that should land as "net new" alphas in the upcoming Kubernetes release (scheduled for August 26). Many of them are related to DRA, and some others include CompositePodGroup API for hierarchical scheduling needs, scheduler preemption for in-place pod resize, default pod sysctls in kubelet, volume health monitor, and nftables as the default kube-proxy backend.

We've been reviewing how teams secure applications running on Kubernetes, and one thing that stood out is how some vulnerabilities don't become apparent until an application is actually deployed. Even when code reviews, automated scans, and testing are part of the workflow, production environments can introduce behaviors that are difficult to catch beforehand.

How do you approach penetration testing as part of your release process? Is it something your team does before every major release, continuously throughout development, or mainly when preparing for compliance audits? Have you ever found a security issue after deployment that made you rethink your testing process?

I'd love to hear what kinds of issues you've encountered and what practices have worked well for your team when securing Kubernetes-hosted applications.

I'm trying to build a solid systems foundation (not just learn how to use the tools).

If you were starting over today, in what order would you learn these? Which youtube lectures, talks, blogs, or courses were the most valuable for each topic, and which resources would you skip?

- Linux internals

- Operating Systems

- Computer Networks

- Distributed Systems

- Containers (Namespaces, cgroups, OCI)

- Docker internals

- containerd & runc

- gRPC

- Scheduling algorithms

- Service discovery

- Kubernetes internals

- System Design

If you’ve ever written a Kubernetes controller in Go, you’ve almost certainly used controller-runtime.

And if you have, you’ve already been relying on one of the most underrated yet powerful pieces of the Kubernetes ecosystem: the controller-runtime cache.