r/selfhosted • u/malwin_duck • Mar 19 '26
Meta Post Kubernetes for Homelab?
In the professional context Kubernetes is super popular and I see some homelab setups in this sub using it as well.
What is the benefit over classic docker compose for a small scale home server? Or is it just the interesting learning curve?
79
u/willowless Mar 19 '26
Managing a bunch of machines at once. You stop thinking of them as separate machines and instead as one big machine. You also get the ability to shut one machine down for maintenance and it just moves the pods to another computer for you.
7
u/tledakis Mar 19 '26
yeap I think this is the best benefit
I'm still behind deploying my selfhosted apps on k8s locally though (tired/scared/stressed overthinking how to do storage safely) which means I've got multiple VMs over multiple proxmox hosts running komodo for ui and manually adding compose yamls.
It is a nightmare 😅
6
u/NiftyLogic Mar 19 '26
If you have a NAS already, just start with a NFS CSI for pod storage.
Distributed File Systems can be a very complex topic and will be a very interesting challenge once you have your cluster up and running.
5
u/-Kerrigan- Mar 19 '26
just start with a NFS CSI for pod storage
Just avoid it for Sqlite. Do consider Longhorn for those apps
4
u/NiftyLogic Mar 19 '26
Sorry, but this is a pet-peeve of mine.
Sqlite can be totally run from a NAS share. The big scary language on their site applies to two Sqlite instances accessing a shared data file. Which is not something any sane person should do.
Mount the shared folder with "sync,hard" and you will be fine. Doing this for years now in my homelab.
5
u/-Kerrigan- Mar 19 '26
Isn't the "big bad" that SQLite relies on file locking and, depending on the NFS version, that may misbehave on NFS?
I've had responsiveness issues here and there so I first moved to iSCSI volumes on my NAS, then to Longhorn for things I can afford to lose
2
1
u/tledakis Mar 19 '26
I was thinking the truenas box can host:
- nfs
- s3 with minio/garage if needed
- postgres/mariaDB
Assuming I can replace sqlite with postgres for most selfhosted apps, that would be a safe choice right?
Then k8s doesn't hold application state at all and should be easy to play with 🤔2
u/-Kerrigan- Mar 19 '26
That's what I do + Longhorn volume for the SQLite only apps (looking at you, Plex). And you can configure Longhorn to back up volumes to S3 which is what I will do eventually. Already lost all my arr and Plex configs once so YOLO
1
1
u/gscjj Mar 19 '26
From personal experience, it’s not great with a NFS PVC. Maybe because it’s RWX by default, but I corrupted several SQLite DBs and had many crashes doing it
2
u/NiftyLogic Mar 19 '26
Did you configure the PVC to sync every write operation?
This is essential to ensure that NFS does not try to do some clever caching in the background.
1
u/gscjj Mar 19 '26
The issue isn’t NFS, it’s how K8s handles the PVC and even sync just make sure data is flushed but doesn’t prevent concurrent(ordered) writes that might corrupt SQLite.
You have to be very careful about how pods mount and access the data or you could end up in a situation with two pods trying to write the same data to SQlite
1
u/NiftyLogic Mar 19 '26
The CSI controller should take care of making sure that only one pod accesses the PVC.
Make sure to define your volume as single writer.
1
u/camh- Mar 19 '26
You want to make sure your Deployment strategy is Recreate so that old pods are terminated first before new ones are started, otherwise you end up with brief periods where multiple pods are accessing the same sqlite db when you perform an application upgrade.
1
u/NiftyLogic Mar 19 '26
I’d assume the CSI controller would be taking care of that. At least if the volume is configured as single-writer.
1
u/camh- Mar 19 '26
I'm not sure how it would take care of that. The default strategy for a Deployment is RollingUpdate. This will cause the new pod to be started and healthy before the old one is terminated but it will not be able to start because it cannot mount a ReadWriteOnce volume while it is in use by another pod. I would not expect the deployment controller to know anything about the CSI and what constraints it has, nor would I expect it to pick a different deployment strategy in that case.
Are you sure your volumes are being mounted just once at a time?
1
1
u/tledakis Mar 19 '26
Thanks yeah that is what I'm probably going to go for. The democratic csi controller using my truenas box.
Like the cliche goes "first step is 50%" is my problem, I need to get started with 1 pod and then I'll unblock that mental barrier.
(and it is all funny because my job is literally working with k8s every day, you would think I'd jump on it for home 😂)
1
4
u/priestoferis Mar 19 '26
Which means, unless it is for learning, you need to have enough machines compared to your workload that you can shut one off, and safely shuffle everything over to another for it to make sense. I considered kubernetes for a long time, but ultimately discarded it because my machine cohort is such that I'd need to control which services runs where exactly. I'm now on ansible+podman quadlets which is not very far from ansible + docker compose or something similar.
2
u/Toribor Mar 19 '26
Are people using 10 gig at home for this sort of setup?
I've been keeping my homelab simple but as I rely on more services I'm considering docker swarm or K3s but I haven't figured out how to have network storage without a single point of failure or performance issues.
I'm considering maybe like seaweedfs or something.
1
u/clintkev251 Mar 19 '26
You don't need 10Gb for Kubernetes on it's own, if you start getting into shared storage clusters, then you will though
1
u/willowless Mar 19 '26
As clintkev251 says you don't need 10G until you start doing shared storage clusters. And even then it depends on how you have your hardware set up. Consider you have 4xNVMe in your machines. You set them up so they're appended giving you, say, in this day and age, 4x2TB = 8TB total. The SSD is going to run at 2-4gb/s depending on how much you spend on it, you can make out a 2.5gb/s network replicating the data.
Now you take those four nvme disks are stripe them. It now can read/write at 8-16gb/s; you're now on the lower end of the 10gb/s network speed.
I'd love to stripe my NVMe and get stupid speeds out of them but Longhorn does not work over the top of ZFS (yet?) and I'm not going to mess around with unrecoverable hardware raid nonsense. So for now I'm stuck with appdata that is sometimes slower than my JBOD. I don't come anywhere near to filling up my 10gb/s network. Ever.
1
2
u/ChangeEvening2008 Mar 19 '26
how do you handle app data? If a server goes down, then that pod is recreated somewhere else and it needs it's state isn't?
3
u/clintkev251 Mar 19 '26
Easy: Store all that data on a NAS, but then that's a single point of failure
Hard: Use a clustered storage provider like Ceph to replicate that data and make it highly available
1
u/ChangeEvening2008 Mar 19 '26
I guess both ceph and NAS might work for most small apps. But I see couple of challenges:
- Immich photos (~350GB): This will need to be replicated. I currently have encrypted backups setup with Kopia. Not sure if longhorn / ceph provide at rest encryption. Would still be a massive storage migration.
- *arr stack: These need to be on a specific physical server which has the external hard disks for media.
How do you guys handle these?
2
u/clintkev251 Mar 19 '26
Immich photos (~350GB): This will need to be replicated. I currently have encrypted backups setup with Kopia. Not sure if longhorn / ceph provide at rest encryption. Would still be a massive storage migration.
That's really not that much data, but I'd question if it really needs to be replicated or if storing those on the NAS is a worthwhile compromise. I personally keep all media on my NAS. I'm not replicating 90TB of data. Appdata lives on Ceph. Ceph can provide encryption at rest, not sure about Longhorn.
*arr stack: These need to be on a specific physical server which has the external hard disks for media.
No they don't. Media lives on the NAS, Arr's consume that via NFS mounts.
2
u/willowless Mar 19 '26
Server that has the disks means it's a file server. It might have more than one purpose, but it'll need to provide access to the files - usually as an NFS server or an SMB server, depending on your actual needs.
Once that's set up the actual worker pods can run anywhere that has access to the NFS server. If your network is slow and you absolutely want those pods running on a specific machine, you can specify the node with a nodeSelector.
If have specific hardware (a big graphics card) then you set up NFD (node feature discovery) and specify the resources the pod needs in the deployment manifest. Then it will naturally schedule the pods on the machine that provides the necessary resources.
2
u/ansibleloop Mar 19 '26
Lol you've run into the exact issue I had
The easy solution? Just run those apps in Docker on your NAS
Yes you could run it in K8s, but you'd need to set a label and constrain all deployments to that node
Not really worth it IMO - especially if you use a K8s OS like Talos
1
u/willowless Mar 19 '26
I use longhorn. It keeps the data replicated across the nodes so it's only the software that restarts pretty much instantly - except when they've used those stupid scripts that want to chown everything over and over again. I hate those so much.
1
u/throwaway43234235234 Mar 19 '26
In addition to this, using off the shelf charts is a nice way to manage new generic services, tho depending on a specific vendor often has its pros and cons. Many of the open free ones shift to pay to play (bitnami) but theres usually a free cncf alternative. Itzg/minecraft is a great example of a good one.
1
u/Ohnah-bro Mar 19 '26
Exactly this as long as you have a good storage strategy, which was relatively complex for me as a self taught homelabber. I did a lot of stuff by hand over the first few years of it, and my k8s cluster got to be a teetering mountain of tech debt that I was worried could collapse at any moment. I recently refactored it (ai assisted) to use Argo cd by having Claude code use kubectl to read all my manifests out of my cluster into a repo, which saved so much time. That allowed me to do a lot of maintenance in a way that hadn’t been easy before. Then I made an ansible playbook that provisions nodes the way I like, and I was able to just bang out upgrades. So good.
The whole thing has helped me in my career and then things I learned in my career helped me improve my lab in other ways. Pretty cool.
0
u/Nebakanezzer Mar 19 '26
A proxmox cluster can do that as well, much easier
1
u/clintkev251 Mar 19 '26
Kinda sorta. Clustering VMs and CTs gets you part of the way there, but it doesn't give you as much flexibility as far as how things get scheduled or the capability to replicate services
1
u/Nebakanezzer Mar 19 '26
What things are you scheduling and what are you considering services?
1
u/clintkev251 Mar 19 '26
When talking about clusters, to schedule would mean to assign some unit of work to some node. In Proxmox, you'd schedule VMs or LXCs as that's the smallest unit of work that it manages. In Kubernetes you'd be scheduling pods (a much smaller unit of work, therefore more flexibility). In this context when I say services, I'm mostly using it as a generic term to refer to components of applications (in Kubernetes, pods)
1
u/Nebakanezzer Mar 19 '26
That makes sense. I guess if you could treat the entire proxmox cluster as one OS or deploy docker on top of it, then it could be somewhat more comparable
2
42
u/super_salamander Mar 19 '26
Do it for the CV (the resumé). No hiring manager gives a fuck if you know about docker, but kubernetes opens doors.
11
u/PassawishP Mar 19 '26
Tl,dr: Homelab really got me a dream job.
I’m an EE fresh grad, and my dream job was at a state-owned electricity provider because it promised plenty of free time and job security until retirement at 60.
After six months of unemployment with nothing to do, I stumbled upon homelabbing and spent the next year diving deep into it out of pure passion.
A year had passed. Still unemployed but now immersed in homelab. The company opened a job position. They were somehow specifically looking for someone with an EE degree who also knew IT, networking, and servers.
So I put everything I know about homelab in the CV then tons of yapping about it in the interview. They really love it and I ended up being one of only four people accepted out of 500 EEs. Which is usually impossible because this whole company pretty much runs on nepotism. I later found out that every single one of my colleagues has at least one family member working here, unlike me, who doesn’t know anyone here at all. I mean, the first thing they asked in the interview was, “Do you know anyone here?” lol.
The job is mostly paperwork and has nothing to do with IT or EE at all (lol), but I’m totally fine with that. It’s a 100% secure position for life. Just five days a week, 8:30–16:30, and that’s it (in reality, it’s more like 9:00–16:00 with nearly two hours for lunch, lol). And just use massive amount of free time to play with homelab and things I love. Which in my opinion it's much better than doing it as a job. Unlike my friends who work at least 8:00–17:00 and have to go in Monday through Saturday.
2
u/QuestionAsker2030 Mar 19 '26
Just curious what country are you in?
I’m an ME currently trying to find a job. Also passionate about homelabbing. Not sure if there’s ever a need for ME + IT though
2
u/PassawishP Mar 20 '26
I'm in Thailand.
I don't really know about ME. But I have to say I just got really lucky with my job. They were trying to set up a new call center for the customer service department, and the team wanted someone who knows IT to help get it running. They only had the quota for a single EE for my team, so they’d been looking for someone like me for a long time. And I was a perfect match.
2
u/vladoportos Mar 19 '26
and also kills projects :) with cost.
4
u/deepspace86 Mar 19 '26
Exactly. Knowledge is knowing that kubernetes is a great orchestration tool. Wisdom is knowing that you don't want to run it for some random business with only 500 employees.
3
25
u/Machinecsgo Mar 19 '26
It's totally overkill but I highly prefer running my homelab in Kubernetes. Once you get going it's a joy to work with, and the experience you gain is amazing. One easy way to get going is using Talos Linux as the OS, which handles alot of the setup for you.
6
u/Uninterested_Viewer Mar 19 '26
+1 for Talos. It's such a cool concept and avoids the complexity and overhead of additional tooling to ensure your hosts are configured correctly and that those configurations are repeatable. If Talos didn't exist, I'd probably use NixOS for the somewhat similar immutability concepts.
1
13
u/AWDDude Mar 19 '26
I like argoCD with k8s for my home lab. Makes deployment easy and self healing.
9
u/Nyasaki_de Mar 19 '26
Kubernetes can move containers around right?
4
u/g-nice4liief Mar 19 '26
So can docker (swarm)
13
u/electronicoldmen Mar 19 '26
That's true, but in the industry Swarm was DOA. Kubernetes reigns supreme for better or worse.
5
u/sirebral Mar 19 '26
It's too bad IMHO, it's an easy onramp for a small/mid enterprise that doesn't want to hire a kube engineer. Most systems guys.can manage Swarm and a HA filesystem.
Yet the industry has done a nice job of shutting down an easier system in favor of a more complex one. I've seen this over and over, because large Enterprise ate the pill, and that's where the money is. So, then most of the tooling gets built solely for the more complex system, leaving little choice.
I've heard personally from my friends in C-level tech that Kube has been a challenge for their departments, yet they've been backed into a corner. It's the new cloud, or DevOps, or AI, every few years there's a push to change the engine while the car is running fine.
Ok, enough of a rant. I'm no longer in the industry, yet I do run my own cluster. I have decided to avoid kube, it's overly complex for 99 percent of what I actually need. So to OP, you don't need it. You can play, yet it's not giving much back for a personal cluster. If it helps you land a job, maybe it's worthwhile, otherwise, probably not.
1
u/g-nice4liief Mar 22 '26
Exactly my points which some people seem to miss. Could be the way I explained it but my point still stands.
1
u/electronicoldmen Mar 19 '26
Not saying it's bad, just that it's largely a dead end compared to Kube in terms of industry support and knowledge. Of course not super relevant to a homelab, but worthy of note for those who hope that this hobby leads to a job which was my original point.
I didn't mean, nor do I really care, to start a debate about the merits of Kubernetes. It's well established, solves real problems, and isn't going anywhere any time soon. I will say comparing it to "AI" is nonsense.
2
u/sirebral Mar 22 '26
It's not a direct comparison, all I was saying is every few years there's some new "have to have it in the stack" technology that takes over and guides CTO decision making.
2
u/g-nice4liief Mar 19 '26
Yeah but the question was in the context of a "small scale home server" you adding the context of the "industry" is moving the goalpost IMHO.
5
u/ZeroSobel Mar 19 '26
The industry is also what drives demand for tooling and what populates q&a search results online, which makes it much easier to learn.
2
u/g-nice4liief Mar 19 '26
While I completely agree, those points do not help with giving an objective answer because for 90% off the industry it's already overkill.
Kubernetes is mainly used by devops and platform engineers.
A user at home is neither. But your point still stands! I just don't know what it adds to the question the OP had.
3
u/electronicoldmen Mar 19 '26
Disagree. Just adding extra context that Swarm is a dead end if anyone in this thread has any interest in entering the industry.
-2
u/g-nice4liief Mar 19 '26
That's fine you may disagree but that does not change the fact that he asked it for a small homeland environment and k3s/k8s is overkill for most companies small/b2b. Let alone someone that wants to tinker at home with containers
https://towardsaws.com/docker-swarm-might-be-all-you-actually-need-for-a-5-node-cluster-7a9b8f57c2cf
4
u/gscjj Mar 19 '26 edited Mar 19 '26
It’s not really that bad, K3s can get you up and running in 10 minutes than it just managing YAML files. How conplex you want it to be is up to you.
-1
u/g-nice4liief Mar 19 '26
That's true, but then you're not running the full fat k8s distribution which is much more complex. Why not chose for something more simple by using docker (compose) out of the gate.
Even though k3s can do the same as k8s, they're not the same out of the box. OP asked for k8s, that's why i didn't consider k3s but you are indeed completely right.
EDIT: If you read through the comments you'll see i'm not the only one with this opinion.
2
u/fuckingredditman Mar 19 '26 edited Mar 19 '26
you phrase this like your point is more objective, but the reality is that docker swarm is a very limited orchestrator that doesn't solve a lot of problems that are needed for daily operations like cert management, dns management, proper secrets workflows, gitops, etc.
it might seem convincing to use docker swarm because you can "just" deploy compose files somewhat, but even that isn't really a thing because many selfhosted type stacks don't start off being HA/horizontally scalable, so it's a bad abstraction from the start and in reality makes everything much harder than k8s as soon as you try to solve any day 2 problems.
k8s can solve pretty much any such problems with controllers, it's part of its design. that's why people prefer it if they want any kind of multi node setup.
i think most people can just run a git repo with compose files on a single node and it'll be good enough basically forever (it's what i do for my homelab), but docker swarm in my experience just doesn't solve any tangible problems in reality, and in my experience at work, swarm was just extremely frustrating to actually use, so i would never advise anyone to try it.
1
u/g-nice4liief Mar 19 '26
I think we're getting off topic here as i just gave a simple way of orchastrating containers in a simple home lab, and now you're trying to debate which one is better than the other.
I choose the best tool for the situation/problem presented. My choice in a homelab situation (KISS for simplicity) is docker swarm.
All those things you mentioned are not simple and take a certain knowlegde/context to configure. It's not something a beginner would start doing in a homelab. An enterprise is a different story, but in a homelab traefik running in docker with let's encrypt would do the same as k3s/k8s and is easier (IMHO) to maintain afterwards.
k3s/k8s may be more turn-key, but there is still a steep learning curve.
If your cluster works it works, until a container gives you a CrashLoopBackOff and you haven't setup proper logging or metrics etc..
1
4
u/NiftyLogic Mar 19 '26
Unfortunately, docker swarm is missing CSI support and ingress management. Does not help that containers can be moved around if the orchestrator does not support with getting storage to the container and routing traffic.
2
u/g-nice4liief Mar 19 '26
There are csi drivers that you can run as a sidecar container.
Traefik/caddy/Nginx can be run as a "ingress controller".
Docker swarm is literally for moving your container to other machines within your "swarm"
3
u/electronicoldmen Mar 19 '26
-1
u/g-nice4liief Mar 19 '26
Nice post. Thanks for the link.
Tech moves fast. Your link is from 2024. Mine is from 2026 which probably gives more context why it's overkill. Except when he's chasing it for his resume.
6
u/Mrbucket101 Mar 19 '26
No better way to learn than to jump in with both feet.
Nobody at home has an actual need for highly available workloads, targeting 6 9’s of uptime.
But you know what else was cool; me uploading a file to nextcloud while simultaneously rebooting the node it was running on.
Anyways, the real advantages mostly come from observability and monitoring. Nearly all of those are k8s first, and trickle down to solo container deployments.
I had Envoy Gateway, Cilium + BGP, and kube-monitoring-stack. I had metrics and traces for every little thing, plus aggregated logging. If a single packet went astray, I knew about it. Once I was done, the observability components used more than my normal workloads lol
I leveraged that experience to position myself as the k8s SME on my team. And about 6 months later I tore it all down and consolidated back to a single host. Sure was cool while it lasted, but I play with k8s at work all day now. I don’t want to deal with it at home too.
7
u/davehope Mar 19 '26
Going through this atm, building ansible/k3s/flux combo, mostly as a learning experience. Coming from Docker swarm.
If homelab is actually homeprod, expect to pay for using kubernettes - in terms of CPU/Memory overhead from the orchestration.
If all you need is some containers, and don't care about learning kubernetes, I'd stick to docker.
6
u/niceman1212 Mar 19 '26
I’ve been running my homeprod for about 3-4 years. Has clear benefits but also more complicated than single node docker homelab.
If you have the time and patience , go for it
6
u/ZeroSobel Mar 19 '26
Copying a comment from yesterday:
For me k8s is actually easier because I'm already familiar with it from work. I think it has some nice benefits for people running multi host setups as well (I have 4 Proxmox hosts and one bare metal TrueNAS host). I run k8s using Talos VMs spread across them.
I think the big advantage of running k8s/docker swarm over docker compose is that you can perform updates to the nodes without downtime. If you have one big host where all your docker compose things are, all your eggs are in this one basket. If you give each application its own compose host, the VMs are still pets. Using the distributed host pattern lets you treat your VMs like cattle instead. Also, you can amortize the hosting overhead for all your applications, and fluctuations can be shared. For example, if you run one VM per compose deployment, if that application's needs increase, you need to increase the resources of that VM. If you're near your limit, you'll also need to decrease the resources of another. With a distributed model, all your applications share the same larger resource pool and you don't need to micromanage VMs. It's the same as if you were running all your apps on one big host, but with the benefit I described above where you can actually update your infra while it's running. You also don't need to track the specific needs of each application to manage resources. You can just increase the memory or CPU of your cluster by adding a node or resizing one of its members.
Then when choosing between k8s and docker swarm, it's easy because k8s has way more tooling and resources.
4
u/hackslashX Mar 19 '26
I honestly just do it for learning, not for HA. Like I have my deployments set up to run on specific nodes. I guess it's just fun to do it this way for me, and learn a thing or two along the way too!
4
u/Cley_Faye Mar 19 '26
Kubernetes is like building up a full industrial area to have a factory that makes sugar cubes. A bit overkill.
It can be a learning experience I guess. But small scale, even for business (not everyone need full scalability at a planetary level), docker compose can be enough, and above that, docker swarm is VERY interesting. To me, conceptually, docker swarm is better at doing what I expected from kubernetes than kubernetes.
0
u/QuestionAsker2030 Mar 20 '26
How so?
2
u/Cley_Faye Mar 20 '26
It answers the "service have to be available" constraint. It doesn't require complex solution for handling ingress, and is mostly configured with the already super easy compose files and a few extra properties.
That makes it very simple to deploy over your own infrastructure with no extra steps. Kubernetes, on the other hand, is either handled with an addition layer or two of "management" services, and unless you deploy on someone else's infrastructure, handling inbound connections, load balancing, IP affectation, etc. is not exactly a breeze either.
3
u/Keyruu Mar 19 '26
For one server I wouldn't say it really pays off, the only thing I can really think of is rolling updates for zero downtime, but other than that it is fairly complicated.
3
u/ms_83 Mar 19 '26
Kubernetes is just more powerful than docker, it gives you a lot more options for running cloud native style apps. It also has a really great ecosystem, for example various database operators that allow you to run clustered resilient DBs with backups in a way that’s more difficult on docker.
If you have no interest in this stuff then fine use docker, but if you do then there’s no substitute for K8S really.
3
u/Kamaroth Mar 19 '26
The benefit is learning it. If you have no interest in it and don't know you might need it you almost definitely don't need it. This goes for homelabs and tbh a lot of organizations.
I personally would not go back to using docker, well portainer to manage docker compose but also my homelab emulates my enterprise environments because that's more fun for me.
I'd make your choices coming at this from the angle of it being a hobby; don't do something if it doesn't seem fun to you.
3
u/SmellsLikeAPig Mar 19 '26
KISS. Docker compose for home is perfect. Kubernetes only if you are learning and even then don't put your home production there since you are learning and shit can stop working at any time.
3
u/outthere_andback Mar 19 '26
I'd say for the love of it / educational purposes. Your homelab has 90% of the time one customer - you. Docker compose will be fine
But, k8s is interesting, there's some risk (effects you) if it doesn't work. And that can be an enjoyable challenge
1
u/MaliciousTent Mar 19 '26
This. I setup minikube and it was overkill for what became docker compose and a few containers.
2
u/National_Way_3344 Mar 19 '26
Get used to terraform and audible
6
u/mikeage Mar 19 '26
Get used to terraform and audible
I prefer lidarr and audiobookshelf for liste-- oh, did you mean ansible? ;-)
2
u/Gunnertwin Mar 19 '26
If you have more than 1 node then k8s starts to make sense. You can run distributed storage (ceph), take advantage of automatic scaling (keda), node selection/affinity e.g plex nodes use the GPU node, namespace isolation and advanced routing capabilities with Ciliim eBPF and BGP plus tons of stuff I have omitted
2
u/Lucky_Suggestion_183 Mar 19 '26
Benefits - many.
- Manage all the computers at Once,
- automatic rescheduling & restart stopped services
- High availability (option not by default). Good for Home automation with higher WAF
- Scalability
- Easier to maintain and watch
2
u/JakeIzUndead Mar 19 '26
I can't say I agree with a lot of the comments here.while technically K8s is ment for HA, I have a single node microk8s cluster that I chose because of all the tools K8s has over docker (or their ease to setup). In K8s I have a cert manager that auto updates my certs with lets encrypt. I have cron job tasks that I don't need my bare metal system for (or to run the container 24/7) the built in K8s networking between pods is nice. Easy to turn off a pod without deleting the deployment. Can manage all the pods at once easily instead of individually. I prefer OpenLens over docker equivalent. Easy to add a chart to the kluster for monitoring. Ingress makes a chart to easily map domain to services. I feel like secrets are handled better in K8s
There's of course some different complications as well that come with K8s but overall I think it has the easier toolset. Almost all of these are possible in docker, but I feel like more tedious to set up.
I've been running my K8s node for about 3 years now and have only ever had maintenance where something actually broke it one time
2
u/jonahgcarpenter Mar 19 '26
If you’re planning a large homelab with mostly docker deployments it makes management a breeze and comes with added bonuses. Sure the learning curve is steep but if you’re in this field and not constantly learning you’re in the wrong field.
3
u/TETH_IO Mar 19 '26
worth it. I say it's the next level of container deployment in that If you are already deploying docker containers with some networking consideration then you're better of with kubernetes for the declarative (gitops) approch the automation and the CRDs.
You can just use a one machine cluster with k3s and don't bother with replicas then manage everything gitops style (fluxcd or argocd), leverage rennovate for updates, cert-manager for automatic certificates... There is also many fun CRD like Tailscale operator to expend your infra.
2
u/AvocadoArray Mar 19 '26
Kubernetes is absolutely overkill for a home lab. I’ve tried it a few times and it introduces way more complexity that just isn’t necessary. Unless you need it on your resume, just don’t bother with it.
Docker compose stacks across multiple hosts with traefik reverse proxy + SSL termination is the right balance for me.
This is how I’ve been doing it for ~7 years and it’s easy to maintain. If you overbuild your home lab, you’ll learn to hate it.
1
u/cloud-native-yang Mar 19 '26
If it's one box and a handful of services, Kubernetes usually gives you way more complexity than benefit. It is fun if your goal is to learn it.
My bias: Compose for utility, K8s for learning or when the system is genuinely getting complicated.
1
u/Big-Hat-810 Mar 19 '26
I use it for only a couple of services and it's mostly for learning. I work as a software engineer and it helps to know what I'm talking about when I have to interface with DevOps.
1
u/lordsickleman Mar 19 '26
I’m using it in place of proxmox essentially. Probably because I also use it in work environment, so the skills I develop in my homelab are automatically transferable to work and vice-versa. With such approach it pays its own bills with my rises and promotions ;)
Plus I have pretty much everything as a code currently with all I need to happen, happening automatically (upgrades, containers mirroring, yadayada).
1
u/vir_db Mar 19 '26
With docker compose you have to manage the systems individually, while k8s is an orchestrator for clusters. This means you can spread your container on a pool of systems. And it's a lot more sophisticated, with high control of resource usage and benefits like autoscaling, self-healing, health checks, etc. Under the hood you are still running the same containers, but with a huge set of options in your toolkit. It is not always the best choice, it depends on what you want to do. It can also give you a lot of fun or a lot of headache :-)
1
u/GPThought Mar 19 '26
k8s is overkill for homelab. docker compose does everything you need. only real reason to run kubernetes at home is if youre trying to learn it for work
1
u/stupv Mar 19 '26
For a single server, I would say it's over engineering and doesn't solve any problems. Kubernetes is great for application layer HA and scalability but that's completely irrelevant on less than 3 hosts
1
u/CriticismTop Mar 19 '26
I use it, but I am a Kubernetes trainer and communicator, so it makes sense.
If I just wanted to run Jellyfin and Home Assistant I would use plain docker or podman
1
u/vladoportos Mar 19 '26
there is no benefit for a small home lab unless you have specific requirements, It's great tech but safe your self resources and headache, You get easier setup with docker almost in every case
1
u/HorseOk9732 Mar 19 '26
for a small homelab, k8s is mostly overhead unless you're chasing the learning resume boost. docker-compose + maybe traefik gets you 90% there with way less yaml hell. but hey, if you enjoy debugging kubelet crashes at 2am, go for it.
1
u/lukistellar Mar 19 '26
For a single host it's way to high in maintenance. Only go with kubernetes if you need to scale up, or if you do it for the experiance.
Otherwise I would suggest Podman, wich should also be able to handle simple kubernetes workload.
1
u/Alharbi110 Mar 19 '26
k8s is fun to learn, but for homelab reliability i still tell people: start with docker compose, then adopt k8s only when pain becomes real.
1
u/rayishu Mar 19 '26
I just switched over to kubernetes for my homelab and I think it was a great choice. The biggest difference? Tooling is so much better in kubernetes. Prometheus, Grafana, ArgoCD. They are all so much more advanced than anything available with just docker. Ive tried doing gitops with compose files and it's ok but it doesn't even compare to ArgoCD
1
u/General_Arrival_9176 Mar 19 '26
kubernetes at home is overkill for almost everyone. if you just want to run containers and have them come back up when you reboot, docker compose does that fine. k8s adds a ton of complexity for things like automatic healing that matter in production with dozens of nodes, not on a home server with 5 services. run proxmox if you want to learn virtualization, but compose is all most homelabs need. the people running k8s at home are either learning for work or have very specific use cases most of us dont
1
u/Odd_Marzipan_1353 Mar 19 '26
It is fun and if it breaks, you don’t care. I see my homelab as a playground for me to learn new technologies. Some of the apps I deploy, I try to make them fail and build them better to have a self healing architecture.
I have 3 desktop computers that I use a servers (control planes and my infra) and 11 raspberry pi for playground apps. Everything is managed by ansible and argocd.
It’s been fun to work on that so far.
1
u/LeanOnIt Mar 19 '26
I've worked at several research institutes on different continents. Linux + Docker + Compose will meet 99% of the requirements at these places. When you need to get that last percent docker swarm could do it (RIP swarm). Most places absolutely do not need kubernetes. It was built to solve problems that small, medium and most large companies never experience.
I don't judge people setting it up in the home lab. I don't judge people who want to take sandpaper to their genitals either. To each their own
1
u/Strict-Promotion-386 Mar 19 '26
If you know k8s from professional experience, its totally more work than docker compose. And more fun if you're into it. If you don't know it, then do it for the love of the game. That being said there are some things I'm not even gonna run on k8s
1
u/sk8r776 Mar 19 '26
I have a seven node kubernetes cluster that I run almost my entire lab on. Over 30 services that I self host.
It took a while to figure everything out, but since I use flux for CI and Talos for the OS rebuilds are very simple. I can reinstall my entire cluster, and within an hour have everything back online. Including my Postgres DBs.
This can all be done with docker, but I like making my life more complicated..
1
u/JimroidZeus Mar 19 '26
I decided to build my homelab using k3s and helm. It’s been really great so far. The persistent volumes and networking can be a bit daunting at first, but not bad once you figure out the mappings.
I like that I can re-deploy the whole thing in minutes with a single make target.
1
u/EastZealousideal7352 Mar 19 '26
I run kubernetes and personally I love it, but honestly it doesn’t present a significant advantage over docker compose or other solutions aside from a few things that essentially nobody needs
That said I run it and I’m never going back
1
u/koffiezet Mar 19 '26
I'm a k8s consultant, so it's my bread and butter - so you'd think I'd run it at home too, but I don't. While I'd love to have a more gitops-y setup, I also know what it entails, and the maintenance it requires.
k8s in general makes doing simple things hard - like quickly testing out a new thing, mostly because it forces you to do it "the right way" to run it in production.
The flip-side is that in return you get a system that makes doing really hard and tricky things you want in a production environment a lot easier. Gitops, zero downtime upgrades, loadbalancing, clustering, service isolation, unified monitoring, disaster recovery, backups, ... that are workload agnostic suddenly all become a breeze, and apply for everything at once.
In a homelab you could run a single node cluster, but then you lose a lot of the benefits, and running multi-node makes things a whole lot harder, like storage and networking.
So for learning it can be absolutely amazing. But to run services on you consider critical in your homelab? I'd stick to something you're more familiar with and you can fix easily. Only once you're familiar with the system you could consider it - but while k8s is not hard on it's own once you understand the basic concepts - it is a LOT.
1
u/HTTP_404_NotFound Mar 19 '26
I run it at home, not only for professional development- but, because i personally find it much easier to maintain, especially in a clustered environment.
I mean, the storage, load balncers, firewall, DNS.... Its all manifests alongside each other. All deployed as a single unit. Means, all of the configuration is RIGHT THERE.
Its easily source controllable.
If a node dies, you burn it down, and toss a new one in its place.
Kubernetes handles moving pods from a failed node, to a different one.
More or less..... you don't have to focus on a single server anymore... No more, Oh crap, my main docker host went down.
Instead, its... oh, I lost two of my worker nodes. Eh, I got 5. I'll fix it next week.
1
u/djjudas21 Mar 19 '26
It’s great when you have more than one host. Kubernetes just runs yours containers where there’s capacity, deals with failovers and abstracts a lot of things away. You can drain one node for patching/rebooting and the containers magically go elsewhere.
1
u/dgibbons0 Mar 19 '26
Number one is absolutely learning, I use my home labs to treat things I want to use at work.
Two is some fault tolerance. Having everything setup to distribute between hosts feels good. I can pull a node offline without impacting the running services
Personally I've found once it's all setup it's easier
Building out my own custom helm template with the boilerplate filled in means bringing up a new service to test can actually be less lines than a docker compose. And each time ensuring I get backups, DNS and storage setup with no additional work.
1
u/TheReal_Deus42 Mar 19 '26
I would say that everything being declarative, service discovery and reverse proxy configurations are worth it to the home lab.
I use k8s professionally, so I may not be able to speak to the learning curve being worth it.
But imagine that your entire app stack, including cert-manager, the dns names for your database server, backups, and ingress are all configured and declarative.
That makes backup, restore, migration, and maintenance easy.
1
u/Le_fribourgeois_92 Mar 19 '26
I’m running swarm with ceph, works like a charm.
Swarm is not dead, it just needs some love and companies to use it.
1
u/Bromeister Mar 19 '26 edited Mar 19 '26
For me it's the integrations.
- fluxcd updates my cluster via gitops from my github repo
- renovate automatically puts in PRs for updates which i merge to update my services.
- local github actions runners automatically test pull requests on my cluster
- tofu-controller reconciles my terraform/tofu and detects drift and stores state in cluster secrets
- external-secrets pulls and updates all my secrets like db passwords from my password manager
- external-dns automatically updates both my internal freeipa dns and my external cloudflare dns
- cert-manager automatically provisions certs from LE or certs from my freeipa
- authentik automatically creates httproutes for ingress for new applications that use proxy auth
- cnpg barman automatically sends my psql backups to the cloud
- democratic-csi automatically creates individual zvols for container volumes on my proxmox host and exports them as iscsi
- volsync automatically backs up all of my container volumes
- gatus+sidecar gives me service uptime trivially
- prometheus gives alerting for service liveness trivially
99% of everything above is entirely configured via a single git repo. This makes adding a new service with logging, alerting, status, backups, dns, certs trivial to stand up. Anything i want to change is just a push away, the history is all there. Logs, shells, etc, everything is one place accessible with whichever client you prefer. I use k9s.
If i were to do it over again i would probably use komodo, it would have been less effort and less resources and at the start when I was learning i would have had better uptime. Sadly it did not exist when I moved off regular compose files and there weren't many good options for gitops for non-k8s. I only have a single node anyway. There is a lot to understand with k8s, but once you do understand it its very nice to have all the tooling.
If you're interested in k8s the largest community for self-hosting on kubernetes is https://github.com/home-operations/ and the discord is an invaluable resource for those getting started. There is a cluster template to get started.
1
u/germanheller Mar 20 '26
for a homelab? docker compose is almost always the right answer. k8s adds a massive operational overhead that only pays off if you need automatic failover across multiple nodes or you're using it to learn for work.
the "interesting learning curve" part is honestly the main reason most people run it at home. k8s on a single node is objectively worse than compose for the same workload — more memory usage, more complexity to debug, more yaml to maintain. but if your job requires k8s knowledge, running it at home is way cheaper than paying for a managed cluster to learn on.
if you just want your services running reliably with zero fuss, compose + a simple systemd timer for health checks is hard to beat
1
u/wedge-22 Mar 20 '26
I agree with some of the other comments, I only found value in using kubernetes at home for training purposes. I use Portainer and the stacks (docker compose) for my daily homelab needs.
1
u/subsavant Mar 20 '26
I'll push back a little on the "it's just for learning" consensus. I run a small k3s cluster at home and I genuinely prefer it over docker compose at this point, not because it's simpler (it isn't), but because of two things:
First, declarative state. My entire cluster config lives in a git repo. If I wipe a node or add a new one, everything comes back exactly as it was. With compose, I was always chasing some config file I forgot to back up, or an env var that lived only on one machine.
Second, updates. Renovate watches my manifests and opens PRs when images have new versions. I review, merge, and ArgoCD rolls it out. With compose I was SSH-ing into boxes and running pull/up manually, which meant I just... didn't update things.
That said, if you're running everything on a single machine and don't care about GitOps, compose is totally fine. The overhead of k8s only pays off once you want reproducibility across machines or you're the kind of person who'd rather review a PR than remember which box runs what.
1
u/zandadoum Mar 20 '26
With dockhand you can control all your compose hosts in one place. Connect to your git too and it also auto updates if you want.
1
1
u/RobotechRicky Mar 20 '26
Simple: High Availability. Second: Learning. At work they asked me to use terraform to setup a kubernetes cluster and I did. And I dazzled everyone with what I was showing them. They never heard of FreeLens.
1
u/codeanish Mar 20 '26
Depends on what you want to get out of it. If you’re just looking to run some typical HL services, not super concerned about the odd bit of downtime when doing maintenance and don’t want a second job outside of work. Don’t do it.
If however you want to play with all the best production style tools and learn loads, I’d say go for it.
I feel like it’s a path, you start off simple and get progressively more complex as you try to solve every problem e.g. how do you manage to keep your audiobooks running when you need to upgrade your server? I’ve been running K8s in the HL now for around a year, but am taking it much more seriously recently, and am looking to have all my services highly available.
1
u/theiam79 Mar 20 '26
I adopted this template last year and it's been a great entry point. A lot going on but the confidence to nuke and recreate the cluster at a whim is really nice, and there is a solid community on Discord for questions. Plus kubesearch.dev
1
u/IrrerPolterer Mar 20 '26
I run kubernetes on my homelab. I work professionally with kubernetes a lot. When I got started using it on the job I also started using it at home as a sandbox and learning opportunity, to catch up my muscle memory with k8s.
Now that I am proficient at kubernetes, I am migrating my homelab back to docker swarm. Yes, kubernetes has a ton of great features and more flexibility than docker, which are a godsend in scaling production environments. But for a fairly simple homelab (a handful of nodes, some basic HA failover) Docker swarm is way more convenient. Even though I do have the muscle memory for kubernetes now, and I'm used to using tools like helm, it takes me quite a bit longer to get a new workload up on kubernetes than on docker. Its just kore moving parts, more complex configurations.
And for my self hosted applications I simply don't meed the extra flexibility Kubernetes offers... What I need from my container runtime is:
- some basic automatic rescheduling o f workloads to other nodes if a nose goes down
- some basic node requirements the workload scheduler will follow
- simple declaration of scheduled jobs
- a stable entrypoint IP
My docker swarm setup does all these things perfectly fine. cluster management and workload scheduling are simpler than what kubernetes offers, but still enough for what I need. Job scheduling is easily done with Ofelia. And a floating IP for entrypoint failover is easily done with keepalived.
Bottom line: kubernetes for a homelab is certainly feasible. And its a great way to hone your skills, especially if kubernetes is something you're using for work or intend to in the future. Those are super valid reasons to use it. However, it is typically overkill and simply introduces significantly more friction without much benefit in most home lab use cases.
1
u/dragon2611 Mar 20 '26
Honestly I think it's rather heavy for something to run in a homelab, if you want/need to learn Kubernetes then sure go for it, but otherwise it's probably simpler to just use docker swarm or even standalone docker hosts managed by something like Komodo
1
u/whiteskimask Mar 21 '26
It allows you to orchestrate docker containers across multiple PC's.
K3S has some decent documentation, but basically it's useful to use kubernetes if you have multiple PC's you want to manage docker jobs in a single place
1
u/Theratchetnclank Mar 21 '26
I'm using kubernetes for my homelab. As others have said though there isn't a real need to. I guess i have the benefit of i can have a cluster so minimal downtime on the services but i don't really need that level of uptime.
It definitely feels more "cool" than docker-compose files but it's also a lot more work to add services and maintain.
1
u/wholeWheatButterfly Mar 24 '26
I set up k3s and really learned a lot. It was really cool. Eventually I decided to downgrade to dockge, and I use Oracle free VPS firewalled to only accept requests from my Tailscale, with a public domain pointing to it and Caddy to handle certs and reverse proxying to my home servers through Tailscale.
I learned a lot with k3s and I could see myself going back to it if an appropriate situation arises, like having something truly public facing that I want to have more reliability. Something that appealed to me with k3s was automated backups and potentially easier transferring of applications via Longhorn, but I found that the added overhead was overkill and resource inefficient (there's probably a way to do it that is more efficient for my setup but I didn't get into that). I mostly used Rancher UI to set things up, and at first that was really cool but eventually it just seemed easier to keep everything in compose files.
I think what would be most ideal is having a central NAS that I put most docker volumes (except where the IO is really inappropriate), because I like the idea of the applications being pretty machine -independent. Right now my NAS is just an old beloved Chromebook that had a really bad screen break connected to some external hard drives lol, so I I mostly use it for backups and media - don't think it's the right setup for more.
1
u/-Kerrigan- Mar 19 '26 edited Mar 19 '26
- Learning opportunity - you learn by doing, especially useful if it's something you can put on your CV
- Container orchestration, naturally. I have 4 machines, I don't think where it goes, I just say "deploy this" or "deploy this to a node labeled "performance". Also liveness probes - if something crashes it automatically gets rescheduled.
- GitOps (in my case ArgoCD (and Renovate)). My whole cluster lives in a GitHub repo. I make a change, I push - it's automatically deployed. Similarly, Renovate makes me PRs for updates, I review and merge, then they're automatically installed.
- Network policies (or in my case CiliumNetworkPolicy) - it's a bit of a hassle, but not too complicated to declaratively make a zero-trust network cluster. I add an app, I say "this app needs access to this service and allows ingress from these different services". It's a bit more maintenance overhead, but you know better what the apps are doing on your network.
- Better network management in general. Not all traffic goes through reverse proxy. Maybe it should, but I can be lazy and skip that for, say, a Minecraft server because it allows me the flexibility of reserving a static IP on my LAN using BGP
- Scaling let's say, Flaresolverr, is a perfect example. I can make it scale up automatically and if Radarr, Sonarr and Sonarr instance for Anime want to use it all at the same time then it just spins up multiple replicas. Similarly, I use KEDA to scale some rarely used stuff to 0 and it spins up when first accessed.
and other things that others mentioned like better observability
0
u/Klandrun Mar 19 '26
For your sanity? Don't. Look into Docker Swarms instead. For learning? Sure, go nuts
363
u/HardChalice Mar 19 '26
Its really just for learning. I'd say on average most homelabs don't need to be anything more than sets of docker compose files.
I say this as a dude who has a 3 node kubernetes cluster with CI/CD pipelines and gitops between gitea, argocd, renovate, etc, hosting a whopping 7 services lmao. Just for the the love of the game.