For many years now I’ve had at least one machine at home which would work as a server to host some apps. In the last couple of years I’ve been getting more into it which has led me to purchase additional hardware. I’ve decided it would be nice to document my journey so I’ll try to make a post like this - detailing the setup that I currently have with both the hardware and the software once a year.
Since this is (almost) the first post on my homelab setup let me start with some background. I am a software developer who likes to dabble in infrastructure. I like hosting things that I use on my own servers. However, as much as I’d like to run them on racks due to the electricity costs in Europe this is prohibitively expensive. What is more - this requires space, which I do not want to sacrifice at home. Therefore, my homelab is on the small side - both physically and from the resource perspective. I also tend to try not to spend too much on hardware so I end up running old and/or second hand hardware.
Hardware
Compute
The homelab consists of 4 machines. These are:
MSI Cubi 3 Silent NUC
| CPU | Dual-core i5-7200U |
| RAM | 2 × 16GB |
| SSD | 240GB KINGSTON SA400S37240G |
This NUC has been running for close to 6 years now. I originally bought it in 2019 with 16GB of RAM, but since upgraded it to 32GB. It still has the original SSD in it and has run faultlessly.
I’ve an old blog post about this machine specifically, if you’re interested.
Currently, this machine acts as k3s server.
2x Lenovo ThinkCentre M910 Tiny
| CPU | Quad-core i5-6500T |
| RAM | 2 × 8GB |
| SSD | Apacer AS350 256GB |
These I bought post lease for around 130 Euros each in 2023. I’ve no clue how much use they saw before becoming part of my homelab, but they have been running flawlessly for the last two years here. These things are tiny and powerful. They are also quiet. This was very important when I was living in an apartment. If I ever need to expand my homelab again, I’m sure to be looking for more post lease ThinkCentres.
These machines are the agents on my k3s cluster.
Mostly second-hand NAS
| CPU | Quad-core i5-4590T |
| RAM | 2 × 8GB |
| Motherboard | MSI H81I-S01 |
| PSU | Silverstone ST30SF 300W |
| Boot Drive | A random 120GB SSD |
| Storage | 4 × 2TB SSD (Mixed: Samsung EVO 870 & Crucial MX500) |
| SATA Expansion | 4-port PCIe card |
| Case | Jonsbo N2 |
Originally assembled at the end of 2023. The new parts on the NAS are the case and the storage drives. The rest of the components I scavenged off of local bulletin boards & local recycling companies working with electronic waste. The whole system(minus the storage) set me back around 250 Euros, with the case being half that. I would have gone for a bigger case(a used one) but living in an apartment space was at a premium and I could not find anything second-hand that would fit the specific place I had in mind for this machine. I would also have gone for HDDs were it not for the noise.
This machine now runs TrueNAS SCALE and serves as durable storage for the k3s cluster.
Networking
I currently live in a location where no wired internet is available. Therefore, I’ve a 4/5G CPE (H352-381) providing the main connectivity. Since the quality of the internet depends heavily on the load of the nearby cellular towers, I see quite dramatic slowdowns during busy hours, such as Sunday evenings. On working days, I’ll get 200-300Mbps down, 50mbps up. On weekends and busy evenings this can drop down to 50 down, 10 up.
Knowing that wireless internet can be a bit of pain, I’ve opted to for a router with dual WAN support. I went for TP-Link’s ER605. The plan is to get a starlink at some point and plug that in.
From there, I’ve 2x TP-Link Deco XE75 wireless access points - one per floor of the house. These have far exceeded my expectations. They provide excellent coverage even in a concrete house and far beyond it.
There are also a couple of unmanaged switches somewhere along the way here as well.
Auxiliary devices
I also own a few shelly smart plugs, those are mostly used for measuring the power consumption of the homelab and for whenever I’m curious about a random device’s power consumption. Besides that, I have 2x Shelly H&T Gen 3 humidity and temperature sensors for monitoring these around the house.
Power consumption
Over the years, I’ve grown quite conscious about how much electricity the devices around the house use. When choosing the hardware I always try to consider energy consumption, to a reasonable degree that is. The homelab uses approximately 49 watts of power in total with the usual load it’s seeing. The NAS is consuming ~22w of that, the Lenovo ThinkCentres take around ~8W each and the Cubi draws ~10W. At the current electricity prices here this comes to right around 7.5 euros a month.
I’ve written in more detail about the power consumption of these machines in my previous blog post if you’re interested.
The “rack”
The homelab neatly fits in a custom built rack with NAS sitting on the bottom shelf, and compute units above that. There’s a power strip attached to the side of it, as well as a small switch. Each of the machines is plugged into a shelly smart plug, which continuously emits updates of the usage to an MQTT server inside the k3s cluster. The rack sits in the utility room of my house. Here’s what it looks like:
The idea here is to put a monitor on top in the future, as well as a keyboard to allow for easy access if the servers would start acting up. This, however, is still in the works. I’ve purposefully left some space to be able to shove more one litre computers if needed.
Software
The NAS
The NAS machine runs TrueNAS Scale. It has a single 4-wide RAIDZ2. 2TB drives each give me a usable capacity of 3.4TB. While this might not sound like a lot, I don’t really store much media here. I store data for applications that I run in k8s and in my case these are usually tiny. I expose the dataset as an SMB share.
The only additional application I end up running on the NAS is netdata for metric collection. This netdata instance only collects the metrics and forwards the data to a netdata parent running on the k3s cluster. I’ve written a blog post on how to do that previously.
Periodically, a job on the k3s cluster will backup the important bits from NAS into an off-site s3 bucket. I use kopia for backups.
The k3s cluster
The cluster consists of a single k3s server and two agents. With the exception of my NAS, all the other hosts run debian. On top of that, I have k3s installed. From there, I use helm with the helm secrets plugin to deploy a whole bunch of things on the kubernetes cluster. The helm charts are stored in a private git repo on github.
Some of the first helm charts I installed on the cluster are the kubernetes csi driver to access the SMB share on the NAS and MetalLB for any services that I need exposed on the local network. I also deploy a PostgreSQL instance to serve as my main database.
For each of the domains I own and use (including the one you’re currently on) I deploy three things - a cloudflared instance for the cloudflare tunnel which forwards traffic to an nginx instance. This then forwards requests further to either a specific kubernetes service or authelia for authentication.
For monitoring I use Netdata. I use their helm chart in the k3s cluster, with a single parent and children running on each node. It will automatically pick up prometheus metrics from pods with the appropriate annotations present.
The actual applications
I think it’s time to move away from the infrastructure and talk about the actual things I have running there:
- Audiobookshelf - I don’t listen to ebooks that often, but when I do I prefer the self hosted option.
- Cyberchef - comes in handy for the work that I do.
- Excalidraw - extensively used for work, self hosting gives me peace of mind.
- Harbor - all the applications I build immediately get Dockerized, and CI builds & pushes them here.
- Jellyfin - this one does not need an introduction, I think.
- Miniflux - let’s me keep up to date with a few blogs I like to read.
- Mouthful - is how I handle comments on this blog.
- Owntracks - a work in progress, helps with home automation.
- Paperless-ngx - I’d say this is my second most popular app on my homelab. An absolute lifesaver. I’ve moved away from paper documentation completely thanks to it.
- EMQX - my MQTT broker of choice. Shellies report their energy consumption here, owntracks uses it as well. I also have some applications hooking into it and emitting events.
- Pi-hole - helps with ad blocking.
- Tandoor - a library of food recipes I enjoy.
- Github action runner controller - allows the cluster to run github actions CI workloads on my own servers.
- Ripe atlas probe - my meager contribution to the RIPE Atlas project. At some point I might even find a way to spend all the credits I’ve accumulated…
- Silverbullet - probably the app that I use the most out of all. I’ve tried many different notes apps, for some reason this one stuck.
- 13ft - for those few times a year when I actually have to read the Medium article.
- Unbound - self hosted DNS resolver, my pi-hole uses it, and my network uses pi-hole as the dns server.
- Umami - self hosted, privacy respecting analytics. There are many like it, but this one uses few resources and is generally pretty good. See my full post on it if you’d like.
- Vaultwarden - password manager of choice.
- Yopass - for when I need to share a secret.
- Custom prometheus exporters. These export things such as shelly statuses, power consumption and various other metrics.
- A few discord bots I’ve written.
- A few web apps & internal APIs I’ve developed over the years, like the embedding api for post recommendations
- And a few cronjobs to back up all the things that need backing up.
The whole cluster uses around 1.1 cores under normal load and around 19GB of RAM.
The future
While I’m generally happy with my homelab, there are a few points that I got wrong or would like to improve.
First, I’d like to run postgres on a completely separate host, or more ideally on two separate hosts with automatic failover. For this, I’d like to use the two M910 Tiny’s I already have. This would require me to remove them from the k3s cluster. I’d like to replace them 2 small machines, possibly with the Intel’s N series processors. However, I’ve struggled to find second hand options for them. I’d use them as k3s servers with the whole cluster consisting of 3 servers. This would allow a failure of a single node with no downtime.
Second, I wish to switch my NAS to an HDD array for extra storage. This will come in handy for future projects. I would also like to move away from SMB to NFS shares, as I’d expect those to play nicer with linux and mac machines at home. I chose SMB for the theoretical use-case of having a windows machine at home, but I’ve never gotten one.
Third, I intend to move away from cloudflare tunnels. I’ll use a VPS and some self hosted tunnel alternative, I’m yet to decide which one. I’ll need to write an ansible playbook to set the server up to my liking so that I can freely move between VPS providers if/when needed.
Fourth, there’s a few applications that are on my to self-host list:
- Changedetection for bargain hunting.
- Frigate to replace the dedicated NVR I currently have running at home.
- A dashboard for the entirety of my home. Ideally it would have links to all the listed applications and allow for custom pages to help with home automation.
Fifth, integrate all the utility appliances at home, such as the recuperator and the heat pump with the dashboard mentioned above to allow for easy control in a single panel, instead of having to use multiple applications.
Sixth, add a UPS to the homelab. This is a nice to have and might not happen any time soon but it’s something I could grab if the right deal comes along.
Seventh, add an outdoor thermometer or a whole weather station to the setup and a prometheus exporter for it so it can be visualized in Netdata.
Eight, actually mount a monitor on top of the rack. A keyboard would help there as well in case I need to directly hook into one of the machines.
And probably a few other things as well…
That’s what my homelab setup looks like. If you made it this far - thank you for reading. If you have any questions, do drop a comment below!