Home lab setup

Published: at 12:33 (10 min read)

It’s been a while since I wrote about my illumos based Kubernetes cluster running in my home lab (home infra), so I felt that its time for a refresher, but inspired of the eCHO episode 114 about Cilium in home labs, I wanted to take the time to describe my home lab in a separate post.

There are a couple of components that are necessary for having the Kubernetes cluster up and running. First and foremost, a facility - but as hardware these days come in all sorts of sizes and shapes (well, it wouldn’t surprise me at all if someone by now managed to put up a tiny edge cluster running on smartwatches) this is probably the least problem. Then, some hardware capable of running the intended workload, I happen to like the Intel Xeon D line of embedded CPUs (in shape of Supermicro X10SDV motherboards) which are relatively power friendly but still rather capable.

Perhaps one of the most important aspects is the choice of software platform in which the software can bloom and being utilized to its fullest. Having every application running within the very same operating system instance would probably not utilize the hardware to its fullest, some instances would even refuse running twice, and above all - the security nightmare it would pose, with an endless park of attack vectors.

OS virtualisation to the rescue? This is actually one (of many) reasons I chose illumos, as I find the solutions available within Linux to be sub-par.

LXC

I had a vm running MythTV and a graphical desktop at home reachable (connecting through fail2ban / jailkit and then using NX to attach/detach the desktop) from my road warrior setup (a tiny Asus EEE PC 701, with which I was editing commercials out of my teve recordings while commuting on train), but my vm was straining my machine as my CPU by then was lacking VT-x and at around year 2008 I found a promising project - LXC.

With LXC I saw opportunities and while, at that time, there was only Debian and later a Fedora (that I had issues getting to work with repositories, I had skipped the whole Mandrake/Mandriva since Red Hat 5.x so it was probably on me) to choose from, but suddenly I had two instances of Debian running rather smoothly. The moment I found out how to mknod devices and making the TUN/TAP available in the instance I had a vm spinning as my road-warrior setup. There was only one big issue, as powerful as I felt in the LXC instance, at the same time I found out that as root in the LXC I was in deep touch with the only running kernel and basically root in the main instance.

By time I replaced hardware to a CPU with the VT-x, I saw that Docker was repackaging whole desktop environments into purpose built containers, but I moved my workloads to VM (virtualbox) instead.

Zones

While working in Solaris based environments I come to realize that the networking in Linux stack left me with something else to desire when compared to the Crossbow implementation of Solaris 11/OpenSolaris. Also, the Zones were designed from the beginning to have various tenants sharing the same resources, with RBAC built-in and connected to the Crossbow networking stack. Not to mention ZFS with its abilities to snapshot/rollback the data. And DTrace to the rescue during troubleshooting.

There were a couple of things that stopped me - first, it was not free software. Second, data would not be able to migrate in-line. Third, I had no spare hardware available. Oracle Solaris needed a support contract in order to be usable and most of the data I had was encrypted with loop-aes and while encrypted ZFS was available I weren’t confident with it as a replacement and it wasn’t time for me to replace hardware anyway.

There seemed to exist option based on OpenSolaris that was called OpenIndiana with the Hipster distribution, or was it illumos. Lots of names that made no sense, but there wasn’t any disk encryption available (and I’ve relied on disk encryption since various different implementations in Windows that I’ve long forgotten, GBDE on FreeBSD, loop-aes and Luks on Linux) and this was not something I wanted to compromise on. Also, I couldn’t find anything about hypervisors.

New hardware

A couple of years passed by and it eventually was time to replace the aging hardware. Even though I couldn’t do disk encryption in the OSS ZFS, I would just enable loop-aes/luks in a VM as it seemed that KVM in a strange way had been ported to illumos by Joyent. The OpenIndiana distribution felt more towards desktop, but I saw that OmniOS looked promising.

I decided to go low power and chose to buy the (rather expensive) X10SDV-2C-7TP4F and a bunch of disks to form a ZFS disk array. I had copies of photos through the years spread out in various media, both internal and external, SATA and USB, encrypted and not encrypted. I read about deduplication in ZFS and decided that none of my photos needed encryption- what would a burgler really do to my family photos anyway? All in all it turned out to be near 200k photos amassing a whopping 4 TB of storage, moving them to my ZPOOL built with deduplication showed a dedup ratio of 1.9x, which meant that nearly every photograph had a copy.

I fell in the trap. I have been in that trap before, with Double Space in MS Windows 95, but while this trap wasn’t that awful - it still aroused details I wasn’t that prepared for. I had in advance calculated that the amount of RAM I bought should be enough to keep up with the DDT (Deduplication Table, and each entry in the table consumes RAM), but the everything dedup utilizes CPU. Also, in combination with my KVM workload I felt that my machine was underpowered. Some of the workload I before had running in a VM was moved to a lightweight zone, but everything encrypted (personal mail, economic documents.. ) was moved to a KVM vm running luks in Linux.

Hypervisors

I scouted Ebay and found a Dell R710, in which I installed a new Debian instance to run KVM/Qemu, but with time I felt that I wanted to explore a bit more into the OmniOS and scouted Ebay for another machine - a Dell R620. The R620 was installed with OmniOS and with a SFP add-on adapter I was now running Fiber Channel having a SAN.

My switch got out of ports and I wanted to do more than two VLANs anyway so I scouted a Brocade ICX6450–48 campus switch. I couldn’t run both Dells 24/7 due to heat/electricity bills so I went back analyzing my needs and decided that I wanted to another X10SDV, but with a bit more cores and found the X10SDV-6C-TLN4F which had better power consumption and eventually bought one from Ebay and started building a new hypervisor. I had recently found out that bhyve was ported from FreeBSD to illumos and got impressed with how it interacted and performed.

I wanted redundancy and a take away a bit of the pressure from the main hypervisor and while scouting Ebay again, I found the X11SDV-8C-TP8F at a too good price to resists. The board, which had support of U.2 NVME, seemed to consume less than the Dells and was way more powerfull than the X10SDV. It turned out to be on the upper side of the consumption and after a long time scouting Ebay again the X10SDV-6C-TLN4F got a sibling.

My Services

I’m running most of my workloads in two 6 core machines (Supermicro X10SDV-6C-TLN4F), and some of my workloads that powers my home are running on them:

There’s a bunch of more services/applications powering the home, but this are the ones I believe makes up the backbone of the household.

Kubernetes

While everything above run perfectly outside of Kubernetes, I have plans to move some of it into the cluster. The control plane is running in illumos, and the workload is running in Linux VMs. At the moment I only have home-assistant and grocy running as stateful (and “useful”) workload within the cluster. One of the issues I’ve had during this time is the persistent storage. Initially I had Longhorn, but the issues they had to keep up with Kubernetes I got to feel the pain - at one moment it was producing near 180k replicas which killed my kube-apiserver at 12G RSS. I’ve since switched to OpenEBS instead. I’m torn about how I want the storage and it would make sense to utilize my ZFS storage array with COMSTAR and deploy storage with ISCSI. In the future I might have the workers running on metal, in my Turing Pi 2, instead of as a Linux VM.

Network 

I’ve not mentioned the network as that probably would do a separate post in the future, but there was a thought to separate things in the home a bit. My son (and his friends) live their digital life in a dedicated network segment (VLAN), IoT in one segment, home alarm in one, I had music in a dedicated but it fel short of Chromecast/Musiccast and multicasting (IGMP proxy solved it partially), OOB in one segment, work related in one segment, labs in different segments, Wifi clients in one segment, remote logging in one segment, offline mode with a LTE-modem and a Raspberry Pi running Home Assistant in one segment, Hypervisor/Global Zone network in one segment. All in all there are a couple of different VLANs for different areas.

Well, that was a bit about my home infrastructure. Please comment with inspiration on what I should run in Kubernetes and that need to have service that should run in every family’s home.