Featured image of post Revamping physical and logical services
Home Lab

Revamping physical and logical services

Creating a new server, creating a new docker compose structure, with 110 TB of total storage. What I learned.

Introduction

For several years, I have had a very nice setup. Google Workspace had unlimited storage, a roughly 20 TB storage NAS that would upload to the Google Workspace Shared Drives storage, automated server dynamic domain name updates, ZFS with encryption, and more. It was my first primary NAS system, full of mistakes.

Out with the old NAS

The old NAS system was originally an AMD system that I installed ESXI on, passed through an LSI HBA into the Guest VMs, and was enormous - it was placed into a full-sized ATX tower. Then, when I moved to the EU, I stayed at my parent’s place and started to have PSOD frequently and rapidly (e.g., once a month). Because there was no power management or IPMI controller attached to this on consumer hardware, I could not do much at such a distance and had to rely on my parents’ good nature and nicety to turn the machine off and on when these things happened. As one can imagine, that good nature only lasts for so long after one year of doing this.

So, something had to change.

The machine itself:

  • AMD Ryzen 7 1700 Eight-Core Processor
  • 64 GB ECC DDR4 Memory
  • ASRock X470 Master SLI/ac

In with the new NAS

I had a few ideas in mind with the new NAS build:

  • Long-Term Usage
  • Consumer components only
  • Small Physical Footprint (ITX or less)
  • Larger Storage Footprint than the old server
  • Components will be purchased in the US and brought to the EU via international flight on carry-on
  • Must have redundancy where possible
  • Minimal initial mistakes
  • Minimize crashing/downtime/issues
  • Must be transportable in case I move
  • Must be able to appeal to a GF / Wife factor
  • Reuse parts that could be reused where possible or made sense
  • As energy efficient as I could be
  • Must be as silent as possible
  • Light enough to be mounted onto a wall shelf and be off the ground
  • Ability to transcode 4K content using Intel Quicksync

The ultimate use case of the system is:

  • ZFS Storage
  • Media Consumption & Automation
    • Plex
    • *arrs
    • Jellyfin, as a backup to Plex
    • Calibre
    • Navidrome / Music Streaming Service
  • Home Services
  • Web Server
  • Automation Tools (n8n)
  • SSO-backed systems and services using:
    • Okta
    • Authentik
  • IRC
  • RSS
  • Document Management
  • System & Service Monitoring
  • Various other things

So after about four months of researching and questions on various forums, I decided on the following components:

Parts & Specifications

Purchased in the EU
Purchased in the US
Reused parts
  • LG Blu-Ray Drive

Collection of parts

This might seem like an overkill machine - usually, the systems I built are. But they achieve the need.

The reason for the single purchase of the case in the EU and the fan - was that after long thought about it, I didn’t think I could bring it back from the US. If it didn’t fit in the space, on the wall next to my tv cabinet, I would need to be able to return it, and shipping it back to the US was not going to be an option.

Warranty Situation

Of all the parts purchased here, only Corsair & Crucial have a global warranty program, which works for me.

All other items have to be shipped back to the originating country. But for Western Digital - Drives must be sent back to the RMA Warehouse from which the original purchase originated, meaning that if one of the drives dies, I need to send it back to the US. However, the cost savings I received by buying the drives in the US, compared to the EU, is worth enough. I am okay with this risk, and I can always bring them back to the US if I need to or ship them.

Testing the hard drives before heading back to the EU

My biggest concern when I first got the drives was ensuring one of them was okay. So to do that, the easiest thing to do is run it through smartctl and bad blocks.

Simple enough, plug in as many drives as I can and run the following:

1
2
3
4
5
6

$ sudo smartctl -xa /dev/sdd >> ~/$SERIALNUMBER.log
$ sudo smartctl -t short /dev/sdd >> ~/$SERIALNUMBER.log
$ sudo smartctl -t long /dev/sdd >> ~/$SERIALNUMBER.log
$ sudo badblocks -wsv /dev/sdd >> ~/$SERIALNUMBER.log
$ sudo smartctl -t long /dev/sdd >> ~/$SERIALNUMBER.log
$ sudo smartctl -xa /dev/sdd >> ~/$SERIALNUMBER.log

Each drive took about nine days to complete the entire process, but none reported errors, so they got shipped back up into the boxes they came in and stuffed back into a bag. I was on a business class ticket, which was a fantastic experience, but this allowed me to have extra space and weight restrictions on my bags to bring back these items without any significant issues.

Building the system

So, after a month of having all the components needed and the desire to build this, I finally got home, unpacked, and set aside some time to get this done and stop dealing with an unreliable server.

Reminder

The plastic film covers most pieces of the motherboard; be sure to take that off before you place everything in the case.

Unpacking the case & components

Jonsbo N2 Case - Front Jonsbo N2 Case - Back - Covers Off Jonsbo N2 Case & PSU Jonsbo N2 PSU Installed

The case is relatively compact, and the back takes advantage of its space quite well.

Unfortunately, as far as I know, Stockholm has no public services where you can provide a 3D printer file to a state or county-owned public service (EG, A library, or an alternative). However, as I was visiting family in the US, the public libraries there have “Maker” areas where you can pay a minimal fee for a 3D Printed file. I spent roughly 2 USD for the extended fan module made of ABS plastic filament. While the printing tolerance could have been slightly improved, overall, it was a fabulous service and worked well for the needs.

Installing the PSU is relatively easy, but it is much easier if you route the cables through the compartment first and leave it “half installed” first (basically, don’t screw it down), as this will provide you enough room to route cables and the like all over.

Installing the power for the backplane

I wanted to keep enough space between the fan that I would install if the 3d printed mount didn’t work and the case power plane itself. So getting 90-degree angle power plugs and the extended fan bracket above would be really useful.

Power Connectors on Backplane Cable Fed to PSU Compartment

This allows the installation of a 25 mm fan over a 15 mm fan to cut down on noise and improve airflow simultaneously.

Installing the CPU and CPU Holder

Because I would mount the case on a wall, I wanted to rely on something other than the CPU holder that came in stock on Intel motherboards, so I grabbed the Thermal Grizzly CPU Mount. If the case falls, the more flimsy stock mount might pop open or off. I have difficulty believing that the Thermal Grizzly holder would do that.

Stock CPU Holder Bare Socket CPU & Thermal Grizzly Mount Installed

Installing the RAM

RAM in Box RAM in Board

Pop the RAM into the board. What else should I say?

Installing the NVME Drives, Riser, and SATA Expander

Unfortunately, I didn’t take as many photos of this process - but the MSI Motherboard has the capability to use 3x m.2 slots in precisely the layout we need.

  • 1x M.2
  • M.2_1 Source (From CPU) supports up to PCIe 4.0 x4 , supports 2280 devices
  • M.2_2 Source (From Chipset) supports up to PCIe 3.0 x4 / SATA mode, supports 2280 devices
  • M.2_3 Source (From Chipset) supports up to PCIe 4.0 x4 , supports 2280 devices

M.2_1 is the unit closest to the top side of the motherboard, M.2_2 is on the heatsink & riser card, which is where the SilverStone Expansion Card will be installed, and M.2_3 is located on the back of the motherboard. In this case, the motherboard placement and the space surrounding it in the case, there is enough space to add or use a heatsink - something I should have looked at when I originally got the NVME drive, as I thought it might not fit. Unfortunately, I need to install this later as I didn’t wait until I got a heat sink for the NVME card on the bottom.

Initially, the plan was to use one of the NVME drives as a boot drive, the other as Docker Cache (EG: Docker scratch space), and a ZIL or SLOG for ZFS. Then I changed my mind while in the US and grabbed the SATA SSDs for the boot drive, and I would use one whole NVME as a Docker Scratch drive and the other NVME as a split SLOG/ZIL.

However, after discussing with some people on the *arr server, reading over more of the discussion on Ubuntu Forums that I had, and some other conversations on IRC, I opted to change this plan to use the SATA SSDs as a mirrored ZFS Boot Drive, the NVME drives as a mirrored ZFS Docker Cache Drive, and the five spinning disks in its original intention as a ZFS RAIDZ Config.

NVME Drive Motherboard Top NVME Slots and Riser Motherboard and SATA Fan Out Connectors

Installing the SATA SSDs (Boot Drive)

The Jonsbo case allows for the installation of one SSD drive on the side of its case here:

SSD Photo

However, if you use the Corsair SF 750, there is enough space in the PSU compartment for all the cables you will use, the PSU, and plenty of spare SATA drives. Unfortunately, I don’t have a picture, as it would have been too dark and too cramped a space to take a photo of - but you can make that space work to your advantage and stick another SSD in there.

Installation on CPU & Heatsink Installation

I have to say, I attempted to use the Thermal Grizzly Thermal Paste, which initially worked well but took much work overall. I tried to warm it up by hand by creating some friction heat by rubbing my hands together around the thermal paste. It seemed to have enough for just one installation without any potential screw-ups.

I recommend installing the mounting brackets first, leaving the heatsink off until you get the motherboard into the case and everything adjusted to how you want. Then, remove the fan’s top and use the included tools to install the heatsink. Then, re-install the fan.

Also, this is another section I should have taken photos of.

Installation of the Backplane SATA Cables

Motherboard and SATA Fan Out Connectors Top Down Shot of Completed Case Side Shot of Completed Case Shot of Sata Fan Out Routing

Using the System

Installing BIOS Update

First things first, update the BIOS. MSI includes a 16 GB USB drive for Windows or BIOS updates. At the time of writing, the BIOS for the system was on something like 11, 12, 13, or 16 (from what I remember, at least). So, anywhere from 6 to 14 months old.

Original BIOS

When I returned to the EU, a BIOS update was released for microcode updates to improve some of the processors’ functionality. So I installed v19.

Beta BIOS

When writing this, it appears that MSI will start adding capacity to the maximum supported RAM from 96 GB to 128GB via a Beta BIOS. Courtesy of the Internet Archive’s Wayback Machine, we can check out the previously supported spec and the current spec.

Old Spec Page New Spec Page

If this interests you and gets pulled for some reason, but you still need the functionality, you can find the beta BIOS here.

Configuring BIOS & Lowering Power Consumption

There were a few things I needed to do to try and get the power consumption down, as well as enable a few extra features.

Primarily:

  • Configure AC Power and Restore Power On
  • Enable vTPM
  • Enable C8, C9, and C10 states
  • I need to hit C8,9, or 10 states, due to some other configured BIOS options.

The rest of the options I found here could also help minimize more power consumption than necessary. However, BIOS management is painful on a headless Linux system. I probably have a bit more to do with this, as the system, while running Plex and streaming movies/shows, runs at 120W idle. So, I need to bring this down.

Installing Firmware Updates on SSD, NVME, and Hard Drives

NVME

I had two different firmware versions on my NVME drives. One of those drives had 3B2QGXA7, which was the firmware version that caused potential data/drive issues. After some digging to find the most recent version and validation on what firmware to pick up on which led me to find this Reddit Post about the differences in the drives as well. These two user manuals may also be helpful for you. Firmware Update Utility User Manual and Samsung Magician 8.0.0 Install Guide.

So, I needed to figure out a way to update the firmware, as I didn’t want to reboot the system as a whole. I was in the middle of finishing up a smartctl test on the drives after the flight.

I grabbed the latest firmware version at the time Samsung_SSD_990_PRO_4B2QJXD7.iso and then found four posts that would explain how to fun fumagician on Linux. One here, here, here, and another here.

This ended up updating all Samsung drive firmware, not just the NVME.

SSD

There were various reasons I chose the SSDs as the same brand as the NVMEs, partially for ease of use and accessibility; the other part is that Samsung arguably makes the best flash storage.

Running the process above, I could also update the SATA drives simultaneously with the NVME drives.

HDD

It seems like firmware updates are impossible, which has been disappointing on the WD Red Pros recently. However, my drives came with Firmware Version: 83.00A83, which seems to be the latest release according to various blog posts, so I was fine with keeping this the way they are. Source 1 & Source 2.

Installing Ubuntu 23.10

I would much rather have Ubuntu 24.04, but it isn’t out yet, and I don’t want to install Ubuntu 22.04 LTS as I think the OS has some key components that I want/need that are missing in the older versions of the software repo there. So instead, I chose 23.10 for now and will make a decision to either upgrade to 24.04 LTS when it is available or 24.04 and keep a rolling release with the newest stuff.

Modifications before getting stuff set

Several modifications need to be made. Since the plan was to use Ubuntu Desktop, and not Server for this so that a GUI could be used when connecting certain devices (like Cameras, BluRay Drives, etc.), I needed to install a few additional packages as well to make sure things we were working well. In the future, if I want to move off of the GUI, I can always install the server edition by:

  • sudo apt install ubuntu-server
  • reboot
  • sudo systemctl set-default multi-user.target
  • reboot
  • sudo apt purge ubuntu-desktop -y && sudo apt autoremove -y && sudo apt autoclean
  • reboot
Installation of additional packages

Several additional packages need to be installed:

sudo apt install zsh trash-cli ubuntu-restricted-extras pwgen gnome-extra-icons vim zfs sysstat git trash-cli fzf opensshd openssh sshd ssh ssh-import-id ssh-askpass-gnome ssh-tools ssh-askpass-fullscreen ssh-askpass ncdu htop powertop gh sensor tree hdparam smartmontools inetutils-tools nano nvme-cli nvme-stas wget smartmontools inetutils-tools nano nvme-cli nvme-stas wget iputils-ping iputils-tracepath iputils-arping iputils-clockdiff fwupd fwupdate gparted lm-sensors speedtest-cli speedtest iotop mediainfo inotify-tools subview filezilla bd_info libbluray libbluray-bin perfmon sysstat top whois inuetutils inetutils inetutils-tools brctl bridge-utils docker-ce docker-compose-v2 docker-ce-cli

And a few other packages.

Docker - daemon.json Modifications

To be able to enable both IPv6 and to reduce the broader scope of how Docker creates IP Address Pools on a host (EG: going from 172. X.X.X range to 192.168.X.X range) that could potentially cause collisions, set the following in the file /etc/docker/daemon.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
    "ipv6": true,
    "fixed-cidr-v6": "2001:db8:1::/64",
    "experimental": true,
    "ip6tables": true,
    "default-address-pools": [
        {
            "base": "172.17.0.0/16",
            "size": 24
        }
    ]
}

I want to be able to host many different services, but only some of them. Only some services need a full 172.X address range; each service will have 254 scaled functions at best. So, this is a much more suitable default configured for my needs primarily as I segment each network and don’t use the default bridge for all services.

rm vs trash

No one wants to delete a file accidentally, and I have had first-hand experience of doing just that after setting up everything correctly only to, out of muscle memory/habit, delete the wrong thing.

After doing a bunch of research, I eventually concluded that switching the rm command to an alias of another command could end in bad things. Instead, let’s just have rm do this: alias rm="echo Use the full path i.e. '/bin/rm', consider using trash"

I took this from this stack overflow suggestion.

oh-my-zsh & zsh over bash

ZSH vs. Bash is debatable, but couple the ease of the convenience of oh-my-zsh, its plugins, and zsh over the default of dealing with Bash, and its mods and the theming system as a whole, I would instead use zsh in that case.

In addition, on top of the ZSH set, I find adding the following to .zshrc helpful.

1
2
3
4
5
6
7
8
9

plugins=(git ssh-agent github docker docker-compose ubuntu vscode zsh-interactive-cd zsh-navigation-tools gh colorize command-not-found alias-finder aliases copy path copyfile history dotenv git-auto-fetch git-escape-magic git-extras github git-prompt sudo)
ENABLE_CORRECTION="true"
HISTFILE="$HOME/.zsh_history"
HISTSIZE=10000000
SAVEHIST=10000000
setopt BANG_HIST                 # Treat the '!' character specially during expansion.
setopt EXTENDED_HISTORY          # Write the history file in the ":start:elapsed;command" format.
setopt INC_APPEND_HISTORY        # Write to the history file immediately, not when the shell exits.
setopt SHARE_HISTORY             # Share history between all sessions.
sysctl modifications

By default, Linux generally caps the watch limit to 4096, which is fine in most day-to-day use cases. However, if I am trying to run Plex, Sonarr, and all these services that watch or manage files, it is necessary to increase that count realistically.

sysctl -p fs.inotify.max_user_watches=524288

1
2

$ cat /etc/sysctl.d/40-max-user-watches.conf
fs.inotify.max_user_watches=524288

Since I am using Tailscale later on, we also need to set this:

1
2
3

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Setting up ZFS

Modifications for ZFS 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

andrew@nas: ~/Downloads
$ zpool status [13:04:41]
  pool: pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:00 with 0 errors on Fri Mar  1 00:00:01 2024
config:

	NAME                                                     STATE     READ WRITE CKSUM
	bpool                                                    ONLINE       0     0     0
	  mirror-0                                               ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_S7EWNJ0WC67566W-part2  ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_S6PXNZ0W803395E-part2  ONLINE       0     0     0

errors: No known data errors

  pool: docker
 state: ONLINE
  scan: scrub repaired 0B in 00:00:09 with 0 errors on Fri Mar  1 00:00:10 2024
config:

	NAME                                              STATE     READ WRITE CKSUM
	docker                                            ONLINE       0     0     0
	  mirror-0                                        ONLINE       0     0     0
	    nvme-Samsung_SSD_990_PRO_2TB_S73WNJ0WB07701M  ONLINE       0     0     0
	    nvme-Samsung_SSD_990_PRO_2TB_S7KHNJ0WA01997H  ONLINE       0     0     0

errors: No known data errors

  pool: pool
 state: ONLINE
  scan: scrub repaired 0B in 11:29:53 with 0 errors on Fri Mar  1 11:29:55 2024
config:

	NAME                                    STATE     READ WRITE CKSUM
	pool                                    ONLINE       0     0     0
	  raidz2-0                              ONLINE       0     0     0
	    ata-WDC_WD221KFGX-68B9KN0_2GG09WAL  ONLINE       0     0     0
	    ata-WDC_WD221KFGX-68B9KN0_2GJZ5NJS  ONLINE       0     0     0
	    ata-WDC_WD221KFGX-68B9KN0_2TG0405P  ONLINE       0     0     0
	    ata-WDC_WD221KFGX-68B9KN0_2TG07LLP  ONLINE       0     0     0
	    ata-WDC_WD221KFGX-68B9KN0_2TG0V65E  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:01:05 with 0 errors on Fri Mar  1 00:01:12 2024
config:

	NAME                                                     STATE     READ WRITE CKSUM
	rpool                                                    ONLINE       0     0     0
	  mirror-0                                               ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_S7EWNJ0WC67566W-part4  ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_500GB_S6PXNZ0W803395E-part4  ONLINE       0     0     0

errors: No known data errors

$ zpool list  [13:04:45]
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   1.88G   248M  1.63G        -         -     0%    12%  1.00x    ONLINE  -
docker  1.81T  16.3G  1.80T        -         -     0%     0%  1.00x    ONLINE  -
pool     100T  28.4T  71.6T        -         -     0%    28%  1.00x    ONLINE  -
rpool    452G  30.0G   422G        -         -     7%     6%  1.00x    ONLINE  -

$ zfs list [13:06:15]
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
bpool                                              247M  1.51G    96K  /boot
bpool/BOOT                                         247M  1.51G    96K  none
bpool/BOOT/ubuntu_adv8kn                           247M  1.51G   247M  /boot
docker                                            16.3G  1.74T   192K  /docker
docker/data                                       16.3G  1.68T  16.3G  /mnt/docker
pool                                              16.8T  42.3T   312K  /pool
pool/downloads                                    8.88T  24.1T  8.88T  /mnt/pool/downloads
pool/media                                        7.95T  12.1T  7.95T  /mnt/pool/media
pool/storage                                       355K  42.3T   355K  /mnt/pool/storage

I consulted ChatGPT for information on ZFS and referenced/fed several man/doc pages on ZFS to figure out the best tactics for the pool/downloads namespace. Since this would be slightly different compared to just general storage, for the most part, the general storage defaults work well for my purpose.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

$ zfs get all pool/downloads | grep -v default  [13:09:20]
NAME            PROPERTY              VALUE                  SOURCE
pool/downloads  type                  filesystem             -
pool/downloads  creation              Tue Jan 30 12:00 2024  -
pool/downloads  used                  8.88T                  -
pool/downloads  available             24.1T                  -
pool/downloads  referenced            8.88T                  -
pool/downloads  compressratio         1.00x                  -
pool/downloads  mounted               yes                    -
pool/downloads  quota                 33T                    local
pool/downloads  recordsize            16K                    local
pool/downloads  mount point            /mnt/pool/downloads    local
pool/downloads  checksum              blake3                 inherited from pool
pool/downloads  compression           lz4                    local
pool/downloads  atime                 off                    local
pool/downloads  createtxg             55                     -
pool/downloads  xattr                 on                     inherited from pool
pool/downloads  copies                1                      local
pool/downloads  version               5                      -
pool/downloads  utf8only              off                    -
pool/downloads  normalization         none                   -
pool/downloads  case sensitivity       sensitive              -
pool/downloads  guid                  2314767234968638898    -
pool/downloads  primary cache          metadata               local
pool/downloads  usedbysnapshots       0B                     -
pool/downloads  usedbydataset         8.88T                  -
pool/downloads  usedbychildren        0B                     -
pool/downloads  usedbyrefreservation  0B                     -
pool/downloads  objsetid              1030                   -
pool/downloads  dedup                 off                    inherited from pool
pool/downloads  sync                  disabled               local
pool/downloads  refcompressratio      1.00x                  -
pool/downloads  written               8.88T                  -
pool/downloads  logicalused           6.67T                  -
pool/downloads  logicalreferenced     6.67T                  -
pool/downloads  encryption            aes-256-gcm            -
pool/downloads  key format             hex                    -
pool/downloads  encryption root        pool                   -
pool/downloads  key status             available              -

Since the download folder will consist of many moving pieces and bits governed by downloaders, this appeared to be the recommended “best results” for this pool’s folder.

Setting up auto-maintenance for ZFS

So now we get to maintenance and verification of the drives.

First off, we need to set up scrubbing. This is a way to keep the ZFS pools healthy. The below contains the looping script to run through each pool at the beginning of every month and a system service to be placed into /etc/systemd/system/ to automate this process.

1
2
3
4
5

$ cat /usr/local/bin/zfs_scrub_all.sh [13:13:25]
#!/bin/bash
for pool in $(zpool list -H -o name); do
    zpool scrub $pool
done

1
2
3
4
5
6

$ cat /etc/systemd/system/zfs_scrub.service [13:13:34]
[Unit]
Description=Monthly ZFS Scrub on All Pools

[Service]
ExecStart=/usr/local/bin/zfs_scrub_all.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ cat /etc/systemd/system/zfs_scrub.timer   [13:14:42]
[Unit]
Description=Monthly Timer for ZFS Scrub

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target

Next, I don’t want the system to hang or wait to run everything until the encryption keys are loaded. I like the drives to mount automatically. I did look at TPM storage options for the security keys, but figured that wouldn’t be worth the time - and given that discrete TPMs are primarily useless for desktops, and I don’t trust an onboard chip TPM on the Intel CPUs to function appropriately (part of that is because it is new tech, but, if a BIOS or update screws up the configuration - all data would be gone) I opted just to keep it simple.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ cat /etc/systemd/system/zfs-load-keys.service  [13:15:40]
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/zfs load-key -a
StandardInput=tty-force

[Install]
WantedBy=zfs-mount.service

Lastly, I need some way to alert people about smart status issues. Scrutiny seems to be a convenient enough project to do just that. Scrutiny

Screenshots of Scrutiny

Scrutiny provides a convenient way to get information on the status of the drives. Coupling this behind Authentik/CloudFlare Tunnels, only specific people could access this in a given situation, which would mean either access to Okta & Authentik or access over the CloudFlare Tunnel. So far, the only issue I have had has been this github issue. It is so minimal that I wouldn’t even consider it a problem with the use case. It’s a great little tool.

Installing Tailscale

I previously tried to get Tailscale set up as a container but, unfortunately - I was never able to get it to work in the way that I wanted it to; either it worked sometimes or never. So, in this case, I resorted to installing it on the host.

Setting up Docker & Services

I had a few requirements for this:

  • I want to be able to use and serve pages over IPv6 from a front-end perspective but not use IPv6 for each container or from a unique server perspective.
  • I also want to be able to use docker-compose over various subfolders that would host each service.
  • I wanted to be able to use multiple YAML files, not a single monolithic file of 3000+ lines.
  • Wanted to be able to add or remove services incredibly easy

An example of this setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ tree
.
├── 00_swag
│   └── docker-compose.yaml
├── 01_db
│   ├── docker-compose.yaml
│   └── .env
│   ...
├── media-automation
├── media-servers
│   ...
├── down.sh
├── global.env
├── pull.sh
├── up.sh
└── VPN

So, how will I split some parts of the configuration across multiple services? A global .env file:

1
2
3
4
5
6
7
8
9

$ cat global.env
BASE_PATH="/mnt/docker"
DOMAIN="your.domain"
PGID=YOUR_ID
PUID=YOUR_ID
OWNER="[email protected]"
EMAIL="[email protected]"
USERNAME="you"
TZ="YOUR/TIMEZONE"

The code above is the shared configuration across all docker containers, with each service using it and a separate env file for its use case, as shown in the .env file in 01_db.

So then, let’s get into the code and setup:

With either the pull or up (or down) file, docker-compose files can be called anything, as long as it has an extension of .yaml or .yml, and any file with an extension of .env will get picked up for recognition in the docker-compose command.

This is then easy enough to expand out to other services that you want to run. Each service can be pulled in to it’s own folder service and then configured as desired according to the service itself.

1 Month Later

For the most part, everything has been working seamlessly. Though I have learned from a few mistakes, that would be fine without any long-term issues so far.

Dynamically updating ddclient vs CNAME in Cloudflare

Previously, I had set up several A records for each DNS entry I needed to use in Cloudflare, and to ensure all of that was automatically managed through the client - I needed to create a script that would automatically add the needed entries into the client’s config. I created this script:

Then, after about a month of this and trying to troubleshoot some of the issues with it, I realized that I was overcomplicating that and it would be much easier to have a single A and AAAA record, with additional CNAMEs that pointed back to the A/AAAA records where Nginx could just response.

That eventually led to exporting all records, manually deleting all of the records in the zone, and modifying the zone export and rearranging some bits to upload.

Now, updating DNS records with the setup is less of a hassle and overhead.

Guacamole/GuacD no longer works with Ubuntu 23.10

I had everything working perfectly… and then some Ubuntu updates broke Guacamole and Gnome’s RDP Session.

Then I found this on the Guacamole Mailing Lists. I found a Gnome Bug Tracker Issue by the same author called “Connecting from Guacamole to Gnome-remote-desktop”, which was closed instantly by the Gnome Developers, as they state “there is no problem from Gnome remote-desktop”.

It’s a bit of a bummer, so basically, this service no longer works as it used to - and the only reason seems to be an updated library or package being included in Ubuntu / Debian.

I believe what might have occurred is something that occurred when gnome-remote-desktop or freerdp had an update released between January 1st and January 29th that broke Guacamole’s support function server side. We can see that a new package version was released on launchpad.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ sudo cat /var/log/apt/history.log.1.gz
Start-Date: 2024-01-29  19:03:02
Commandline: /usr/bin/unattended-upgrades -v
Upgrade: ... libfreerdp2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1, 2.10.0+dfsg1-1.1ubuntu1.1), libfreerdp-server2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1, 2.10.0+dfsg1-1.1ubuntu1.1), libfreerdp-client2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1, 2.10.0+dfsg1-1.1ubuntu1.1), ...
End-Date: 2024-01-29  19:03:35

Start-Date: 2023-10-16  10:47:07
Commandline: apt-get --yes -oDebug::pkgDepCache::AutoInstall=yes install ... libfreerdp-client2-2 libfreerdp-server2-2 libfreerdp2-2 ...
Install:  libfreerdp-client2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1)
End-Date: 2023-10-16  10:47:51

Start-Date: 2023-10-16  10:37:40
Commandline: apt-get --yes -oDebug::pkgDepCache::AutoInstall=yes install ... libfreerdp2-2 ...
Install: libfreerdp2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1), libfreerdp-server2-2:amd64 (2.10.0+dfsg1-1.1ubuntu1)
End-Date: 2023-10-16  10:40:56

This was the last freerdp upgrade since the issues occurred. I can roll Guacamole back to 1.4.X-1.5.X and am still unable to connect to Ubuntu. So, unfortunately, this has been hopeless. I am honestly not even sure where to file a bug report since the initial bug reports were closed - but the problem exists in libfreerdp. Somewhere.

Docker Compose v2 Issues

The only other issue I had appeared about a week before writing is that docker containers randomly fail to join a network. This also seems to be happening to others based on this Github Issue. This should hopefully get fixed soon, so not too worried about it in the long term.

Power Consumption

I managed to get power consumption down to an eventual 70-80 watts per hour at “idle”, and for me, idle would be low-usage, eg: Nothing playing on Plex, minimal to no disk activity related to user usage (nominal disk usage due to downloads, media, or otherwise).

Plan for the next several months

I would say the next steps for me are to figure out how to restrict and reduce more of the power consumption from the system.

© 2015 - 2025
Thanks for stopping by!
Built with Hugo