lfcode.ca notes compiled for future reference

Setting up client certs for secure remote access to home lab services

Because I have some masochistic tendencies at times, I decided that it was a totally good idea™ to set up client certificate authentication to secure remote access to my lab services such as Grafana or Guacamole.

Unsurprisingly, since it's a rather uncommonly used finicky authentication method, there were problems. There were quite a few.

I'm writing this post mostly just for myself if I ever do this again, because it felt like it took too long to accomplish.

First, the list of requirements:

  • Should allow access without certs on the local network

  • Should use nginx

The latter was pretty easy, since I'm most familiar with nginx, however the former was rather interesting. I realized that, to implement this, I need to set verification as optional, then enforce it manually. This meant modifying the back ends (meaning maintaining patches, nope!) or doing it within nginx.

One issue is that nginx has if statements that are rather strange, presumably due to simplistic grammar while parsing the configuration. There is no way to do an and statement without hacks. The hack that I chose to use was some variable concatenation (which cannot be done in a single line on the if statement, it must be in its own separate if statement). Here's how I enforce certs from non-LAN hosts:

if ( $ssl_client_verify != "SUCCESS" ) {
    set $clientfail "F";
if ( $client_loc = "OUT" ) {
    set $clientfail "${clientfail}F";
if ( $clientfail = "FF" ) {
    return 401;

$client_loc is defined in a geo block:

geo $client_loc {
    default OUT; IN; IN;

But defining ssl_client_certificate and setting up the clients would be too easy. In setting this up, I learned that nginx has an error message: "The SSL certificate error". Yes. That's an error message. It's so bad that it could be written by Microsoft. Fortunately, it's very simple to just write an error_log logs/debug.log debug and get some slightly less cryptic details.

The big thing that tripped me up with the server setup was that ssl_verify_depth is set by default such that with a Root→Intermediate→Client hierarchy, clients fail to be verified. Set it to something like 3 and it will work.

Next, for the certificate setup:

The server directive ssl_client_certificate needs to point to a chain certificate file, or else it will fail with an error that suggests problems with the server certificate (thankfully).

The clients (for now, Chrome on Linux), need a pkcs12 file with some chain like stuff in it. Generate one with something like:

openssl pkcs12 -in client-chain.cert.pem -out client.pfx -inkey client.key.pem -export

where client-chain.cert.pem is a full chain from client to root CA and client.key.pem is a key file.

The other issue with the clients was that they didn't trust my CA that was imported as part of the pfx file to authenticate servers. This was quickly solved with a trip to the CA tab in the Chrome cert settings.

The client certs used in this were from my CA and have the Client Authentication property enabled.

Tags: homelab, nginx, tls

NUT not finding my UPS + fix

I use a CyberPower CP1500AVRLCD as a UPS in my lab. I'm just now getting more stuff running on it to the point that I want automatic shutdown (because it won't run for long with the higher power usage of more equipment). So, I plugged it into the pi that was running as a cups-cloud-print server and sitting on a shelf with my network equipment. The problem was that the driver for it in NUT didn't want to load. As is frighteningly common, it's a permissions problem:

Here's the log showing the issue:

Jul 09 16:49:58 print_demon upsdrvctl[8816]: USB communication driver 0.33
Jul 09 16:49:58 print_demon upsdrvctl[8816]: No matching HID UPS found
Jul 09 16:49:58 print_demon upsdrvctl[8816]: Driver failed to start (exit status=1)

Here's the udev rule that fixes it:


What this does is, when udev gets an event of the device with USB product id 0501 and vendor id 0764 being added to the system, it changes the permissions on the device files (think /dev/bus/usb/001/004 and /devices/platform/soc/20980000.usb/usb1/1-1/1-1.3) to allow group nut to read and write to it, allowing comms between the NUT driver and the device.

Tags: homelab, linux, raspberry-pi, udev

Windows folder extremely slow to load

Due to some weirdness and presumably thumbnail rendering, if a folder is set to "Optimize for Pictures", it takes 10+ times as long as it should to load. This was happening for my Downloads folder. It seems to only apply when it's accessed through "This PC".

Anyway, to fix it, in the properties of the folder in question, under customize, change Optimize this folder for: to General Items and it will work much better.


Tags: windows, small-fixes

Human error is the root of all problems, especially with network bridges

When in doubt, the problem is directly caused by one's own stupidity.

I was trying to run an LXD host in a Hyper-V VM and went to set up bridged networking (in this case, notworking). Twice. The good old rule that it's caused by my stupidity rang very true. The problem was caused by the network adapter in the VM not having the ability to change the MAC address of its packets. The toggle is in the VM properties under advanced settings in the child node on the NIC.

This is why you should have a routed network.

Tags: homelab, hyper-v, lxd, containers, networking

nftables: redirect not working + fix

Recently, I made the somewhat-rash decision to switch to nftables from ufw-managed iptables on this VPS.

It's been a fun ride. The man page doesn't even document the redirect feature. It doesn't even acknowledge its existence, nor what it really does.

That's irrelevant however, because it does the same thing as the REDIRECT target in iptables, documented in the iptables-extensions man page. This allows the functionality of redirect in nftables to be inferred as "change destination address to localhost, and change the destination port to the one specified after to".

I, however, was a bit too dense to go looking through there and didn't read the wiki too well about redirection. I figured "hey, just need to put redirect at the start of the chain hooked into nat prerouting to enable it, then add a rule specifically redirecting the port". Later, I wondered why it wasn't working. After some tcpdump, copious quantities of counters everywhere, and netcat instances, I figured that out.

Note that you need to allow the packets with dport 11113 in your filter. Your filter table will never see any packets on port 113 unless something has gone horribly wrong, as all of them will have dport changed to 11113 in the nat table. If, for some reason, you want to drop these, you probably can do it in a chain with type mangle hook prerouting priority 0, but I have no idea why you would want to do that.

Here's the functional config:

table ip nat {
  chain prerouting {
    type nat hook prerouting priority 0;
    tcp dport 113 counter redirect to 11113

  chain postrouting {
    type nat hook postrouting priority 0;

table ip6 nat {
  chain prerouting {
    type nat hook prerouting priority 0;
    tcp dport 113 counter redirect to 11113

  chain postrouting {
    type nat hook postrouting priority 0;

Tags: nftables, linux

Introducing my new theme

Recently, I had enough of the Arabica theme for Ghost. Put simply, it was ancient, didn't look that great anyway, and was missing a bunch of newer Ghost features.

Its replacement is a fork of lanyon-ghost, itself a fork of lanyon (a theme for Jekyll).

Currently, all I've changed is the fonts, and I switched the homepage to display full posts, as it's quite irritating to have to click on each one to read it (while I'm at it, it would be great if Ghost allowed to put a mark where the fold in the page is, so that longer posts don't eat up all the space on the page).

The fonts in use are the beautiful Charter (main content), Fira Sans (headings, other text), and Source Code Pro (monospace/code).

There's also an author page that shows the author's description, image and such along with their posts.

Here's the code: https://github.com/lf-/lanyon-ghost

Tags: meta, ghost

How to have a functional dhcrelay

I'm dumb. Or ignorant. Or inexperienced. I haven't decided which.

dhcrelay only gets proper responses if it's listening on both the interface that it's actually listening on for requests and the one where it will get the responses.

My command line for it to forward dhcp requests to my Windows dhcp server in my virtual lab is:

/usr/bin/dhcrelay -4 -d -i eth1 -i eth2 10.x.x.x

eth1 is the interface with the Windows dhcp server on its subnet

eth2 is the interface with the clients on it

10.x.x.x is the address of the Windows dhcp server

This is run on my arch (yes, I know. Debian took longer than Windows to install. The only stuff on it is in base, vim, and dhcp) gateway VM. I could also stand up a Windows box and have it do NAT, but that doesn't use 512MB of RAM nearly as happily.

Tags: Windows Server, dhcp, linux, homelab

Swapping Back and Menu/Overview buttons on Android

I use a OnePlus One as my daily driver. Unfortunately, like nearly every phone on the market with capacitive buttons, they're backwards! I could enable software keys, but that's admitting defeat. CyanogenMod doesn't allow swapping the keys in the settings, because it would result in some pretty horrible user experience.

None of this is relevant however, because this is Android, and I have root:

In /system/usr/keylayout/Generic.kl, you can see the key mapping for all keys on the system. Simply swap the stuff in the rightmost column: BACK and MENU.

MENU is at key 139 and BACK is at key 158.

I use this on the latest Cyanogen OS based on Lollipop. It works perfectly. If you want to revert this, simply do the reverse of what's written.

A little note: my blog is just stuff I need to write down for easy reference later. It's on completely random themes, although centered around technology. I should probably make a wiki for this stuff.

Tags: android, cyanogenmod, oneplus

Setting up DHCP on a DC with secure dynamic DNS

So, in my virtual homelabbing, I decided I was going to get a Windows based network set up with more or less only PowerShell. In these efforts, I discovered a pretty poor pile of documentation (such as this insanity where they tell you to create credentials with netsh, restart the service, then delete the credentials and restart again [optional step: wonder why it doesn't work]).

Here's how I set it up:

Create AD account:
# Get username and password for the new account (remember to include your domain!)
$cred = Get-Credential

# Create the user (it needs no special permissions)
New-ADUser -Enabled $true -SamAccountName $cred.UserName -AccountPassword $cred.Password
Make the DHCP server use it:
# Set the credentials for the DHCP server
Set-DhcpServerDnsCredential $cred

# Restart the DHCP Server
Restart-Service DhcpServer

You're set!


Also remember to set the DNS server to only allow secure updates!

Set-DnsServerPrimaryZone -DynamicUpdate Secure

Tags: PowerShell, Active Directory, dhcp, dns

General Network Error when running Install-ADDSForest

When I was messing about with AD DS a bit on Windows Server 2016 TP 2, I encountered the error General Network Error, with error ID 54. This is obviously a very unhelpful error. In troubleshooting, I noticed that the VM was being assigned an address in 169.254.x.x. This wasn't part of my intended IP range, so I started investigating.

It turns out that 169.254.x.x is a reserved range for APIPA (Automatic Private IP Addressing), where an operating system automatically assigns an IP when there is no DHCP available (which there wasn't because I intended to set up Windows DHCP). After disabling this, the AD setup worked correctly.

You may be wondering how to disable this problematic system. Here's how you do it (in PowerShell):

# Disable DHCP
Get-NetAdapter | Set-NetIPInterface -Dhcp Disabled
# Disable APIPA
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name IPAutoconfigurationEnabled -Value 0 -Type DWord
# Reboot to apply

Tags: PowerShell, Windows Server, Active Directory