Running services

I'm running the following HTTP-based web services on my server:

• Caddy as reverse proxy, terminates SSL and routes to other services
• Apache2 with some mods (like CGI) as web server for the homepage
• Navidrome for the music streaming service
• Gitea as the git server
• Immich for the photo/video management solution
• SearXNG for the aggregated search service
• OpenCloud for the file server

Additionally, I'm running a mail server via Postfix and Dovecot, and an ad-blocking DNS server (Pi-hole). As the firewall, I'm using ufw – though note that there's another firewall on the router.

Service setups

Caddy, apache2, ufw, postfix, and dovecot run directly on the machine, installed as system packages. The rest runs as podman containers, via a bunch of podman-compose.yml files that expose the services' plain-text HTTP interfaces on some port only to localhost, e.g. 127.0.0.1:9000. Adding new web services is just a matter of spinning up new containers and telling caddy to reverse-proxy incoming traffic for some subdomain to their HTTP port. Scheduling a crontab (1) job to update the container images periodically (e.g. weekly) might also be a good idea.

Lingering users

In case the system uses systemd-logind for managing login sessions, systemd will kill all of a user's processes (via a per-user scope unit) when they log out. This obviously interferes with some utilities that are meant to outlast a login session, like tmux or podman containers. Luckily, there's a way out by telling systemd-logind to start a user manager for individual users at boot that is kept around after logouts: loginctl enable-linger ${username}

Punching pi-holes

To allow pi-hole to act as a DNS server, I had to stop systemd-resolved from listening to port 53 by setting DNSStubListener=no in /etc/systemd/resolved.conf.

Custom ufw rules

Because podman typically runs as an unprivileged user, using port 53 (which is pretty much required for DNS servers) is out of the question by default. Instead, I've added an iptables redirect via the ufw config – similar to the approach from this blog post. On my server, I added a few lines to the top of /etc/ufw/before.rules, under the "*nat" section:

# redirect port 53 to 5353 (pi-hole)
:PREROUTING ACCEPT [0:0]
-A PREROUTING -p tcp --dport 53 -j REDIRECT --to-port 5353
-A PREROUTING -p udp --dport 53 -j REDIRECT --to-port 5353
COMMIT

To be on the safe side, I also allowed the port: sudo ufw allow 5353

Extra virgin snakeoil

I've symlinked /etc/ssl/{certs,private} and /etc/dovecot/private, pointing to ~caddy/.local/share/certificates/.../max-moser.dev/..., to give postfix and dovecot access to actually valid certs instead of self-signed snakeoil certificates. Give Google Mail & co one reason less to junk-jail your emails.

Some fresh software

Debian and Ubuntu aren't exactly known to be bleeding edge distributions; it's quite common that the versions of packages are a bit dated (if they're available at all). Neither jless nor yazi are available, and the version of rust is too old to compile them from source. Luckily, this can be sidestepped by installing rustup with apt and using that to install the latest version of rust, which in turn can be used to easily cargo install the desired commands.

Editing files on the server

I've found that editing files on the server with good old vim can be a bit of a slog, I actually miss the LSP integration from my nvim setup! Fortunately, sshfs (1) exists and can solve this exact problem. Mounting the server's file system to a local directory server-mount is as simple as this command: sshfs -o follow_symlinks user@server: ./server-mount