Don't make the same mistake I did if you're self-hosting a Pi-hole

Self-hosting has been a very fun and rewarding journey for me. I've kind of settled down in terms of trying new services, but my NAS is now powering a lot of cool stuff, including cloud storage, an Office suite, and a streaming server. Recently, I also decided to self-host Pi-hole in order to block ads across my entire Wi-Fi network.

It was all going great until TrueNAS alerted me to a Pi-hole update being available. I tried updating Pi-hole and... my entire network went down. If you know networking, you probably already know what went wrong, but let me explain.

What Pi-hole does

I really should have known

Pi-hole is often touted as an ad blocker, which was the primary reason I wanted to set it up for myself. In more concrete terms, though Pi-hole is a self-hosted DNS resolver. It takes all the DNS queries from devices on your networks and matches them to the appropriate servers, so you can connect to all the websites you usually do.

Typically, your network's router or modem will provide a preset DNS resolver that's somewhere else on the internet. This preset is typically your ISP's DNS, though not always. For example, before I set up Pi-hole, my router would connect to Google's DNS servers (8.8.8.8 and 8.8.4.4). With Pi-hole, after setting up the self-hosted app, you just change the DNS resolver on your router to the IP address of your Pi-hole instance. Pi-hole will use an upstream resolver such as Quad9, Google, or Cloudflare, then mix it with the lists you've added to your Pi-hole. Your PC and any other devices will then make DNS requests to your Pi-hole, and it will respond with requests for a blocked domain with a dummy address, such as "::" for an IPv6 address or "0.0.0.0" for an IPv4 address.

Blocking advertisements and tracking is made possible through the fact that any DNS requests to known ad servers are simply not connected to their intended destination and instead are essentially dropped, so the ads never make their way into your network. It can actually save you a good bit of data usage, too, so the benefits are twofold.

How my network went down

I really should have had a secondary server

A week ago, I was looking at my TrueNAS instance, and pretty much all of my self-hosted apps had updates available. I figured that updating my Pi-hole might cause network issues for a couple of minutes while it updated and devices couldn't contact it, so I left that for last. With that said, I clearly did not anticipate what would happen next. When updating Pi-hole, TrueNAS shuts down the app, pulls the updated package from the repository, and then deploys it. The problem was, after shutting down Pi-hole, there was no DNS resolver to help establish the connection, so TrueNAS was stuck in a loop attempting to download the updated Pi-hole package. Of course, while this all happened, my network was also non-functioning because there was no DNS resolver in the network. To make matters worse, because an update was triggered, I couldn't just start the container back up again.

I say all this in retrospect because at the time, I had no idea what was happening. In a situation like this, you can work around the issue by using a specified DNS server on your devices, but I forgot this was possible, so when I was in a meeting using my phone and a 5G connection, my colleague Adam Conway reminded me of that. So I set Google's DNS servers for my Mac Mini so I could still work, and it was at this point that I showed Adam what error I was getting, and he started laughing when he realized what was happening.

Thankfully, the solution was fairly simple. I simply had to go into my TrueNAS network settings and add a separate DNS entry, like I had done on my Mac Mini, so that the NAS could still resolve addresses. My Pi-hole was my only DNS resolver defined in my DHCP, which my NAS had also dutifully followed. Once I defined an additional, external DNS server on TrueNAS, I was finally able to update Pi-hole and get my network running again.

Redundancy is important

Especially for low-resource services like Pi-hole

Now, you may have noticed that when I referred to Google's DNS servers above, I mentioned two IPs, but with my Pi-hole, there's only one. Indeed. Google provides both a main DNS resolver and a "secondary" one, and really, this is crucial. In many cases, a device will poll both DNS servers to resolve a query, and the faster one to respond will be used, which helps ensure a faster connection. Plus, if one of the servers is backed up or unresponsive, the other can come in clutch.

With Pi-hole, you don't really get that, at least if you only have one instance. The other day, my colleague Adam wrote about how it's a great idea to have a second Pi-hole instance for this very reason. If one of your Pi-hole servers stops responding, your network can continue running as normal, as the other one will take up the brunt of your entire network in order to resolve all of your queries. Now, this isn't mandatory, but there are clear benefits, as I found out during this whole incident.

You could just use a cloud-based DNS resolver like Google as a backup, but if your goal is to block ads, any time the secondary server is used, that won't happen because that server will still serve ads as usual. So, I set up my single Pi-hole instance without any redundancy; there was no backup server that my devices could fall back on. And then the update happened, and of course, everything went sideways. To avoid issues like these, Adam told me that when his Pi-hole was deployed on TrueNAS, he specified Cloudflare's DNS in his TrueNAS network settings. In his experience, his TrueNAS instance (according to his Pi-hole log) never accesses blocked domains anyway, and defining an external DNS provider at the OS-level avoids any other unforeseen issues. Imagine being away, trying to remotely access your server, and then discovering that you can't connect because Pi-hole suddenly started acting up?

While I personally haven't had it happen yet, your main Pi-hole instance may slow down or run into issues (and there are many reports of this with Pi-hole V6), so having a secondary DNS server is important for redundancy reasons. As Adam noted, a Pi-hole uses very little by way of resources, so deploying two of them on separate machines, if you have the resources, can help ensure your network always stays functional. And, as I learned, this is especially true when you're updating one of your Pi-hole instances.

Self-hosting can be tough for newbies

Self-hosting is certainly a fun and rewarding journey, but it's also a challenging one if you don't look into things ahead of time or if you don't know what you don't know. Issues like what I had with Pi-hole are bound to happen, but in the end, it's still worth the trouble. Having no ads served while I'm on Wi-Fi — plus all the other benefits of self-hosting — has been totally worth it, and I highly recommend it. And while I'm using it on a NAS, you can use a cheap Raspberry Pi (or even an old phone) to host Pi-hole at a much lower cost overall.