Keeping your home network or home lab safe is essential to securing private data and your personal devices. While I've got a prebuilt hardware firewall on my main network these days, my home lab often has a custom firewall built to test plugins and new configurations. It's a good way to learn cybersecurity best practices, how intrusion detection and intrusion prevention systems (IDS/IPS) work together to spot unauthorized traffic, and how networking works underneath it all.
That level of granularity and control is fantastic once you know what you're doing, but it's also a ticking time bomb in the wrong hands with incorrect settings. Not to dissuade anyone, as installing a dedicated router OS or building a router from a bare-bones Linux installation teaches a lot about security and routing. Still, as with any high-level skill, a lot could go wrong, and I wish I hadn't needed to learn these points the hard way.

Related
5 reasons you should use an old PC as a dedicated firewall in your home
Why waste money on an enterprise firewall when you can build your own from old PC parts?
Your time is worth more than you think
Setting up a custom hardware firewall can be easy, depending on how many things go wrong and how long it takes to search for the answers. There is always a hidden cost, and if you can't see the cost up-front, it'll either be a subscription that you'll find out about after installing everything, or something similar. And if you can't see the hidden cost anywhere, you are the cost, both in time invested in searching for forum answers or reading manual pages and how-tos, so you get a grasp on the mechanics behind everything you're building.
Suppose you're using old PC hardware to install your firewall on. In that case, you'll need at least one more PCIe Ethernet card, and possibly two, because the Realtek NICs used on many motherboards don't work correctly on the operating systems used for firewalls. Intel NICs are still the easiest to get running, as are Intel CPUs. The firewall software might be free to use. Still, things like timely security updates might be on a subscription model, or dedicated support that isn't forum-based, and many of the security plugins you'll want to use also have subscriptions for faster upgrades.

Related
Should you build your own router with OPNsense or pfSense?
If you're wondering whether you need to use a custom platform to manage your internet, here's everything you need to think about.
4 I'm the weakest link
Did you know 95% of firewall breaches are due to user error?

The hardware and software used to create your firewall are only part of the equation. You, as the system admin, are responsible for how well that firewall works, from how it handles suspicious traffic to which devices can access the network, which protocols can be used to pass through, which devices on the network can talk to each other, and so on.
Misconfigured settings in the packet-filtering stage could grant all traffic free rein, overriding any rules you've painstakingly set up. Other overly permissive settings could let Remote Desktop Protocol through the firewall, when it really should be disabled unless necessary. You have to figure out how file-sharing applications will work, and which devices are allowed to access them.
If you're going to use Telnet, SSH, or FTP, will you change the ports allowed and only allow access when inside the secured network? It's also a fantastic idea to set something up that reads the firewall logs and sends you reports of unauthorized devices or irregular traffic coming from approved devices, so that you can drill down and see what the matter is.

Related
5 essential firewall rules every home lab should have
Before you start playing with services and tools, here's what you should set up first.
3 You'll have compatibility issues
Hardware bottlenecks and driver incompatibilities abound
While sourcing the hardware for your dedicated firewall, there are a few hardware components that you absolutely should not skimp on. The first is high-quality NICs, because all your network traffic will flow through these, so they need to be up to the task. Intel-based NICs are better, because they're more likely to have drivers that work on FreeBSD and/or Linux, while Realtek NICs are pretty much not supported. You'll also want at least two NICs, because while you can use a network switch to get more ports on the LAN side, you need a WAN and LAN port for the firewall to work. You could get more NICs to add LAN ports, but you could run out of PCIe slots, and it's often cheaper to get a network switch and do it that way.
Advanced features require single-threaded performance, because the code is mostly written for single-threaded use, although some security features now use multi-threading by default. It's still better to assume single-threaded speeds are key, even if the CPU you pick has multiple cores to work with. Again, Intel CPUs are better supported on both FreeBSD and Linux, so you're better off looking in that direction. It doesn't have to be an expensive CPU, though. A Pentium Gold 8505 can be fine if you're dealing with gigabit connections, although you might want a Core i3 or i5 if you're using 10GbE at home.
The last thing is that RAM doesn't have to be fast, but you should aim for a decent amount. If we look at hardware firewall manufacturers or the operating system requirements, you can be safe with 4GB of RAM, but I'd opt for 16GB or even 32GB if you can fit it into your device. RAM is a cheap upgrade, and having a little extra over what you think you need is always better.

Related
5 reasons I virtualize OPNsense in Proxmox
It may seem strange, but virtualizing your router and firewall can be better than running it on bare metal.
2 Maintenance never stops
Better block some calendar time for updates
One of the first tasks after installing your firewall software is to update the firmware and the software packages and plugins it ships with. But this won't be the last time you have to do this, and you'll want to keep on top of updates to ensure that security bugs are patched so your firewall is working at optimal levels. OPNsense and a few other firewall OSes notify you when you log into the management pages, so set a timer to remember every month, and you'll be fine.
But it's not as simple as just checking on system updates. Any plugins need checking as well (if your firewall OS doesn't do that all at once), and it's good to do periodic speed tests and other checks on the physical network wiring to make sure nothing is out of place or becoming damaged.

Related
You can virtualize your router and firewall - here's how
Virtualizing a router isn't anywhere nearly as difficult as you might think, provided you've got some spare network interfaces
1 Advanced features aren't optional
You'll need some plugins for a layered approach to security
The operating system on your firewall is fine for basic usage, but to get the most out of it, you'll want to add some advanced features with plugins. Think about how many other programs you install on your computer or phone to use daily, alongside the basic install. That's the same with your firewall, except the plugins and tools you add will be security-focused.
You'll want an IDS/IPS, whether Zenarmor, Suricata, Snort, or any of the other commonly used packages. You'll want a monitoring solution, whether ntoping or others. CrowdSec stops known threats from being able to scan your external ports, and you can add remote access tools like Tailscale or NetBird, reverse proxies like Nginx, or any useful plugins. Some of these will come with associated subscription costs to get the best version of the security rules, but it's worth it if they protect your home network from intrusion.
Being your own sysadmin is a rewarding task, but it does take up a lot of time
Building (and maintaining) your own firewall is a fairly hefty task, but it's not out of reach of anyone given the inclination and time to learn. Every hour you spend configuring, troubleshooting issues and installing hardware components gets you a better understanding of how networking works, which is highly valuable. You'll end up with a more secure home network because of it, as long as you double-check every setting and keep backups in case of an emergency.