Why hot-patching is awesome
Yesterday we released a security fix for UnrealIRCd with no restart needed. It's called hot-patching and we have been doing it for 20 years. I think it's awesome.
IRC is special
As usual, IRC is a bit special. Sessions are normally long-lived TCP/IP connections and, unlike other services, on IRC restarting the server daemon creates quite a mess. Mass disconnects, connects, and so on. That's why not all IRC server admins are too fond of upgrades.
Now, there are workarounds to this if you don't want that mess. You could take a server out of DNS Round Robin so no new users connect to it and then wait for days. You can use load shedding to actively disconnect users over time and make that period shorter. Or accept the pain and try to find a moment where it hurts the least, like at night. But for a network with people from all over the globe there is not really a clear "night".
There's another option: putting code in modules. Let me tell you about how that all started.
Module support
UnrealIRCd, which is an Open Source IRC Daemon written in C, added module support all the way back to the UnrealIRCd 3.2.x days in March 2001. Credit for this goes to Stskeeps (Carsten Munk) and codemastr. I only joined the team a year later. Having code in modules allows adding new features, making enhancements, but also fixing bugs. You simply reload the module on the fly. No user disconnects. Zero downtime. Very neat.
The first hot-patch in 2005
In January 2005, we had a crash issue in UnrealIRCd that could be triggered by any ordinary user. Very bad. In the announcement of the 3.2.2b release I offered a "hot patch" for the first time. It was a shell script and we guided people to download that and run it. It worked great and people responded:
That's how I like patches... Thank you! Good work!
I stuck with the shell script idea for quite a while, until January 2020 when it got integrated in our ./unrealircd CLI tool. So yesterday when we pushed a new release out, we could simply tell admins on older versions to run:
./unrealircd hot-patch webserver-header-dosThe tool downloaded the patch, verified its GPG signature, did a dry-run, applied it, recompiled the code and rehashed the server. Server patched!
How good was hot-patching in the past 20 years?
Now is a good time to look back on how often vulnerabilities in UnrealIRCd's source could be hot-patched and why sometimes we couldn't.
More than 80% of the vulnerabilities could be hot-patched
| Date | Version | Issue | Hot-patch |
|---|---|---|---|
| 2005-01 | 3.2.2b | User-triggerable crash (NULL deref) | Yes |
| 2009-04 | 3.2.8.1 | Buffer overflow in noident handling | Yes |
| 2010-06 | 3.2.8.1 | Backdoor on the download mirrors | No (core) |
| 2013-11 | 3.2.10.2 | Remote crash and read-after-free | No (core) |
| 2015-08 | 3.2.10.5 | Crash when SASL is enabled | Yes |
| 2016-07 | 4.0.5 | Remote crash, flood, IPv6 ban bypass | No (core) |
| 2016-09 | 4.0.6 | SASL certfp spoofing, account takeover | Yes |
| 2016-12 | 4.0.9 | Ghost users and memory leak | Yes |
| 2017-01 | 4.0.10 | Buffer overflow, not exploitable | No (not urgent) |
| 2017-02 | 4.0.11 | ISON overflow, any user could crash | Yes |
| 2017-10 | 4.0.15 | Two crashes, one pre-authentication | Yes |
| 2018-06 | 4.0.18 | Memory leaks and out of bounds read | No (not urgent) |
| 2020-01 | 5.0.2 | Halfop desync, ghost user reads along | Yes |
| 2020-02 | 5.0.3 | Labeled-response flood between servers | Yes |
| 2021-07 | 5.2.1 | Ban exception bypassed kline and gline | Yes |
| 2022-01 | 5.2.4 / 6.0.2 | DCC crash, hits the default config | Yes |
| 2023-12 | 6.1.4 | CVE-2023-50784, websocket overflow | Yes |
| 2026-05 | 6.2.5 | IPv6 /64 bypassed maxperip | Yes |
| 2026-09 | 6.2.7 | CVE-2026-90668, webserver DoS | Yes |
A lot could be hot-patched. Two of them were not urgent enough to warrant a hot-patch, so they were addressed in a regular release. Now, let's walk through the ones that could not be hot-patched.
Two times vulnerabilities could not be hot-patched
Not everything is or can be in modules. Roughly speaking:
- 63% of the code is in modules: all IRC commands, the webserver code, most user and channel modes, IRCv3 extensions, etc.
- 17% is the config file parser and the module API (core)
- 20% is the rest of the core, of which some is end-user-reachable
Some of that 20% still has potential for user-triggerable vulnerabilities.
The first vulnerability that could not be hot-patched was in 3.2.10.2. This was an issue in the SSL/TLS handling in the I/O engine.
The second time that things could not be hot-patched was with several vulnerabilities in 4.0.5. One of them was in the I/O engine again, one was in connection flood handling, and another one was in a user ban matching routine.
The I/O engine is really the core of the IRCd so this will always be difficult or impossible to modularize. The user ban matching routine and most anti-flood code were moved to modules over the years, so are hot-patchable today.
The UnrealIRCd 3.2.8.1 backdoor in 2010
Oh yeah, that embarrassing Some versions of Unreal3.2.8.1.tar.gz contain a backdoor situation. Better known as CVE-2010-2075. This was also a problem in the core, but not a problem with the code in the (CVS) repository. On various mirrors the download tarball was replaced with a backdoored version that allowed anyone to execute shell commands. Ugh, those were not good times.
Funny enough, this is where a lot of security researchers got to know UnrealIRCd from, and it wasn't so much because they used IRC. In 2012 Metasploitable 2 was released, which was a VM running all kinds of vulnerable software on purpose. It was used for training pentesters and it included the backdoored UnrealIRCd. In fact, it lived on in Metasploitable 3 and in HackTheBox machines, which even ended up in OSCP learning material. So a whole generation of pentesters got one of their first shells on UnrealIRCd.
Comparing with other software
Linux kernel
Having code in modules that can be reloaded is not new. The most well known example is probably the Linux kernel, which has it since 1995. Linux kernel modules are very handy but they have one problem if you want to hot-patch an issue: you cannot reload a module if it is in use. That was often a bummer if you wanted to fix things that affected you: after all, the module is almost always in use then.
There was unofficial hot-patching for the Linux kernel starting with the year 2008, it was called Ksplice. Then Oracle took it over in 2011 and made it a paid service. In 2014 Red Hat released kpatch and SUSE kGraft. In 2015, Linux 4.0 came out with Livepatch support and it finally became official. However, it should be noted that all these systems are more complex, they don't really just swap a module, like UnrealIRCd does. They have to do clever but complex things like code swapping and have to ensure that no thread is running in that code at the same time. Still, it's interesting that official hot-patching in Linux arrived in 2015, while we already had it for 10 years in UnrealIRCd.
IRC daemons
In the IRC world various other IRC servers also have modules. In fact UnrealIRCd was not the first. I did some archeological digging to get a timeline:
ejbadded initial module support in ircd-hybrid in November 2000- UnrealIRCd added module support in March 2001
- InspIRCd has module support since the beginning (January 2003)
- Bahamut also added module support in February 2003
The same module system from hybrid lives on today in ircd-ratbox, charybdis and Solanum.
As for InspIRCd, they actually offered module reloading to patch a vulnerability in 2021.
What's here to learn
Is hot-patching useful for everyone? No. A lot of internet applications nowadays work with stateless requests and responses. Retrying is cheap and often safe, so it doesn't really matter if a connection is closed. And, because requests take a short time, you can use connection draining for the upgrade process: a new server takes on new requests and the old server dies off after its last in-progress request finishes.
But for those working on systems that have to deal with long term sessions, where disconnections actually hurt, I think there's a lesson to learn here. Consider making a module API that things can plug into. That's the thing that enables reloading and hence hot-patching. Now I'm going to be honest here: the cost is not zero. We had to create a module API for all kinds of subsystems: commands, user modes, channel modes, extended bans, and so on, and we had to move existing code over from the core to modules. All of this is a lot easier if you design from the ground up. Doing it afterwards, like us, costs a lot more time and effort, but it is doable.
At UnrealIRCd we are not done yet. We're doing well with the many issues we could and did hot-patch... but we can do better. There's still quite a chunk of code in the core. Some of it is hard to move out, but other parts simply haven't been done yet. My plan is to bring the percentage in the core further down, so we can hot-patch more issues if there is ever a problem in those areas.