This blog post has a few hints [1]. Apparently they had to rewrite a load of code when they moved to a multicore processor, so it definitely seems to be in-house software.
It's absolutely wild to think about a suite of software this sophisticated that exists outside the realm of Unix, Windows, or any of the long-term players in the embedded networking device market. I know there are boutique embedded IP stacks out there but it still boggles my mind that a small company like this has had sufficient revenue to keep up with the "churn" in the networking space for 20+ years w/o leaning on free/open-source software.
Piling-on to my own comment: Searching the site https://www.revk.uk for the keyword "firebrick" returns a ton of results detailing the development. It looks like it's exactly what it says on the tin-- a proprietary operating system with its own implementations of a ton of protocols.
I wish I had a reason to interview RevK about the Firebrick code. I think it would be an immensely interesting topic.
Which kernel are they using?