A flurry of articles went out today on security concerns in open networking and the ONIE (http://onie.org) open source project, sparked in anticipation of Gregory Pickett’s upcoming talk at Black Hat: “Staying Persistent in Software Defined Networks.”
As active members of the security community, we’ve worked with Greg in the past (see his talk last year), so he was kind enough to reach out recently to discuss this issue when he first observed the concern. As part of our discussion, Greg agreed with me that the issue is not SDN (or even networking) specific, and actually affects other open networking switches, non-open networking switches as well as servers (see BIOS hacking on servers for the same attack vector).
As a result, it’s not an actionable security issue for more than 99% of customers. Similar to what’s on the server side, we also discussed some interesting trusted hardware-based (e.g., TPM) solutions that we are developing for these extreme network security needs. Security has always been paramount for Big Switch Networks as evidenced by our focus on hitless upgrades for rapid deployment of security patches and the fast turnaround we provide to our customers for security vulnerabilities like the recent OpenSSL issues. Let me provide some more detail on the issue Greg has identified and also describe the initiative to develop a hardware based solution for the most security conscious customers.
What is ONIE?
ONIE is a piece of software that ships with bare metal switches from the factory. When the switch boots up for the first time, ONIE software activates the switch's management port and helps it download a switch OS (e.g., Big Switch's Switch Light, Cumulus Linux, or any third-party OS) from a server, typically on the data center management/ILO network. The ONIE software itself is open source, has been contributed to Facebook's Open Compute Project, and has increasingly been a part of customer requirements as it dramatically simplifies the deployment of large numbers of switches.
In other words, ONIE is very similar to PXE. ONIE provides the same security approach for networking that PXE offers in the server world: it’s a tool to auto-install and provision open networking switches in the same zero-touch manner that the server folks have been doing for years. And once a network operating system is booted, if it’s compromised, it’s possible to overwrite and abuse any persistent system code, be it ONIE, a lower level boot loader (like uBoot), or a system level BIOS. The issue Greg will be discussing at Black Hat is not unique to SDN, open networking, non-open networking devices, or even servers.
Air Gap The Management Network: Standard Practice
If this issue affects all devices on the network, why isn’t it a bigger concern? With all of our customers, standard practice is to deploy an independent and physically separate management network that is part of the data center infrastructure plumbing. You may notice that most switches have a dedicated management port, separate from the high bandwidth “dataplane” ports. Most servers also have a small management port connected directly to the motherboard, separate from the powerful 1G/10G/40G NICs that carry application traffic. The management network in all of our customers runs on cables and switches that are physically separate from the rest of the data center to achieve strong isolation needed for security. The management network is used to bootstrap the plumbing of the data center, typically during initial install, capacity increases or infrastructure software upgrades, and is a small fraction of the size, speed and complexity of the primary data center network.
In every one of our customers, the management network has strong separation (e.g., non-routable IPs + firewall +IDS, air gap, etc.) from the rest of the data center. The number of people who can access it, the services that run on it, and the PXE and ONIE software that can access it, are by design extremely limited. The purposeful simplicity and physical separation of these management networks running PXE and ONIE traffic typically lend a level of security via good practice rather than additional software.
Just like there are organizations or deployment scenarios which are so concerned about security that they won’t run PXE- the standard out-of-the-box ONIE may not be right for a small number of organizations that must maintain an extreme risk-averse profile and Stuxnet-level security concerns. We're working towards a hardened version of ONIE to be used in the same environments as secure PXE in the future. Using already built-in trusted hardware, like the popular TPM chips, the hardware would verify that ONIE and the network operating system match cryptographically signed signatures, i.e., that they have not been compromised.
In summary, these same security issues and solutions exist in the server world. For the small percentage of customers who run with trusted hardware operating systems, we will match that high-level of security with hardened ONIE. But for the vast majority of customers who don’t, ONIE absolutely meets their security needs.
– Rob Sherwood, CTO
– Kyle Forster, Co-Founder