On the trail of the SuperMicro BMC hack

By now most people in the infrastructure business will know about Bloomberg’s story about the alleged hacking of Super Micro hardware by the Chinese.
But what’s really going on here? As this story unfolds, what do we know so far?
First of all let’s assume we know that the Chinese / PLA is involved in hardware hacking for purposes of espionage. China produces a very large part of the computer hardware that is currently running in the data centers of the world. A large chunk is still manufactured in Taiwan (another good reason for China to invade, I guess) and elsewhere, but still. Hardware hacking has always been a part and parcel of the spook agencies’ toolbox, there is no reason to assume the Chinese would not try to leverage the incredible advantage they have here. Assuming the opposite, that PLA is not involved in hardware hacking for intelligence purposes, would not be credible IMHO.

Here is a couple of previous examples of hardware hacking for intelligence gathering purposes. And here’s a random peek into the war among spooks that is apparently going on on a daily basis. This is most certainly a big issue, and something that is being taken very seriously.

But what’s up with Super Micro? For many years Super Micro was considered a “white-boxer”, a server provider for outfits that did not want to pony up the extra dough to get “real” brand-name servers from HP or Dell. In recent years this has changed. The big “hyperscalers”, i.e. Amazon, Google and Microsoft long ago started eating into enterprise server sales – a natural consequence of servers moving into big public clouds. However this struck enterprise servers vendors like HP and Dell extra hard, because the hyperscalers at the same time stopped buying servers from them, and instead started purchasing directly from “white-boxers” or simply had Taiwanese manufacturers produce servers directly from their own designs. This also to some extent changed the perception of white-boxers like Super Micro. If their servers were good enough for the hyperscalers, why would they not suffice for enterprise data centers? So Super Micro has been on the rise, mostly at the expense of HP and Dell. Just to add to the array of possible conspiracy theories, Super Micro is an American / Taiwanese corporation according to Wikipedia. I am not sure what this means, but most likely it means that Super Micro’s three founders are Taiwanese, that manufacturing is done in Taiwan and China and sales in the US. Then after years of meteoric rise, something strange happened. In 2018 Super Micro got hit by an SEC audit/investigation after several years of messy bookkeeping, and in August it reported that it would be unable to file its 10-K annual report, and probably would be delisted from Nasdaq, whereupon the stock plunged 21%.

Like all modern servers, servers from Super Micro include a BMC for remote access and installation, without requiring physical access to the server. The BMC has been a ticking bomb for many years. In fact the BMC’s have in general been so completely compromised, that just succesfully getting a single remotely accessible computer on the management network (VLAN) of a major player like Apple, Amazon or Facebook would likely be an epic win for any intelligence agency in the world. So, PLA is most probably actively trying to infiltrate and hack BMC hardware and firmware of all server manufacturers with this exact purpose, and has probably succeeded in doing just this several times – we usually just dont’ hear about it for obvious reasons. According to Bloomberg, Super Micro’s portal for download of BMC firmware was apparently hacked in 2015. Rather than, or in addition to, hacking the BMC firmware of servers, which are in most places overwritten anyway, with the newest firmware downloaded from the vendors website as part of the installation procedure in datacenters, apparently the PLA or some other player, had successfully replaced the firmware downloads on Super Micro’s portal with hacked versions. For how long this had been going on is hard to tell, but the number of infected servers must have been, and probably still is, massive – downloading and flashing your BMC firmware is a very common operation in data centers, especially upon deployment. We only have Bloomberg’s word for this pretty epic hack, but at least Facebook and Apple seem to confirm it. Whether the website was in fact hacked we do not know – an operator could have found some other way of placing infected firmware on the site. Most likely Super Micro does not write the firmware themselves, but acquires this from subvendors. Super Micro’s own refutal of the hardware hack seems to confirm, that Super Micro inadvertently got and spread the infected firmware from a subvendor:

Furthermore, Super Micro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.

To me this sounds awfully much like “OK, we were hacked, but look at how this works – we were probably not the only ones”. If that is indeed the case, well, Houston, we may have an even bigger problem on our hands. It may be, that BMC firmware hacking and associated hardware hacking is all but very common. It may be, that so much firmware is infected that some intelligence bureau is simply drowning in information. The “chip” in Bloombergs Big Hack story may simply be a “marker” implanted in only the very interesting batches of motherboards destined for interesting locations, and the hacked firmware may be instructed to only bother phoning home, if this marker is detected. A lot of maybes obviously. What does seem evident though is, that Super Micro, and specifically the firmware in their BMC’s appears to have been the battle ground for some unknown intelligence agencies. Maybe the NSA was aware of the firmware being hacked, maybe they used this to feed the PLA false information, who knows? What we do know is who appears to have taken the biggest blow from this so far. Super Micro’s stock never did get delisted, and is still actively traded on Nasdaq. However they certainly appear to have lost Amazon, Apple and Facebook as customers, and after the “Big Hack” article by Bloomberg, their stock dropped a further almost 50%. This has got to be excellent news for some, at the very least for HP and Dell.