Bloomberg Businessweek has dropped a bombshell: Chinese intelligence — agents of the People's Liberation Army — forced factories in China to add tiny spy chips to server boards being manufactured for industry-leading Super Micro, to be sold to industry giants like Apple and Amazon. Their boards and servers literally provide the hearts and minds for many of the world's data centers, large and small. And, the report says, they've been hacked at the hardware level.
December 11, 2018: Super Micro: No 'Big Hack' malicious chips found in motherboards
A third party audit of Super Micro motherboards, old and new, has found zero evidence of the 'big hack' hardware spy chips Bloomberg alleged were sold to Amazon, Apple, and dozens of other tech companies.
From Reuters:
Computer hardware maker Super Micro Computer Inc told customers on Tuesday that an outside investigations firm had found no evidence of any malicious hardware in its current or older-model motherboards.
Given Apple and Amazon's strong denials and the lack of any corroborating reporting from other outlets like The Washington Post or the New York Times, this is looking worse and worse for Bloomberg.
October 7, 2018: Named source in "The Big Hack" has doubts about the story
A new episode of RISKY.BIZ reveals that the ' "Big Hack" technical source Joe Fitzpatrick has concerns about Bloomberg's reporting...'
October 7, 2018: Apple VP of Information Security tells Congress no signs of Big Hack
https://twitter.com/reneritchie/status/1049329985968922625
From Reuters:
Apple Vice President for Information Security George Stathakopoulos wrote in a letter to the Senate and House commerce committees that the company had repeatedly investigated and found no evidence for the main points in a Bloomberg Businessweek article published on Thursday, including that chips inside servers sold to Apple by Super Micro Computer Inc (SMCI.PK) allowed for backdoor transmissions to China."Apple's proprietary security tools are continuously scanning for precisely this kind of outbound traffic, as it indicates the existence of malware or other malicious activity. Nothing was ever found," he wrote in the letter provided to Reuters.
October 6, 2018: DHS says it has 'no reason to doubt statements' on Big Hack from Apple & Amazon
From the U.S. Department of Homeland Security:
The Department of Homeland Security is aware of the media reports of a technology supply chain compromise. Like our partners in the UK, the National Cyber Security Centre, at this time we have no reason to doubt the statements from the companies named in the story. Information and communications technology supply chain security is core to DHS's cybersecurity mission and we are committed to the security and integrity of the technology on which Americans and others around the world increasingly rely. Just this month – National Cybersecurity Awareness Month – we launched several government-industry initiatives to develop near- and long-term solutions to manage risk posed by the complex challenges of increasingly global supply chains. These initiatives will build on existing partnerships with a wide range of technology companies to strengthen our nation's collective cybersecurity and risk management efforts.
October 5, 2018: Former Apple General Counsel, Bruce Sewell: Nobody at the FBI knew what the SuperMicro story was about
Bruce Sewell retired earlier this year after a long and successful career culminating in his time as Apple General Counsel. Here's what he had to say about the Super Micro story as reported by Bloomberg.
From Reuters:
Apple's recently retired general counsel, Bruce Sewell, told Reuters he called the FBI's then-general counsel James Baker last year after being told by Bloomberg of an open investigation into Super Micro Computer Inc , a hardware maker whose products Bloomberg said were implanted with malicious Chinese chips."I got on the phone with him personally and said, 'Do you know anything about this?," Sewell said of his conversation with Baker. "He said, 'I've never heard of this, but give me 24 hours to make sure.' He called me back 24 hours later and said 'Nobody here knows what this story is about.'"
According to Bloomberg, the hardware hack was discovered when Amazon decided to buy Super Micro customer, and streaming video disruptor Elemental Technologies, but first had sample servers sent to Canada for a security evaluation.
The result, again according to Bloomberg:
Nested on the servers' mtherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn't part of the boards' original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental's servers could be found in Department of Defense data centers, the CIA's drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines.
If true, it's impossible to downplay the severity of this: Compromised steaming servers running in the centers of not only the world's biggest technology companies but the intelligence and defense apparatus of the U.S. Government.
(Bloomberg doesn't state whether any other countries use these servers in similar ways but, given Super Micro's position in the market, it's difficult to imagine they don't.)
Now, hardware attacks are nothing new. We've seen everything from Juice-Jacking, which compromised USB ports to inject malware into any device that tried to connect to them, to interception attacks where agencies, including U.S. intelligence agencies according to Edward Snowdown, grabbed devices during transit and compromise them before they got to their destination.
What this alleges, though, is deeper and far wider ranging than any of that.
Here's how the attack supposedly worked:
- A Chinese military unit designed and manufactured microchips as small as a sharpened pencil tip. Some of the chips were built to look like signal conditioning couplers, and they incorporated memory, networking capability, and sufficient processing power for an attack.
- The microchips were inserted at Chinese factories that supplied Supermicro, one of the world's biggest sellers of server motherboards.
- The compromised motherboards were built into servers assembled by Supermicro.
- The sabotaged servers made their way inside data centers operated by dozens of companies.
To get the chips into the motherboards, Bloomberg says an ages-old bride/threat model was used. Plant managers at the factories where production had been outsourced were offered money and, if that didn't work, threatened with business-closing inspections.
And here's what Bloomberg says they did:
In simplified terms, the implants on Supermicro hardware manipulated the core operating instructions that tell the server what to do as data move across a motherboard, two people familiar with the chips' operation say. This happened at a crucial moment, as small bits of the operating system were being stored in the board's temporary memory en route to the server's central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects.Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device's operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.This system could let the attackers alter how the device functioned, line by line, however they wanted, leaving no one the wiser.
There's been some debate about the technical accuracy and acumen of Bloomberg's reporting. So much so, with something this important, I wish they'd engaged a high-level information security expert as technical editor before publishing.
Whether a chip, as described, can do what's being described and whether or not the group being described could produce such a chip are among the debate topics.
Bloomberg alleges these compromised broads found their way into over 30 U.S. companies, including banks, U.S. military and defense agencies, Amazon, and similarly right up there in the headline, Apple.
Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally. Government investigators were still chasing clues on their own when Amazon made its discovery and gave them access to sabotaged hardware, according to one U.S. official. This created an invaluable opportunity for intelligence agencies and the FBI—by then running a full investigation led by its cyber- and counterintelligence teams—to see what the chips looked like and how they worked.
Now, issues between Apple and Super Micro have been reported before.
In February of 2017, The Information wrote:
In early 2016, Apple discovered what it believed was a potential security vulnerability in at least one data center server it purchased from a U.S.-based manufacturer, Super Micro Computer, according to a Super Micro executive and two people who were briefed about the incident at Apple. The server was part of Apple's technical infrastructure, which powers its web-based services and holds customer data.Apple ended up terminating its yearslong business relationship with Super Micro, according to Tau Leng, a senior vice president of technology for Super Micro, and a person who was told about the incident by a senior infrastructure engineering executive at Apple. The tech giant even returned some of Super Micro's servers to the company, according to one of the people briefed about the incident.There is conflicting information about the exact nature of the vulnerability and the circumstances surrounding the incident. According to Mr. Leng, an Apple representative told its account manager at Super Micro via email that Apple's "internal development environment was being compromised" because of firmware it downloaded to certain microchips within servers it had bought from Super Micro.
At the time, Apple's response to The Information was:
Apple was "not aware of...infected firmware found on the servers purchased from this vendor."
The servers were described as being used by the Apple-aquired Topsy Labs team to improve App Store and Siri Search, something echoed by Bloomberg.
Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.
Why Apple would wait so long to take action, given the severity of the circumstances alleged, isn't addressed by Bloomberg.
Apple's response to Bloomberg was, in a word, savage. I've been covering Apple for a decade and I can't recall ever seeing anything as aggressive or encompassing as this.
Here's what Apple shared with me and other outlets — and, yeah, I know, so much reading so far.. so much… but this is important and really has to be presented in full to be understood in full:
Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg's story relating to Apple.On this we can be very clear: Apple has never found malicious chips, "hardware manipulations" or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement.In response to Bloomberg's latest version of the narrative, we present the following facts: Siri and Topsy never shared servers; Siri has never been deployed on servers sold to us by Super Micro; and Topsy data was limited to approximately 2,000 Super Micro servers, not 7,000. None of those servers have ever been found to hold malicious chips.As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections. We did not uncover any unusual vulnerabilities in the servers we purchased from Super Micro when we updated the firmware and software according to our standard procedures.We are deeply disappointed that in their dealings with us, Bloomberg's reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.While there has been no claim that customer data was involved, we take these allegations seriously and we want users to know that we do everything possible to safeguard the personal information they entrust to us. We also want them to know that what Bloomberg is reporting about Apple is inaccurate.Apple has always believed in being transparent about the ways we handle and protect data. If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement. Apple engineers conduct regular and rigorous security screenings to ensure that our systems are safe. We know that security is an endless race and that's why we constantly fortify our systems against increasingly sophisticated hackers and cybercriminals who want to steal our data.
Apple has since greatly expanded on that, including denying any gag order or secrecy obligation is in place, in a Newsroom post.
Just as I was about to post this, Amazon also pushed out a refutation every bit as aggressive and encompassing. I'll spare you the full text of that, but will share the best part here and link to the full statement above.
There are so many inaccuracies in this article as it relates to Amazon that they're hard to count. We will name only a few of them here. First, when Amazon was considering acquiring Elemental, we did a lot of due diligence with our own security team, and also commissioned a single external security company to do a security assessment for us as well. That report did not identify any issues with modified chips or hardware. As is typical with most of these audits, it offered some recommended areas to remediate, and we fixed all critical issues before the acquisition closed. This was the sole external security report commissioned. Bloomberg has admittedly never seen our commissioned security report nor any other (and refused to share any details of any purported other report with us).
Here you have what should be one of the most respected business publications in the industry with a years-long report that, presumably, had it's fact checkers fact checks fact checked, and on the other side, the biggest tech companies in the world, public companies that are subject to the SEC and shareholder lawsuits, issuing statements that contradict it in the strongest terms possible.
About the only thing everyone agrees on is that there's no evidence any customer data — any of our data — has been compromised.
Now, just as I pointed out The Information had previously reported on Apple and Super Micro, I'd be remiss if I didn't point out that Bloomberg has gotten Apple wrong in the past, including and especially its reports that iPhone X wasn't selling — something that I called at the time a failure verging on malpractice that, combined with similar coverage from similar outlets, needed to be carefully vetted for potential market manipulation by the usual hedge fund suspects.
Bloomberg also holds the distinction of drawing the previous aggressive PR response record when it claimed Apple had sacrificed Face ID security in order to increase manufacturing yields. Something that was almost Steve Jobs-ian in its terse fury.
So, where does this leave us?
One, Bloomberg could have gotten this catastrophically wrong. Through some mix of broken telephone, rumor mutation, and the constant need to get Apple into headlines, the story as written could have elements of truth but in broad strokes and details simply not have gotten it right. For a major publication, that would be a bloody nose to say the least. Though, we now live in a day and age where previously career-ending incidents sometimes aren't even remembered a few hours later.
Two, Apple and Amazon could be lying. A gag order would result in no comment, compartmentalization — where executives know things PR does not — may fly for a standard rebuttal but not anything as extreme as we're seeing. This isn't PR in the dark. This is PR unleashed, Kraken style. They're not even parsing words or hiding attribution. They're closing holes and stamping their names. And, as public companies, that's more than risking a bloody nose. It's risking the liver shot of federal investigation and civil lawsuits. There's no crime that we know of here to cover up. Apple, Amazon, and others are victims. No risk assessment makes that make sense.
Three, something else entirely could be going on. As with iPhone X sales reports being manipulated for stock shorting purposes, there could be elements at play trying to manipulate companies, markets, and sentiments in aid of or againt anything and everything from trade agreements to security agendas. That's an incredibly conspiratorial stance to take on any of this, but given how media can and will be manipulated these days, it's better leaving nothing on the table.
No matter what you choose personally to believe, the risk is so great here because eventually the truth will come out. If there is or was an FBI investigation, that will come out. And that's where none of this makes any sense.
I'm an optimist. I like to believe Bloomberg would fact-check the hell out of all of this before printing world one. That they would have it cold. But I also like to believe no public company would risk refuting it this strong if they weren't dead sure it was wrong.
The various accounts can't be reconciled. There are no multiple truths here. Someone got it wrong under circumstances where getting it wrong is catastrophic.
