Fallout from the SolarWinds supply chain attack will plague us for years, even if not in the headlines.
FireEye founder, Kevin Mandia, first broke the news that somebody had attacked FireEye in a Dec. 8, 2020 blog post. Mandia may have suspected this would be just the opening chapter in the most far-reaching cyberattack in history.
The attackers tailored their world-class capabilities specifically to target and attack FireEye. They are highly trained in operational security and executed with discipline and focus. They operated clandestinely, using methods that counter security tools and forensic examination. They used a novel combination of techniques not witnessed by us or our partners in the past.
https://www.fireeye.com/blog/products-and-services/2020/12/fireeye-shares-details-of-recent-cyber-attack-actions-to-protect-community.html
If Mandia suspected the FireEye attack was part of something much larger, he was right.
On Dec. 13, FireEye announced the attackers used a compromised version of SolarWinds Orion to invade its network, and warned that all SolarWinds customers were vulnerable. Microsoft also published a blog post with tips for businesses to protect themselves. The world exploded. Krebs on Security published stories on Dec. 14, Dec. 15, Dec. 16, and Dec. 18. Barrons published a Dec. 14 story summarizing the FireEye and original Microsoft blog posts. The US National Security Council held an emergency meeting. Reuters published its story on Dec. 15. CISA issued a warning on Dec. 17. And on Dec. 18, Microsoft published another blog post with more more technology details.
Press outlets and politicians across the country picked up the story and for a few days, it was everywhere, mixed with 2020 election politics. Researchers who dug deep attributed it to Russia. President Trump wanted to blame the Chinese and tried to tie it to his theory about rigged voting machines.
As usual with news of a major attack, there was lots of noise, but little information. Some got the attack facts wrong. NPR published an article on Dec. 15 that originally said, “The hack hinged on a vulnerability on a software monitoring product from SolarWinds, a company based in Austin, Texas.” To its credit, NPR fixed that mistake in a Dec. 21 update.
On Dec. 26, investigators discovered a second piece of malware inside SolarWinds software, apparently unrelated to the first attack. Both attacks now have names; Sunburst and Solar Flare.
But the popular press never told the public the real reason why all this is a big deal. It might be the biggest big deal we’ve ever seen.
Why It’s a Big Deal
Supply-chain attacks over the internet confuse people. So forget the internet for a minute. Visualize a scene from a favorite spy movie where good guys meet in a room to plan how to save the world. But the good guys don’t know the room is bugged. Cut to another room where bad guys eavesdrop on the good guys and and hatch a plot to spoil the good-guy plans.
Bad guys want to plant bugs in all good-guy rooms. But visiting every good-guy room is not practical, and so bad guys need to get creative. So they take over the light-bulb factory and plant bugs inside every new light-bulb. When good guys change light-bulbs, they plant bugs in their own rooms. That’s a supply chain attack. Instead of attacking targets directly, attack a common supplier and turn those supplies into a weapon.
SolarWinds is the light-bulb factory in my metaphor above. But instead of light-bulbs, SolarWinds builds network management software. Its customers include all US military branches, several US Government departments, most major internet service providers, several colleges and universities, and thousands of the world’s most influential companies.
Around March, 2020, somebody compromised the SolarWinds’ update servers and planted sophisticated malicious software, detailed in the Dec. 13 FireEye blog. And every SolarWinds customer who applied that SolarWinds update brought the attackers inside their networks. From March through mid-December, 2020 when FireEye found them, attackers roamed freely across networks inside many of the most influential organizations in the world.
So, yeah, this is a big deal.
What Went Wrong?
Questions are swirling about how attackers penetrated SolarWinds. In one sign of weak security at SolarWinds, apparently, the Solarwinds build server password was “solarwinds123” and it may have been accessible to the public. But attackers did more than just compromise a build server. Attackers also signed tainted SolarWinds updates with valid digital signatures, which means attackers must have also penetrated the SolarWinds’ signing servers. This Microsoft blog post also suggests attackers were inside SolarWinds as early as October, 2019.
Signing servers and build servers should have been among SolarWinds’ most protected assets. If an attacker could steal access to a signing server and apply good digital signatures to tainted software, what good are digital signatures? That leads to the logical next question, how did attackers penetrate them?
As of this writing, Dec. 27, 2020, answers are scarce. The best SolarWinds has offered is a security advisory with little information, and an SEC filing.
SolarWinds’ executives are silent. But this Bloomberg story says that former security adviser, Ian Thornton-Trump warned SolarWinds’ top managers that “the survival of the company depends on an internal commitment to security.” Thornton-Trump presented a security plan, but everyone ignored it, and so he left. Here is a PDF in case the Bloomberg link goes bad.
Hopefully, SolarWinds’ leaders will come clean, soon. The world needs to know what went wrong and why.
Now What?
The SolarWinds incident is not the first and won’t be the last devastating supply chain attack. Supply chain attacks first burst on public consciousness back in 2017 when Russian attackers compromised the software updates for a Ukrainian accounting package and exploited other Windows vulnerabilities to launch NotPetya. That attack went out of control and crippled companies around the world. NotPetya even made its way back into Russian oil company, Rosneft. I talked about notPetya on WCCO Radio with Roshini Rajkumar shortly after it hit.
Back in 2017, NotPetya flew through global shipping giant, Maersk, and destroyed all its application servers and Windows domain controllers. With no domain controller backups, all that separated Maersk from the stone age was a power outage in Africa. People at Maersk and other victim companies poured superhuman effort into recovering. And then life went on. And the world forgot. But the attackers didn’t forget. The attackers improved their tactics. And in 2020, they found paydirt. They owned many of the world’s most important networks for around nine months.
SolarWinds customers are pouring effort into mitigating the damage. But customers of SolarWinds’ customers are also potentially in trouble. Hostile attackers were inside most major internet service providers, courtesy of tainted SolarWinds updates, which means they could have planted back doors in internet routers, set-top boxes, and elsewhere. Or they could have gone after Zoom or other video meeting services and recorded meeting streams for later decrypting and playback.
This attack touched everyone, which means everyone needs to examine their internet activity since Match, 2020, around the same time COVID-19 forced much of the world to work from home.
Lessons?
The Russians did us a backhanded favor in 2017. We should have learned from it. The attackers learned. We didn’t. But better late than never. Let’s learn from the 2020 SolarWInds attack.
To ensure this never happens again, we all need to step up our security game. Had Maersk segregated its network and backed up its domain controllers in 2017, that outage would have been minor. But without that African power outage to preserve the last good Windows domain controller, recovery would have taken months and maybe even killed Maersk.
Prior to NotPetya, Maersk had plans to improve its security, but nobody’s bonus depended on it, and so it was a low priority. Evidence suggests security was also a low priority at SolarWinds in 2020. This needs to change.
We need to ensure the software updates we apply are good. One tried-and-true defensive tactic – stage updates in a test area first before rolling them out in production. Unfortunately, that isn’t good enough, because the SolarWinds malware stayed dormant for two weeks. SolarWinds’ customers would have tried it a lab, declared it good, and then rolled it out in production as part of a routine patching cycle.
We trust digital signatures to verify updates. But the SolarWinds attack proved digital signatures aren’t good enough. We need something else. Checksums might be helpful. Every software company could publish a checksum for each update; every customer could calculate a checksum and compare it to the published one before applying an update. But if attackers can get deep enough into a supplier’s network to forge digital signatures, they can also forge checksums.
Ultimately, no technology can guarentee integrity. And so, that leaves non-technical tools.
Reputation could do it. Instead of blindly accepting updates, customers could insist on only spending software money with companies who prove their updates are good. The incentives make sense. If an attacker poisons a software update, it could destroy that software company’s reputation. SolarWinds might not survive this debacle, and might not deserve to survive. Perhaps other software companies will watch and learn. But this only works if enough customers with enough spending power demonstrate they’re serious about demanding quality.
Software companies could earn a good reputation by embracing open tactics in their development and build practices. Document every piece of the build process; maybe even make videos of new-version builds and updates for public examination. It would be a gutsy move. But if it earns and keeps trust, it could pay handsome rewards.
Here is a presentation I delivered in 2019 about embracing open security practices. The same concept should apply to building software.
SolarWinds fought embracing open. And now it has a trust problem. Maybe this is an opportunity for SolarWinds to reevaluate its stance on open source methods and open in general.
I look forward to SolarWinds publishing an honest analysis of how attackers penetrated its defenses, and how it plans to ensure this never happens again. A public discussion would help repair trust. But I won’t hold my breath.
I also talked about the SolarWinds attack on AM1280 The Patriot and 830 WCCO Radio.
Recent Comments