Dec 8, 2021 6:23 PM

A Year After the SolarWinds Hack, Supply Chain Threats Still Loom

The Russia-led campaign was a wake-up call to the industry, but there's no one solution to the threat.

A year ago today, the security firm FireEye made an announcement that was as surprising as it was alarming. Sophisticated hackers had silently slipped into the company's network, carefully tailoring their attack to evade the company's defenses. It was a thread that would unspool into what is now known as the SolarWinds hack, a Russian espionage campaign that resulted in the compromise of countless victims.

To say the SolarWinds attack was a wake-up call would be an understatement. It laid bare how extensive the fallout can be from so-called supply chain attacks, when attackers compromise widely used software at the source, in turn giving them the ability to infect anyone who uses it. In this case, it meant that Russian intelligence had potential access to as many as 18,000 SolarWinds customers. They ultimately broke into fewer than 100 choice networks—including those of Fortune 500 companies like Microsoft and the US Justice Department, State Department, and NASA.

Supply chain attacks aren't new. But the magnitude of the SolarWinds crisis significantly raised awareness, sparking a year of frantic investment in security improvements across the tech industry and US government.

“If I don’t get a call on December 12, I’ll consider that a success,” says SolarWinds president and CEO Sudhakar Ramakrishna. On that date a year ago, SolarWinds itself learned that Orion, its IT management tool, was the source of the FireEye intrusion—and what would ultimately become dozens more. Ramakrishna did not yet work at SolarWinds, but he was slated to join on January 4, 2021.

While this week marks the one-year anniversary of cascading discoveries around the SolarWinds hack, the incident actually dates back as early as March 2020. Russia's APT 29 hackers—also known as Cozy Bear, UNC2452, and Nobelium—spent months laying the groundwork. But that very dissonance illustrates the nature of software supply chain threats. The hardest part of the job is upfront. If the staging phase is successful, they can flip a switch and simultaneously gain access to many victim networks at once, all with trusted software that seems legitimate.

Across the security industry, practitioners universally told WIRED that the SolarWinds hack—also called the Sunburst hack, after the backdoor malware distributed through Orion—has meaningfully expanded understanding about the need for transparency and insight into the provenance and integrity of software. There had certainly been other impactful software supply chain attacks before December 2020, like the compromise of computer cleanup tool CCleaner and Russia's infamous distribution of the destructive NotPetya malware through the Ukrainian accounting software MEDoc. But for the US government and tech industry, the new campaign hit especially close to home.

“It definitely was a turning point,” says Eric Brewer, Google's vice president of Cloud Infrastructure. “Before I would explain to people that the industry has a challenge here, we need to deal with it. And I think there was some understanding, but it wasn’t very highly prioritized. Attacks people haven’t seen directly are just abstract. But post-SolarWinds that message resonated in a different way.”

That awareness has also begun to translate into action, including building out the software equivalent of ingredient lists and ways to better monitor code. But it's slow work; the supply chain problem requires as many solutions as there are types of software development.

Keeping tabs on proprietary systems like MEDoc and Orion is challenging because security tools need to foster transparency and validation without exposing competitive secrets or intellectual property. The problem becomes especially complicated for open source software, where developers are often volunteers and projects may not have stable funding—if they're even maintained at all anymore. On top of that, developers often repurpose useful chunks of open source code, which in turn means that a supply chain attack that compromises an open source tool could push malicious updates to far-flung systems. Or tainted code could circulate freely online and get pulled into other software without a second thought.

An executive order in mid-May was one tangible sign of progress. The Biden White House addressed numerous aspects of government cybersecurity, with a specific section dedicated to the supply chain. It outlined requirements for federal agencies to generate guidelines, conduct evaluations, and implement improvements.

“The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors,” the order states. “There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended.”

The US government has a poor track record when it comes to actually following through on fixing its cybersecurity weak spots. But Dan Lorenc, a longtime software supply chain security researcher and CEO of the startup Chainguard, says he's been pleasantly surprised to see federal agencies actually adhering to the timelines set by the White House, perhaps an early indicator that the software supply chain security epiphany will have some staying power.

“I think the White House set some very aggressive time frames, which raised eyebrows both in the private sector and among government agencies,” says Allan Friedman, a senior advisor and strategist at the Department of Homeland Security's Cybersecurity and Infrastructure Security. “But I think because it has been such a clear priority, agencies have been able to meet the deadlines thus far, and I think it’s also helped the broader software community understand that the whole administration is serious about this.”

The federal software supply chain security initiative also has a major focus on public-private cooperation. At a White House cybersecurity meeting with major tech companies at the end of August, Google announced $10 billion in security investment over five years, listing software supply chain as a high priority focus. Brewer and his colleagues, for example, have spent several years working on a project called OpenSSF, a scorecard framework that allows developers to assess the potential risks of open source software. Other initiatives from companies like GitHub, which is owned by Microsoft, aim to automatically spot security vulnerabilities and other weaknesses in open source projects. A decentralized project known as Sigstore, launched in June, is working to make it simple for open source projects to implement “code signing," an important integrity check used in proprietary software that open source projects often omit. And researchers at Google also created a software supply chain integrity framework for developers known as SLSA (pronounced “salsa”).

“It’s been a crazy year,” says Chainguard's Lorenc, who previously worked at Google and worked on Sigstore and SLSA. “After the SolarWinds incident it almost was a night and day shift in awareness and momentum. Last December and January were a huge wake-up moment, and there was a lot of panic with everyone trying to figure out what to do. But ultimately that’s better than nobody paying attention at all.”

CISA has been working to expand a 2018 project to develop and popularize “SBOMs,” or software bills of materials. The idea is to create a sort of “nutrition facts” reference for software that provides insight and inventory about what’s in a finished product and what potential exposures it may have as a result. And the May executive order specifically mandates that the National Institute of Standards and Technology develop guidelines for SBOMs.

Next week, CISA will host a virtual “SBOM-a-rama” event as part of its efforts to facilitate public-private collaboration on software bills of materials.

“This is Cybersecurity 101; the most basic thing you can do is say, ‘what do you have?’” says CISA's Friedman. "If you’re thinking about software, there’s typically not enough information to know what’s under the hood. We don’t have the data. No one can instinctively look for the allergens on the ingredient list. But we’re already seeing organizations and startups building out the tools.”

SolarWinds CEO Ramakrishna says the company itself has undergone a massive security overhaul this year, changing the way it approaches its own internal security, reexamining how it interconnects with partners and customers, and taking steps to promote software supply chain security best practices. The company has particularly embraced open source as a way to bring added transparency and flexibility in its own supply chain.

Even with all of these initiatives and improvements across the industry, software supply chain insecurity is still a very real and current problem. For example, a breach this spring that compromised a software development tool from the company Codecov impacted hundreds of the firm’s customers, and a hack of the IT managed services provider Kaseya spawned a number of damaging ransomware attacks in July. In recent years, numerous open source projects have been compromised.

Meanwhile, the attackers behind the SolarWinds intrusion haven’t been resting on their laurels. Nobelium has continued to target prominent companies, government entities, and nonprofits in the US and around the world for espionage. Throughout 2021, the group has mounted aggressive phishing attacks and other campaigns to steal credentials, infiltrated email accounts and other systems, and even attacked resellers and cloud customization service providers in attempts to compromise other parts of the tech supply chain.

“Looking back over the past year, Nobelium’s wide-scale attacks are difficult to overstate," Vasu Jakkal, corporate vice president of security, compliance, and identity at Microsoft, told WIRED in a statement. “It has been a moment of reckoning illustrating how technology has become both a defensive tool and an offensive weapon.”

So for all the progress over the last year, software supply chain security experts emphasize that the risks and exposures are still very real and can't be solved with any one solution.

“A SolarWinds-type attack could happen at any point and may actually be in the process of happening right now," says Charles Carmakal, senior vice president and chief technical officer of the cybersecurity firm Mandiant, which was a division of FireEye during the company’s breach last year. “I don’t want to be the guy who’s negative, I also want to celebrate the wins this year, but it’s still an effective way to break into a target.”

After decades of the issue being overlooked, though, at least the right people are finally paying attention to the supply chain threat.