SolarWinds & Securing the Software Supply Chain – One Year Later Retrospective
Posted by: Ingrid Olson
Software supply chain threats have sat on the collective radar of security professionals across government, industry, and academia for over a decade. These cyberattacks target vulnerabilities in the supply chains of products and vendors with the intention of targeting customer organizations further down the supply chain who use the compromised software. Software supply chain security must be prioritized by every organization, as oftentimes it’s the downstream organizations and their customers who are most impacted.
One year ago today on Tuesday, December 8, 2020, just after the closing bell, cybersecurity vendor FireEye first publicly announced that “a highly sophisticated state-sponsored adversary” had gained unauthorized access to its Red Team tools. Less than a day later, after the next closing bell, FireEye would see its stock price plummet by more than 13%. Over the next few days FireEye’s incident response and forensics teams would trace the source problem to a supply chain compromise that had been hiding in plain sight, undetected by any organization for months.
We would ultimately come to learn that FireEye was one of over 18,000 customers to download a legitimate patch to SolarWinds Orion Platform software produced and maintained by SolarWinds. Unbeknownst to SolarWinds and their customers, however, the software patches released from March to June of 2020 included malicious code that threat actors stealthily slipped into each release at the end of the build process. On Saturday, December 12, 2020, FireEye executives informed SolarWinds of the source of the supply chain compromise, malware that would henceforth be known as SUNBURST.
Included amongst these 18,000 potentially infected customers were multiple federal agencies, at least nine of which were subject to further, targeted compromise. By Sunday, December 13, 2020, it was “all hands on deck.” The Cybersecurity & Infrastructure Security Agency (CISA) issued Emergency Directive 21-01 informing all federal agencies about the SolarWinds Orion compromise with instructions for required emergency mitigations. CISA, the Federal Bureau of Investigations (FBI), the National Security Council (NSC), and the Director of National Intelligence (ODNI) joined forces to create Cyber Unified Coordination Group (UCG) to streamline the investigation, remediation, and analysis of the compromise across both the public and private sectors.
On January 5, 2021, the FBI, CISA, ODNI, and the National Security Agency (NSA) released a joint statement asserting that “an Advanced Persistent Threat (APT), likely Russian in origin” was “responsible for most or all of the recently discovered, ongoing cyber compromise of both government and non-government networks.” By the end of February, the dust had settled enough for public retrospection. On February 23, 2021, representatives from SolarWinds, FireEye, Microsoft, and CrowdStrike voluntarily testified to the Senate Select Committee regarding the SolarWinds hack. A similar joint house committee meeting would take place a few days later. Congressional hearings in March, April, and May saw government officials test SolarWinds publicly testifying about the SolarWinds compromise.
On May 12, 2021, President Biden signed Executive Order 10428: Improving the Nation’s Cybersecurity. Though the name of the Executive Order sounds generic, the content therein describes policies based on the lessons learned from the SolarWinds incident. It lays out a detailed and ambitions roadmap as we start to round out the end of the first quarter of the 21st century.
As we reflect back over the year since FireEye first encountered what we would come to understand was only the tip of the largest cyber espionage icebergs in our nations history, it is natural to ask: “What, if anything, has changed with respect to securing the software supply chain?”
At the highest level we have witnessed a shift in, from both government and industry, on prioritizing securing the software supply chain and an acknowledgement of the grave risks associated with not doing so. Section 4 of EO 10428, “Enhancing Software Supply Chain Security” emphasizes a renewed focus on prioritizing software supply chain security:
There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended. The security and integrity of “critical software” – software that performs functions critical to trust (such as affording or requiring elevated system privileges or direct access to networking and computer resources) is a particular concern. Accordingly the Federal Government must take action to rapidly improve the security and integrity of the software supply chain, with a priority on addressing critical software.
Perhaps the biggest change over the last year as it pertains to securing the software supply chain, is the growing consensus regarding the need to shift towards a secure-by-design mindset. Whether that be rethinking the best practices for Identity Access Management by implementing Zero Trust Architectures or establishing a minimum baseline for establishing a common Software Bill of Materials (SBOM), baking security in up-front with respect to, people processes, technologies, and policies will yield dividends in terms of time, money, security, and opportunity.
This isn’t to say that we must or should completely throw the proverbial baby out with the bathwater. There will always be a need to validate existing security postures through traditional activities such as: source code reviews, manual and automated application security testing, penetration testing & vulnerability management. But as Benjamin Franklin famously advised Philadelphians with respect to the threat of physical fire in 1763: “An ounce of prevention is worth a pound of cure.”
SolarWinds Timeline
October 2019
- First evidence that threat actor was in SolarWinds’ environment – “benign” proof-of-concept .dll included in build by threat actor.
March – June 2020
- Malicious code included in SolarWinds packages – downloaded by 18,000+.
December 2020
Tuesday, December 8, 2020
- FireEye announces that “a highly sophisticated state-sponsored adversary” had gained unauthorized access to its Red Team tools.
Saturday, December 12, 2020
- FireEye executives inform SolarWinds about the SUNBURST malware that threat actors were able to sneak into the Orion product produced by SolarWinds via their build processes.
Sunday, December 13, 2020
- The Cybersecurity & Infrastructure Security Agency (CISA) issues Emergency Directive 21-01.
- Cyber Unified Coordination Group (UCG) forms in response to the incident.
Tuesday, December 22, 2020
- FBI issues Private Industry Notification regarding SolarWinds Orion compromise.
January 2021
January 5, 2021
- Joint Statement by FBI, CISA, acknowledging Russia is the likely threat actor behind the campaign.
February 2021
February 23, 2021
- SolarWinds, FireEye, Microsoft, and CrowdStrike testify at a Senate Select Committee Hearing on the SolarWinds Hack.
February 26, 2021
- Longer joint house committee meeting with FireEye, SolarWinds, Microsoft, this time including the CEO during the breach.
March 2021
March 18, 2021
- Government officials testify to the Senate Homeland Security and Government Affairs Committee about the SolarWinds attack.
- CISA & NSA Release CHIRP tool for public use to help organizations look for threat actors.
April 2021
April 6, 2021
- Most recent updated version of SolarWinds Security Advisory.
May 2021
May 12, 2021
- President Biden signs Executive Order 10428: Improving the Nation’s Cybersecurity.
June 2021
- NIST virtual event Enhancing Software Supply Chain Security: Workshop and Call for Position Papers on Standards and Guidelines.
- CISA Cybersecurity Advisory Board Established.
July 2021
July 8, 2021
- Initial draft of NIST publishes initial draft of Guidelines on Minimum Standards for Developer Verification of Software (NISTIR 8397 final version published in October 2021).
July 21, 2021
- DHS Software Supply Chain Risk Management Act of 2021(HR-461) introduced to House of Representatives.
September 2021
September 30, 2021
- Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities draft published.
October 2021
October 13, 2021
- NIST publishes Definition of Critical Software Under Executive Order (EO) 10428.
October 20, 2021
- DHS Software Supply Chain Risk Management Act of 2021 introduced to House of Representatives passes almost unanimously in the House (412-2).
November 2021
November 8, 2021
- Virtual NIST Workshop Executive Order 14028: Guidelines for Enhancing Software Supply Chain Security.
November 15, 2021
- Rollout of Cyber Talent Management System.
November 18, 2021
- Federal Reserve approves mandatory 36 hour limit disclosure for any major incidents related to banks will go into effect on May 1, 2022.
December 2021
December 1, 2021
• CISA announces first 23 person panel to its Cybersecurity Advisory Board.
December 10, 2021
February 2022
February 6, 2022
- NIST to publish finalized version of guidance for best practices regarding standards, procedures, and criteria enhancing software supply chain security.
May 2022
May 1, 2022
- Date by which U.S. banks must start reporting hacks.
May 8, 2022
- NIST required to publish guidelines for reviewing and revising guidelines at regular intervals.
Ingrid Olson
Application Security Consultant,
GuidePoint Security
Ingrid Olson has over a decade of experience in software engineering and information security. She is currently an application security consultant at GuidePoint Security, where she has worked since 2016 in a number of capacities. Before re-joining the AppSec team in November 2021, she worked as a security developer for the Research and Development practice, and as the DevOps Lead for the internal IT team. Prior to coming to GuidePoint she worked as a software engineer and scrum master. She previously served as the Chapter Lead for OWASP Maine.
Ingrid holds a B.A. in Physics from Reed College, and a M.S. in Computer Science from the University of Southern Maine. She recently earned a Professional Certificate in Quantum Computing Fundamentals from MIT.