How a Tiny Code Change (Almost) Exposed Millions to Remote Attacks

Over Easter weekend, more facts came to light about the unauthorized and subsequent planting of a backdoor in the open-source XZ project. First, some background on the issue:

• XZ utils is a set of data compression tools used in Unix and Linux operating systems, and since version 5.0, in Microsoft Windows as well.
• Microsoft software engineer Andres Freund discovered malicious code hidden within versions 5.6.0 and 5.6.1 and posted about it on March 29, 2024.
• Freund found the backdoor somewhat by accident when he was investigating slow secure shell (SSH) logins on a Debian Linux system. He found the latency was due to a system call made by SSH to the liblzma compression library, which is included in XZ utils.
• The backdoor allowed remote access and execution of code with root (administrator) privileges.
• Even though the code was not part of the original GitHub source code for the library, it was included in the installation archive and would attempt to run via an obfuscated script executed during the configuration setup.
• The code changes were committed to GitHub back in February and made their way into test releases of Fedora, Alpine, and Kali Linux. No one noticed the issue until Freund reported it. Since you can find OpenSSH running on some 20 million IP addresses currently (nearly 10 times more prevalent than Microsoft Remote Desktop Protocol (RDP)), had the attacker successfully compromised these systems, the damage could have been widespread and potentially catastrophic.

The backdoor’s discovery prompted a swift response from the Linux community. Security patches were released to address the vulnerability, and users were urged to update their systems immediately:

• Red Hat: Not affected.
• Fedora: Impact on Fedora 41 and Fedora Rawhide.
• Debian: Not affected.
• Kali Linux: Any installations updated between March 26th and March 29th.
• openSUSE Tumbleweed: A safe snapshot (20240328 or later) was compiled to remove the backdoor.
• Amazon Linux: Not affected.
• Alpine: Versions 5.6.0 and 5.6.1 are affected.

People are advised to take the following steps:

• Downgrade to an uncompromised XZ utils version (earlier than 5.6.0).
• Follow the guidance provided by each Linux distribution.
• Use CVE-2024-3094 to identify the risk in your environment.

Open-source software is a two-edged sword. On the one hand, it’s cost-effective and theoretically transparent, allowing for quicker identification of bugs and vulnerabilities due to community scrutiny. However, the reality is that many open-source projects are under-supported, lacking volunteers with the time and expertise to address problems and leading to wide variance in project quality and stability. The result is that wily threat actors, particularly those with nation-state backing (knowledge, resources, time), can insinuate themselves into a project, ingratiate themselves to the project managers, and ultimately leverage this trust to create mischief.

Reality check:

“Tens of thousands of unpaid developers are writing the core of the world’s most critical infrastructure nowadays.” – Kevin Beaumont

Such seems to be the case here. The backdoor appears to have been introduced by someone using the alias “Jia Tan” (UserID JiaT75 on GitHub). Jia Tan apparently gained maintainer status over several years – likely through social engineering and trust-building tactics – belying a sophisticated and patient threat actor. According to security pundit Kevin Beaumont, Jia Tan “hijacked the open-source XZ project by social engineering the volunteer developer into handing over maintainer access after they cited some mental health issues…” Additionally, a request was opened to make the threat actor a Linux kernel module maintainer for XZ Embedded (variant of XZ utils specifically designed for embedded systems).

As of this writing, the sophisticated malware is still being reverse-engineered by a number of parties, but according to Beaumont: “The backdoor uses a five-stage loader to try to hide and includes a function where future updates can be placed in extra files without modifying the original XZ code changes.”

According to Cloudflare cryptographer Filippo Valsorda, “This might be the best-executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.”

Security researcher HD Moore summarized for ArseTechnica how the attack works:

• When verifying an SSH public key, if the public key matches a certain fingerprint function, the key contents are decrypted using a pre-shared key before the public key is actually verified. The decrypted contents are then passed directly to the system.
• If the fingerprint doesn’t match or the decrypted contents don’t match a certain format, it falls back to regular key verification and no one’s the wiser.
• The backdoor is super sneaky. It uses a little-known feature of the glibc to hook a function. It only triggers when the backdoored XZ library gets loaded by a /usr/bin/sshd process on one of the affected distributions. There may be many other backdoors, but the one everyone is talking about uses the function indirection stuff to add the hook. The payload was encoded into fake XZ test files and runs as shellcode effectively, changing the SSH RSA key verification code so that a magic public key (sent during normal authentication) lets the attacker gain access.
• (The attacker’s) grand scheme was:
  • Sneakily backdoor the release tarballs (code archives), but not the source code.
  • Use sockpuppet accounts (fake online identities) to convince the various Linux distributions to pull the latest version and package it.
  • Once those distributions shipped it, they could take over any downstream user, company system, etc.

One of the tools for addressing software supply chain security is a software bill of materials (SBOM). An SBOM provides a comprehensive inventory of software components used in an application and details dependencies, versions, and licensing information of code libraries used in building an app.

The XZ project backdoor exemplifies SBOM-related challenges, namely the following:

• Undetected Backdoors: The malicious code in the XZ utility was discovered far upstream in the Linux release cycle.
• Supply Chain Risk: The compromised XZ utility could propagate to potentially vulnerable downstream systems.
• Lack of Visibility: Without an accurate SBOM, it’s challenging to trace the impact of vulnerabilities.

Just as with the commercial software development community, the Linux community should leverage the use of SBOMs, automated scanning tools, and regular audits to help prevent future incidents. Collaboration among maintainers, developers, and security experts is crucial.

It also highlights the need for a Zero Trust approach to computer security. Since even “trusted” software can’t be trusted, you always need layers of security in place so that if one user/device/process begins misbehaving, you can limit the damage it can do.

Zero Trust Architecture is all about putting the following in place: Never Trust the System:

• Assume that any part of the environment, including the local area network or cloud computing resources, may be compromised.

Rely on strong authentication and authorization regardless of location or asset ownership. Identification and Authentication Everywhere:

• Verify user identities and tie them to role-based access controls (RBAC).
• Ensure that only authorized users can access requested resources.

Segmentation:

• Divide the environment into segments based on functions, groups, and identities.
• Granularly control access, privileges, and traffic flow.

Contain breaches within specific segments to prevent lateral movement. Visibility and Continuous Monitoring:

• Collect security audit data on all users, devices, applications, systems (even cloud, SaaS, etc.)
• Correlate the data to look for indicators of compromise (IOCs)
• Continuously monitor behavior, device risk posture, and user risk.

Remediate issues to lower risk and minimize the attack surface. Abacode can help mitigate the risks from supply-chain issues such as the XZ compromise by enabling you to implement a Zero Trust Architecture and providing 24x7x365 Security Information and Event Management (SIEM) monitoring with incident response. Please contact one of our solutions advisors today for help.