UPDATE!!(12/15/22)
Another Critical SSL VPN Vulnerability

The following is an update to a blog post we initially published on April 8, 2021. What prompted it was a security advisory that came out from Fortinet on December 12, 2022, in which the company urged customers to take immediate steps to mitigate a critical flaw that had been disclosed earlier by French cybersecurity firm Olympe Cyberdefense.

Just as with the vulnerabilities described in our earlier blog post, the victim was FortiOS SSL-VPN, and the vehicle was a heap-based buffer overflow flaw. The bug, which is being tracked as CVE-2022-42475, affects multiple versions of the FortiOS sslvpnd and ranks 9.3 for criticality – meaning that it could lead to an unauthenticated attacker being able to execute arbitrary code on your system.

When exploited, this type of bug can be difficult to track down, because it is in firewall’s firmware – which may not be part of the regular vulnerability patching program. In the case of the 2021 Fortinet bugs, the exploit led to dumping admin credentials in plaintext from the firewall, and we found it being actively exploited in the wild.

Fortinet said in its advisory that it is aware there is active exploitation of CVE-2022-42475 now, as well. Therefore, updating your firewall’s firmware immediately is critical.  The good news is that patches are available.

As before, Abacode’s advice remains the same:

• Disable the SSL VPN service from your firewall if not needed.
• If you absolutely require it, implement the guidance of NIST SP 800-113.
• Make sure you have continuous network surveillance via a Security Information and Event Management (SIEM) or eXtensible Detection & Response (XDR) solution with 24/7 eyes-on-glass monitoring from a qualified analyst team. Abacode provides this through our Cyber Lorica

A Wakeup Call

It’s not often that we get missives about specific cyber-threats from the FBI Special Agent coordinating our local InfraGard chapter. But earlier this month, we received the following, highlighting an important Joint Cybersecurity Advisory issued by the FBI and DHS’s Cybersecurity and Infrastructure Security Agency (CISA):

I realize you receive many of these types of advisories every week, but I encourage you to pay particular attention to this one.  The potential exposure is very specific and provides a discrete, limited area to check (as opposed to an open-ended hunt).

Our FBI agent goes on to state that Advanced Persistent Threat (APT) actors (i.e., likely well-funded and coordinated nation-states), are scanning three specific ports, named in the Joint Advisory, and exploiting three Common Vulnerabilities and Exposures (CVEs), all in the same product (Fortinet FortiOS):

Please ensure your FortiOS systems are patched to ensure protection against these CVEs, look for evidence of exploitation of the CVEs or malicious activity targeting those ports, and immediately investigate any potential intrusions if any are found.

These vulnerabilities are not particularly new, but sometimes it takes a while before attackers are able to weaponize an exploit and scan for it on a mass scale. Evidently, that has happened now, elevating this to a critical global threat.

Note that there are patches available for all of these vulnerabilities; however, updating firewall firmware sometimes escapes the notice of companies – even those with a regular patching routine. We have seen this many times before: the focus is on Windows and desktop application updates, while network infrastructure devices go ignored.

A proper vulnerability management program must include all devices (e.g., firewalls, routers, Wi-Fi, switches, printers, cameras, VoIP phones, etc.) and software you may have forgotten you had, such as virtualization/hypervisor, remote desktop access, remote server access, database, and Web services.

Patch Now!

In September 2020, fully a year after the patch had come out, Abacode was called in to do a Digital Forensics and Incident Response (DFIR) project for an online retailer that had been hit with one of these FortiOS exploits to devastating effect. Unfortunately, they were not a managed services customer of ours, or else we could have been able to spot the attack and stop it before they found themselves riddled with ransomware. Abacode has sent out a notice to all of our managed services clients about these CVEs and worked with them to close out the threat. If you are a FortiGate customer and need help with this, please contact us today.

This is not the first issue we have seen with Fortinet products. Recall the very popular (with hackers, at least) 2016 SSH backdoor under CVE-2016–1909? That was due to a hard-coded authentication token. Well, guess what? We’re going to find this to be the case with this latest batch of vulnerabilities as well – begging the question of whether the FortiOS source code has ever undergone a thorough static analysis. One must wonder if there are more hardcoded backdoors to be found.

The three CVEs in question are as follows:

• CVE-2018-13379 – is a directory traversal attack, in which Fortigate firewall’s SSL VPN Web portal allows a specially crafted HTTP request (pre-authentication) to access restricted directories and download files. In particular, one can retrieve the /dev/cmdb/sslvpn_websession file, which contains usernames and passwords in plaintext!

• CVE-2020-12812 – is an authentication bypass in which a user can login successfully without being prompted for the second factor of authentication (the “FortiToken”) – but only if they have taken the trouble to change the case of their username!

• CVE-2019-5591 – is not quite as critical, more of an information leak that could be leveraged for further exploit. Basically, there’s a default configuration in FortiOS that allows an unauthenticated attacker on the same subnet to intercept sensitive information by impersonating as the LDAP server.

A Broader Issue

The first two vulnerabilities were made public in a 2019 Black Hat Briefings USA talk by Cheng-Da Tsai (aka “Orange Tsai,” Twitter handle @orange_8361) and Tingyi Chang (aka “Meh Chang,” Twitter @mehqq_), both of the Taiwanese security consulting firm DEVCORE and members of the HITCON Capture the Flag (CTF) team that impressively finished third in the DEFCON 28 (2020) and second in the DEFCON 27 (2019) CTF competitions.

Here are their PowerPoint slides and a video of the DEFCON version of their talk. For the uninitiated, DEFCON always follows Black Hat Briefings in Las Vegas (well, unless both are virtual due to COVID), and many speakers choose to give the same or similar talks at both conferences – the difference being, of course, that Black Hat costs about $3,000 to attend, while DEFCON costs about $300. Also, while Black Hat may frown on speakers doing shots during their talks, this is more or less expected behavior at DEFCON.

Anyway, Tsai & Chang have found numerous bugs in SSL VPNs (with Chang saying they are like slot machines – from a bug bounty standpoint). Compared to IPSec VPN, SSL VPN is easier to use, because you don’t need a special client – you only need a web browser – and thus, they’re very popular worldwide. If you can find an exploit, you’re going to get a lot of bang for your buck. For example, Tsai got a record (at the time) $20,000 for one bug he brought to the attention of Twitter. Fortigate SSL VPN is found on nearly 500,000 servers on the Internet.

Some of the firms that Tsai & Chang found using SSL VPNs:

Since SSL VPNs are typically proprietary, and there’s no source code out there in the wild for them, how do you attack them?

If you have an unencrypted virtual disk image, it’s trivial to get root and make changes. Just as with any Unix-based system, you can boot in GRUB or LILO to single user mode, mount the .VMDK, and modify the filesystem. On an encrypted disk image, it’s harder, but not impossible.

Tsai & Chang showed that when you boot a VMware image, it allows you to interrupt the initialization process: “Press <Enter> to view or update your appliance settings.” Once you hit enter, VMware spawns a process to do this editing. By suspending that process and doing in-memory forensics, Tsai & Chang found that a Perl script called /home/bin/dsconfig.pl was being called, and then, by patching a few bytes into memory during execution of the script, they could pop a shell (i.e., gain command line access).

Well, blow me down, I thought, this looks very much like a DEFCON CTF challenge!  Who’d have thought there would be real-world applications for this stuff <wink>?

With a shell, now they could analyze that otherwise “black box” of SSL VPN functionality and find some workarounds. The concentration of their analysis was around WebVPN. Because this is a portable, clientless technology that is super powerful and must proxy all traffic, support various protocols, and handle all Web resources, the code is notoriously difficult to write, implement, and maintain – thus prone to memory bugs and other vulnerabilities.

There are some open-source options available, but if a company wishes to write their own WebVPN from scratch, they are most likely not going to reinvent the wheel, and so they have two choices:

1. Start with an open-source code and modify it. However, if your starting code had bugs, you are now just propagating these to other products.

2. Utilize standard code libraries. However, your software needs to be maintained to reference the latest updated library versions, and that requires regression testing. So, we tend to see a lot of outdated and deprecated libraries being used (some decades old, according to Chang). And those libraries are deprecated for a reason – usually security-related.

Tsai & Chang found that most WebVPN implementations use native script language extensions:

Here, the developers are trying to extend the functionality of the language in which the code is written, but at the expense of type mismatches, string operation errors, and other fun stuff that CTF players love to exploit with buffer overflows and format string exploits.

There’s also the matter of multi-layered architecture issues, where because developers are using pieces/parts borrowed from multiple code bases, there’s inconsistency between the layers that could lead to exploit. Tsai showed this in his 2018 Black Hat talk “Breaking Parser Logic: Take Your Path Normalization Off and Pop 0Days Out.”

In that 2018 talk, Tsai explained that path normalization bugs between a customized C/C++ Web server front-end and a RESTful API back-end could be leveraged to do directory traversal, read passwords, and so forth. In fact, he published numerous CVEs and gained lots of bug bounty $$ from these:

These sorts of errors would prove to be useful to Tsai & Chang in exploiting FortiGate’s SSL VPN as well as Pulse Secure’s SSL VPN. Note that this intrepid team responsibly disclosed all of these CVEs, and vendors had produced patches for them before they publicly spoke or blogged about them. However, as noted above, there’s always a lag between when patches become available and when they are applied. According to the Ponemon Institute, that average is about 102 days.

Upon reverse engineering FortiOS, Tsai & Chang found that all executables in the OS are really just symbolic links pointing to /bin/init. That’s one giant binary (500 MB in size with 85,000 functions) to rule them all. Of course, all symbols had been stripped, making it a joy to step through—not. Upon further examination, they found that it was using a web service daemon derived from a 2002 version of Apache (Chang joked that it was as old as her grandma).

I don’t know about you, but I have run across that simplistic FortiGate web login screen many, many times over the years during penetration tests, and I always felt there must be something I could do to exploit it; I just didn’t have the right tool. Well, now I do (a Metasploit script based on the exploit).

How did they do it? In a function to read and parse JSON files, Tsai & Chang found an insecure implementation of snprintf() that (unsuccessfully) tried to limit the call to be used only on JSON files: snprintf(s, 0x40, “/migadmin/lang/%s.json”, lang);

However, when they naughtily tried the string: “/lang migadmin/lang =/../../../..//////////////////////////////bin/sh //../../../..//////////////////////////////bin/sh.json”, the function gladly took it, and since it exceeded the allowable buffer size, it stripped the “.json” off the end of the string, essentially allowing them to traverse to whatever directory and file they wanted.

After that, it was a (somewhat) simple matter to poke around and look for interesting files to read. They found what they dubbed the “SSL VPN Mystery” – something that appeared not only in FortiOS, but several other vendors’ software. The mystery being that it stored excessively detailed session logs in /dev/cmdb/sslvpn_websession, serving as a treasure trove of information. Wonderful nuggets in that file include the following:

• Sessions tokens
• IP addresses
• Usernames
• Plaintext passwords(!)

All of this is fine and dandy, but how do you get to the point where you are able to execute that function, do the directory traversal, and read all of passwords? This unfortunately gets a bit technical.

Tsai & Chang did some sleuthing to look for the possibility of a heap overflow vulnerability. At first, it was uncontrollable, i.e., they couldn’t redirect program flow to execute a shellcode of their choice, and there was no memory free() until the connection ended (this is an Apache feature). Refusing to become despondent, they ultimately found a good candidate: the handshake_func pointer inside the SSL structure would allow them to modify program flow. By sending numerous “good” requests with one “bad” one interleaved, they were able to abuse the Java parser and overflow the SSL structure with a bunch of AAAAAAAAAAAAA’s, thus overwriting the call to the ssl_accept() function and crashing the program. The ultimate goal was to segfault to a known address such as the beginning of system(), and then use that as an offset to execute their own shellcode. Cutting to the chase: it worked! After several tries, they could reliably get a shell without permanently crashing the server.

This is all well and good for the 0.001% of the people on earth that are good with binary exploitation, but what about the rest of us? Well, Tsai & Chang also poked around and found the bane of FortiOS code over the years: a hard-coded “magic” parameter, intended for password resets:

Final Thoughts

The sloppy SSL VPN implementations of Fortinet and other vendors are not going away any time soon. When vulnerability researchers can make $20,000 off a single bug bounty, many others besides Tsai & Chang will try to cash in. The issues appear to be endemic to the entire class of products – i.e., using deprecated code that’s full of format string and other bugs. The same issues likely apply to myriad Internet of Things (IoT) products as well. Companies need to be wary of using these convenient technologies, or at least keep close tabs on them in the enterprise and include them in vulnerability management programs.

In general, Abacode strongly encourages disabling the SSL VPN web console for all SSL appliances. If you need to use this service for remote access, we recommend implementing all of the guidance of National Institute of Standards and Technology (NIST) on the secure use of SSL VPN as published in NIST SP 800-113. Finally, as previously mentioned, having continuous network surveillance via a Security Information and Event Management (SIEM) solution with 24/7 eyes-on-glass monitoring is a best practice for immediately recognizing and responding to these threats. If you do not have this capability, Abacode can assist with this today.