Integrity Checking - an Integral Part of Cyber Security
Table of Contents
by:
Tomi Koski and
Joona Hoikkala
During the last twenty years there has been endless talks and discussions regarding the importance of Transport Layer Security (TLS) and the potential threats emerging from failing to implement TLS correctly. Even with all the talk, the real life examples of impactful man-in-the-middle (MitM) exploitation are rare. This post aims to shed some light on one of these cases.
A Prelude to a 0-Day #
It was a dark and stormy night normal workday for us working in an internal
red team. A large part of our job is to map potential attack surfaces, and
while large companies often have a huge external attack surface there’s still a
lot of other things going on under the surface as well.
One of these things are the employee devices and the installed enterprise software on them, typically provisioned by the company. Looking through the devices that are used daily by thousands of our colleagues we saw what we expected; reputable EDR solutions, inventory management software and so on.
IT needs to Keep Track of All the Devices #
As a company grows in terms of employee count, the IT department can quickly find themselves needing a helping hand to keep track of the devices used by the company employees. This is especially true in a workplace where working remotely is an option, as you are not just able to walk into an office or look at your network data to see which devices are being used.
The software solution built for this purpose used in our company is called FreshService, an inventory management solution by FreshWorks. It works by contacting a central service over the internet every now and then to report details about the device it is installed on. This helps the IT to see which devices are actively used, what types of software is installed on them and so on.
FreshService itself is one of the largest inventory management software solutions globally, and is being used by a large portion of the world’s enterprise sized companies.
Something is Rotten in the State of TLS Integrity Checking #
While monitoring the network traffic from the corporate laptop and getting familiar with the pieces of installed software previously unknown to us, there was something that caught our eye. A live log file by FreshService Agent software, with a line:
With this information, we decided to look into this application in more detail. As a next step, we set up a proper TLS interception proxy to see if the validation really is disabled and to inspect the data communicated over the wire by the FreshService agent.
Surely enough, our concern turned out to be valid and we saw the agent software happily accepting a self-signed MitM TLS certificate and proceeding to communicate with the (now malicious) endpoint.
An Investigation Ensues #
As we found out while FreshService Agent software communicates back to the central service, the typical pingback payload it sends is the device details, an example request can be seen below. The subdomain name (companyname) it connects to is unique to each customer.
POST /itil/scan_agents/add_device.json HTTP/1.1
Host: companyname.freshservice.com
RegistrationKey: eyJ0e...<REDACTED>
AccessKey: eyJ0e...<REDACTED>
AgentVersion: 4.2.0
User-Agent: FreshServiceAgent
Connection: Keep-Alive
Content-Type: application/json
Content-Length: 3423
{
"items": [
{
"Device": {
"name": "JohnDoesPuter",
"type": "Laptop",
"serial_number": "C1234567890",
"uuid": "C0FFEE-DEAD-BEEF-DEAD-C0FFEEEE",
"model": "MacBook Pro",
"manufacturer": "Apple Inc.",
"ip_address": "1234:🔢1234🔢1234%en6"
<...CLIPPED FOR READABILITY...>
{
"softwareName": "App Store",
"softwareVersion": "3.0",
"softwarePublisher": "-",
"softwareLocation": "/System/Applications/App Store.app",
"softwareInstallDate": "-",
"operatingSystem": "false"
}
]
}
}
]
}
From this we can already see that the missing TLS verification can potentially disclose a lot of somewhat sensitive information about the device and its user if the employee connects through an untrusted network.
While a lot of sensitive information is transmitted daily through these devices, the transport usually happens through an additional layer of network security - VPN. A properly configured VPN client refuses to connect over a TLS intercepted hostile network but as FreshService Agent communicates through the public internet this missing additional layer of security does not stop it from sending the data.
What’s the worst thing that could happen? #
It is the natural next question. To answer this, we decided to look deeper into the software at hand; what does it collect from the device and how, are there other endpoints it talks to? The language that FreshService Agent is written in (.NET) is fortunately trivial to decompile and to inspect, so we decided to do exactly that. To top of the potential exploitability, the software seems to be contain a self-update daemon as well…
As can be seen from the single log line posted there earlier, the application does log debug level messages and that made our work a lot easier as we could see the exact modules and functions where the debug messages originated from. It all becomes easier to understand by looking at a larger snippet from the log file:
------------------------------
| AGENT VERSION : 4.2.0 |
------------------------------
2022-03-14 16:48:49,944 [1] DEBUG FSDiscovery.FSAgent - StartAgentTasks invoked
2022-03-14 16:48:49,944 [1] DEBUG FSDiscovery.FSAgent - RetainAgentPresentPath invoked
2022-03-14 16:48:49,945 [1] DEBUG FSDiscovery.FSAgent - PersistPortalUrl invoked
2022-03-14 16:48:49,946 [1] DEBUG FSDiscovery.FSAgent - test url0: https://company.freshservice.com
2022-03-14 16:48:49,954 [1] DEBUG FSUtil.HttpHandler - ************************overriding https certificate check tls 1_1
2022-03-14 16:48:49,954 [1] DEBUG FSUtil.HttpHandler - https certificate check overrided
2022-03-14 16:48:49,954 [1] DEBUG FSUtil.HttpHandler - IsAlive invoked.
From the decompiled source code, it became obvious that the TLS checks were disabled globally, and as such our focus immediately turned towards the self-update daemon. Being able to execute code remotely on the victim machine gives the keys to the kingdom after all. It seemed even more serious; the self-update daemon was running as root on macOS and Linux systems, and with SYSTEM privileges on the Windows installation.
The next step in our research process was trying to go from the normal software update flow pictured above, to the exploitation flow where we could intercept the connection and run our own malicious code on the victim machine as pictured below.
While the FreshService Agent connects to companyname.freshservice.com to send its operational data, the software update is handled through a static subdomain fstools.freshservice.com. The self-update daemon checks for software updates (on default settings) every seven days, but there were ways to trigger the functionality manually so we could effectively continue with our research.
More Integrity Checks to Overcome #
FreshService Agent has an additional integrity check for update packages downloaded from the internet.
It is calculating custom checksum values of the downloaded binary before executing it, so it seems to be hard to exploit, right?
Well, the checksum value is fetched over the interceptable TLS connection as well, so we are able to modify that on the wire as well. We just needed to figure out how it was calculated and if we could reverse engineer the algorithm to provide valid checksum for our malicious payload. The request made by FreshService Agent to discover if there’s an update package available alongside with its checksum can be seen below:
GET /agent/mac-manifest.json HTTP/1.1
Host: fstools.freshservice.com
User-Agent: FreshServiceAgent
Connection: close
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 253
Server: Werkzeug/2.0.3 Python/3.8.9
Date: Mon, 14 Mar 2021 17:25:24 GMT
{"AvailableTo":"*","CheckSum":"EAAAAKF9Oy5AblYUYTOi9Cdqy6s36JpQ9r1AnghfHVhJdpo5V1tZno19ysYCVsq8yuwiAuN7JNI/g8BMqqkO6au0lU5078FqUbnrtBUxUZBQNXgKgtiQUZBR9mleOo7AwhEg5w==","OldVersion":"4.2.0","ProductName":"fs mac agent","ProductVersion":"4.2.1"}
Surely enough, the source code revealed that the checksum method used was definitely there, and was using a static, hardcoded salt and a secret key, which allowed us to create a simple CLI application that we would use to calculate valid checksums for our malicious update packages.
So basically we were able to do the following to publish a legitimate package to all clients using the following procedure:
- Generate malicious archive files (.tar.gz / .zip) including our payloads (these can be part of normal distribution to not raise awareness of end-users)
- Calculate SHA-256 checksum for our archive
- Run our own CSharp application (copy-pasted from extracted code) and receive an legit, FS Agent approved checksum to be distributed globally
So here we have our second vulnerability, using hard-coded secrets (static SECRET_KEY and _salt) which can be trivially extracted from reversed CSharp -source code.
With this information, we now had everything we needed to proceed with a full exploitation workflow as follows:
- Creating a malicious update package
- Calculating a checksum for it that will be accepted by the FreshService Agent during the process.
- Setting up TLS interception that will modify the FreshService responses for requests to https://fstools.freshservice.com/agent/{mac,linux,win}-manifest.json and to make our malicious update package at https://fstools.freshservice.com/agent/{mac,linux,win}-installer-4.2.1.tar.gz available for download.
- Waiting for the client to connect and watch the show.
- Observing the mayhem
- Watching the network flow to validate our proof of concept shows that everything is working as we expected.
Exploit or Go Home #
- First, the FreshService Agent fetches information about a potential update, which we gladly provide with a valid checksum for our malicious package:
GET /agent/mac-manifest.json HTTP/1.1
Host: fstools.freshservice.com
User-Agent: FreshServiceAgent
Connection: close
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 253
Server: Werkzeug/2.0.3 Python/3.8.9
Date: Mon, 14 Mar 2021 17:25:24 GMT
{"AvailableTo":"*","CheckSum":"EAAAAKF9Oy5AblYUYTOi9Cdqy6s36JpQ9r1AnghfHVhJdpo5V1tZno19ysYCVsq8yuwiAuN7JNI/g8BMqqkO6au0lU5078FqUbnrtBUxUZBQNXgKgtiQUZBR9mleOo7AwhEg5w==","OldVersion":"4.2.0","ProductName":"fs mac agent","ProductVersion":"4.2.1"}
- After being told that there’s a new version available, the agent proceeds to download and to execute the malicious package.
GET /agent/mac-installer-4.2.1.tar.gz HTTP/1.1
Host: fstools.freshservice.com
User-Agent: FreshServiceAgent
Connection: close
HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 121
Date: Mon, 14 Mar 2021 17:27:01 GMT
...
...
(binary content here)
- And presto, there we have our Calculator.app executing as root.
Concluding the Research #
We reported the vulnerability to the vendor through their responsible disclosure channel and because of the impact and potentially affected parties amidst a geopolitically unstable time, we decided to send the report to the Nordic CERTs at the same time as well.
Because of the nature of the vulnerability, we decided to wait for a good amount of time before publishing the results of this research to ensure the self-update to be able to secure the affected parties.
We are fortunate to have a really awesome blue team within our organization and in collaboration with them we were able to mitigate the exploitability of this vulnerability in a very short period of time. However due to the sensitive nature of the issue, we could not provide this information outside of our organization (with the exception of the vendor and CERTs of course).
We truly believe in celebrating the incidents that did not happen, so cheers for no breaches in any of the many fortune 500 companies due to this Zero Touch Zero-day vuln….. No, it is not that anymore. It is now just “a secure piece of software”.
A Timeline of Actions #
2022-03-15: Research concluded, and report sent to the vendor (FreshWorks)
2022-03-15: Research disclosed to Finnish, Swedish and Norwegian national CERTs
2022-03-17: The vulnerabilities were confirmed by the vendor
2022-04-13: Fix released by the vendor, as a tiered rollout
2022-04-18: First tier of customers started receiving the fixed version through self-update
2022-04-29: Last tier of customers started receiving the fixed version through self-update
For the checksum issues the vulnerable versions of FreshService Agent were:
- Mac agent < 4.2.0
- Windows agent < 2.11.0
- Linux agent < 3.3.0
And for the missing TLS certificate validation:
- Mac agent < 4.4.0
- Windows agent < 2.12.0
- Linux agent < 3.4.0
- Probe < 4.11.0
Final Thoughts #
Integrity validation is crucial, and should not be overlooked. We do have theories why the TLS verification was disabled in this case, but going into that opens a whole new can of worms, and is a battle that we’ll fight on another day.
The communication towards the vendor was easy, they responded fast and fixed the vulnerabilities during a reasonable timeframe. Still we are somewhat disappointed by their practice of not requesting CVE numbers for vulnerabilities found in their software. The fix was addressed in the release notes plainly as “Security fix: added TLS certificate verification” as if it were an additional feature. This can potentially leave some of their existing or previous customers vulnerable if they don’t get the updated version.
These vulnerabilities require the attacker to have control over the network, but the exploitation scenario isn’t that far-fetched in the modern day remote work environment. It’s becoming more common for people to work from different locations - hotel rooms using hotel WiFi, cafés around the block or shared public workspaces.
Credits #
The research was done by the Visma Red Team: Tomi Koski and Joona Hoikkala and the fast local mitigation plan was implemented by Florin-Alexandru Pătruța from Visma GlobalSOC.
Update: 2022-09-07 #
The two vulnerabilities detailed above have been assigned the following CVEs by MITRE: CVE-2022-36173 and CVE-2022-36174.
Update: 2022-09-15 #
The vulnerabilities were published by NIST on 2022-09-12. After analysis, they describe the vulnerabilities as follows:
| CVE | CWE | CVSS Base Score |
|---|---|---|
| CVE-2022-36173 | CWE-295 | 8.1 High |
| CVE-2022-36174 | CWE-354 | 8.1 High |