Contrast Security discovers Netflix OSS Genie bug that can lead to RCE during file upload

Genie is a federated big data orchestration and execution engine developed and open sourced by Netflix. 

Genie provides REST-ful application programming interfaces (APIs) to run a variety of big data jobs such as Hadoop, Pig, Hive, Spark, Presto, Sqoop and more. It also provides APIs for managing the metadata of many  distributed processing clusters and the commands and applications that run on them. 

From the perspective of the end user, Genie abstracts away the physical details of various  (potentially transient) computational resources (like YARN, Presto, Mesos clusters, etc.). It then provides APIs to submit and monitor jobs on these clusters without users having to install any clients themselves or having to know details of the clusters and commands.

Contrast Assess is an Interactive Application Security Testing  (IAST) solution that  continuously  detects and prioritizes vulnerabilities and guides development teams on how to eliminate  risks. We discovered a vulnerability in Genie recently as part of research conducted by Contrast, using Assess. 

Contrast was rated the #1 leader in the G2 Summer 2023 Grid® Report for IAST, with a satisfaction score of 93 out of 100. Download the report.

Specifically, the open-source version of  Genie  had a path traversal  vulnerability  — also known as directory traversal  — that attackers could  leverage to accomplish remote code execution (RCE). If successful, such an attack could fool a web application into reading and consequently exposing the contents of files outside of the document root directory of the application or the web server, including credentials for back-end systems, application code and data, and sensitive operating system files. 

The vulnerability — designated CVE-2024-4701 — has been rated critical, at a 9.9 out of 10 on the CVSS severity rating standard.  It is important to note that this issue did not affect any Netflix products or services, nor Netflix’s own internal Genie service. However, other Genie users without the same mitigating controls that Netflix had in place for its own use and whose instances use the local filesystem to store the affected file uploads would have been impacted. Any instances of Genie not using the local filesystem to store these file uploads (e.g. using AWS S3 for storage) would not have been vulnerable.

Genie’s path traversal vulnerability 

Found by Contrast Assess

This issue was discovered by instrumenting the Genie application with Assess, also independently by another researcher, jmoritzc53, who also reported this privately to Netflix.

Blocked by Contrast Protect

With Contrast Protect enabled, the attack is successfully blocked and the user is notified.  On Contrast’s own Genie instance, we’ve enabled Protect to block the path traversal attack. We should note that Netflix is not a user of Contrast Protect and is not affiliated with it.

Contrast Protect is Contrast’s Runtime Application Self Protection, or  RASP, technology: a solution that protects running applications and APIs by stopping attacks that exploit logic flaws or zero days that would otherwise bypass other first-line defense tools.  

Note that Contrast Protect often pre-blocks attacks even before the vulnerabilities are known, such as it did with Log4Shell and this growing list of zero days:

​​CVE-2023-22527 Atlassian Confluence – Template Injection

CVE-2023-34040 Spring/Kafka – Unsafe Deserialization

CVE-2023-22965 Spring4Shell – Malicious Data Binding

CVE-2021-44228 Log4Shell – JNDI Injection RCE

CVE-2021-26084 Atlassian Confluence EL Injection

CVE-2020-17530 Apache Struts2 – EL Injection

CVE-2020-11651 Python Salt – Authentication Bypass

CVE-2020-11652 Python Salt – Directory Traversal

CVE-2020-9484 Apache Tomcat – Unsafe Deserialization

CVE-2019-2725 WebLogic – Unsafe Deserialization

CVE-2019-0230 Apache Struts2 – EL Injection

CVE-2018-11776 Apache Struts2 – EL Injection

CVE-2016-0792 Jenkins XStream – Unsafe Deserialization

It does this by looking for execution behaviors where untrusted input (like the files and JSON manifest for the files) get to a dangerous function (like the storing of those files to the host file system) without proper sanitization. 

Contrast Protect blocks the CVE-2024-4701 path traversal attack as of Version 6.5.0.

Applications and libraries that we use every day are full of latent vulnerabilities

In this case, there was no evidence that this vulnerability had been attacked, but often, the first time vulnerabilities are known about is when they are exploited in a zero-day attack. Even the most robust detection combined with the most rapid resolution possibly leaves you open to a window of being hacked, and most companies cannot afford the expense of the “most robust” and “most rapid.” With Contrast Protect, your use of these applications and libraries are pre-protected, leaving no risk window. Think of it as negative-day protection for zero days.

Vulnerability explanation

Within the Genie application is a REST API that, among other things, allows you to submit SQL queries using Spark SQL. As part of this process, you can upload a SQL file containing the SQL to be run.

This upload is done using a multipart file upload, as shown in the HTTP request below:

The HTTP request shown above would upload two files — named query.sql and query2.sql — that would then be run as a job specified in the JSON at the beginning of the request.

However, the filename parameter was vulnerable to a path-traversal attack from

genie/genie-web/src/main/java/com/netflix/genie/web/services/impl/LocalFileSystemAttachmentServiceImpl.java at 99cd72286b6fc813d87afc379dd8a83450391368 · Netflix/genie. 

When the uploaded file is written to disk, you can see this code:

Here, attachmentPath is the path to which uploaded files should be sent, which is /tmp/genie/attachments/{uuid}/,

Where uuid is generated for each request, giving a unique directory path.

For the above request, you get a path of

/tmp/genie/attachments/{uuid}/query.sql

However, if you change the filename parameter in the http request to ../../../../../root/query.sql,

The file would have been uploaded to /root/query.sql, as shown below:

Meaning that we were able to break out of the expected upload location, which in turn meant that we could have leveraged the path traversal for RCE. 

Leveraging path traversal for RCE

So we can upload files to the filesystem of the Genie App. But what can we do with that?

To answer that, we can look to the fact that the file upload has only these restrictions: 

1. You can only upload to locations to which the web server has access. Luckily, we are using the docker-compose from the genie documentation, where the Genie application runs as root, so no issues there.
2. You can only create new files; you cannot overwrite existing files. 

Using these conditions, we can upload a shared object file /tmp/pe.so.

This shared object contains the following code:

Which …

1. Unlinks the ld.so.preload (more on that later), 
2. Prints “HACK HACK HACK” to the Standard Out and 
3. Executes a bash command to create a new file to prove that the code was executed.

If you want to try this, it can be compiled using the following:

gcc -fPIC -shared -o pe.so pe.c -nostartfiles

Once done, you get a pe.so file: a shared object (i.e., a computer file) that contains executable code designed to be used by multiple computer programs or other libraries at runtime.

The pe.so files can then be uploaded to any location on the filesystem by using the above vulnerability.

Executing the shared object

With the shared object uploaded, we need to execute it.

To do this, we upload another file, to /etc/ld.so.preload.

This file just contains “/tmp/pe.so”.

The ld.so.preload file is read (if it exists) during the execution of ld.so, which is the dynamic linker and which is run before any other program is executed. The dynamic linker also loads any shared libraries required for that program before any other program is executed. That means that ld.so runs before any other program, as does anything in ld.so.preload.

Using this, we can upload both the malicious shared object as well as the file to ensure it is executed the next time any program is run.

And on the Genie App OS … 

… you can see that before the payload was sent to Genie, /tmp/ contained neither the pe.so file nor the command.out file.

After upload, the next command that ran on the host was ls. This triggered ld.so and thereby executed the code within the pe.so library as well, leading to the text HACK HACK HACK being output to the terminal and the file written to disk.

Demo

Before Netflix resolved the issue, this exploit could be reproduced using the following steps:

1. Download the genie docker file from: https://netflix.github.io/genie/docs/4.3.0/demo/docker-compose.yml
2. Run:
   docker-compose up
3. Download the example exploit payload found here:
   loadlib
4. Run the following command: 
   nc -w 5 localhost 8080 < loadlib
   This will send the payload to the genie app running in docker, which uploads the pe.so and ld.so.upload files.
5. Then login to the genie-app docker container:
   docker exec -it genie_demo_app_4.3.0 /bin/bash
6. The act of logging in triggers the execution of the malicious library.
7. Run:
   ls /tmp/
   To see the file command.out written to disk.

Current status

I submitted the vulnerability to Netflix via Bugcrowd. Netflix fixed the path traversal and released a security advisory. The path traversal issue is fixed in Genie OSS v4.3.18. This issue was fixed in #1216 and #1217, and a new release with the fix was created. Netflix is asking users to upgrade Genie OSS instances to the new version.

Despite the fact that mine was a duplicate of a previous report of the path traversal, Netflix  took the unusual step of awarding me $250.

Contrast Security customers didn’t need to be concerned

Contrast Assess discovered this path traversal vulnerability during our independent assessment,  while Contrast Protect  Version 6.5.0 and greater blocked the attack on our own Genie instance.

If you would like to see how Contrast Assess detects vulnerabilities in custom code and in third-party libraries and how you can protect running applications, please reach out for a  demo call. Contrast Security’s approach to Application Security (AppSec) is designed to eliminate entire classes of vulnerabilities. Let us show you how we do that. 

For more information about path-traversal vulnerabilities, check out our glossary entry:

Related:

• Blog: What’s a basketball got to do with Application Security instrumentation?
• Blog:  It’s time to replace our broken AppSec tools with something that actually works: Runtime Security
• Glossary: What is a Path Traversal Attack or Directory Traversal Attack?
• Blog: The Top 10 app-attack trends in the financial sector
• Blog: Authenticated Remote Code Execution in OpenMRS
• Contrast Labs Blog: Mapping Risk Profiles for Select OWASP Top 10 Vulnerabilities to Understand Their AppSec Risk
• Blog: Security Observability: Intelligent security assessment = seeing what others can’t
• Blog: Attack-path mapping your applications