Serverless (in)security

TL;DR

• Most of vulnerabilities existing in traditional applications can also appear in serverless applications. The most common ones are described in OWASP Serverless Top 10.
• There are also threats which are specific to serverless, like event injection or overwriting the code stored in S3 bucket.
• It’s quite common, that Lambda’s execution role has more permissions than it’s required.
• The malicious code can be also smuggled to your serverless application via used dependencies.

What is AWS Lambda?

FaaS (Function as a Service) model allows to build applications and services without the need to manage physical or virtual servers. It is the provider who is responsible for the security of networks, servers, operating systems, their configuration, and update. On the other hand, the developer’s responsibility is to keep the code, logic and application configuration secure. The division of responsibility is defined by the shared responsibility model.

To understand attack vectors in an application built in a serverless architecture, we must first realize that Lambda is actually a runtime environment for our code, operating in a container built on Amazon Linux image and run on microVM (Firecracker).

In theory this environment is run separately for each call, just for the time of handling our code and then it is removed. In practice, launching a new Lambda instance takes a relatively long time (so called cold start). To speed up the performance of subsequent Lambda calls, “old” instances are not immediately closed and are used to handle other incoming calls, while new instances are still being launched. Taking into account that the calls are not completely separated from each other, and the Lambda instance file system is not completely in a read-only mode (the folder /tmp has write permissions) then under favourable circumstances it is possible for an attacker to “take over” Lambda instances and execute malicious code in the context of a new call 😮 You can read more about it here.

If you want to see Lambda from the inside and without digging in the code, I recommend you to check out the project: lambdashell.com.

OWASP Serverless Top mistakes

[MISSING GIF]

Mistakes in the code can always be made, regardless of whether you are developing a web or a serverless application. The 10 most common types of errors have been published as OWASP Serverless Top 10 project. The reason of the most common errors is the lack of proper input data validation. In addition to well-known vulnerabilities, like cross-site scripting (XSS), XML External Entity (XXE) or insecure deserialization, there’s also a new attack vector in serverless, called: Event Injection.

Event injection

AWS Lambda can handle event sources coming from other AWS services such as e.g. S3, DynamoDB, SNS etc. Let’s suppose our test application allows users to upload files to the S3 bucket. The idea here is simple: uploading a new file creates an event, what should trigger the Lambda function and in turn it adds a new file name to the MySQL table.

A Lambda developer needs to take out the name of the uploaded object (let’s call it ”uploaded file”) and put it in the MySQL table. The following simple code would do a job:

Imagine that one user would send a file named: `1");(delete * from uploaded_files_table`. In this case, the following query will be sent to the database, which of course will delete all entries in the table:

INSERT INTO uploaded_files_table (`file`) VALUES ("1");(delete * from uploaded_files_table)

Remote Code Execution in serverless app

Let’s examine another case. An application https://www.serverless-hack.me/ takes a document URL as a .doc parameter, which then is converted accordingly using Lambda. However, it turns out that by passing a suitably modified request, we can execute remote code in the environment where Lambda is running😲 What may an attacker do in this situation? For example, it’s possible to take over the role of Lambda, which permissions are defined in function policy. Lambda stores access keys and the session token of the role in environment variables. So, if the attacker can execute a remote code, the keys can be easily displayed:

Privilege abuse

After getting the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN the attacker can do as much as the implemented policy allows him to. Practice shows that developers usually grant higher privileges than actually it’s required for their applications to work properly. For example in lambdashell project the S3 write access was initially granted to the Lambda execution role, what allows for project’s page hijacking. In some cases it is also possible to escalate privileges of a compromised role and further penetrate the AWS environment. For readers interested in this topic, I especially recommend to check out the CloudGoat project.

It’s worth mentioning here that excessive permissions are not just the result of the developer’s “laziness” (after all, the line ”Action” policy: “*” in IAM policy guarantees no problems with the lack of required permissions, doesn’t it?). For example, in the Serverless framework, one role is used for all functions by default 😮 In this approach, the role privileges must be wide enough to meet the needs of all functions. Remember, to always apply one policy per function!!! If you use a Serverless framework, a dedicated plug-in, like Serverless IAM Roles Per Function can help you to do that.

In addition to the principle of least privilege, you should also consider an account-level separation, i.e. using (at least) a separate account for development and a separate production account. You can easily manage multiple AWS accounts using the AWS Organizations service. Moreover, the Service Control Policies may help you to limit permissions at the organizational level (e.g. usually there’s no need to give all IAM or CloudTrail permissions to development account). Read more about best practices in AWS security in my free guide.

Vulnerabilities in external libraries

Even if your code is bug-free and nobody can it doesn’t mean it is safe. Vulnerabilities may also appear in external libraries. It is worth mentioning here the research, in which it was possible to take control over the code of 14% of all NPM packages 😮

Defining a package version as:

"package name NPM": "^1.0.0."

means using the latest compatible version e.g. 1.1.0 or 1.0.1. Now imagine the situation when a repository owner modifies the NPM code to send to attacker’s server environment variables of Lambda in which you use his package… And not to remain groundless — there is a researcher who has put this scenario in practice — note the number of downloads despite a meaningful package name ”do-not-download-this-package” 😂

Of course, you can find a whole range of vulnerabilities in dependencies. Let’s assume that our serverless application uses a package humanize-ms to convert time into milliseconds. In the old version of this package one could find a ReDoS vulnerability — a bug in a regular expression. This vulnerability consists in the fact that an attacker can send a value which processing time exceeds the allowed timeout of Lambda (that is max 15 minutes). Given that we pay for Lambda’s execution time, sending thousands or millions of such requests can significantly increase costs, thus causing real financial loss. So, you can say that a known vulnerability the Denial of Service in serverless has evolved into Denial of Wallet 😉

Monitoring vulnerabilities in dependencies is quite a difficult task, especially in large systems. However, it is worth mentioning that we can check NPM packages for known vulnerabilities using the tool npm audit. This tool asks for known vulnerabilities every time we run npm install (it can also be used to scan local packages manually). After vulnerability detection, you can install recommended updates with npm audit fix command.

Summary

Without a doubt, serverless opens up a number of benefits for us, but we can’t forget about its threats. Even if you “believe” in security of your code, you must remember that external libraries may imperceptibly compromise your AWS environment, including its all resources. The “pay as you go” model used in serverless — although full of undoubted advantages — may prove to be a nail in your business coffin when you allow your user for too much. Therefore, regardless of whether you are using a classic or serverless architecture, remember about a multi-layered approach to security.

Feel free to reach me out. You can find me on Twitter and LinkedIn.

Paweł Rzepa Senior IT Security Consultant