ACM.79 When a process leaves sensitive data accessible to users or malware and what you can do about it.
This is a continuation of my series on Automating Cybersecurity Metrics.
In the last post we created some SSH credentials for a user but the credentials ended up in a files in the home directory on our EC2 instance. You can search the home directory for these files using ls and grep.
Well, we probably don’t want someone coming along and obtaining that key who shouldn’t have it. We also wanted to create our key in such a manner that our administrators can’t access it so we have non-repudiation for actions taken with that key as explained in the last post. What are our options?
Who might be able to access that file?
On a shared host, anyone who has access to the host might have access.A vulnerability in some part of AWS might allow an attacker to access the host and the file.If the server is hosting an application exposed to the Internet that has a vulnerability and attackers get on the host or leverage an attack that allows them to read the file system they might gain access to the file.If someone creates an image of the host and launches it and can login and read the data they could access the file.If backups exist for the EBS volume on which this files resides, anyone with access to the backups and related permissions can attack the volume to their own instance and read the files.
That’s what I can think of off the top of my head. You can check out the various forms of attack on MITRE if you want to try to find more:
We may or may not have a problem if we just left that file sitting there, but I’d rather not leave it hanging around on the host. What can we do? You might think, oh, we can just delete the file, right? Maybe that will help us. Maybe it won’t.
Retrieving deleted contents on Linux
Have you ever been working on a file in vi or some other text editor on Linux and accidentally delete the file?
I have. Oops.
But… there is a way to potentially get your content back. It’s not as simple as pulling the file out of source control, so I highly recommend frequent check-ins to your source control over this process, but here’s how it works.
Let’s say I create a file named yada.txt in my home directory with these contents:
Now I delete that file:
rm yada.txt
OK it’s gone, right? Maybe. It depends on how many other commands you’ve executed since then.
Before we run our command, navigate over to the EC2 instance you’re using and click on the storage tab. Notice the device name: /dev/xvda.
Run this command:
sudo grep -i -a -B10 -A10 ‘yada yada’ /dev/xvda
I grabbed the command off this page because I always forget the exact syntax and it has some additional information:
https://www.cyberciti.biz/tips/linuxunix-recover-deleted-files.html
In the command above, I am searching for some unique text in my file, that exists in the specified drive. If you run this before the information is overwritten, you should get back the contents of your deleted file.
You will see a lot of other information as well as you will get various parts of deleted files. I also got back part of a private key that was recently created (and will be deleted prior to this post getting published).
So even if we create and delete our sensitive file it is still there. What can we do about that?
Use a temporary host and terminate it.
We could run this process on a temporary host that is immediately and permanently deleted. AWS had stringent methods of ensuring data gets deleted when a host is destroyed that were documented in this white paper, since archived:
That white paper was one of the sources of information that convinced me that Amazon was serious about security. Now the architecture of Amazon EC2 instances is completely different (in a good way) so some aspects of that paper are obsolete. Now the paper has a link to other resources. I want to know how Amazon handles deleting data on my EC2 instance when it terminates. Let’s head over to the link.
Security, Identity & Compliance | AWS Architecture Center
It’s a huge list of resources that doesn’t specify what Amazon does for security but rather has a list of best practices that customers should be following. Where can we find out how to be sure that Amazon deletes our data when we terminate an instance?
Searching around on Google didn’t lead to anything too in depth. I did find this related to EC2 instance termination:
After an instance is terminated, resources such as tags and volumes are gradually disassociated from the instance and may no longer be visible on the terminated instance after a short while.When an instance terminates, the data on any instance store volumes associated with that instance is deleted.By default, Amazon EBS root device volumes are automatically deleted when the instance terminates. However, by default, any additional EBS volumes that you attach at launch, or any EBS volumes that you attach to an existing instance persist even after the instance terminates.
“Deleted.” Like the deletion that occurs in this post that isn’t really deleted? Or something more permanent? How exactly is the data deleted on termination? I can no longer find this answer. But at this point I need to move on and work on some other things. Hopefully the documentation will be updated to something more specific. #awswishlist.
Our data in this case is on the root device so it would be immediately deleted. Whatever that means.
Encrypt the drive
We can encrypt the drive on which the key is stored. Does this solve all our problems? First of all, I’ve written before about the encryption fallacy:
If we use the AWS default encryption, anyone with permission to run EC2 instances in our account and attach EBS volumes might be able to still decrypt the contents of our EC2 instance and volumes.
What if we use a customer managed key? That’s a better solution. We can lock that key own to the people who are supposed to use the instance or drive using a key resource policy. Let’s say someone went rogue in our account and was trying to attach EBS volumes to their own instance. They wouldn’t be able to if they didn’t have permission to use the key. Let’s say someone inadvertently shared their credentials with AWS support as explained in this post:
AWS Credentials in Boto3 and CLI Debug Output
If the support person then used those permissions to try to access and expose EBS volumes in the account publicly or to another account, they would not be able to get into the contents of the EBS volumes even if they could take those actions.
This solution doesn’t protect us against malware that obtains access to the host. Malware that infiltrates the host and can read the files could get access to our key.
Network Controls
If we implement proper network controls and malware gets onto the host and accesses the key, they still wouldn’t be able to use it if they can’t get to the hosts where the key is used via port 22. That prevents use of the stolen key potentially, but it still lets the attacker get the key.
MFA
As mentioned it is possible to add MFA to our SSH controls. As long as the attacker doesn’t have access to change the MFA settings on the host, they wouldn’t be able to login because they wouldn’t have the second factor associated with the key. This presumes that the attacker cannot simply automate making numerous requests to try a whole bunch of numbers and simply brute forcing their way past the MFA. MFA plus rate limiting plus and alert on this type of activity and auto-blocking IPs associated with rate limiting failures might help — as long as you don’t lock yourself out in the process.
Don’t write the file to disk
What if we don’t want the malware to have access to the key though? Well, it would be better if we never wrote the file to disk in the first place. We could extract the key in memory and process it that way. But in this case the data still resides in memory.
Does that actually solve our problem alone? If we don’t write the file to disk but it is accessible in memory, some malware might be able to grab it from there. How would it do that? Let me count the ways. Malware might hijack a process that has access to that portion of memory. Malware might find a way to crash the system in such a way that it dumps out the memory or use some other method to get the system to leak memory.
One of the things I’ve spent a lot of time on in various security classes were different ways to create system crashes or use some other method of getting a memory dump. There are many tools that try to help you do this as well. Mimikatz is a tool that will grab Windows credentials out of memory, for example. What are the chances someone would be able to do this? It may not be a simple attack but it might be possible. For our purposes if we never write the file to disk and immediately terminate the instance for now, perhaps we would be OK. All these tactics are things I can use on penetration tests — and those are tactics attackers can use in the real world.
Use a Nitro Enclave
There are steps we can take to further protect data in memory such as with a trusted execution environment (TEE) or AWS Nitro Enclaves:
The purpose of an Enclave is to create a secure compute environment where only your authorized code can run. If you need to encrypt and decrypt data, only your code would have access to the encryption key in this environment, even in memory.
That’s interesting. We’ll take a look at that option in another post.
Shred
We can delete a file in a more permanent manner using a utility called shred.
Create a new file with different unique contents you can search on. I used this term in my file:
Blahdeblahbah
OK now instead of rm to delete the file we’ll use shred:
shred -zvu -n 5 yada2.txt
Now search for our keyword to see if we can find the file:
sudo grep -i -a -B10 -A10 ‘Blahdeblahbah’ /dev/xvda
Nope. It’s gone. I get back some gibberish but not my deleted file.
What is also very interesting is that in the gibberish — I still get a copy of the private key that was recently created. It’s probably not a good idea to keep the key hanging around. It’s stored in Parameter Store as explained in the lsat post. Let’s shred it and run the command above again (because I am curious).
Deleting the two files from the last post:
Up arrow a few times to get to the command to search for my deleted file.
I no longer see the key files when I run that command or similar.
One note on shred is to be aware of the different options which affected how thoroughly shredded your data will be.
How to Securely Delete Files in Linux Using shred
The following has some other methods for deleting files as well, not all of which exist on Amazon Linux.
3 Ways to Permanently and Securely Delete ‘Files and Directories’ in Linux
The last item on the list for deleting memory doesn’t work. Your mileage may vary.
Modifying our script that creates user keys
Well, as an immediate and simple improvement, I am going to use shred to delete the files generated by key generation script. I also add an encryption key to all EC2 instances. I could try to only create the information in memory but it is on the host for such a short time — if I want to go through the trouble to do that I would probably want to try to use an enclave. We’ll save that for another post.
Here’s the updated script where I added shred. Note that I change this script even more in later posts so stay tuned.
This is not a perfect solution. But it is certainly better than leaving files with keys laying around on hosts. An administrator who has access to the code could alter it to get the key value unless we run it on in an isolated environment where the administrator has no access, such as an automated batch job or better net a Nitro Enclave. Follow for more on both those topics over the course of this series.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
When What You Deleted is Not Really Deleted was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
