ACM.89 Using a KMS customer managed key (CMK) to limit access to data on EC2 Instances and EBS Volumes
This is a continuation of my series on Automating Cybersecurity Metrics.
Encrypting Volumes when you deploy an EC2 instance is a security best practice on AWS. In fact, you will probably want to enforce this throughout your organization. If you use the default AWS encryption, anyone who has permission to use KMS in your account can decrypt the contents of the volumes (drives) attached to your VM.
If you use your own customer managed key you can apply restrictions to who can use the key that encrypts and decrypts the volumes associated with the EC2 instance. By doing so, someone who doesn’t have permission to use the key cannot attach the volumes to their own instance and view the data it contains.
Encryption
When you start an EC2 instance it may have multiple volumes: a root volume on which the operating system and ephemeral data exists and one or more data volumes where you can store your application code and data. When you encrypt an EC2 instance you want to make sure you encrypt all the volumes. The method for encrypting the root volume is described in this blog post from AWS:
Set root volume properties for Amazon EC2 instances created with AWS CloudFormation templates
From the above:
To set the properties of the root volume for an EC2 instance, you must identify the device name of the root volume for your Amazon Machine Image (AMI). Then, you can use the BlockDeviceMapping property of an AWS::EC2::Instance resource to set the properties of the root volume.
What does that mean? Check out the CloudFormation options when creating an EC2 instance.
One of the properties is called BlockDeviceMappings:
Click on that property to view the details of it:
From there click on the Ebs property. EBS stands for Elastic Block Store or in normal nomenclature, a drive. Don’t get all technical with me I’m try to explain this in a way people can understand because when I first started using AWS, “EBS” through me for a loop until I realized it was basically a virtual drive. I fried a few physical drives in my lifetime — much rather be dealing with virtual ones. Check out the Ebs properties:
Now our description one more time:
To set the properties of the root volume for an EC2 instance, you must identify the device name of the root volume for your Amazon Machine Image (AMI). Then, you can use the BlockDeviceMapping property of an AWS::EC2::Instance resource to set the properties of the root volume.
This configuration above is where we can override the defaults that were used to deploy our EC2 instance in the last post because we didn’t specify anything.
Device Names
We need to specify the volume for the EC2 instance and override the configuration. In order to do that we need the device name. What’s that?
Head over to the EC2 dashboard and click on the instance we just created. Click on the Storage link and look at the “Device name” column. In this case we only have one device and it is the root device.
Here’s a bigger image so you can see the name of the device is /dev/xvda. You can also see in the Encrypted column that this device is not encrypted. You can also see the root device name at the top of the tab contents. There is only one device and it is the root device.
I have another instance in my account where I added two drives. You can see which one is the root drive, and that both are encrypted.
Now go check all the EC2 instance volumes in your account. What? Your volumes are not encrypted?? Better get on that before you hire me for a cloud security assessment or cloud penetration test as that’s one of the things I’m going to check. 🙂
You can encrypt your EBS volumes by specifying by creating a device block mapping. If you want to encrypt the root volume set the name of the device mapping to the name of the root volume. Set encrypted to true. Assign a KMS Key ID to use a CMK (recommended).
Encrypted: Indicates whether the volume should be encrypted. The effect of setting the encryption state to true depends on the volume origin (new or from a snapshot), starting encryption state, ownership, and whether encryption by default is enabled.
So…what is the effect? The documentation could be a bit clearer.
The pertinent information from the AWS documentation:
Amazon EBS encrypts your volume with a data key using industry-standard AES-256 data encryption. The data key is generated by AWS KMS and then encrypted by AWS KMS with your AWS KMS key prior to being stored with your volume information. All snapshots, and any subsequent volumes created from those snapshots using the same AWS KMS key share the same data key. For more information, see Data keys in the AWS Key Management Service Developer Guide.
Note that if you try to share an encrypted volume or AMI, the users that need to use it must have permission to use the KMS key that encrypted the volume.
Create a KMS Key
Alright, we already have our Developer encryption key used to encrypt secrets. We will also use that to encrypt our VMs. In a production environment I would probably create a separate KMS key for each critical application and maybe each customer depending on the number of customers that need to be supported and the sensitivity of the data. The downside, as mentioned, is the cost of each KMS key. If you have millions of customers, that’s going to add up fast.
At least segregating by application would help limit the blast radius in a data breach like Capital One. The applications that had buckets encrypting data with a separate key that the role on the firewall EC2 instance did not have permission to use were not affected by the breach (based on the account from someone who used to work there I spoke to recently).
Next add our Block Device Mapping properties to our EC2 CloudFormation template:
Add the KMS Key Export Name to the parameters and default it to the Developer Resources key we created earlier to encrypt our KMS key.
Here’s where the cryptic error messages start. When you try to deploy this template you’ll see an error like this. If you didn’t know that the KMS key was the only thing you changed you might struggle to interpret this error. This is the reason it’s good to deploy things small pieces at a time so you can test them.
I remember how this error drove developers crazy at Capital One. Instance i-xxxxxxxx failed to stabilize. Current state: shutting-down. Reason: Client.InternalError: Client error on launchThe company enforced that all EC2 instances were launched with encryption. The only problem was that we had 11,000 developers who didn’t all get the message. We had a lot of internal channels to get help and this question came up over and over when they could not launch images. Why it has to be a secret that you can’t launch the image due to a specific KMS error is beyond me. It wasted tons of our time and caused the developers a lot of grief.
Head over to CloudTrail to see what sort of error message we get over there. Remember we are using the AppDeploy role.
Now you might think you will find the error by looking at the EC2 event source but no.
CloudFormation? No. If you think about what we just configuring it was KMS. So search for the KMS event source. Click on the log entry that says AccessDenied (remember we added the Error column in a prior post).
Here we get a more reasonable and helpful error message:
“errorMessage”: “User: arn:aws:sts::xxx:assumed-role/AppDeploymentGroup/botocore-session-xxx is not authorized to perform: kms:GenerateDataKeyWithoutPlaintext on resource: arn:aws:kms:xxx:xxx:key/xxx because no resource-based policy allows the kms:GenerateDataKeyWithoutPlaintext action”
We need to give our AppDeploy role permission to perform the following KMS action:
kms:GenerateDataKeyWithoutPlaintext
Give this role permission to encrypt data in our KMS key policy. We can simply look up the ARN and add it to our comma-separated list in our deploy script:
Add the permissions to the AppDeploy role policy also.
I get the same error. Why? Our condition….we have specified that our key can only be used with Secrets Manager. Now we have a dilemma. We can either create separate keys for secretsmanager and EC2 instances, or we can generically allow the DeveloperResources KMS key to be used with any service.
Let’s look at the request that is getting denied in a bit more detail:
“eventSource”: “kms.amazonaws.com”
If we create an EC2 instance key and pass in that service name our key policy should work. Let’s just create a new key. It will cost me another dollar but that’s not breaking the bank.
Before deploying that I tried to delete everything relate to the DeveloperResources key in CloudFormation. But we need to update a few other things first before we can do that.
So..so I gave the two new keys different names and deployed those first.
Then I fixed the policies that reference the old key to use the new key. I changed the name of the export to DeveloperSecrets instead of DeveloperResources in the AppSec and IAMAdmins role policies:
Then I had to update the ImportValue for the new DeveloperComputeResources key in the AppDeploy role:
Then I could delete the other key.
We also need to redeploy our SSH secret.
I had to also update the key reference in the User Secret Policy.
Then we can redeploy our VM…Getting KMS error with AppDeploy Group. Nothing is every simple…
User: arn:aws:sts::xxxxx:assumed-role/AppDeploymentGroup/botocore-session-xxxx is not authorized to perform: kms:GenerateDataKeyWithoutPlaintext on resource: arn:aws:kms:xxx:xxx:key/xx because no resource-based policy allows the kms:GenerateDataKeyWithoutPlaintext action
Here’s the pertinent portion of the policy:
Clearly the kms action is present. The AppDeploymentGroup role is correct. The only thing left is the eventSource condition.
Clearly the event source is KMS:
Well, let’s try removing the condition.
Yes, removing the condition works. That seems like a bug for AWS to fix. Clearly the eventSource is kms.amazonaws.com. At any rate let’s get this working.
Now we get a different error.
“errorMessage”: “User: arn:aws:sts::xxxx:assumed-role/AppDeploymentGroup/botocore-session-xxxx is not authorized to perform: kms:CreateGrant on resource: arn:aws:kms:xxx:xxxx:key/xxxx because no resource-based policy allows the kms:CreateGrant action”,
We do not have that action in our policy:
Let’s add it. We could try to conditionally add it somehow but for the moment I am just adding it to see if we can get this working.
And..that works.
Running:
And encrypted!
Phew took quite a few blog posts to get here. We still need to figure out why the KMS key policy condition isn’t working right. I’ll take another look at that in the next post and we’ll test our SSH key to see if we can login to our EC2 instance.
Since I’m taking a break now — I’ll stop that instance to save some money. Don’t pay for resources when you’re not using them!
Follow for updates.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
Deploy an EC2 Instance with a KMS Encryption Key was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
