AWS Changing ARNs in Trust Policies — Big Problems
ACM.94 Trying to restore things after a user gets deleted leaves you in a malformed state for which there is no simple recovery
This is a continuation of my series on Automating Cybersecurity Metrics.
While updating my code in prior posts, the KMSAdmin user got inadvertently deleted so I couldn’t manage KMS keys by assuming the associated KMS role. The user was removed from the KMS Admin group. The Developer user was also removed from the AppDeployment group which hindered deployments by that role.
I tried re-running the CloudFormation templates that add the users to the groups. Since the template hadn’t changed re-running it has no effect. CloudFormation only deploys templates that changed, not things that are out of sync with what should be deployed.
If you have been following along you know that I added an output to force updating every time a KMS key gets deployed to solve this problem when AWS magically mangles ARNs in key policies. I tried that approach again for this problem.
Since I need a timestamp twice now I created a shared function for it (the principle of abstraction I’ve been telling you about repeatedly — don’t repeat yourself or the DRY principle):
I added the timestamp to my add_user_to_group function:
Unfortunately, that did not work for this scenario. I’m still getting an error that the KMS Admin cannot assume the group role.
Head over the trust policy. Aha. AWS does the same thing in trust policies that they do in KMS key policies, and it’s not updating for the same reasons. The deleted user ARN was apparently replaced with some sort of logical ID and the policy is no longer correct, nor does anything related to it work anymore.
Over to the role to add the timestamp to force an update same as above. Good thing I made get_timestamp a common function. 🙂
OK things are getting weird. My role is definitely re-deploying with the force update. I can see the new parameter and output in the template in CloudFormation. I can see the correct KMS Admin user passed in as a parameter. The stack shows updated. No errors in the deploy script.
And yet…..the trust policy has not updated. This is a problem.
If I try to delete the role in CloudFormation that will fail because all the key policies reference it. And lots of things reference all the keys.
UGGGHHH.
So now I could manually update the trust policy, but that will be bad. And it is at this point that I realize this is going to be a single blog post because this is very, very problematic.
I really don’t think Amazon should be changing customer policies.
So what can we do about this…I can try to force the trust policy some other way since the force update parameter is not working. I can temporarily add another user to the trust policy and then remove them again maybe.
What if I rename the group?
Well maybe the group alone doesn’t update the trust policy…
What if I change allow to deny?
Luckily, my IAM role is in a separate deploy script otherwise I would lock out my IAM admin if I had delete that user and group addition as well. I could also pull out the KMS admin into a separate script since I don’t want this change to apply to every other trust policy while trying to force this update. That seems safer. Let’s do that.
test.sh:
Well, something happened but not what we wanted. The updated failed. Here’s an error:
Here’s why:
Let’s change it back to Allow and try to figure something else out.
And now we have a huge mess:
This is the kind of nightmare you can get into with CloudFormation and the fact that AWS is changing these policies unbeknownst to the customer is a huge problem in my opinion. This does not seem like the correct solution to whatever problem it was intended to solve. Please stop doing this. #awswishlist
Let’s think about this for a minute. If a user is deleted and a policy references an ARN for a resource that does not exist, what is the risk? Nothing can use that policy because no related user exists to use the permission. There is no need to delete the user in this policy in that case.But …if someone does re-add the user back in with the same ARN, that user can now use the permissions in the policy. But is it really the same user? Someone could delete a user and add back in one they have credentials for to gain access to some permissions they shouldn’t have. That is the risk AWS is trying to protect you against.However, I would argue that it would be better to warn the user before making the change and disallow the change, or optionally allow the user to add a deny statement for that ARN to the policy rather than just change the policy and mangle a customer’s whole stack of resources in the process. I’m sure someone at AWS can think of a better solution based on how things work behind the scenes than what is happening above.
Luckily I’m just in a POC environment and I can literally delete everything and start over. I should probably write a script for that…
Next post.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
AWS Changing ARNs in Trust Policies — Big Problems was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
