91. Granting AWS Console access for Secrets Manager Secrets to address IAM Policy Limitations
This is a continuation of my series on Automating Cybersecurity Metrics.
If you recall from a prior post we had some complications when trying to protect a user-specific secret due to the way AWS policies work. We couldn’t fully achieve our objective of requiring MFA for a user-specific secret.
We had the following problems to overcome:
When we enforced MFA, the secret was not available via the command line by a user with long-term credentials (A developer secret key and access key). Those type of credentials don’t support MFA at the time of this writing.We can’t create a user-specific secret policy for a role, unless we were to give ever user their own role and allow only that developer to assume it.Due to secret naming conventions, we can’t enforce a user-specific secret in a group policy.
That leaves us pretty much with a user policy and console access to enforce MFA when retrieving and setting user-specific secrets.
What I’ve decided to do is try to enforce MFA and then give the Developer user in my framework console access, so they can login and get the SSH key using the secrets manager console. The MFA enforcement should work with the console. They just won’t be able to access the secret programmatically, and that may be fine.
If a developer wanted to get the secret programmatically using their local laptop top, they would have to install the AWS CLI on their local machine, and that is exactly what we are trying to avoid. We can reduce the local attack surface on a developer’s local machine by having them install tools on a cloud VM. We can avoid storing cloud credentials on the same host where a developer reads emails (phishing being one of the most common forms of attack used for stealing credentials). So it actually makes more sense to have them retrieve the credentials from the AWS console that they require to login to that remote host.
Warning: this post is long because I’m explaining a lot of issues I faced trying to get to the end result that others may face. The end result code is towards the bottom and in the GitHub repo.
Enforce MFA in the User Secret Policy
Recall that we created a user-specific secret policy in a series of posts. It turned out to be more complicated than I expected.
Thankfully I made a note for myself in the above policy because this is all very inconsistent and I already forgot exactly what the issue was.
MFA is not enforced if you add the “ifexists” option recommended in the AWS documentation when a user is using AWS developer credentials without a role for Secrets Manager (i.e. using developer credentials with the AWS CLI) because the MFA indicator in the request is not present in that case.
I wrote about that here:
User-Specific Secrets on AWS: IAM Policies
Well, if we enforce MFA, hopefully we can restrict access to the AWS console since MFA should be present. We’ll test it all out to make sure.
Add User Access to AWS Console using a CloudFormation LoginProfile
Next we want to add the ability for a user to login on the AWS console.
We will give our developers a password via a login profile which will enable console access:
The login profile requires a password.
Recall that if we want to pass in a value via a CloudFormation parameter it will be visible in the AWS console or elsewhere. And yes, even if you use the NoEcho option it may be accessible — that’s not recommended:
We can use Secrets Manager to generate a random password and reference that in our password property.
AWS::SecretsManager::Secret GenerateSecretString
The problem: getting the password to the user and only the user
So now we have a tricky problem. We have a random user password and somehow we need to get the user their initial password to login.
Do we want to email them the password? Email is generally not that secure. However, if it is a temporary password they are going to change immediately that might be ok. Unfortunately there’s no way to add an email that I can see to the user profile here to have AWS do it automatically for us.
What if we put the password in the outputs? Then anyone in the console can see it. Hmm. What if we lock it down to an AWS account only accessible to AWS IAM admins? Maybe.
What if we used AWS SNS or AWS pinpoint to send an email or text to a user? Seems a bit complicated and possibly pricey with per-user resources. I haven’t thought about it much because I think it’s overkill for my particular use case. If you are at a large organization and want to develop a fully-baked workflow maybe you would look into that.
We could also store the password in a secret and encrypt it with an IAM Admins KMS key. But a user can’t get a secret without access to the console and we’re trying to get the user the password to access the console.
AWS could add the option to add an email to this configuration to send the user their password. Maybe that’s not included here because emails are considered PII and would be visible as parameters in the AWS Console.
A bad approach — CloudFormation outputs
There are numerous ways to solve this problem but as a first cut, I’m going to start by generating the password and included it in the stack outputs and immediately change it. This is not a good approach, but at the moment I’m the only one in this account and I’m just verifying that I can get a password and that it works.
So here’s my first cut at the new user template:
And I get this error:
An error occurred (ValidationError) when calling the CreateChangeSet operation: 1 validation error detected: Value ‘[AWS::IAM::User, AWS::SecretsManager::Secret GenerateSecretString]’ at ‘typeNameList’ failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 204, Member must have length greater than or equal to 10, Member must satisfy regular expression pattern: [A-Za-z0-9]{2,64}::[A-Za-z0-9]{2,64}::[A-Za-z0-9]{2,64}(::MODULE){0,1}]
Wut.
I copied the type off the page and there’s a space in the name. There’s also no example code at the bottom of the page like there is for most CloudFormation resources. I’m guessing the space is the problem. None of the properties are required. I remove the space and try again.
“Unrecognized Resource Type”
Nope. Apparently you cannot use the password generation functionality standalone. That would be very nice. Someone else has requested it here:
Storing the new password in Secrets Manager
Perhaps AWS doesn’t want you outputting passwords as what I’m doing is not a great solution anyway. We can just generate a password and store it in a secret as explained here:
We can reference the value of the secret using “resolve” as explained here:
Retrieve an AWS Secrets Manager secret in an AWS CloudFormation resource
Conditionally including the console login in CloudFormation
While testing, I added a condition to the LoginProfile but that is not allowed. I really wish you could put conditions on properties.
Encountered unsupported property Condition
Instead, I used an Fn::If operator and used the condition with that as shown below to add a secret for a user who is configured to have console access.
I won’t bore you with all the details abut that is not what I started with. I had to add the Join operator to concatenate the values to correctly retrieve the AWS Secret.
Issues with resolve:secretsmanager functionality
I think the AWS documentation here is wrong for resovle:secretsmanager in at least one place because it says you can reference your logical ID of your resource in the CloudFormation template if I understand it correctly. But you can’t.
When you do that you get this error. Huh?
Secrets Manager can’t find the specified secret. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: xxxx; Proxy: null)
I wish this error would print out the secret name it cannot find. I also wish that it would tell you when you are not using a valid ARN and one is required. That ended up being the problem.
As the CloudFormation documentation states, when you use !Ref with a secret, you get the ARN for a secret. There’s no way to use GetAtt to get any other value. That is your only option.
As it turns out, resolve:secretsmanager requires an ARN to reference the secret so luckily it worked out. It just took a lot of trial and error to figure this out because this doesn’t seem to be documented on the AWS site. I figured it out in a roundabout way from a stack exchange post.
I then figured out that I could create the resolve statement with the ARN by concatenating the values I needed using Fn::Join:
I got some other errors as well…this was all very time-consuming, and I wrote the next post in between all this because I got tired of manually deleting stacks while trying to figure this out. The code will be updated to handle that for you when they are in a ROLLBACK_FAILED state.
Invalid JSON for the SecretStringTemplate
At one point I was getting this error for the SecretStringTemplate.
CloudFormation SecretTargetAttachment return SecretString is not valid JSON
The original issue is that I was using a Sub and it wasn’t correctly inserting the NameParam value in the correct place. Not sure if I had a typo but I changed to Join to concatenate the values and then my quotes were off. I ended up with this:
Permission to call GetRandomPassword
I had to add permission to generate the random password for the IAM User:
User: arn:aws:sts::xxx:assumed-role/IAMAdminsGroup/botocore-session-xxx is not authorized to perform: secretsmanager:GetRandomPassword because no identity-based policy allows the secretsmanager:GetRandomPassword action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxx; Proxy: null)
I’ve shown how to do that numerous times in other posts. I updated the IAMAdmins role in our framework and deployed it.
DeleteSecret concerns
I also noticed that when the stack fails I also get this error:
User: arn:aws:sts::xxx:assumed-role/IAMAdminsGroup/botocore-session-xxx is not authorized to perform: secretsmanager:DeleteSecret on resource: xxx because no identity-based policy allows the secretsmanager:DeleteSecret action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxx; Proxy: null)
It would be nice if AWS would make it easy to implement the the following rule logic:
A user can only delete a secret that user created or a specific group created.
But it’s not. The way we could fix this would be to prefix our secrets and only allow access to secrets with a certain prefix. It’s not a perfect solution (I explained the problem with prefixes in other posts) but it helps. The code in the GitHub repo uses a similar to restrict actions on CloudFormation stacks.
For now, I’m going to just add the delete permissions to the IAM user role, because what I hope to do in the future is move IAM administrators to their own account and they would then be restricted to secrets in that account for the delete operation. Secrets can be shared across accounts so hopefully that will work for secrets that must be shared read only to people in other accounts. More on all that later.
Share Secrets Manager secrets between accounts
Update the role and deploy it. Then try again to deploy the secret.
Initial Successful State — which didn’t actually deploy the Login Functionality
I got to a successful state — but as you may have noticed, I did not yet explain how to pass in a parameter to tell the template to grant console access to any of our users. So this was prior to trying to use the LoginProfile.
Updated deploy script to pass in a parameter to allow console access
Next I updated my deploy script to pass in console access, true or false:
Then I added the parameter if console is set to true in my deploy_user function:
Now run the script again. The Developer user is the only one that gets console access and that user gets deployed first now.
Can’t delete a user when it’s in a group
Cannot delete entity, must remove users from group first. (Service: AmazonIdentityManagement; Status Code: 409; Error Code: DeleteConflict; Request ID: xxx; Proxy: null)
I went head and just deleted the stack that added the user to the group and the developer user.
Also…CloudFormation does not delete IAM users. So you’ll have to delete the existing developer user from IAM manually. That means we have to set up MFA and credentials again as well. So I guess if you try to add a login subsequent to initial creation of the user you have to delete them first? You could automate that. I didn’t do it for this post.
Ah, of course. Now we have the problem that the IAM user has to get the secret value to assign to the user.
secretsmanager:GetSecretValue — more consideration of secrets access
Next I got an error because IAMAdmins don’t have GetSecret access. This was by design if you refer to prior posts.
User: arn:aws:sts::xxxx:assumed-role/IAMAdminsGroup/botocore-session-xxxx is not authorized to perform: secretsmanager:GetSecretValue on resource: Passwd because no identity-based policy allows the secretsmanager:GetSecretValue action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxx; Proxy: null)
Now, if we simply add that permission to our IAM user then they can also see the SSH keys in AWS Secrets Manager without other restrictions. We are trying to prevent them from seeing and all our hard work to limit permissions could be for naught.
However, as I explained in a separate post already, I created a separate User to create the resource policies on those secrets and the IAM user cannot change the resource policy. The resource policy also does not allow the IAM User to get the secret value. So I think we are OK. I plan to review all these policies again later.
AppSec Role for Secrets Management
This complexity and understanding how all your IAM and resource policies work together is why you need teams and architects dedicated to IAM. It is probably one of the most complicated things to get right in the cloud. I think it more complex than networking — if you really think through your permissions and segregation of duties and do it right.
Anyway, I added that permission to the IAM Admins role and redeployed. I don’t really like this part of the framework or code and have different ideas how I would implement it in a large organization but this will get us to the next step. This is taking forever. Some of this functionality could definitely be simplified by AWS and still get to the same result I am after.
The final template — create a user and store the new password in a Secret
Finally, after hours, this is what I came up with (and I don’t consider it complete, but it does what we need in a better than no security manner):
What are the problems that still exist?
No encryption key on the temporary passwordNo resource policy on the secretNo deletion of the temp password after the user has reset their password (we could write a Lambda Function or Batch Job to clean those up to save money and reduce risk if someone fails to reset their password in a timely manner. For now the IAM Admins can delete the secret after providing it to a user.)I would prefer that the IAM administrator never had the secret in the first place.
Re-creation of all the users
Now the kind of annoying thing is that this template is forcing re-creation of other users, not just the ones to which I want to add a password. What is up with that? Do I need to delete all the users and re-deploy all the credentials?
That, right there, is a really strong case against using CloudFormation to create users, unless you are sure you have the script you want to use long term. Yikes!
However, as it turns out, I think the issue was that I initially changed the logical ID of my resource in the CloudFormation template. Then I changed it back. I had partially deployed the KMSAdmin user with the new logical ID. Every other user was untouched because I killed the script before it went further or maybe it had an error.
I deleted the KMSAdmin user and it re-deployed just fine. Then all the other users deployed just fine as well. I highly, highly, highly, recommend testing all IAM changes in an account that mirrors your production environment before you roll them out to production.
That means you have the branch in your source control repository that gets you to the exact production state. Then you deploy your new branch on top. Also deploy small changes at a time.
Anyway, make sure you don’t change the logical ID for your user (I think that was it) in your CloudFormation template if you don’t want to have to delete and re-create all your IAM users, or perhaps give them new names.
Seeing if our password works
OK, we have a user name and password but what I didn’t think about is that I also use SSO in this account. Can I still login to the console with IAM after all that? I have various accounts in various states for testing and never actually done this before. Yes you can…get the URL off the IAM dashboard.
Get the new password from Secrets Manager:
This is where I realize I forgot to put a condition on my Secret in the CloudFormation template so it created a password for every user. However, even though the secret is created, I checked the permissions for users that should not have console access on the IAM dashboard, they do not.
But out Developer does:
That’s great, but the password doesn’t work. I’m guessing that the resolve secret functionality doesn’t actually work and it’s sticking that whole string in the password field instead of the actual password.
Easy enough to find out. Put it in the outputs:
Yes. I don’t know how this function is supposed to work but nothing I tried works. That is a post for another day. For now I’m going to manually create a new password for my user and test it.
Using this option we also do not have non-repudiation since administrators can see the initial password on reset.
Let’s see if I can login, at least. Yes. And I have to change the password.
Maybe I’ll try to fix that secret issue later but for now I need to move on.
I had problems with this functionality because I had logged in using a different use and then logged out. Even after logging out I could not properly login to the console. Always test your applications for logging in and out as different users and make sure no remnants of the prior user remain. Don’t force people to clear cookies.
I closed and re-opened a new incognito window which effectively cleared all the cookies and then I could reset the password and login. I mention those issues because they caused some of my students grief in classes.
Monitoring New User and New Credential Creation
It will be important to monitor the use of new credentials to make sure no one gets into them during the process of distributing accounts and console access. It’s event better if you can restrict access to your account and the AWS console using IP conditions in AWS IAM policies — which alone are not adequate at all, but combined with strong passwords and MFA, offer multiple layers of protection.
This is where a VPN can help you. Many people misunderstand one of the useful purposes of a VPN. Yes it can encrypt your traffic, but in an enterprise environment — or even a e-commerce startup like the one I ran years ago when I first started using a VPN for this purpose — you can set up restricted access to your network and then once authenticated to the network, people can access other resources on it. It’s harder to create IP restrictions when you have people coming from many different IP addresses. I’m going to skip the IP address restriction for the moment but be aware that it exists and how it can help.
If you want to create alerts when new users are created check out this post:
Send a notification when an IAM user is created
Add the users back into Groups where removed
At any rate I can login. Now, however, I need to go re-run my group scripts to add the users back into their respective groups. Once I do that and I
Recreate SSH secret and EC2 instance
Now, it is possible that my user can still access the SSH secret because their ARN is the same, but I’ll go ahead and recreate it. The other thing is that if I delete the SSH key associated with an EC2 instance, then applying a new key to it is not exactly simple. It can be done but since we have nothing on our EC2 instance I will simply delete and recreate it.
At this point, we should be able to get the SSH key from Secrets Manager as the Developer and try to log into our EC2 instance. We’ll try that out in a couple of posts, after I show you how I fixed the issue with deleting and recreating stacks.
Stay tuned for more.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
Github Repo
GitHub – tradichel/SecurityMetricsAutomation
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
Autogenerated Passwords in CloudFormation for AWS Console Access was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
