90. Formulating test scripts to deploy resources with dependencies in the correct order (and with minimal hassle)
This is a continuation of my series on Automating Cybersecurity Metrics.
Order of Operations and Dependencies with CloudFormation
When you use CloudFormation by creating a mammoth script with everything in it CloudFormation tries to handle all your dependencies for you. It misses a few things but for the most part it does a good job. However, then you have a large, unwieldy CloudFormation template that is difficult to troubleshoot.
As I’ve shown a number of times in my blogs a single space can cause the most obscure error messages. Would you rather find that space in 1000 lines of code or 25?
The reason I don’t use massive CloudFormation templates despite the dependency management that provides is for maintainability and easier troubleshooting. We can also independently deploy different resources if we need to. It’s easier to track and find the stack with the associated resource by keeping our templates small and use the naming convention I described previously.
I also explained that I don’t like the dependency on S3 for stack sets. There’s a whole catch 22 — I haven’t even created an S3 bucket yet and look at all the things we needed to deploy first. You’d have to manually create a bucket without encryption to create the stack with the KMS user to deploy the key and then deploy the bucket and …you get the idea. I prefer to avoid S3 for CloudFormation deployments.
Because I’m handling the deployment of individual resources myself, I also have to manage dependencies and deploy things in the correct order. It’s not too difficult if you break up your templates the way I’ve done in my GitHub repo.
Dependencies in my latest changes that affect my test scripts
Right now, I want to check in all that code I just worked on. I should have checked it in a lot sooner but I didn’t want to break anything for users trying out the existing code. Before I check it in again I need to update and test all my test scripts. I should ideally delete and re-deploy everything as well, but I’m going to initially just make sure it works.
As I explained in a prior post:
Before we can store a new SSH key in a secret we need to create the secret.Before we can create the secret we have to create the KMS key used to deploy the secret and encrypt it.We need to create our IAM users, roles, and policies before we can take any of the above actions using KMS or Secrets Manager.
For this reason I have two test scripts in the IAM directory now. I’ve broken apart the creation of KMS keys from the deployment of users, roles, and groups.
I think I mentioned before it is also possible to let users create those credentials for themselves, but at the moment I have not given my users ability to use the console. In this account, everything is supposed to be deployed in an automated fashion — no button clicks!
Edits to initial IAM admin creation in our test script
By the way, I also edited the default creation of the first IAM admin user, role and group to use a profile named IAM to simply things. The script tells you when you run it that you need to create that IAM profile. It asks you if you want to continue. Otherwise you can exit and set up the profile with any user.
Initially you might manually create a user to deploy the first automated user. Perhaps you keep that user around as a backup in case all the CloudFormation templates get deleted and require two people to use it later if it has a very powerful role.
After creating the initial IAM user, group and admin, the script pauses again and allows you to exit to revise your IAM profile to use the new IAM user and role that just got created with MFA.
Then, you can delete any temporary users and roles you created to execute the initial IAM administration script (after you validate the permissions are working correctly.)
Then you can proceed to run the rest of the test script which creates all the other resources we’ve created to date (and anything else I checked since this post.)
New and modified test scripts
As I mentioned I changed the admin user to use the IAM profile to simplify the code so that was removed from the test.sh script in the IAM directory:
I created a test_ssh.sh file in the IAM directory.
We need a test.sh script in our AppDeploy directory.
Also in our VMs directory.
Now that we have all that we can deploy things in the correct order in the test.sh file in the root directory of our GitHub repo code.
After testing that once with the existing stacks to confirm I had no typos, I deleted and re-deployed everything again.
At this point I realized I had a problem with my batch job credential scripts. Rather than redeploying one long test script with all my resources over and over, I could simply deploy the individual script in that folder and test it until I fixed the problem. After that I could continue trying to run the longer script.
After that one fix, everything seems to create correctly.
And now…I can check in the code!
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
Updating Test Scripts With Dependencies in Mind was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
