ACM.98 Using our SSH key stored in Secrets Manager to log into an EC2 instance
This is a continuation of my series of posts on Automating Cybersecurity Metrics.
When I first started using AWS it was very confusing initially to understand how to set up an AWS EC2 instance and the related networking so I could log into it. I knew I had to open up the firewall rules so I could connect on port 22 but there was a bit more to it. This post covers the thins you need to do to be able to connect to an EC2 instance on port 22 — and configurations you can remove if you don’t want people to connect to a virtual machine on AWS from the Internet
In our last post we started our EC2 instance using the AWS Console.
Allowing Users to Start Encrypted EC2 Instances in the AWS Console
We previously stored an SSH key in secrets manager to use to log into this host. We’ll need this key to log into our EC2 instance.
Creating and Storing an EC2 SSH Key in Secrets Manager
I wrote a few posts after that about properly security credentials in Secrets Manager for a single use as well.
I also wrote many posts and have the code available in GitHub to deploy the related networking.
GitHub – tradichel/SecurityMetricsAutomation
If you stopped your instance you’ll need to start it again.
Now let’s try to log into it. Based on the networking and EC2 instance we created do you think we will be able to? I already know the answer.
Retrieve the SSH Key from Secrets Manager
Login as your Developer user and navigate to AWS Secrets Manager. As I explained in the post where I was having the KMS issue, redeploying that policy would fix the ARN and grant our user access the secret. And yes, it does.
Now open up that secret and save it to your local machine and name it developer.pem.
I’m presuming in this post that you are familiar with how to log into an EC2 instance with an SSH key. If you are not, there are many other posts covering that topic, including one I wrote that explains how to do it from a Google Chromebook. If you’re using a Mac you can use the command line to login as I’m going to do in this post. If you’re using Windows you can use the Windows Subsystem for Linux (WSL) with the commands in this post or a tool like Putty.
In my case I take these steps:
Edit the secret.Select all the text. Copy it.Open a terminal window.Navigate to the folder where you want to store the private key.Type this command:vi Developer.pemPaste in the contents of the text.
Well this doesn’t look right (and of course I’m no longer using this key):
Secrets manager is not properly preserving the formatting so fix it.Change the permissions of the key file to read only or it won’t work.chmod 400 Developer.pem
Attempt to SSH into your EC2 instance
Once your instance is running:
Click on it and then choose Actions > Connect from the top right menu.
Click on SSH client.
(Yes if you know networking you see the problem but humor me.)
Click on the box next to the ssh command and copy it.
Open a terminal window (again assuming you are on a Mac or Linux, otherwise follow the instructions for Windows.) Make sure you are in the folder where your pem file is located.
Most likely your system is going to hang out for a while like this until it finally fails.
What’s the problem? That IP address is private IP address not a public IP address. What does that mean exactly? Recall that when we created our VPC we defined an IP range.
That IP range was as private IP range. When resources that exist in your VPC try to communicate with each other they will try to connect via their assigned private IP addresses. When you try to connect to that private IP address from your laptop at your home or in your office, your laptop is try to connect to something at that IP address within your own network. If you happen to have something in your local network at that IP address running an SSH server, you might get a response, but it’s not from AWS and your key won’t work.
The local route in an AWS VPC
Recall that we deployed this instance in our Remote Access VPC. Navigate to that VPC and on the VPC dashboard and click on the route table.
Look at the routes. There’s a “local” route that references the CIDR block we assigned to the route table. That’s not a route we added. It was automatically added by AWS and you can’t remove it. It allows resources in your VPC communicate with each other. As explained in the post on NACLs you can deny the traffic, but you can’t remove the route.
Automated Creation and CIDR Allocation for Subnets on AWS
Public IP Address for an EC2 instance
If we were trying to connect to our EC2 instance from within our VPC we would be able to connect but when connecting over the Internet we need to connect a public IP address.
What’s the public IP address for our EC2 instance? Well, let’s go take a look at it. On the EC2 Dashboard click on your EC2 instance.
There is no public IP address.
On AWS (and GCP) you cannot SSH into a host with only a private IP address via the Internet. (Azure creates a magic outbound public IP address when you don’t create one — what?!)
You could connect to your EC2 instance if you have set up a hybrid connection with a VPN or Direct Connect to connect your local network to AWS via a private connection, but we haven’t done that.
Add a public IP address to your EC2 instance
How do we get a public IP address? Let’s revisit the CloudFormation documentation for an EC2 instance.
There’s no property that indicates adding a public IP:
That’s because the Public IP address actually comes from the network interface.
Click on NetworkInterfaces. This property is a list of network interfaces. Click on Network Interface.
There’s our public IP address property:
Return to the prior page. Scroll down and AWS provides a working example to add a public IP to an EC2 instance automatically. This is what we need to add to our EC2 instance to get a public address.
Add the Network Interfaces property to the EC2 instance template. Move the ImportValue statements for the security group and subnet to the Network Interface and remove them the EC2 instance properties. When you add a network interface you add the networking to that instead.
Some EC2 instance types can support multiple network interfaces residing in different networks. This can be a security problem so you shouldn’t do that unless you really know what you are doing. Some network and security appliances do this intentionally to monitor and route traffic.
You can find out how many interfaces each EC2 instance type supports here:
Notice that when we add a network interface, CouldFormation deletes our existing EC2 instance and creates a new one.
When that update completes, head back over to your EC2 instance and you should see that now it has a public IP address:
Now return to the Actions > Connect on the top right.
Now we have a public IP address. Repeat the steps above to connect to your instance.
Network Troubleshooting
Now let’s say you cannot connect to your EC2 instance. Then what? In my case I’m getting a network timeout.
Navigate to the VPC dashboard. click on Flow Logs. Click the the link to view your flow logs.
Under Log stream you’ll see a list of values starting with eni. That stands for Elastic Network Interface. Each Network Interface will have its own log stream and related logs. Why so many here? I deleted and recreate the EC2 instance a number of times. Click on the eni with the most recent timestamp to view the logs for your active EC2 instance.
We have a whole bunch of rejected traffic. Uh-oh what is that?
I know from memory that is an AWS address but if you want to verify you can look it up at Arin.net.
Well, something is reaching out to some AWS service over the Internet on port 443. Recall that we blocked all outbound traffic by default for ephemeral ports. I’m not sure what that is but that is not my concern at the moment.
Well, I am a bit curious. I can’t help it. I have to look.
Head over to the AWS IP ranges:
https://ip-ranges.amazonaws.com/ip-ranges.json
We’re not going to find that exact IP address. It’s one of the IP addresses in one of those ranges. I can start by looking for 52.219.104 without going into a detailed explanation of CIDR blocks. I don’t find anything so I have to go more broad an search for 52.219.104.
There are multiple entries. Here’s one:
Well 104 is after 60 but it is in that range? If you don’t know CIDR block ranges at a glance (I know some, not all) head over to ARIN’s CIDR calculator.
ARIN – American Registry for Internet Numbers
Enter the prefix and range from list item above.
Our IP address in question is not in that range. Keep searching.
This one look about right:
Yes, our IP is in this range.
Interesting. The instance is trying to reach out to S3 in the us-west-2 region. I’d have to do more inspecting to find out why. Perhaps it is trying to automatically update itself.
For now I want to know why I cannot connect on port 22. I should see rejected traffic from my local network to port 22 being rejected.
Let’s search for port 22:
Nothing. Well what that means is that my traffic never even got to AWS. It’s being blocked by my local network. I happen to have what my housemate seems to think is an overzealous firewall at times.
If this happens to you, then you would need to open up your firewall to allow you to connect to the public IP of your EC2 Instance — or you can configure a private connection using a VPN or direct connect to connect to the private IP.
For now, I know I have what is called an Elastic IP address that my network will allow through. I’m going to assign it to my EC2 instance. More on EIPs later just know that I have allowed access to that specific IP address through. my firewall.
Well I can see traffic on port 22 in the logs. But I know that’s not my traffic. It’s not my IP Address. I can see that these remote IP addresses are trying to get to my EC2 instance IP address (note that it logs the private IP address) from ephemeral ports to port 22.
I can look the IP addresses up in ARIN just like before.
Interesting but not not helping me connect to my EC2 instance.
I have two EIPs and I don’t think that was the right one so let me try the other. Oh my goodness, what is this:
Scary, huh? Well the problem is I as connected to this IP and it was assigned to a different VM. I took if off that VM because I know I can get through my firewall with it and assigned it to my Developer machine. This message is warning me that the IP I’m connecting to is not going the same host it was before. Perhaps someone was able to deploy their own VM and get me to put my user name and password in on their host so they could steal it.
In this case it’s just me swapping the IP address. In order to clear this message I can delete my known hosts file on my local machine.
Navigate to your home directory and type this command (on a Mac or similar command on Windows to view hidden files).
ls -alcd .ssh
Delete or rename known_hosts to something else.
Now what’s this:
We’ll explore that message in the next post. For now type ‘yes’ and enter.
Success. We’ve logged into the EC2 instance we created.
In some upcoming posts I’ll show you how to add a permanent IP address and configure rules on a local firewall and GitHub to restrict traffic.
Follow for updates.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)GitHub – tradichel/SecurityMetricsAutomation
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
Connecting to an EC2 instance via SSH (and when you can’t) was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
