ACM.75 An alternative to NATs and Internet Gateways
This is a continuation of my series on Automating Cybersecurity Metrics.
We looked at adding a VPC configuration for our Lambda function in an earlier post in this series where I demonstrated how a Lambda function that had Internet access may be abused. I also mentioned two options for allowing resources without direct Internet access to access Internet resources (or resources in other networks): NATs or Private Link with VPC Endpoints.
We covered how we might use VPC endpoints in our last couple of posts for our batch job architecture but first let’s look at the service in a bit more detail.
What is Private Link?
Rather than reword everything I’ll just refer you to the AWS documentation to answer that question:
AWS PrivateLink is a highly available, scalable technology that enables you to privately connect your VPC to services as if they were in your VPC. You do not need to use an internet gateway, NAT device, public IP address, AWS Direct Connect connection, or AWS Site-to-Site VPN connection to allow communication with the service from your private subnets. Therefore, you control the specific API endpoints, sites, and services that are reachable from your VPC.
You can create your own service and host it on AWS and allow people to access via Private Link. That way you could offer the ability for people in other AWS accounts to create a private connection that stays on the AWS network (as opposed to traversing the Internet) to access your service. From the AWS documentation above, you would construct your service with a load balancer and a defined endpoint that your users or customers would access.
AWS has set up many of their own services to work this way. You can create a VPC endpoint in your account to allow your resources to access a particular AWS Service.
What is a VPC Endpoint?
A VPC Endpoint is essentially a configuration that allows your services in your VPC to access a service made available to you via Private Link. Depending on the requirements of the service to which you are connecting, your VPC endpoint will leverage a network interface, load balancer, or gateway to access the service. The configuration of your VPC endpoint specifies the type of endpoint you are creating.
I am going to look at these endpoints in more detail in upcoming blog posts so we can understand exactly how they work and how to inspect traffic related to these endpoints.
How does it work?
The answer to this Stack Exchange question has a nice explanation of how AWS Private Link and VPC endpoints work. The answer is provided by Chris Williams who is an AWS Solution Architect so it seems to be a reliable source and the answer provides more clarity than some of the documentation at the time of this writing.
It does a good job distinguishing between routing and the fact that you end up with a network interface associated with your subnet when you create a VPC endpoint. There’s also a link to a great video from AWS re:Invent which covers the underlying mechanism of how it works under the hood in more detail.
How does the AWS Inteface VPC endpoint actually route traffic to regional service?
When you have a network interface associated with a subnet in your VPC, you should be able to inspect traffic to and from that interface using VPC Flow Logs. I explained what VPC Flow Logs are and why they are important here (anyone that works in incident response or performs threat hunting already knows this):
I’ve heard people say you don’t need to inspect network traffic if you’re using a VPC endpoint because you can just use CloudTrail. CloudTrail won’t give you visibility into traffic beyond HTTP requests. Any traffic that is not an HTTP request or that gets accepted or rejected on another port will not be visible in CloudTrail.
Examples:
DNS queries, tunnels, and exfiltrationICMP traffic and ICMP tunnels such as those used in the Target Breach.An attacker scanning all the network interfaces in your account.A network attack where an attacker is inserting bits in OSI layers below the application layer.
Just because your endpoint is only supposed to allow HTTP traffic doesn’t mean that is what it is doing. You need to monitor it for a possible misconfiguration. Additionally, if you see someone hitting that endpoint on other ports and protocols, you may have an attacker in your cloud network who is scanning for vulnerabilities. You won’t know that if you are not inspecting those logs. When I perform a penetration test I scan all network interfaces in the account.
Additionally, as I explained already in a post on Lambda networking, attacks are possible in layers prior to the application layer, where HTTP operates. Network logs can be captured at a lower layer in the network stack to obtain visibility into headers stripped off before getting to the application layer. On the other hand, the logs at the application layer can show you things that can’t be detected at lower layers by appliances that don’t piece your requests back together. You need both.
This is a big topic unto itself and not the main purpose of why I’m writing this post. It’s just that people keep coming up with this idea that they don’t need network logs or networking at all so I continue to address it to some degree. Hopefully I can provide more insight into this as we progress through the series but really, I just wanted to get this code implemented so I can use it!
AWS Services that work with Private Link
AWS provides a list of their own services work with Private Link on here:
AWS services that integrate with AWS PrivateLink
We are going to try out Private Link in our public and private VPCs in the upcoming posts and take a look at some network traffic.
Private Link Pricing
The costs for AWS Private link vary by the type of endpoint you need to create:
AWS PrivateLink Pricing – Amazon Web Services
At the moment the cost in us-west-2 is .01/hour for an Interface endpoint.
Data Transfer:
For a gateway endpoint per hour per AZ (you’ll generally want two):
Data transfer:
As you can see the Gateway option costs a lot more than the Interface option for data transfer and the option you can use depends on the service to which you are connecting and what it requires.
NAT Pricing
Logically Isolated Virtual Network – Amazon VPC Pricing – Amazon Web Services
As you can see the hourly charge for a NAT is 4.5 times higher for hourly alone. But then you’ll need to consider if you are only using one Private Link or multiple to different services and how much it adds up.
Per GB pricing is slightly higher than the Gateway Private Link option.
Which one is really cheaper?
What if you need to use Private Link for multiple AWS services. Do you pay or each service you use? The documentation says:
There is no cost to the number of endpoints you are deploying for PrivateLink.
So it sounds like you just pay one hourly rate plus whatever data you send but of course you will want to verify any assumptions with a beta test or POC.
https://aws.amazon.com/privatelink/faqs/
For the other factors the NAT is generally more expensive so it seems like any way you look at it AWS Private Link should be cheaper.
Throughput
Of course cost is not the only consideration. Some companies will need certain throughput or in other words they need to send a whole bunch of data all at once through the pipe while other organizations just send a smaller amount of data constantly over time. You’ll need to consider how much data each option can send at once.
Private Link — From the AWS documentation:
By default, each interface endpoint can support a bandwidth of up to 10 Gbps per Availability Zone and automatically scales up to 100 Gbps. If your application needs higher throughput, contact AWS support.
NAT Gateway
A NAT gateway supports 5 Gbps of bandwidth and automatically scales up to 100 Gbps. If you require more bandwidth, you can split your resources into multiple subnets and create a NAT gateway in each subnet.
So it seems like a NAT Gateway can support higher bandwidth needs out of the gate but maybe you can get more from Private Link from AWS Support. If you have that high of bandwidth needs you probably should be talking to an AWS TAM (account manager) anyway who may connect you with AWS solutions architects and possibly product managers to discuss your needs in more detail.
Performance
With a NAT your performance may rely in part again on your design:
A NAT gateway can process one million packets per second and automatically scales up to ten million packets per second. Beyond this limit, a NAT gateway will drop packets. To prevent packet loss, split your resources into multiple subnets and create a separate NAT gateway for each subnet.A NAT gateway can support up to 55,000 simultaneous connections to each unique destination. This limit also applies if you create approximately 900 connections per second to a single destination (about 55,000 connections per minute).
You’ll want to read all the requirements here and more importantly — beta test any solution. That is the beauty of the cloud. You can try something out for a while and turn it off when you’re done and stop paying for it.
I couldn’t find specifics with a quick search for Private Link but I presume with less appliances to traverse, Private Link will be faster. It’s touted as being “fast” but without specifics as compared to a NAT. It’s always best to test this out anyway with your specific architecture.
You also want to make sure your architecture works with whatever solution you choose if you are using Direct Connect, Peering, a Transit Gateway, hybrid or multi-cloud connectivity. Also, ensure that you will not bump into any quotas or limits. Definitely talk to AWS if you do to see if they can accommodate.
What about security?
Both options say they will keep traffic on the Internet but are both options equally secure? What later in the OSI model does each operate at? Can you capture packets on a Private Link? Will we be able to see the outbound logs for a Lambda function that were missing in our last test for a Lambda associated with a VPC but no NAT?
The best thing to do in this case — is test things out and make sure we get what we need from a particular implementation. That’s what we’ll do in some upcoming posts. But before we get to those, we need to think about our architecture a bit more and implement a few additional resources we’ll need for testing purposes.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
Automating Cybersecurity Metrics (ACM)
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
AWS Private Link and VPC Endpoints was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
