The limitation in this scenario was a network appliance in the datacenter, a site-to-site (STS) VPN is operational between this appliance and AWS. All traffic from AWS to the datacenter resources use this VPN for connectivity. The principle of a Network Address Translation (NAT) Gateway would fit in this scenario, as when the NAT Gateway forwards network packets, it replaces the source IP address with its own IP address. Until recently the NAT Gateway was only available for public/internet traffic, recently a private/internal version came available, this looked right for the challenge we were facing here.

First a test setup was created in AWS as to avoid risks of disturbing the production environment. The Netgate pfSense appliance is available on the AWS Marketplace, so we can emulate a potential on-premise environment which makes it possible to test the complete STS VPN setup within a single AWS Account. PfSense is chosen because it is a rather inexpensive Firewall/VPN/Router appliance with a clear and easy to use GUI.

Setting up the destination environment

The destination environment can be another account or another VPC. The EC2 PfSense appliance from the Marketplace will need to be installed here and will act as the customer gateway for the STS VPN.

Instruction on installing:

• Create a new EC2 instance
• Use the PfSense AMI from the marketplace, you will need to accept the offer before the instance is created.
• For testing purposes a T3.micro instance should be sufficient to get the tests done.
• You can continue without keypair but then you won’t be able to SSH into the server, this is not needed for the test since we are configuring using the web GUI.
• Edit Network settings and place the instance in a public subnet and make sure a public ip address will be assigned.
• We recommend using the wizard to create the security group, and update it afterwards when necessary. Please note : We'll create a new security group called 'Netgate pfSense Plus Firewall/VPN/Router-22.05.1-AutogenByAWSMP--2' with the following rules: , this security group can best be edited as soon as the service is deployed and then replace 0.0.0.0/0 for SSH, HTTP and HTTPS with your own external IP. Inbound connections are only used for management.
• Deploy the instance.

Write down after deploying the Auto-assigned IP address. This is the customer gateway address.

Setting up the source environment

Make sure you have a tiered VPC, with public and private subnets, Deploy STS VPN setup including a VPN Gateway, Customer Gateway and VPN Connection and corresponding routes.

I’ve found that the Terraform code from Cloudposse https://github.com/cloudposse/terraform-aws-vpn-connection works very good.

Example config:

module "vpn_connection" {   source = "cloudposse/vpn-connection/aws"   # Cloud Posse recommends pinning every module to a specific version   # version = "x.x.x"   namespace                                 = "eg"   stage                                     = "dev"   name                                     = "test"   vpc_id                                   = "vpc-xxxxxxxx"   vpn_gateway_amazon_side_asn               = 64512   customer_gateway_bgp_asn                 = 65000   customer_gateway_ip_address               = "18.184.85.65"   route_table_ids                           = ["rtb-xxxxxxxx", "rtb-yyyyyyyy", "rtb-zzzzzzzz"]   vpn_connection_static_routes_only         = "true"   vpn_connection_static_routes_destinations = ["10.112.160.0/20"] }

replace the customer_gateway_ip_address with the public ip of the PfSense instance and vpn_connection_static_routes_destinations with the CIDR in which the PfSense instance resides. route_table_ids with the route table id’s of the private subnets.

After deploying you can download a configuration file for PfSense. Basically any configuration file will be ok as long as you retrieve 1 remote gateway address and the Pre-Shared Key, these are required in the next step.

Initiating STS VPN connection from PfSense

The STS VPN connection can’t be initiated from AWS VPN service so we need to do this from the PfSense appliance. Get instance screenshot of the PfSense setup from the EC2 dashboard to validate if it has started correctly and is ready for setting up VPN:

This should show something like :

You can now access the login page for PfSense using your webbrowser using https://<public ip>. We will get an SSL Error since we haven’t configured a hostname and created a certificate, for now ignore SSL errors. Login with ec2-user and password provided in the screenshot. If the screen shows events instead of the password then restart the instance and try again. You can’t copy / paste so have fun with that

When logged in Select VPN → IPsec:

Select the green Add P1 button.

Fill in information, change Description if you like. Make sure you use the Remote Gateway and Pre-Shared Key retrieved from the configuration in the previous step.

All settings in the downloaded VPN configuration file from AWS VPN shows the least secure options possible, although they are used in this example I would advise to use more secure options in public exposed or production environments.

More secure would be :

• Phase 1: Encryption Algorythm : AES
• Key Length : 256bits
• Hash : SHA2 512bits
• DH Group : 21

Click Save or Add.

Click the blue Show Phase 2 Entries.. Will have zero and click the green Add P2 button.

Fill in the form as below. Change description if you like. Local Network, type Network with CIDR range of destination network. Remote Network CIDR Range of remote network

All settings in the downloaded VPN configuration file from AWS VPN shows the least secure options possible, although they are used in this example I would advise to use more secure options in public exposed or production environments.

More secure would be :

• Phase 2: Encryption Algorithms : AES
• Key Length : 256bits
• Hash Algorithms : SHA2 512bits
• PFS key Group : 21

Save when done.

Go to status → IPsec

Click Connect P1 and P2s, this should connect almost directly:

We now should have a working VPN Connection.

Creating an instance in source environment for testing

Create a simple bastion EC2 Amazon Linux 2 instance so we can test. Make sure the EC2 is deployed in a subnet with internet access ( public would be fine ). AmazonSSMManagedInstanceCore IAM role is added so we can connect using SSM. We start with testing if we can reach the internal IP of the PfSense instance:

Connect to the instance using SSM and ping the internal ip of the PfSense instance. in this example 10.112.160.206.

As you can see below this does not work with default routes on the subnet since 10.112.160.206 will be matched to the 0.0.0.0/0 destination and routed through the internet gateway. ( 10.112.160.206 is not accessible from the internet since it is a private IP address).

Routes:

If we add a route for the network 10.112.160.0/20 to the virtual private gateway, this should route traffic through the VPN. Ping again:

We now do get a connection, start a continuous ping and head over to the PfSense interface to find out out the source IP:

In PfSense goto Diagnostics → Package Capture.

Start, wait a few seconds, press stop and View Capture. Output should be something like this:

Source IP here is 10.30.0.89 which should match the Private IP4 address of the instance from where the ping is running.

Add the private NAT Gateway

We now get to the interesting part to see if we can get a single outgoing IP for multiple hosts using NAT. Create a private NAT gateway, it needs to have its own subnet. After creating it, note the Private IP Address: in this example this is : 10.30.9.65

The subnet in which the NAT Gateway runs should have a route to the virtual private gateway:

We once more edit the routing table of the subnet where the test instance resides, change the route from Virtual Private Gateway to the NAT Gateway:

Start a continuous ping and run a Package Capture in PfSense:

Result: We see now 10.30.9.65 as a source from our ping and no longer the IP address of our test EC2, which means the NAT Gateway is working as intended.

Conclusion:

We received a customer request that initially required some testing before we were able to implement it in the production environment. By using a simple and cost-efficient appliance like the pfSense, we were able to simulate the customer environment within AWS, and perform the test without customer efforts. This way we were able to verify the behavior of running workloads directly, or behind a private NAT Gateway in a fairly easy to setup test environment. Basically simulating an on-premise environment can be a very good way to get hands on experience with potential on-premise appliances without having to rely on the physical environment, with potential risks of disturbing workloads. There are a lot of appliances available from almost every hardware vendor: FpSense, Cisco, Arista, Fortinet are just a couple of examples. Most offer trial licences so should give a great way of testing setups quick, cheap and easy. Once our tests were finished we were able to implement this setup in the live customer environment with minimal downtime.

We hope to have provided some insights into how we approach such a challenge, and how far some creative thinking and testing can bring you. In this case we did not have to bother the customer Aiosyn for testing, and were able to implement the working solution very efficiently. Secondly we hope this blog will help you get insights in the required changes on AWS services to build such a custom networking solution.