In some circumstances there may be a requirement to setup an IPSEC Site-to-Site VPN tunnel into an AWS VPC using something other than the AWS VPN Service.  This usually means configuring an EC2 Instance based VPN endpoint and is what AWS refer to as a “Software VPN”.

Unlike the AWS VPN Service (which is managed as a service by AWS) this EC2 Instance is managed by the customer.  As these Site-to-Site VPN connections tend to be mission critical it is a good idea to build a setup with some self-healing, monitoring and configuration backups.

This example used EC2 Instances running CentOS 7 with LibreSWAN (an Open Source project based on OpenSWAN).  Focus is only on one side of the VPN and it is assumed that the “remote” side is a generic IPSEC capable device (Cisco ASA etc).

This example has the Software VPN running in an EC2 Instance configured in Auto-Scaling Groups, using secured s3 Buckets to hold the configuration files.  These buckets will be accessed by the EC2 Instances using IAM roles.  The config files in the buckets will be protected using Versioning and Lifecycle Rules.

Create the base AMI Template.

Spin up CentOS 7 EC2 instance from the AWS Marketplace and ssh in.

Remove / disable any firewalls (iptables / firewalld).  By default the firewall should be completely open but in the event it isn’t then do the following (REF : https://www.liquidweb.com/kb/how-to-stop-and-disable-firewalld-on-centos-7/)

To disable firewalld, run the following command as root:

# systemctl disable firewalld

To stop firewalld, run the following command as root:

# systemctl stop firewalld

And finally, to check the status of firewalld, run the following command as root:

# systemctl status firewalld

Disable SELinux

# vi /etc/selinux/config

Change SELINUX=enforcing to:

SELINUX=disabled

Update the system:

# yum update -y

Shutdown the Instance and create an AMI from it.  This AMI will be used as the reference from the Auto-Scaling Group and can also be shared across regions or even different accounts (depending on the use case).

Once the AMI is created, Terminate the EC2 Instance.

Create s3 Bucket(s)

Create the s3 Bucket (or Buckets depending on the use case) to house the config files that the Auto-Scaling Group EC2 Instances will use to configure the VPN.  We will create the Bucket and then place a policy on it to lock down access.

Click Create Bucket and give it a name.

To protect against accidental deletes from this Bucket, enable Versioning.

If there are concerns about cost over the long term then enable Lifecycle rules (although for the size of files and frequency of change this is probably not required).  Do this by clicking Lifecycle > Add rule.

Further info on s3 Lifecycle rules can be found at http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-with-versioning.html.

Configure an access policy for the Bucket(s).  Below is an example for access from a single account and access from multiple accounts.  Make sure to use the AWS Account number in the appropriate field.

Example for single account access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SingleAccountVPNAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::112233445566:root"
      },
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::dkb-test1-vpn-asg",
        "arn:aws:s3:::dkb-test1-vpn-asg/*"
      ]
    }
  ]
}

Example for multiple account access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MultiAccountVPNAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::112233445566:root",
          "arn:aws:iam::223344556677:root"
      },
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::dkb-test1-vpn-asg",
        "arn:aws:s3:::dkb-test1-vpn-asg/*"
      ]
    }
  ]
}

Click Permissions > Add bucket policy.  

Copy and paste the appropriate policy into the Bucket Policy Editor and click save.

IAM Role and Policies

Create an IAM role so the Instance in the Auto-Scaling Group can perform some actions automatically without having to store AWS keys on the AMI.  The Instance will need to be able to:

• Automatically grab an Elastic IP
• Disable the source/destination check on the Instance (to allow traffic to traverse across it)
• Alter the route table (so Instances inside the VPC know where to route traffic to the other CIDR block).

IAM Policies

In IAM create a new 2 new policies .  The first policy will allow changes to routing tables and also allow access to claim Elastic IP Addresses.

Click IAM > Policies > Create Policies

Click Select on Create Your Own Policy

Create the 2 news policies with the details below.  Take care to change the s3 Bucket references to the one created in the s3 Bucket step.

Policy Name : alternetworking
Description: Allow access to alter routing table and claim EIPs
Policy Document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1447243435000",
      "Effect": "Allow",
      "Action": [
        "ec2:AssociateAddress",
        "ec2:AttachNetworkInterface",
        "ec2:DescribeRouteTables",
        "ec2:ReplaceRoute",
        "ec2:CreateRoute",
        "ec2:ModifyInstanceAttribute"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Policy Name : accesstovpnbucket
Description : Allow read access to s3 Bucket
Policy Document :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
      "s3:GetObject",
      "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::dkb-test1-vpn-asg/*",
        "arn:aws:s3:::dkb-test1-vpn-asg"
      ]
    }
  ]
}

IAM Role

Create an IAM Role and associate the 2 new policies with that role.  Click IAM > Role > Create Role.  Give the role a name.

Select AWS Service Roles > Amazon EC2.

Attach the 2 policies we created in the previous step.

The final role should look something like this.

Elastic IP assignment

For the VPN to run in the VPC you will need a static IP which means creating an Elastic IP.  Click on VPC > Elastic IPs and Allocate New Address.  Make a note of the Allocation ID.

LibreSWAN Config files

LibreSWAN requires 2 config files to attempt to create a connection to a VPN peer.  These files are the tunnel .conf and .secrets files and reside in the /etc/ipsec.d folder.  In this example the tunnel is called AWSVPNTEST1 and next the various config files will be uploaded to the s3 bucket created earlier (this includes a sysctl.conf which is also required for system config changes to enable LibreSwan to run).

Create the .conf file:

• conn : The tunnel name (i.e. AWSVPNTEST1)
• leftid : This is the local externally visible IP (i.e. the Elastic IP Address)
• leftsubnet : Local subnet (i.e. VPC CIDR Block)
• right : Remote Peer IP Address
• rightsubnet : Remote subnet CIDR Block

Upload the various config files used by cloud init to the s3 Bucket.

NOTE: I have experienced issues with uploading from Windows machine may corrupt your config files and cause LibreSWAN to not start but will not throw any error messages.  If you do run into problems then try to edit and upload the config files from a Linux or MAC.

AWSVPNTEST1.conf:

conn AWSVPNTEST1
 type=tunnel
 authby=secret
 left=%defaultroute
 leftid=52.32.85.227
 leftnexthop=%defaultroute
 leftsubnet=172.16.100.0/24
 right=52.43.239.134
 rightsubnet=192.168.0.0/24
 phase2=esp
 phase2alg=aes256-sha2_256;modp1024
 ike=aes256-sha2_256;modp1024
 ikelifetime=3600s
 salifetime=3600s
 pfs=yes
 auto=start
 rekey=yes
 keyingtries=%forever
 dpddelay=10
 dpdtimeout=60
 dpdaction=restart_by_peer

AWSVPNTEST1.secrets:

52.32.85.227 52.43.239.134 : PSK "mysecretpass"

sysctl.conf

# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
 
# Disable Source verificyation and send redirects to avoid Bogus traffic within OpenSWAN
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.eth0.rp_filter=0
net.ipv4.conf.lo.rp_filter=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.lo.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.lo.accept_redirects=0
 
# Enable IP forwarding in order to route traffic through the instances
net.ipv4.ip_forward=1

Security Groups

Create the security groups needed for the EC2 Instances that will be running the VPN.

To allow all IPSEC traffic from the Remote Peer, open the following ports:

CUSTOM UDP RULE : UDP : 500
CUSTOM PROTOCOL : ESP (50) : All

Allow all traffic from VPC.  This is to allow traffic from the VPC to hit the VPN Tunnel.

Allow SSH from your IP for admin access.

Auto-Scaling

LaunchConfiguration

Click EC2 > Launch Configurations > Create launch configuration and select the AMI created from My AMIs:

Choose Instance type (this really depends on how much traffic is expected to traverse the tunnel).  This example uses t2.micro but Production a larger Instance will be required (i.e. an M4.large).  If high traffic is expected then maybe the Network enhanced Instance is more appropriate.  More details here:

• https://aws.amazon.com/ec2/instance-types/
• https://aws.amazon.com/ec2/faqs/#What_networking_capabilities_are_included_in_this_feature

Give the Launch Configuration a name (use a naming convention that allows for version numbers will help to keep track of which one is the latest).

Select your IAM Role from the drop down.

Input the initial EC2 Instance User data. This is a set of instructions that the Instance runs at boot time and in this example runs all the configuration for the tunnel on the fly from the the s3 hosted config files.  More information on AWS User Data.

The bits that need to change for your config will be the hostname, Elastic IP Allocation, the route table id, region, the s3 Buckets and the config file names.

#!/bin/bash -x
sleep 30
hostname DKB-TEST1-VPN
echo 'hostname' >> /etc/hostname
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
my_eni_id=$(aws ec2 describe-network-interfaces --region us-west-2 --filters Name=attachment.instance-id,Values=${instance_id} Name=attachment.device-index,Values=0 --output text | grep NETWORKINTERFACES | cut -f5)
yum install unzip libreswan tcpdump -y
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
export AWS_DEFAULT_REGION=us-west-2
/usr/local/bin/aws ec2 associate-address --instance-id $instance_id --allocation-id eipalloc-c5ba63a2 --region us-west-2
aws ec2 replace-route --route-table-id rtb-f9c82d9e --destination-cidr-block 192.168.0.0/24 --network-interface-id ${my_eni_id} --region us-west-2
/usr/local/bin/aws s3 cp s3://dkb-test1-vpn-asg/AWSVPNTEST1.conf /etc/ipsec.d/
/usr/local/bin/aws s3 cp s3://dkb-test1-vpn-asg/AWSVPNTEST1.secrets /etc/ipsec.d/
/usr/local/bin/aws s3 cp s3://dkb-test1-vpn-asg/sysctl.conf /etc/sysctl.conf
sysctl -p
systemctl enable ipsec
systemctl restart ipsec

Tick Assign a public IP address to every instance (this allows the EC2 Instance to be able to communicate with the outside world prior to grabbing the Elastic IP).

Next on Add Storage tick the Delete on Termination box so when the Instance is shutdown and terminated it doesn’t leave volumes behind on your account (they cost $).

On Configure Security Groups, Select the VPC from the dropdown and assign the Security Groups created earlier.  Click through to review and finish the creation of the Launch Configuration.  Select the correct ssh keys at the final step.

Autoscaling Group

Click EC2 > Auto Scaling Groups > Create Auto Scaling Group > Create and Auto Scaling group from an existing launch configuration > select the new created configuration.

Give the group a name, choose the VPC and select the public subnets available in the VPC (in this example there are 2 public subnets configured in different Availability Zones for High Availability).

In Configure scaling policies just select Keep this group at its initial size as this will only be using ASG for HA (i.e. only require a single Instance running at any one time).

Skip over Add notifications as this will be covered in the next section about monitoring.

In Configure Tags set the Key Name to be something useful so that all Instances started up have to same tag.

Click Review and then Create Auto Scaling Group.  The Instance should now start up.  Check the status from Activity History in the ASG.

Once the Instance is booted it should run all the commands in User Data and attempt to establish the connection with the other side (as long as everything on both sides is configured correctly).

Troubleshooting

There are a number of elements that can go wrong with this type of setup so it is important to verify configuration settings.

Firewall / Security Groups

The first thing that is usually checked is a ping across the tunnel from devices / VMs on each subnet.  Ensure that on the AWS side the EC2 Instance does have a Security Group that allows traffic inbound from the remote subnet.  For example, a very open Security Group may open ALL traffic from 192.168.0.0/24.  The same needs to happen on the remote end.

Obviously, in the real world you should lock down traffic to what is required rather than opening everything.

EC2 System Log

A common issue is with the commands running on boot via User Data.  Verify Instance startup from the System Log by click EC2 > Running Instances.  Right click on the EC2 Instance > Instance Settings > Get System Log:Check to see if any obvious errors occur.  In the example below the EIP allocation failed!  This was probably caused by a typo or maybe someone deleted the EIP…….

Source / Destination Check

If an EC2 Instance needs to route traffic across itself then it needs to have the Source / Destination Check flag disabled.  NOTE: This should be done on the fly at boot by the Instance itself (via the AWS User Data settings).

Click EC2 > Running Instances.  Right click on the EC2 Instance > Networking > Change Source/Dest. Check:

Should look like this:

Route Table check

To verify that the new VPN instance has altered the route table correctly click VPC > Route Tables > click the appropriate route table and click the Routes tab.  The example below shows the Route Table used by the VPC has a route for 192.168.0.0/24 via the EC2 Instance running LibreSwan.  NOTE: This should be done on the fly at boot by the Instance itself (via the AWS User Data settings).

LibreSwan Tunnel Status

To check what the tunnel is doing on the EC2 Instance side ssh in and check the following:

tail /var/log/secure

The main log file by default.

ipsec auto --status

Shows the status of the tunnels configured.  The above example everything is working.

ipsec look

This gives more detail on configured tunnels.

journalctl -xe

This error often indicates a corrupted config file.

Monitoring

More to come on Monitoring the status of your EC2 Instance and tunnel itself ……..