Building a Debian Based VPN Router - Part 3 - VPN Tunnels and Policy Based Routing


Introduction

We have a few requirements for VPN:

  • We want to aggregate VPN tunnels so we can use multiple of them on a per stream basis, mainly we want this because with more streams we can distribute CPU load.
  • Push all DNS traffic over the tunnels
  • Push all traffic from the protected VLAN (192.168.1.0/24) over the tunnels.
  • Any traffic going via the Unprotected VLAN goes directly out over the WAN.
  • I want to use two VPN providers, PIA and AirVPN for different purposes.

What we're basically going to be setting up here for the last 4 items is some policy based routing rules to allow us to distinguish between traffic and then completely change the routing table based on that policy.

Routing Tables

Before we go any further, we're going to set up some additional routing tables for our router.

We're going to be setting up two separate tables for PIA and for AirVPN.

So lets start by making our routing tables

1
2
echo "100 pia" >> /etc/iproute2/rt_tables
echo "200 airvpn" >> /etc/iproute2/rt_tables

Now we have those routing tables lets start by adding some ip rules that specify under what conditions we would want to use these tables.

Firstly lets look at the rules you currently have set up:

1
2
3
4
5
root@router-1:~# ip rule

0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

What this tells us is we have 3 rules set up by default, firstly it will use the local table, then the main table, then the default table.

The format is the following:

1
<priority>:  <predicate> lookup <table>

So the first column is the order that they are evaluated, "from all" effectively means we push all rules through this (so we have no predicate for this table, it is always used) and then we use a specific table.

So what we want to do is add some rules into this, lets start by adding in an ip rule for PIA.

1
2
ip rule add from 192.168.1.0/24 lookup pia priority 50
ip rule add from 192.168.1.0/24 prohibit priority 51

So, what does this actually mean? Lets look at our output again to see what the table looks like now:

1
2
3
4
5
6
7
root@router-1:~# ip rule

0:  from all lookup local
50: from 192.168.1.0/24 lookup pia
51: from 192.168.1.0/24 prohibit
32766:  from all lookup main
32767:  from all lookup default

So we now have two other rules that says if the source is 192.168.1.0/24 then use the protected lookup table. What happens in this case is those rules are evaluated then it will fall through to the next rule so we add an explicit prohibit as priority 51. This means that no additional tables will be used after this evaluation.

So now any requests from 192.168.1.0/24 will use the protected table and nothing else. So it can't use the default gateway of the box, that's perfect, we in no way want our protected traffic from escaping over the WAN!

So what we now need to do is we need to script this up, ip rules are not saved and we need to ensure a way that these are brought back in if we restart the server. So lets add in some additional rules for other traffic into a script:

/opt/synroute/setup-rules.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/bash

# This sets up the IP rules

# Force all traffic from a specific host to AirVPN
ip rule add from 192.168.1.246/32 lookup airvpn priority 30
ip rule add from 192.168.1.246/32 prohibit priority 31

# Force protected traffic to go via pia
ip rule add from 192.168.1.0/24 lookup pia priority 50
ip rule add from 192.168.1.0/24 prohibit priority 51

So this script will now set up some other rules. We have a special case where we want a single host (192.168.1.246/32) to use AirVPN, so we create a new rule that uses the airvpn table. Anything else in that subnet will use the pia table.

We want to execute this when the network comes up so lets create a new service that will execute this script when the network interface comes up:

/etc/systemd/system/iprules.service

1
2
3
4
5
6
[Service]
Type=oneshot
ExecStart=/opt/synroute/setup-rules.sh

[Install]
WantedBy=sys-subsystem-net-devices-enp1s0f1.device

All this basically does is runs the setup-rules script when the enp1s0f1 device comes up. Perfect now we have the rules on bootup, make sure you reload systemd and enable the service!

1
2
systemctl daemon-reload
systemctl enable iprules

Currently our pia and airvpn tables are completely empty, now lets take a look at our default table:

1
2
3
4
5
root@router-1:~# ip route show
default via 1.2.3.4 dev enp1s0f0
default via 1.2.3.4 dev enp1s0f0 proto dhcp src 1.2.3.5 metric 1024
192.168.1.0/24 dev enp1s0f1.100 proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev enp1s0f1.200 proto kernel scope link src 192.168.2.1

So although we don't want to use the default route here, we do still want to be able to route to the other local subnets, so we need some way of populating the airvpn and pia tables with these rules when the interfaces come up.

If we were using our traditional networking scripts here we could use a post-up to handle the adding of these rules. But we're using systemd-networkd, luckily we can just use some oneshot service scripts to do this in the same way we did with our ip rules setup script:

/etc/systemd/system/setup-enp1s0f1.100.service

1
2
3
4
5
6
[Service]
Type=oneshot
ExecStart=/opt/synroute/setup-enp1s0f1.100.sh

[Install]
WantedBy=sys-subsystem-net-devices-enp1s0f1.100.device

/etc/systemd/system/setup-enp1s0f1.200.service

1
2
3
4
5
6
[Service]
Type=oneshot
ExecStart=/opt/synroute/setup-enp1s0f1.200.sh

[Install]
WantedBy=sys-subsystem-net-devices-enp1s0f1.200.device

And as usual lets make sure we enable these services.

1
2
3
systemctl daemon-reload
systemctl enable setup-enp1s0f1.100
systemctl enable setup-enp1s0f1.200

You'll notice that these are calling scripts located in /opt/synroute, so lets go ahead and create those scripts, ensure they're executable!

/opt/synroute/setup-enp1s0f1.100.sh

1
2
3
4
5
6
7
8
#!/bin/bash

# Set up the base routing table for AirVPN and pia
# AirVPN
ip route replace 192.168.1.0/24 dev enp1s0f1.100 scope link table airvpn

# PIA
ip route replace 192.168.1.0/24 dev enp1s0f1.100 scope link table pia

/opt/synroute/setup-enp1s0f1.200.sh

1
2
3
4
5
6
7
8
#!/bin/bash

# Set up the base routing table for AirVPN and pia
# AirVPN
ip route replace 192.168.1.0/24 dev enp1s0f1.200 scope link table airvpn

# PIA
ip route replace 192.168.1.0/24 dev enp1s0f1.200 scope link table pia

So what is this actually doing? When the network link comes up we want to add routing rules into the airvpn and pia tables. So this creates those base routing rules, this mimics what happens by default for the main table.

If you don't want to reboot to try this, run the scripts that you just created and now check your pia and airvpn routing tables.

1
2
3
4
5
6
7
root@router-1:/etc/synroute# ip route show table pia
192.168.1.0/24 dev enp1s0f1.100 scope link
192.168.2.0/24 dev enp1s0f1.200 scope link

root@router-1:/etc/synroute# ip route show table airvpn
192.168.1.0/24 dev enp1s0f1.100 scope link
192.168.2.0/24 dev enp1s0f1.200 scope link

Perfect, we have our default routing tables that will be persisted across reboots. The problem is, we don't have any default gateways, so lets move on to setting up our openvpn clients so we can set up those default gateways!

OpenVPN

Lets first start by installing openvpn, and we want to disable the default service, because I'm going to be adding a custom service script to start the clients.

1
2
apt install openvpn
systemctl disable openvpn

Now lets create some folders to manage our client config structure, and keep those configs away from our server implementation.

1
2
mkdir /etc/openvpn/client
mkdir /etc/openvpn/client/{certs,auth,scripts}

You'll also want to create a custom systemd templated service script for openvpn-client that will ensure you can load the configs from those directories.

/etc/systemd/system/openvpn-client@.service&

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[Unit]
Description=OpenVPN client for %i
Wants=network-online.target
After=network-online.target

[Service]
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn/client --config /etc/openvpn/client/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
ProtectSystem=yes
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_READ_SEARCH CAP_AUDIT_WRITE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This is basically just like the server implementation with a couple of differences, mainly we point --config and --cd to the config directory /etc/openvpn/client instead of /etc/openvpn and we add in a Restart=Always and a RestartSec=10 so that if our service dies we try to start it up again with a 10 second delay.

So now we have that setup lets start by configuring PIA, you'll want to probably grab the base config from the PIA website rather than relying on my config, but you'll want to make a couple of changes to ensure you don't mess up your routing rules. After all, you don't really want OpenVPN messing with your routing table in this case.

So lets create a PIA client connection to London:

/etc/openvpn/client/pia-london.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
client
auth-nocache
route-nopull
dev tun0
proto udp
remote uk-london.privateinternetaccess.com 1198
resolv-retry infinite
nobind
persist-key
persist-tun
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass auth/pia-login.conf
comp-lzo
verb 1
reneg-sec 0
crl-verify certs/pia-crl.rsa.2048.pem
ca certs/pia-ca.rsa.2048.crt
disable-occ
script-security 2

So the main things to take note of here are the following that differ from the PIA default config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# We explicitly set the device to tun0, this will become apparent why
# we do this soon.
dev tun0

# Don't change the routing table, don't pull routes
route-nopull

# Read the authentication credentials from
# /etc/openvpn/client/auth/pia-login.conf
auth-user-pass auth/pia-login.conf

# Allow us to run scripts on up/down, we're going to add this in soon!
script-security 2

# Put the certs in the certs dir instead of being in the same dir as the
# config
crl-verify certs/pia-crl.rsa.2048.pem
ca certs/pia-ca.rsa.2048.crt

Ok so we also need to grab the certs from PIA and pop those in the /etc/openvpn/client/certs dir. We also need to create the auth-user-pass file that will allow us to log in without prompt.

/etc/openvpn/client/auth/pia-login.conf

1
2
username
password

Replace this with your real username and password.

Ok so, now we have this we should be able to start up a tunnel by starting our templated service, if you'll notice it takes the name of the service and then loads the config by the same name:

1
2
systemctl start openvpn-client@pia-london
systemctl enable openvpn-client@pia-london

so at this stage you should see that we have the service running, we also have a /dev/tun0 device. However we're missing one part, currently we have absolutely no default gateway set up for these tunnels. Obviously we didn't want OpenVPN messing with our routing table so we're now going to go ahead and implement these routing table changes ourselves!

Automatic Tunnel Aggregation

So lets take note of what we want, we currently have a single tunnel set up to PIA but we have no default route for our 192.168.1.0/24 subnet. So we want to do this automatically.

We're going to create a new script that modifies the routing rules for us, this is fairly specific to my use case but you'll see the power of this soon.

So lets first of all lets install a utility we need just to make things a little simpler for our script

1
apt install pcregrep

Then lets create our script:

/etc/openvpn/client/scripts/iproute-changed.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/bash

TUN_INTERFACES=`ip link show | pcregrep -o1 "(tun[0-9]):"`
PREFIX="/bin/ip route replace table pia default"
COMMAND="${PREFIX} "

for dev in $TUN_INTERFACES
do
    COMMAND="$COMMAND nexthop dev $dev"
done

eval $COMMAND

AIR_INTERFACES=`ip link show | pcregrep -o1 "(tun[1][0-9]):"`
PREFIX="/bin/ip route replace table airvpn default"
COMMAND="${PREFIX} "

for dev in $AIR_INTERFACES
do
    COMMAND="$COMMAND nexthop dev $dev"
done

eval $COMMAND

Ok so this is actually fairly simple, what we're doing is we're looking for all tunnels that start with tun[0-9], and we're adding those to the pia routing table.

Then we're looking for all tunnel devices that start with tun1[0-9] and we're adding those to the airvpn routing table.

So now it should be pretty apparent why we named the tunnel device manually in the VPN config, by naming the tunnel tun0 we added it to the pia routing table.

If we spin up another openvpn client to airvpn and name the tunnel device /dev/tun10 then we can add this to the airvpn routing table. You can modify this script to do whatever you want to add the tunnel devices to whatever routing table you wish, this is just the simplest option for me.

What this basically means is that we can dynamically spin up openvpn tunnels and they will be automatically aggregated together and streams will be pushed over tunnels. This is effectively Equal Cost Multipath Routing, we have multiple tunnels that all have the same metric.

Hopefully this will become obvious soon as to why this is beneficial.

So now we have that we want to get our vpn client to execute this script when the tunnel comes up or is torn down. Lets edit our pia-london.conf to add this in:

1
2
echo "up /etc/openvpn/client/scripts/iproute-changed.sh" >> /etc/openvpn/client/pia-london.conf
echo "down /etc/openvpn/client/scripts/iproute-changed.sh" >> /etc/openvpn/client/pia-london.conf

Now lets take the tunnel down and start it up again.

1
2
systemctl stop openvpn-client@pia-london
systemctl start openvpn-client@pia-london

Now lets check our routing rules:

1
2
3
4
5
root@router-1:~# ip route show table pia
default
    nexthop dev tun0 weight 1
192.168.1.0/24 dev enp1s0f1.100 scope link
192.168.2.0/24 dev enp1s0f1.200 scope link

If everything has worked correctly we now have tun0 as the default gateway for the protected vlan. However we're still missing something, we can't actually get any traffic out because we don't have NAT set up, we need to change our iptables rules so we can get out over this tunnel.

1
2
3
4
5
iptables -t filter -A FORWARD -i tun+ -o enp1s0f1.100 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t filter -A FORWARD -i tun+ -o enp1s0f1.200 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t filter -A FORWARD -i enp1s0f1.100 -o tun+ -j ACCEPT
iptables -t filter -A FORWARD -i enp1s0f1.200 -o tun+ -j ACCEPT
iptables -t nat -A POSTROUTING -o tun+ -j MASQUERADE

Here we're using a wildcard in iptables so that if the interface matches the name tun* we automatically NAT. As normal remember to iptables-save after you've made these modifications.

Adding More Tunnels

That was quite a bit of work to get that set up, but now you can see how simple it is to add another tunnel to your routing table. Lets simulate setting up a new tunnel to a PIA German node.

First thing we want to do is add another config file:

1
2
3
cd /etc/openvpn/client/
cp pia-london.conf pia-germany.conf
vim pia-germany.conf

Now you'll want to change two things in this config, first lets change the dev to tun1 and then change the remote to germany.privateinternetaccess.com.

1
2
3
4
5
6
7
8
3c3
< dev tun0
---
> dev tun1
5c5
< remote uk-london.privateinternetaccess.com 1198
---
> remote germany.privateinternetaccess.com 1198

Now we have that lets start it up!

1
2
systemctl start openvpn-client@pia-germany
systemctl enable openvpn-client@pia-germany

And lets check our routing table for protected.

1
2
3
4
5
6
root@router-1:~# ip route show table pia
default
    nexthop dev tun0 weight 1
    nexthop dev tun1 weight 1
192.168.1.0/24 dev enp1s0f1.100 scope link
192.168.2.0/24 dev enp1s0f1.200 scope link

We now have another openvpn tunnel set up that was automatically added to the routing table. We have redundancy on it and we can balance streams between them, that's pretty easy!

AirVPN

I'm not going to go into a great amount of detail with this, but effectively for AIRVPN you need to to the same steps as you did with PIA. Make sure you set up the client config in the same way ensuring you set all the client options like route-nopull and set up the script execution. There is a slight difference in the setup in that AirVPN uses certificate based auth rather than user/password, so you won't need to create an auth file like in the PIA example. I would start off by logging into the AirVPN portal and downloading a config which will include your cert.

And importantly you must ensure that you set the tunnel device names to tun10-19, that will ensure that the tunnels are instead added to the airvpn table.

Call your config file something like /etc/openvpn/client/air-uk.conf and start it up with the same templated service file:

1
2
systemctl start openvpn-client@air-uk
systemctl enable openvpn-client@air-uk

And that's it! You should have your airvpn routing table update in entirely the same way.

So the question is why are we doing this? Why do we have PIA and AirVPN?

Well the great thing is that with AirVPN you can forward ports. So say you wanted to host a service via the VPN you could do that, you could forward port 80 in their panel and then set up DNAT rules in iptables that forwarded that port through to one of your servers that perhaps hosts a website.

Lets look at the iptables rules that would be required for us to do this:

1
iptables -t nat -A PREROUTING -i tun10 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.246

Here we're saying that port 80 from tun10 (the AirVPN tunnel) is being forwarded to 192.168.1.246. If you look back up at the top of this post you'll see that we already set up an ip rule that forces all traffic from 192.168.1.246 out over the airvpn gateway. So that's all we need to do!

DNS Requests

So one thing we also want is we want to ensure that all DNS requests go out via the PIA tunnel. We've seen that we can add source rules that say all traffic with this source prefix goes out over the tunnel, but how can we be more specific, so how can we say something like, if the packet destination port is 53 then use this routing table.

Well the answer is we can't do this solely with iproute2. So lets have a look at some of the other options for ip rule that are available to us to identify this traffic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
root@router-1:~# ip rule help
Usage: ip rule { add | del } SELECTOR ACTION
       ip rule { flush | save | restore }
       ip rule [ list [ SELECTOR ]]
SELECTOR := [ not ] [ from PREFIX ] [ to PREFIX ] [ tos TOS ] [ fwmark FWMARK[/MASK] ]
            [ iif STRING ] [ oif STRING ] [ pref NUMBER ] [ l3mdev ]
ACTION := [ table TABLE_ID ]
          [ nat ADDRESS ]
          [ realms [SRCREALM/]DSTREALM ]
          [ goto NUMBER ]
          SUPPRESSOR
SUPPRESSOR := [ suppress_prefixlength NUMBER ]
              [ suppress_ifgroup DEVGROUP ]
TABLE_ID := [ local | main | default | NUMBER ]

You can see in terms of SELECTOR we can use a variety, but the most interesting one for us in this context is fwmark. With fwmark we can use iptables to MARK interesting packets, then apply a different routing table based on this MARK.

That's a really powerful feature for us, we can force certain types of traffic out over the VPN. So lets add a rule into our iptables that will mark packets destined for port 53.

1
2
itpables -t mangle -A OUTPUT -p tcp --dport 53 -j MARK --set-xmark 0x1/0xffffffff
itpables -t mangle -A OUTPUT -p udp --dport 53 -j MARK --set-xmark 0x1/0xffffffff

So what are we doing here? We're identifying traffic that is either tcp or udp on port 53 and we're telling iptables to MARK that packet with a 1.

As usual iptables-save this to your rules file.

So now we have our DNS traffic being marked with a 1, we can use that to create some new ip rules that will force that traffic out over PIA:

1
2
3
4
5
ip rule add fwmark 1 lookup pia priority 40
ip rule add fwmark 1 prohibit priority 41

echo "ip rule add fwmark 1 lookup pia priority 40" >> /opt/synroute/setup-rules.sh
echo "ip rule add fwmark 1 prohibit priority 41" >> /opt/synroute/setup-rules.sh

Now you should find that all of your DNS traffic, regardless of if your client is using your internal DNS or an external DNS server like 8.8.8.8 will all be going over the PIA tunnel.

Booting Up Problems

So we still have one last issue to solve, if you'll notice we're using DNS names for our OpenVPN tunnels. But this is a bit of a race condition:

  1. The box comes up and OpenVPN tries to resolve the DNS name in it's config
  2. The tunnel isn't up so the routing table is blank, it has no way of resolving that DNS name.
  3. The tunnel fails to come up and will never come up.

So how do we solve this, well we can use the same method we used for ad blocking. How about every so often we resolve the DNS names for our tunnels and then we add them to our local unbound server.

/opt/synroute/openvpn/update-dns.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/bin/bash

set -e

# This effectively caches the DNS entries in unbound for AirVPN and PIA
# This means when the box comes up it can still initate the connection.
CONF_FILE="/etc/unbound/unbound.conf.d/vpns.conf"
TMP_FILE="/tmp/vpns.conf"

hosts=("uk.vpn.airdns.org" "uk-london.privateinternetaccess.com" "uk-southampton.privateinternetaccess.com")

echo "" > $TMP_FILE

for hostname in "${hosts[@]}"
do
    ips=$(/usr/bin/dig +short $hostname)

    for ip in $ips
    do
        echo "local-data: \"$hostname. A ${ip}\"" >> $TMP_FILE
    done

done

mv $TMP_FILE $CONF_FILE

/bin/systemctl restart unbound

What this is doing is fairly simple, we list all the hosts we want to cache locally on our unbound server and we dig them and add them to a local zone with the IP address and format them into the unbound config format.

The reason we copy it into a temp file is that if a dig fails for whatever reason we don't want to overwrite what is potentially working in the local unbound server.

Lets add this as a cron to run every 2 hours:

1
0 */2 * * * /opt/synroute/openvpn/update-dns.sh

A fairly hacky solution, if you want to avoid this you could always use the IP addresses in your config files but there's no guarantee your provider will always use this IP.

Summary

So that was a fairly long process but what we have now is a really easy way to add additional tunnel gateways, we have redundancy on our tunnels, and we have policy based routing that allows us to specify which tunnels we wish to use for our traffic.

The final part in this article is around the performance of this setup.

Comments