Multi-WAN + Multi-LAN + No-NAT routing with pfSense 2.0.1

This notes summarise how to run multiple No-NAT LAN and WAN connections using version 2.0.1 of pfSense (an excellent open-source routing/firewalling appliance operating system).   My setup didn't work out of the box initially, so I thought it was worth writing up a summary of the settings that are now working here.

If you are running NAT (boo!), or if you want to do load-balancing rather than policy-based routing, then these notes are probably not for you.    The official pfSense "Multi-WAN 2.0" documentation is at http://doc.pfsense.org/index.php/Multi-WAN_2.0

The beauty of pfSense 2.0.1 for multi-WAN setups is that you can define as many Gateway Groups as you like.  You use these Gateway Groups in your outbound firewall rules to define your outbound routing & failover policies.  In my case, there are several internal LANs, each with its own outbound Gateway Group.  But in a simpler setup, you could have just one internal LAN, but define your outbound firewall rules to use different Gateway Groups for different computers and other devices (by source IP address) and/or different applications (by destination TCP port).


Summary of Connections

My setup uses a Soekris NET5501 low-power computer with 4 Ethernet ports as the combined firewall/router running pfSense.  To conserve Ethernet ports while allowing separation of different internal networks, all the internal networks leave the firewall on a VLAN trunk.  This trunk connects to a Cisco Small Business SG200-26 switch.  Both the Soekris and the Cisco switch are fanless, which makes for total silence and very low power consumption.   The Soekris boots pfSense from a CompactFlash card, so hard disk failures do not occur.

No other routers are used here: my pfSense system controls all the internet uplinks itself, using two ADSL modems plus a 3G dongle.

My WAN links are as follows:

    ▪    1 x ADSL PPPoE via BT 20CN to ISP AAISP ( http://aaisp.net.uk/broadband.html )
    ▪    1 x ADSL PPPoE via BT 21CN to ISP AAISP
    ▪    1 x 3G/UMTS PPP via 3UK   to ISP AAISP ( http://aaisp.net.uk/telecoms-mobile-data.html )

AAISP customers: there is a Bonding Lines KB article at http://aaisp.net.uk/kb-broadband-bonding.html

My LAN networks are as follows:

    ▪    VOIP subnet - where my Asterisk telephone system lives - with full ingress and egress filtering
    ▪    HOSTING subnet - where my web & mail servers live - with full ingress and egress filtering
    ▪    INTERNAL subnet - where all the household computers and handheld devices live - full ingress filtering
    ▪    PENTEST subnet - for running security audits - with no filtering.

All three WAN links terminate at Andrews & Arnold (www.aaisp.net.uk) who offer true unfiltered no-NAT connections, with free blocks of public IPV4 and IPV6 addresses.    For now I'm only using IPV4, as the IPV6 support in pfSense 2.0.1 isn't ready for production use.   The sister project Monowall offers full IPV6 support but currently lacks the multi-WAN capabilities of its sibling pfSense.

My 3G/UMTS link is just an old HUAWEI USB 3G modem with an AAISP Data SIM card in it.  This is used as a failover connection, in case both the ADSL lines fail at the same time.   The SIM card only costs £2 per month to rent, but the data costs 2.5p per megabyte, so it's only used during failover.

Each LAN subnet has its own static IP address block, so there is no NAT anywhere in the system.  NAT is evil: it tends to break certain applications, and makes debugging unnecessarily difficult.  You don't need NAT if you use a proper ISP that doesn't charge for IP addresses.   (And no, NAT is not a security feature.  For security, you need a firewall that offers ingress and egress filtering, not simply address rewriting.)


Routing Principles: Multi-WAN, Multi-LAN, No-NAT...

My setup depends on all three Internet connections correctly routing the same static IP address blocks.  AAISP has a control panel that lets you specify which IP blocks should be routed down each of your lines.  You can specify primary, secondary, tertiary (etc) routing to say what you want to happen when any given link or links fail.   This only affects the downstream direction (from AAISP to you).  The upstream direction is of course up to you: that's where pfSense's Multi-WAN policies come into play.  AAISP will accept packets from any of your links for any of your IP addresses.  

You can choose whether to spread all your traffic across all of your lines (depending which boxes you tick).  If you do that, then AAISP will weight the traffic according to the speed of each line.   In my case, I like to reserve one ADSL link for VOIP traffic, and use the other ADSL line for everything else.  That way, I normally get perfect VOIP quality, but if one ADSL line fails, then all applications will share the remaining ADSL line.  The 3G link is only used if both ADSL links fail.

In addition to the static IP blocks, each WAN connection has its own static IP address which is where the PPP session terminates.  You wouldn't normally do much with these individual static IPs, but you might choose to terminate VPN endpoints on them in pfSense.

The ADSL lines connect to Draytek Vigor 120 modems, which convert ADSL frames to PPPoE frames (phone line in, ethernet out).  This reduces the usable MTU to 1492 bytes.  pfSense does TCP MSS clamping by default, so there's no need to adjust MTUs on the computers. 

In theory, using ADSL modems rather than ADSL routers means that the pfSense firewall knows all about the state of each ADSL line (as it's doing all the routing itself).  In practice (at least with pfSense 2.0.1) when an ADSL line loses PPP sync, this doesn't seem to trigger the failover policy rules in pfSense, so some ICMP ping targets are necessary to make the failover policy rules fire.   I'll come onto this shortly.



Settings used in ISP control panel on www.aaisp.net.uk

NOTE: If you can't see all of the in-line screen shots, you need to make your window bigger (thanks Blogger)...

On each ADSL line (click on the telephone number) :-
    ▪    Tick "Rate: 90%" (reserves space on downlink for short UDP frames i.e. VOIP)
    ▪    Tick "MTU1492" (does MSS clamping to avoid creating pMTUd blackhole on misconfigured websites)
    ▪    Tick "FastTimeout" (speed up failover on loss of sync on one ADSL line)

On each IP address block (click the IP address block e.g. 217.x.x.x/27) :-
    ▪    Tick the lines under "IP Routing", "IP Routing3", and "IP Routing3" to show where to route each IP Block

Settings used in pfSense firewall/router web GUI

NOTE: If you can't see all of the in-line screen shots, you need to make your window bigger (thanks Blogger)...

GUI Section: "Interfaces/Assign"

    ▪    Set up Ethernet ports and VLANs ...

    ▪    Define the ADSL lines (PPPoE) …

    ▪    Define the 3G/UMTS link (PPP) ...

GUI Section: "Interfaces", "Interface Groups"

    ▪    Define one group called INTERNET for all internet-facing links.

    ▪    This group is then used for setting up firewall rules for all incoming traffic

GUI Section: "System/Routing/Gateways"

    ▪    Name each outbound gateway (I just used the underlying Interface name)

    ▪    Edit each Gateway in turn, and set Monitor IPs and advanced threshold parameters (latency, packet loss, etc) for each gateway. 

    ▪    DO NOT select any of the gateways as the Default Gateway (see below).

    ▪    Note that different Monitor IPs are needed for each line.  (You might not need Monitor IPs if each gateway has a different IP at the ISP end, but that's ISP-specific.)

    ▪    The Latency (etc) parameters will be different for 3G dongles, compared with ADSL lines.

GUI Section: "System/Routing/Gateway Groups"

    ▪    Define named gateway groups, for outbound routing purposes.  Each named Group defines an order for trying to get outbound packets to the Internet...

    ▪    Failover only works if the right parameters are set, for example:-



GUI Section: "Firewall rules" (incl 'internet' gateway set)

    ▪    Set outbound egress firewall rules using named gateway groups in the Gateway column (see below).  This ensures that outbound traffic takes correct route(s) so that different kinds of traffic go out through the interfaces you require.  This also makes failover work!

    ▪    Remember to set the inbound firewall rules under the INTERNET interface group (we created that earlier).  This means you don't need to bother setting inbound firewall rules for each internet-facing interface.

    ⁃    for example, my main LAN uses "Surfing_GW" (ADSL line 4, failing over to ADSL line 1, failing over to the 3G dongle) except for certain hosts such as digital TV boxes :-

    ▪    To give another example, here are the egress ACLs (outbound firewall rules) for my VOICE network.  Notice how the rules can use alias names for particular internal and external hosts.  These aliases map to fixed IP addresses in the pfSense GUI.
   

Overall, it all seems to work quite well.  Failover isn't totally transparent though.  There are a few seconds of disruption during failover.  I'm not sure whether the stateful firewall rules allow replies to continue to arrive following a failover event, so things like audio streams may need to be restarted.

Failover seems to work fairly well but there are still some occasional problems with Gateway Group priorities not being restored after all the gateways come back up.   So for example, if my voice traffic fails over to ADSL_L4, it won't necessarily revert to ADSL_L1 after L1 comes back up.

UPDATE: It seems that in pfSense 2.0.1's Multi-WAN system, the Policy Routing framework can get confused in setups where both WAN links have the same next-hop IP address.   Such setups will pass traffic but they're not fully supported.  Hopefully a later version will fix this.  2.1 is not far away, but apparently this is mainly for IPv6 support.

There is a tick box under "System / Advanced : Miscellaneous", called "Gateway Monitoring States":

"By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections."

It may be best NOT to tick this box, otherwise the firewall rules may still try to force traffic down gateways that are down.

Hopefully there may be some subtle Multi-WAN improvements to come in pfSense 2.1 and later versions.