Monday, 2 January 2012

Multi-WAN + Multi-LAN + No-NAT routing with pfSense 2.0.1

This notes summarise how to run multiple No-NAT LAN and WAN connections using version 2.0.1 of pfSense (an excellent open-source routing/firewalling appliance operating system).   My setup didn't work out of the box initially, so I thought it was worth writing up a summary of the settings that are now working here.

If you are running NAT (boo!), or if you want to do load-balancing rather than policy-based routing, then these notes are probably not for you.    The official pfSense "Multi-WAN 2.0" documentation is at

The beauty of pfSense 2.0.1 for multi-WAN setups is that you can define as many Gateway Groups as you like.  You use these Gateway Groups in your outbound firewall rules to define your outbound routing & failover policies.  In my case, there are several internal LANs, each with its own outbound Gateway Group.  But in a simpler setup, you could have just one internal LAN, but define your outbound firewall rules to use different Gateway Groups for different computers and other devices (by source IP address) and/or different applications (by destination TCP port).

Summary of Connections

My setup uses a Soekris NET5501 low-power computer with 4 Ethernet ports as the combined firewall/router running pfSense.  To conserve Ethernet ports while allowing separation of different internal networks, all the internal networks leave the firewall on a VLAN trunk.  This trunk connects to a Cisco Small Business SG200-26 switch.  Both the Soekris and the Cisco switch are fanless, which makes for total silence and very low power consumption.   The Soekris boots pfSense from a CompactFlash card, so hard disk failures do not occur.

No other routers are used here: my pfSense system controls all the internet uplinks itself, using two ADSL modems plus a 3G dongle.

My WAN links are as follows:

    ▪    1 x ADSL PPPoE via BT 20CN to ISP AAISP ( )
    ▪    1 x ADSL PPPoE via BT 21CN to ISP AAISP
    ▪    1 x 3G/UMTS PPP via 3UK   to ISP AAISP ( )

AAISP customers: there is a Bonding Lines KB article at

My LAN networks are as follows:

    ▪    VOIP subnet - where my Asterisk telephone system lives - with full ingress and egress filtering
    ▪    HOSTING subnet - where my web & mail servers live - with full ingress and egress filtering
    ▪    INTERNAL subnet - where all the household computers and handheld devices live - full ingress filtering
    ▪    PENTEST subnet - for running security audits - with no filtering.

All three WAN links terminate at Andrews & Arnold ( who offer true unfiltered no-NAT connections, with free blocks of public IPV4 and IPV6 addresses.    For now I'm only using IPV4, as the IPV6 support in pfSense 2.0.1 isn't ready for production use.   The sister project Monowall offers full IPV6 support but currently lacks the multi-WAN capabilities of its sibling pfSense.

My 3G/UMTS link is just an old HUAWEI USB 3G modem with an AAISP Data SIM card in it.  This is used as a failover connection, in case both the ADSL lines fail at the same time.   The SIM card only costs £2 per month to rent, but the data costs 2.5p per megabyte, so it's only used during failover.

Each LAN subnet has its own static IP address block, so there is no NAT anywhere in the system.  NAT is evil: it tends to break certain applications, and makes debugging unnecessarily difficult.  You don't need NAT if you use a proper ISP that doesn't charge for IP addresses.   (And no, NAT is not a security feature.  For security, you need a firewall that offers ingress and egress filtering, not simply address rewriting.)

Routing Principles: Multi-WAN, Multi-LAN, No-NAT...

My setup depends on all three Internet connections correctly routing the same static IP address blocks.  AAISP has a control panel that lets you specify which IP blocks should be routed down each of your lines.  You can specify primary, secondary, tertiary (etc) routing to say what you want to happen when any given link or links fail.   This only affects the downstream direction (from AAISP to you).  The upstream direction is of course up to you: that's where pfSense's Multi-WAN policies come into play.  AAISP will accept packets from any of your links for any of your IP addresses.  

You can choose whether to spread all your traffic across all of your lines (depending which boxes you tick).  If you do that, then AAISP will weight the traffic according to the speed of each line.   In my case, I like to reserve one ADSL link for VOIP traffic, and use the other ADSL line for everything else.  That way, I normally get perfect VOIP quality, but if one ADSL line fails, then all applications will share the remaining ADSL line.  The 3G link is only used if both ADSL links fail.

In addition to the static IP blocks, each WAN connection has its own static IP address which is where the PPP session terminates.  You wouldn't normally do much with these individual static IPs, but you might choose to terminate VPN endpoints on them in pfSense.

The ADSL lines connect to Draytek Vigor 120 modems, which convert ADSL frames to PPPoE frames (phone line in, ethernet out).  This reduces the usable MTU to 1492 bytes.  pfSense does TCP MSS clamping by default, so there's no need to adjust MTUs on the computers. 

In theory, using ADSL modems rather than ADSL routers means that the pfSense firewall knows all about the state of each ADSL line (as it's doing all the routing itself).  In practice (at least with pfSense 2.0.1) when an ADSL line loses PPP sync, this doesn't seem to trigger the failover policy rules in pfSense, so some ICMP ping targets are necessary to make the failover policy rules fire.   I'll come onto this shortly.

Settings used in ISP control panel on

NOTE: If you can't see all of the in-line screen shots, you need to make your window bigger (thanks Blogger)...

On each ADSL line (click on the telephone number) :-
    ▪    Tick "Rate: 90%" (reserves space on downlink for short UDP frames i.e. VOIP)
    ▪    Tick "MTU1492" (does MSS clamping to avoid creating pMTUd blackhole on misconfigured websites)
    ▪    Tick "FastTimeout" (speed up failover on loss of sync on one ADSL line)

On each IP address block (click the IP address block e.g. 217.x.x.x/27) :-
    ▪    Tick the lines under "IP Routing", "IP Routing3", and "IP Routing3" to show where to route each IP Block

Settings used in pfSense firewall/router web GUI

NOTE: If you can't see all of the in-line screen shots, you need to make your window bigger (thanks Blogger)...

GUI Section: "Interfaces/Assign"

    ▪    Set up Ethernet ports and VLANs ...

    ▪    Define the ADSL lines (PPPoE) …

    ▪    Define the 3G/UMTS link (PPP) ...

GUI Section: "Interfaces", "Interface Groups"

    ▪    Define one group called INTERNET for all internet-facing links.

    ▪    This group is then used for setting up firewall rules for all incoming traffic

GUI Section: "System/Routing/Gateways"

    ▪    Name each outbound gateway (I just used the underlying Interface name)

    ▪    Edit each Gateway in turn, and set Monitor IPs and advanced threshold parameters (latency, packet loss, etc) for each gateway. 

    ▪    DO NOT select any of the gateways as the Default Gateway (see below).

    ▪    Note that different Monitor IPs are needed for each line.  (You might not need Monitor IPs if each gateway has a different IP at the ISP end, but that's ISP-specific.)

    ▪    The Latency (etc) parameters will be different for 3G dongles, compared with ADSL lines.

GUI Section: "System/Routing/Gateway Groups"

    ▪    Define named gateway groups, for outbound routing purposes.  Each named Group defines an order for trying to get outbound packets to the Internet...

    ▪    Failover only works if the right parameters are set, for example:-

GUI Section: "Firewall rules" (incl 'internet' gateway set)

    ▪    Set outbound egress firewall rules using named gateway groups in the Gateway column (see below).  This ensures that outbound traffic takes correct route(s) so that different kinds of traffic go out through the interfaces you require.  This also makes failover work!

    ▪    Remember to set the inbound firewall rules under the INTERNET interface group (we created that earlier).  This means you don't need to bother setting inbound firewall rules for each internet-facing interface.

    ⁃    for example, my main LAN uses "Surfing_GW" (ADSL line 4, failing over to ADSL line 1, failing over to the 3G dongle) except for certain hosts such as digital TV boxes :-

    ▪    To give another example, here are the egress ACLs (outbound firewall rules) for my VOICE network.  Notice how the rules can use alias names for particular internal and external hosts.  These aliases map to fixed IP addresses in the pfSense GUI.

Overall, it all seems to work quite well.  Failover isn't totally transparent though.  There are a few seconds of disruption during failover.  I'm not sure whether the stateful firewall rules allow replies to continue to arrive following a failover event, so things like audio streams may need to be restarted.

Failover seems to work fairly well but there are still some occasional problems with Gateway Group priorities not being restored after all the gateways come back up.   So for example, if my voice traffic fails over to ADSL_L4, it won't necessarily revert to ADSL_L1 after L1 comes back up.

UPDATE: It seems that in pfSense 2.0.1's Multi-WAN system, the Policy Routing framework can get confused in setups where both WAN links have the same next-hop IP address.   Such setups will pass traffic but they're not fully supported.  Hopefully a later version will fix this.  2.1 is not far away, but apparently this is mainly for IPv6 support.

There is a tick box under "System / Advanced : Miscellaneous", called "Gateway Monitoring States":
"By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections."
It may be best NOT to tick this box, otherwise the firewall rules may still try to force traffic down gateways that are down.

Hopefully there may be some subtle Multi-WAN improvements to come in pfSense 2.1 and later versions.


  1. Great tutorial on multi WAN pfSense. Thank you for this!
    I am working on a project where I want to have 1Voice and 1Data subnet with a fat Fibre connection. Could you please detail the way you setup the voice and lan as Vlans on your example? Why is DMZ chosen for the voice?
    Kind regards

  2. Hi, not entirely sure I understand what you wish to know, but I'll explain a bit more about the implementation of my network.

    The Voice server and the Hosting web/mail server are set up on different subnets, so that I could add VOIP handsets and more servers (and maybe HTTPS instances) at a later date, while keeping Voice and Hosting apart for security reasons - so that if someone ever compromises one machine, that doesn't give them a platform from which to attack the others - in other words, defence in depth. You could do this more simply if you didn't have plenty of IP address blocks to play with, or if you weren't as paranoid as me.

    You could implement this using normal LANs if your firewall had enough Ethernet ports. But I thought it would be neat to create a VLAN trunk from pfSense, containing all the subnets I needed. So the two ADSL PPP modems use normal Ethernet ports on the pfSense firewall, but all the other subnets (LAN, Voice, Hosting, Pentest) exist as VLANS via the Trunk port. I have a Cisco Small Business SG200-26 managed switch in the rack. This is a small, cheap, fanless alternative to the classic Cisco Catalyst switch that you see in datacentres. It seems entirely adequate for small-scale use.

    So, the first Ethernet port on the pfSense firewall is a VLAN trunk to the Cisco switch. Frames to and from the various subnets are tagged with the relevant VLAN number and sent down the trunk. The switch then uses the VLAN tag number to forward the packet to the correct switch port(s) for the VLAN in question. The VLAN tag is stripped off before the packet is retransmitted onto the relevant switch ports. You just use the switch management GUI to define which switch ports you want to be in which VLAN. For debugging or intrusion detection, there's also a SPAN (Mirror) port: this is a read-only port that sends out a copy of all packets for audit purposes. Annoyingly the packets that are output seem to be lacking the VLAN tags though, and I'm not certain that I have the option of enabling tags on that port.

    All nice and simple, but of course it does create a SPOF (Single Point Of Failure), i.e. I'd be a bit screwed if the switch died. There are ways to create a redundant switching infrastructure of course, but you have to stop somewhere for domestic installations!

    BTW there appears to be an issue with the 3G backup link dropping its PPP session and then not coming back up unless you log into pfSense and press 'Connect' (under Status / Interfaces). I'll report that on the pfSense forum.

  3. Thanks Martin,

    Your setup is very advanced... :), I was looking for something easy to implement and maintain and I wasn't sure how to add a second subnet to my network...

    I've decided to have 2 separate networks that join in pfSense.
    What I did was to add a total of 3 Nics to my server, 1 for the WAN and 2 for LANs. One of the LAN was renamed to VOICE. The LAN was set as and the VOICE as The Trixbox is Now I have separate traffic on the LANs sharing a single access point feeding on Fibre.


  4. I'm trying to remotely setup an Orange 3G+ usb modem and am wondering if we can directly plug it into a pfSense's USB, supermicro atom, port and start configuring it? Good article and thanks.

  5. Hi, there's a good chance your Orange card will just work, especially if it's something common like a Huwaei device. Definitely worth a go!

  6. Great tutorial on multi WAN pfSense. Thank you for your great input!

  7. Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.

  8. Hi Martin,

    Just though to take your valuable advice as your blog is somewhere near to what i am looking. Please have patience as i am trying my best to explain

    whole situation.

    First my present condition :-
    ISP(Internet Service Provider):-
    I have two ISP which are totally different from each other.
    a) Tikona Digital network:- giving me
    2Mbps Dedicate 1:1 connectivity with 115.193.XXX.193 - where this is WAN IP
    115.193.XXX.194 - Used in pfsense gateway using this i am able to browse my pfsense on internet.
    115.193.XXX.195 - usable IP
    115.193.XXX.196 - Usable IP
    115.193.XXX.197 - Usable IP
    115.193.XXX.198 - Usable IP

    b) Tata Communication Ltd.:- This i have recently purchased as I need more bandwidth and alternative if one ISP goes down due to local disturbance

    such as wire cut etc.,
    2Mbps Dedicate 1:1 connectivity with A - where this is WAN IP
    Another difference is they have different gateway for usable IP's & off-course the series of IP is also different. That i didn't understand.
    B - Is this also going to point same Pfsense machine or i can use this in some server ?
    C - usable IP
    D - Usable IP
    E - Usable IP
    F - Usable IP

    Purpose of having all this:-
    I want to create & run a small data center where i will provide servers, cloud hosting, website hosting etc., using physical / virtual boxes. Or i say more in professional manner i want to become IaaS provider.

    To start with, i host my own company website & its needed infrastructure within my premise as most essentially 2 name-servers(2 Windows DNS 2008 R2 in

    virtual environment).
    a) For this i have my own domain(.com) registered with go daddy, where i registered my two nameserver hosts as {115.193.XXX.195 - (} and

    {115.193.XXX.196 -(}. In local network these server has IP as following &
    b) I have also one web server (1 windows web edition 2008 R2 in virtual environment) binded to IP as {115.193.XXX.197 - Usable IP- (Web1)} which

    locally have IP as
    c) Still i left with one more IP where i put my another Linux Vm for trials to my customer. runs on 115.193.XXX.198 & locally

    In PFsense i created virtual IP so that it start acknowledging the IP provided by the first ISP. So, by looking above setup you can guess that i use

    NAT based routing to allow traffic coming from outside world to my servers.

    I started with first ISP 4-5 months back & using various Google links etc., some how i setup my Nat based routing. Nat based routing was not my

    obvious choice as that point of time i didn't know where to start or even i don't know is there any other way exist.

    BY above setup you can also guess that i have three interface configured in PFsense:-
    a) Wan-1 Interface (Onboard Ethernet Adapter) - Incoming <-- First ISP connection
    b) Wan-2 Interface (PCI Card based Dlink Ethernet adapter) <-- Second ISP connection
    c) Lan-0 Interface (PCI Card based Realtek Ethernet adapter) --> Connected to local Dlink Switch

    From switch i have connected my all machines example discussed above servers.

    Last thing is taking idea from few Google links, i implemented my Multi WAN setup and result is if i loss my first connection then traffic from my internal network to outside world switch over other ISP automatically or Vice Versa also.

    Now, but what i actually want is if my one ISP goes down let suppose first ISP, then my most important server like 2 name-servers & one web server

    should remain accessible from outside world. Or more precisely IP given by first ISP should remain accessible even if its own WAN goes down.

    My pin point Questions are :-
    a) What i can correct in order to first rescue from NAT based routing to public IPs.
    b) How i can achieve always on access to my servers even when one of my ISP goes down.
    c) Any other suggestion which i should take care as i am going to grow further as IaaS provider in terms of PfSense.

    1. Taking the easy one first...

      a) How to turn off NAT...

      From the pfSense docs.. To completely disable NAT to have a routing-only firewall, do the following.

      * Go to the Firewall -> NAT page, and click the Outbound tab.
      * Select the option "Manual Outbound NAT rule generation (Advanced Outbound NAT (AON))" and click Save.
      * Remove all automatically generated NAT rules at the bottom of the screen.
      * Apply changes

      It could make the routing easier to configure in pfSense if you ask the ISP to give you a subnet of public IPs that are separate from the WAN IP of the router itself. That way, your pfSense has a single IP on the WAN interface, and a block of public IPs on the LAN interface.

      But, turning off NAT is the easy thing!

      b) How to keep your servers accessible from the web, even when one ISP link goes down.

      pfSense Multi-WAN is not enough in your situation. By default, the public IP block for your web servers only has one upstream route for incoming connections. pfSense Multi-WAN can give you resilient *outbound* internet access, but not resilient *inbound* internet access (unless both uplinks use the same ISP, and the ISP does failover routing).

      For true resilient routing (inbound and outbound) with two ISPs, you need to use BGP. I've never done it myself, but you need to get your router (Cisco or pfSense or whatever) to send BGP route announcements down both your links. Then ISP routers all over the internet learn the best upstream router IPs to use to reach your public IP block at any time. If one leg fails, the BGP announcements stop going down that leg, and the ISP routers stop using the dead leg.

      You would need a block of 256 IPs (i.e. a /24 block) as that's the smallest block that will get BGP announced round the Internet. I think you also need to get an AS (autonomous system) number. And you must make sure that both ISPs allow you to send BGP announcements (as some ISPs filter them for security).

      I think there is a way to publish a list of valid upstream AS numbers (that is, to say what your normal upstream ISPs are), to prevent a third party attacker from sending fake BGP announcements and thereby stealing your traffic and your /24 IP block!

      c) Any other suggestions...

      If you want to provide real resilient commercial hosting, you need to buy a good book about routing protocols.

      You might be better off using two routers for your BGP (one router for each ISP). Those routers could be Ciscos, or for development & testing they could be old PC's running the free routing package Quagga.

      You could host Quagga on pfSense, but to begin with it is probably a lot easier to do a minimal install of FreeBSD or Linux, and use that. So, for example you could have two PCs doing the BGP routing in Quagga on FreeBSD. Then you would have an Ethernet switch. Then your pfSense firewall, running some kind of routing protocol (OSPF perhaps?) to control which of your Quagga routers it should use for outbound connections. Or you could use three Quagga routers and just forget about pfSense (or just use it in Layer 2 mode as a transparent packet filter in front of your actual Web servers).

      This is all quite complicated, and certainly beyond my practical experience. It would be great fun to try though, if you can get a /24 IP block allocated to you! You just need to learn about routing protocols first. You can do that in a cheap test lab (just some old PCs, ethernet switches, etc) before you commit yourself to spending the money to do it for real.

      For starting out on a small scale, I would suggest testing with Quagga (open-source) routers for all the routing protocols. Unless you have lots of money, Cisco routers often work out too expensive, because Cisco charges lots of money for firmware upgrades and access to the more useful parts of the Cisco website. Old PCs are cheap - you just need reliable kit, and good network cards like Intel.

      Hope this helps - and good luck!

      - Martin

    2. Dear Martin,

      Thanks for your reply, This paves way to my future research.
      I will try to put off the NAT, but does this affect my accessibility of my servers ? As right now for particular ports i have defined NAT for example 115.193.XXX.197 : 80 >> :80 (Web Server). So should i assume the will continue to work properly ?

      Another query is, i just quickly dome some Google search could i use PFSense as BGP( and is it necessary to have three different machines running to get BGP done for Two ISP.

      I Don't want to use Cisco as right now i am start-up and can't afford such costly affairs.

    3. You need a bigger IP block allocation, in order to lose the NAT.
      115.193.XXX.193 - router WAN IP ?
      115.193.XXX.194 - pfsense gateway WAN IP ? (You should NOT be able to browse the pfSense GUI from the Internet!)
      115.193.XXX.195 - usable IP
      115.193.XXX.196 - Usable IP
      115.193.XXX.197 - Usable IP
      115.193.XXX.198 - Usable IP

      In No-NAT mode, your pfSense would have a WAN subnet (outside) and a LAN subnet (inside), and you would configure a static route on the router so that it knew the public LAN IP range was reached through the pfSense WAN IP.

      But your existing IP block is too small to split into LAN and WAN subnets.

      With only your existing IP block, you could try using pfSense as a Layer 2 Transparent Bridge:

      But if you want BGP (for full resilience), then you need an extra /24 IP block anyway (in addition to your router and firewall WAN IPs).

      BGP is difficult to secure (attackers can send fake BGP adverts claiming your IP block), so if it was me, I'd be tempted to have two lines to the same ISP (and just accept that ISP as a single point of failure). Perhaps the ISP can offer private BGP or some other private routing protocol to manage the dual feed. Then you could use a smaller IP block, perhaps a /27 or a /28 to begin with.

      You could do all your BGP on one box. But in your place, I would use several cheap old boxes (ideally booting from Compact Flash cards) to give a redundant failover setup, where the network continues working even if some boxes break down. You should read up about CARP and VRRP. I believe CARP is well supported on pfSense these days, but I haven't used it myself.

      You need to do some serious reading to grow your technical knowledge. Maybe set up a test network with some cheap old boxes from EBay. pfSense and Quagga will run on really old hardware, if you are just testing.

      Sorry, I don't have time to help you any further.

  9. Great project you had

  10. Awesome dude... Martin you just ROCK man......!

    Great Stuff,


    1. For financial reasons, I have terminated the second ADSL line now :-(

      Looking back at the experience, the limitations were...

      1. Two uplinks to same ISP. Buggy (not fully supported) in pfSense 2.0.1 if both uplinks have same next-hop IP.

      2. Two uplinks to different ISP's. Supported in pfSense. However, for a hosting setup this is useless unless you have a public /24 IP block. Because you need to run BGP if you want to make your hosting IP block accessible via two different ISPs.

      But, if you simply want to provide a resilient form of Internet access, then pfSense multi-WAN is just great - and using two different ISPs is the way to go.


      - Martin


  11. Hey Im using Two WAN connections..My Question is I wanna use these two WAN connection separately...One is for Emails and Other connection for surf Internet..When the One connection get down automatically one should share for both services...Is pf-sense help for this network??

  12. Hi, yes that's easy. Define your two WAN gateways in pfSense, then define two Gateway Groups giving the gateways the right way round for each traffic group.

    Then in your LAN egress firewall rules, above the default allow rule, stick in some TCP-port-number-specific egress rules that specify the correct Gateway Group for that class of traffic.

    I may have got the terminology a bit wrong but that's the gist of it. Pretty easy really.

  13. Hi Martin,

    Firstly thanks for the great tutorial.

    I am trying to setup a kind of hosting environment. What I want to do is I have several class C ip addresses each with its own gateway and I have several virtual server. I have connection to provider with a Ethernet connection.

    I want to use static nat for each virtual server and I want to use VLANs on both WAN and LAN port on pfsense.With the policy based routing each virtual server will use their seperate gateways.

    I am not even sure if pfsense support this kinda setup but as far as I know pfsense is only option for advanced stuff.

  14. Hi,

    VLANs are no problem. I seem to recall once running pfSense with just one physical NIC by using VLANs.

    There's a good chance it will do what you want. I know you can set the NAT mappings up manually (though I don't use NAT myself).

    The software is free so just give it a go. Almost any old PC is good enough for testing.

    The pfSense message boards are good for support if you get stuck.

    - Martin

  15. hi guys is there a limit for the maximum number of NIC's or (ISP's) supported to set up with multi wan?

    1. I don't know for sure, but I would think the practical limit would be the number of physical NICs (or VLANs) in your system. If in doubt, ask on the pfSense forums.


Spammers: please stop wasting my time. All comments are moderated before publication.