Sunday, 27 September 2009

Using PPP over UK ADSL using pfSense

Most home broadband connections use an ADSL router with a built-in NAT firewall. For more complex networks, the next step is to place a dedicated firewall behind your ADSL router. But there's a third way: put your ADSL device into Bridge mode. Then it's just a dumb modem, so your firewall can run the PPP session to your ISP....

First some background: just why would you want to ditch your ADSL router and run PPP straight from the firewall?

Advantages:
  • There's one less thing to go wrong, because there's one less router between you and your ISP.
  • You don't waste a static IP address, because your first Internet-visible IP terminates on the firewall, not the router. 
  • If you want to adjust the services you publish to the Internet, you only have one device to reconfigure.
  • You can use a cheaper ADSL modem instead of a pricey ADSL router.  This can be quite a saving if you need something more sophisticated than basic NAT.
  • For multi-WAN configurations, if the firewall runs both ADSL PPP sessions, then it may be easier to arrange automatic fail-over when one ADSL line fails, or goes down for two minutes to re-sync.
  • If you want to support IPV4 + IPV6, there's probably more chance of getting everything working properly if there's just one key device to configure, not two.
Disadvantages:
  • In the UK, most ADSL providers offer PPPoA (point-to-point protocol over ATM) which, through an ADSL router, delivers a full-size MTU of 1500 bytes.  But if you switch to an ADSL modem/bridge delivering your traffic via Ethernet, then 8 bytes of each frame is lost due to PPP headers, so you end up with an MTU of 1492 bytes.  This should not be a problem, except for some broken websites which suffer from the "pMTUd blackhole" problem.  These misconfigured websites set the Don't Fragment bit on their outgoing IP frames, whilst simultaneously discarding any ICMP "Frag Needed but Don't Frag was Set" responses from routers which are unable to deliver large unfragmented frames across low-MTU links. As a workaround, pfSense automatically implements MSS Clamping on PPPoE interfaces, meaning that it fiddles with the TCP options field in SYN / SYN-ACK frames to ensure that the MSS isn't set higher than PPPoE can cope with.  This is slightly dirty (the firewall is changing the packets passing through it) but it seems to work well enough.
  • Some security people take the view that an Internet-facing security perimeter should have several devices, placed in series, each with an ACL (access control list) set up to filter incoming traffic.  The converse view is that firewalling should be done at the firewall, and routers should be left alone to get on with routing.
At home, for some years I've been running a Cisco 877W ADSL router on the outside, with a dedicated pfSense firewall on the inside.  pfSense is a free and easy-to-use open-source firewall software package, based on the rock-solid FreeBSD Unix operating system but with a user-friendly web-based management screen.  pfSense will run on just about any Intel-based hardware, but I've chosen a diskless Soekris NET5501 low-energy rackmount server, purchased from the excellent KD85.com.  Whilst Soekris computers are quite expensive, they use almost no electricity, so they probably pay for themselves if you run them 24*7 for a few years.

The Cisco router isn't really designed as a consumer-friendly device.  Even for a network geek, it can be a bit of a pain to configure, and to maintain with security updates, even with a Cisco SmartNet contract. By contrast, the pfSense firewall is much simpler to update.  You just save the configuration using your web browser, load the new firmware onto a fresh Compact Flash card, boot it up and restore the saved config from your web browser.  In the event of problems, you just pop the old Compact Flash card back in.

So for now I've unplugged the Cisco router, and connected a cheap ADSL modem in Bridge mode.  There are at least half a dozen UK ADSL modems that offer Bridge Mode, whereby your ADSL link still runs PPPoA, but this is presented to you as PPPoE.  I've tested three of them.  They are all nice and cheap compared to some of the fancier ADSL routers out there....
  • D-Link DSL-320BSummary: Best avoided unless the firmware improves. This is a small device offering one Ethernet port and one ADSL port.  It runs from 9 volts AC. Annoyingly, even after flashing it to the latest UK firmware, I never did get it working in standard PPPoA mode (as an ADSL NAT router) though it did work with an Apple Mac in Bridged mode (as an ADSL modem). For bridged mode on UK BT 20CN ADSL2 lines, you must select "1483 Bridged IP LLC" mode, as shown in the picture below (click to enlarge). The whole experience didn't really inspire confidence, so I didn't bother testing it with pfSense. Rather disappointing, as I had expected better from D-Link.

    • Draytek Vigor 120.  Summary: works like a charm.  This gadget is sold as a PPPoE ADSL modem, not as an router than can also be used as a modem.  The vendor claims you can just plug it into your router/firewall and use it without any configuration at all.  I took the time to assign a static IP address for management purposes: this isn't strictly necessary, but does facilitate statistics display:-

      The management IP address does not need to be related in any way to the IP subnets used by the firewall.  It's only there so that you can plug in a laptop and see what's going on with the ADSL link itself.  The device is still acting only as a modem, not a router.
        The only minor gripe I found with the Draytek is that the web management interface was needlessly confusing. The default status page shows "DISCONNECTED", which probably relates to routing being switched off.  The status page marked "Online Status" (shown above) is what you want, but you can't always access that page because the web menu screen is buggy (and has a clashing colours making it hard to read). But refreshing the page usually makes the menu render properly, or you can just open the status page directly at /doc/online.sht.
          In case it helps anyone... My system is in the UK, on a standard but rather faint BT 20CN ADSL line.  My Draytek Vigor 120 arrived with the current v3.2.4.1 firmware but the modem code seemed to be out-of-date, so I updated it from the maker's website using the file v120_a4_v324 (modem code 332201) as this seemed to be the recommended modem code for the UK.  It's been running flawlessly for 10 days now without re-syncing once, despite the high attenuation on my line, maintaining a good stable 11 dB SNR (with the BT end set for extra stable mode).
            So far then, the Vigor 120 seems perfectly compatible with pfSense 1.2.3, and highly stable even on marginal BT ADSL limes.  It also claims compatibility with all the new ADSL standards, so with a bit of luck it should still work when BT upgrade my local exchange to 21CN ADSL 2+, making it good value for the price.
              • Billion BiPAC 5200Summary: works pretty well, but tends to re-sync every few days on my noisy ADSL line.  This again is a very small device.  It offers one ADSL port and four Ethernet ports, and runs from 12 volts DC (which is handy because that's what all my Soekris servers and Netgear switches use - so in theory I could use a true battery backup circuit instead of a wasteful mains UPS).     
                The Billion device worked straight away as a plain old ADSL NAT router, but upon switching to Bridge mode, it just didn't seem to want to work with the PPP client in pfSense until I tried switching it to an unlikely sounding connection mode: "1483 Bridge IP LLC".  This sounded wrong, because the normal connection mode in the UK is VC rather than LLC, but it works.  The correct setup is as shown above (click picture to enlarge).  Note that VPI/VCI needs to be set to 0/38 for normal UK ADSL lines (BT 20CN ADSL2), whilst the other settings can be left at defaults...
                    The Billion device comes in two models: the 5200 and the 5200S.   I went for the 5200, which has 4 Ethernet ports, whilst the slightly cheaper 5200S only has a single Ethernet port.  Used in Bridged Mode, only one Ethernet port can carry Internet traffic (because everything's wrapped up inside PPP frames).  However, the web management service remains active when Bridge Mode is used, so it may be useful to have more than one Ethernet port in case you want to monitor things like ADSL line conditions without disconnecting the PPP link.
                      So much for the different ADSL modems.  What about configuring pfSense?  For test purposes, I downloaded the CDROM installer ISO image for pfSense 1.2.3, burnt it to CD, booted it up and installed it on a 10 year old Dell desktop PC fitted with two network cards.  Predictably, pfSense found and auto-configured all the hardware: it even recognised the power switch, so that it does a clean shutdown when you switch off.  One network interface became "LAN" (local area network), the other "WAN" (wide area network).   So far just standard stuff, the default install equivalent to a simple NAT firewall to begin with.

                      For PPP over ADSL, we just need to go into Interfaces / WAN in the pfSense web GUI, and set it up as shown below (click image to enlarge):-
                      At the top, you set "Type" to be "PPPoE".  This means the interface doesn't have an IP address or Gateway, as everything should get set up automatically over Ethernet using PPP.   You just need to fill in your ADSL ISP username and password lower down under "PPPoE Configuration".  Then just hit the Save button at the bottom, wait a minute for pfSense to apply the changes, and the link should come up (assuming that you got the username and password correct).

                      If you have problems, the first thing to do is look at the pfSense log.  However, this isn't very detailed, so it may help to configure pfSense to send your system log to another machine using the SYSLOG protocol on the LAN interface.  See my blog post from earlier today, for details of how to configure a FreeBSD server to receive SYSLOG messages from a remote host.

                      Another debugging technique is to plug the pfSense firewall into the ADSL Ethernet modem via an Ethernet hub (not a switch!).  Then plug a laptop into the Ethernet hub and run Wireshark to watch the packets going past.

                      Whilst we're just messing about testing stuff, if you have a recent Apple Mac to hand, you can plug that in instead of the pfSense box to control the PPP link.  On the Mac, go to "System Preferences" - "Network", then press the "+" button at bottom left.  In the pop-up box, for "Interface" choose "PPPoE" and for "Service Name" enter something like "IPV6 PPPoE Test".   In the box that appears, leave the PPPoE service name blank, then enter your ADSL ISP account name and password.   Under "Advanced", just make sure "Configure IPV4" is set to "Using PPP" whilst "Configure IPv6" is set to "Automatically".  Save your settings and press the Connect button.  Once the PPP link comes up, for IPV6 access you'll need to correct the IPV6 address and gateway, as outlined in my earlier post today (PPP over ADSL and PPP over dialup are pretty similar from the Mac's point of view).  If the link doesn't seem to be working, once again you can run Wireshark to watch the PPP protocol start up over the Ethernet port.

                      My next step will probably be to switch from pfSense to Monowall, with a view to adding native IPV6 support to my networks.   Another option would be to stick with IPv4 and pfSense, but add a second ADSL line for higher speed and redundancy.  Apparently pfSense 2.0 alpha supports true WAN link aggregration, as does my ISP, aaisp.net.  

                      UPDATE: Currently running Monowall 1.32 using PPP over ADSL with a Traverse Viking PCI ADSL card.  This has given me native IPV6 over ADSL for the first time, thanks to my ISP, www.aaisp.net .

                      11 comments:

                      1. Interested in your assertion re. PPPoA MTU being 1500. My Draytek 2820 defaults to MTU of 1442, with a maximum of 1492. If I ping out from my network I seem to get fragmentation above 1492 (reported by Wireshark on the receiving end, even with router MTU set to 1442). Setting MTU on the router doesn't seem to make *any* difference to fragmentation (one would expect it to start rejecting packets with "Don't Fragment set", but it doesn't). I've looked at loads of uninformed posts on the Internet, and yours is very well informed! Still can't fathom out my network though :-)

                        ReplyDelete
                      2. Hi Dave,

                        If you're using the Draytek for routing, then 1500 MTU should be possible for PPPoA (at least on BT 20CN ADSL lines in the UK). But if you're using the Draytek just as a modem (doing the PPPoE from pfSense) then you lose 8 bytes per frame, so 1492 is the correct MTU in that case.

                        I believe some ISPs work differently to BT, so 1492 may be correct for them even if your router thinks it's running PPPoA.

                        It's handy that pfSense does MSS Clamping, as that avoid fragmentation without the need to reduce the MTU on all your computers.

                        Of course there's nothing wrong with fragmentation except that some badly-configured websites break path MTU discovery, which will cause problems if your machines try to establish TCP connections using an MTU that's too large for the link.

                        ReplyDelete
                      3. "There are at least half a dozen UK ADSL modems that offer Bridge Mode, whereby your ADSL link still runs PPPoA, but this is presented to you as PPPoE."

                        Not quite. Standard BT-based ADSL can work with either PPPoA or PPPoE, most of these modems really are just the simple ATM/Ethernet bridge, so you are talking PPPoE over the line. The Vigor120 is different and really does convert PPPoE to PPPoA.

                        Real pity it is limited to 1492 MTU though. Many ethernet chips allow "baby jumbos", if the draytek supported this, and if combined with suitably configured interfaces on the router, you could get the full 1500 through...

                        ReplyDelete
                      4. Hi Dave,

                        Given you seem to have done some fairly thorough investigation into this I was wondering if you could shed some light on an issue i'm having?

                        Using a Draytek 2820n on a UK ADSL line with the PPPOA -> PPPOE bridge setup and passed through to the pfsense box however I have issues with MSTSC, whenever I try and RDP in from outside the network I get the login screen, enter my password and hit enter then get a black screen. Apparantly this is an MTU issue as RDP is sensitive to MTU issues but I've tried every combination of settings on the router mtu (not that im even sure that matters in bridge mode but I tried nontheless!) and the MSS clamping value in pfsense and even setting the mtu value on the wan connection using pfsense commandline...

                        The Draytek says max MTU is 1492 and with everything left at default settings, if I do a ping -f l I get utter failure at anything over 1464 bytes and 1464 works perfectly.

                        I nuked all firewall rules that might intefere, setup NAT correctly for RDP (evidently as I was getting to the login) and tried setting firewall to conservative and turning off scrubbing to no avail.

                        The 2nd thing keeping me from using pfsense is a Vodafone Sure Signal fem-to-cell box that does not seem to work at all under PPPOE for reasons I cannot fathom that may well also be related to MTU problems (it uses IPSEC and vpns to vodafone that never seem to even connect using PPPOE).

                        Any help would be greatly appreciated as I need something that can handle a large amount of NAT sessions and do good QoS, both of which pfsense was brilliant at until I realised I couldn't use RDP or the Sure Signal box.

                        ReplyDelete
                      5. Hi Owain

                        I don't know whether Dave is watching this thread. But my own 2 cents worth is that you'd expect the MTU to be 8 bytes less than normal (and pfSense sets 1492 automatically for PPPoE links). Since RDP uses TCP, it should see the clamping setting during TCP session setup, and it should all just work.

                        For troubleshooting purposes, I would try (at least for testing) the following...

                        1. Clamp the MTU to something nice and low like 1400 on the pfSense WAN page. In theory you can go far lower than this: that's what IP fragmentation is for...

                        2. If pfSense has a global setting to disable Fragment processing, make sure that's not ticked. Also make sure that the actual firewall rules on LAN and WAN don't have Fragments blocked under Advanced options. (Can't remember if there are such options, as I'm not running pfSense just now.)

                        3. Lots of MTU-related problems are caused by over-zealous ICMP filtering. Whilst I think MSS clamping ought to render this irrelevant, nonetheless for test purposes it's worth setting the firewall rules so that ICMP is allowed in without restrictions on both LAN and WAN interfaces. You can always lock things down later once you've got it working.

                        4. I would (at least for testing) turn off the Windows firewall completely, and stick a rule on the LAN interface that lets traffic go out without any restrictions.

                        I would have thought the Vodafone gadget ought to cope with NAT traversal and reduced MTU, if you make sure ICMP isn't being filtered. But I don't know what protocols it uses.

                        Good luck! Let us know how you get on.

                        ReplyDelete
                      6. Thanks for the response I will give it another go this evening and if I can get everything working i'm hovering over buying a small atom machine to run it on.

                        A few more things I neglected to add are that my ISP allows me to connect via PPPOE on the Draytek however I didn't seem to be able to get it to pass this off to pfsense when I tried..

                        Finally a rather large omition - this is all running inside a virtual machine using vmware workstation on a Windows 7 host using bridge virtual NICs but I had everything up and running fantastic except for RDPing from outside the LAN to the Win7 host and the fem-2-cell box...

                        ReplyDelete
                      7. Ah, I wasn't reading closely enough..

                        The 2820n Draytek device is designed to offer routing and wireless LAN functions. If you want pfSense to do MSS clamping and firewalling, then you don't want routing and WLAN in the router. So you might be better off with a Draytek Vigor 120 that just acts as an ADSL modem. But maybe you can set up the 2820n Draytek to act as an ADSL modem, if you disable the WLAN and routing functions - though it seems like a waste!

                        I guess it all depends what you want pfSense to do. For example, you could have a small semi-firewalled hosting DMZ between the router and the firewall, with a more secure LAN inside the firewall.

                        ReplyDelete
                      8. Hi Martin, another Martyn here albeit spelt differently!

                        I came across your blog post while trying to resolve an issue I am having with pfSense 2.0 and DSL modems. The issue being it seems to take ages if at all to connect to the DSL this has happened on at least 5 DSL lines and modems so far, I usually use the Draytek modems but have recently tried the Linksys modems with the same effect.

                        Any ideas?

                        ReplyDelete
                      9. You rock man..i was using FreeBSD to connect to ISP and do the Routing and FW etc. Everything was command line based and my other colleague who comes from MS background had trouble doing anything. Just gave pfsense a shot and was just wondering how to make it connect PPPoE. You answered it.
                        Thanks

                        ReplyDelete
                      10. Thanks i was looking for this

                        ReplyDelete

                      Spammers: please stop wasting my time. All comments are moderated before publication.