Problems with FTTP on PFsense

[abyssunderground]

Hi all,

I wonder if anyone can give me some help with this as I'm currently at a loss as to what the problem is. I have spoken to BT support but they were not able to help with 3rd party routers and directed me here.

I can successfully connect to the Internet using PFsense and running speedtests brings back the expected results, but after that is where issues begin.

If I try to download from my dedicated server in Germany, I am lucky to get 5Mbps.

If I download from my friend in Sweden I am lucky to get 2.5Mbps.

Even downloading from PFsense's own website gives me only 5Mbps but on my dedicated server I get 400Mbps so there is no bottleneck on their end.

However if I connect my BT Smart Hub back in place I can get 190Mbps from my server in Germany and 35Mbps from my friend in Sweden, and I can get good speeds from PFsense's website too.

I don't understand why I would get slower speeds to Germany/Sweden but still get full speed on speedtests simply by changing to PFsense rather than using the BT Smart Hub, unless there is some setting that I have misconfigured. PFsense is up to date and I have even tried a completely fresh install too to rule out anything other settings may have caused.

Here are the settings I am currently using:

PPPoE

Username: bthomehub@btbroadband.com

Password: BT

MTU: 1470 (default, and not entered also auto-negotiates to this value)

I have read some people using VLAN 101 but trying this results in no connection being established.

My advanced settings are all set to the defaults as I have no idea what I would change here. I haven't found any information online regarding those.

Here is my PPPoE connection log, in case this gives any further information that you may need:

Spoiler

Dec 3 20:39:05 ppp Multi-link PPP daemon for FreeBSD
Dec 3 20:39:05 ppp process 22871 started, version 5.8 (root@pfSense_v2_4_4_amd64-pfSense_v2_4_4-job-09 11:56 16-Nov-2018)
Dec 3 20:39:05 ppp waiting for process 70115 to die...
Dec 3 20:39:05 ppp caught fatal signal TERM
Dec 3 20:39:05 ppp [wan] IFACE: Close event
Dec 3 20:39:05 ppp [wan] IPCP: Close event
Dec 3 20:39:05 ppp [wan] IPCP: state change Opened --> Closing
Dec 3 20:39:05 ppp [wan] IPCP: SendTerminateReq #4
Dec 3 20:39:05 ppp [wan] IPCP: LayerDown
Dec 3 20:39:05 ppp [wan] IPV6CP: Close event
Dec 3 20:39:05 ppp [wan] IPV6CP: state change Opened --> Closing
Dec 3 20:39:05 ppp [wan] IPV6CP: SendTerminateReq #2
Dec 3 20:39:05 ppp [wan] IPV6CP: LayerDown
Dec 3 20:39:05 ppp [wan] IFACE: Down event
Dec 3 20:39:05 ppp [wan] IFACE: Rename interface pppoe2 to pppoe2
Dec 3 20:39:05 ppp [wan] IPCP: rec'd Terminate Ack #4 (Closing)
Dec 3 20:39:05 ppp [wan] IPCP: state change Closing --> Closed
Dec 3 20:39:05 ppp [wan] IPCP: LayerFinish
Dec 3 20:39:05 ppp [wan_link0] LCP: rec'd Terminate Request #117 (Opened)
Dec 3 20:39:05 ppp [wan_link0] LCP: state change Opened --> Stopping
Dec 3 20:39:05 ppp [wan_link0] Link: Leave bundle "wan"
Dec 3 20:39:05 ppp [wan] Bundle: Status update: up 0 links, total bandwidth 9600 bps
Dec 3 20:39:05 ppp [wan] IPCP: Close event
Dec 3 20:39:05 ppp [wan] IPV6CP: Close event
Dec 3 20:39:05 ppp [wan] IPCP: Down event
Dec 3 20:39:05 ppp [wan] IPCP: state change Closed --> Initial
Dec 3 20:39:05 ppp [wan] IPV6CP: Down event
Dec 3 20:39:05 ppp [wan] IPV6CP: LayerFinish
Dec 3 20:39:05 ppp [wan] Bundle: No NCPs left. Closing links...
Dec 3 20:39:05 ppp [wan] IPV6CP: state change Closing --> Initial
Dec 3 20:39:05 ppp [wan_link0] LCP: SendTerminateAck #3
Dec 3 20:39:05 ppp [wan_link0] LCP: LayerDown
Dec 3 20:39:05 ppp [wan] rec'd unexpected protocol IP
Dec 3 20:39:05 ppp [wan] rec'd unexpected protocol IP
Dec 3 20:39:05 ppp [wan_link0] PPPoE: connection closed
Dec 3 20:39:05 ppp [wan_link0] Link: DOWN event
Dec 3 20:39:05 ppp [wan_link0] Link: giving up after 0 reconnection attempts
Dec 3 20:39:05 ppp [wan_link0] LCP: Close event
Dec 3 20:39:05 ppp [wan_link0] LCP: state change Stopping --> Closing
Dec 3 20:39:05 ppp [wan_link0] LCP: Down event
Dec 3 20:39:05 ppp [wan_link0] LCP: LayerFinish
Dec 3 20:39:05 ppp [wan_link0] LCP: state change Closing --> Initial
Dec 3 20:39:06 ppp waiting for process 70115 to die...
Dec 3 20:39:07 ppp waiting for process 70115 to die...
Dec 3 20:39:07 ppp [wan] Bundle: Shutdown
Dec 3 20:39:07 ppp [wan_link0] Link: Shutdown
Dec 3 20:39:07 ppp process 70115 terminated
Dec 3 20:39:08 ppp web: web is not running
Dec 3 20:39:08 ppp [wan] Bundle: Interface ng0 created
Dec 3 20:39:08 ppp [wan_link0] Link: OPEN event
Dec 3 20:39:08 ppp [wan_link0] LCP: Open event
Dec 3 20:39:08 ppp [wan_link0] LCP: state change Initial --> Starting
Dec 3 20:39:08 ppp [wan_link0] LCP: LayerStart
Dec 3 20:39:08 ppp [wan_link0] PPPoE: Connecting to ''
Dec 3 20:39:08 ppp PPPoE: rec'd ACNAME "acc-aln2.lv-wal"
Dec 3 20:39:08 ppp [wan_link0] PPPoE: connection successful
Dec 3 20:39:08 ppp [wan_link0] Link: UP event
Dec 3 20:39:08 ppp [wan_link0] LCP: Up event
Dec 3 20:39:08 ppp [wan_link0] LCP: state change Starting --> Req-Sent
Dec 3 20:39:08 ppp [wan_link0] LCP: SendConfigReq #1
Dec 3 20:39:08 ppp [wan_link0] PROTOCOMP
Dec 3 20:39:08 ppp [wan_link0] MRU 1492
Dec 3 20:39:08 ppp [wan_link0] MAGICNUM 0xbe79b0b4
Dec 3 20:39:08 ppp [wan_link0] LCP: rec'd Configure Request #13 (Req-Sent)
Dec 3 20:39:08 ppp [wan_link0] MRU 1492
Dec 3 20:39:08 ppp [wan_link0] AUTHPROTO CHAP MD5
Dec 3 20:39:08 ppp [wan_link0] MAGICNUM 0x40bd5d5d
Dec 3 20:39:08 ppp [wan_link0] LCP: SendConfigAck #13
Dec 3 20:39:08 ppp [wan_link0] MRU 1492
Dec 3 20:39:08 ppp [wan_link0] AUTHPROTO CHAP MD5
Dec 3 20:39:08 ppp [wan_link0] MAGICNUM 0x40bd5d5d
Dec 3 20:39:08 ppp [wan_link0] LCP: state change Req-Sent --> Ack-Sent
Dec 3 20:39:08 ppp [wan_link0] LCP: rec'd Configure Reject #1 (Ack-Sent)
Dec 3 20:39:08 ppp [wan_link0] PROTOCOMP
Dec 3 20:39:08 ppp [wan_link0] LCP: SendConfigReq #2
Dec 3 20:39:08 ppp [wan_link0] MRU 1492
Dec 3 20:39:08 ppp [wan_link0] MAGICNUM 0xbe79b0b4
Dec 3 20:39:08 ppp [wan_link0] rec'd proto CHAP during establishment phase
Dec 3 20:39:08 ppp [wan_link0] LCP: rec'd Configure Ack #2 (Ack-Sent)
Dec 3 20:39:08 ppp [wan_link0] MRU 1492
Dec 3 20:39:08 ppp [wan_link0] MAGICNUM 0xbe79b0b4
Dec 3 20:39:08 ppp [wan_link0] LCP: state change Ack-Sent --> Opened
Dec 3 20:39:08 ppp [wan_link0] LCP: auth: peer wants CHAP, I want nothing
Dec 3 20:39:08 ppp [wan_link0] LCP: LayerUp
Dec 3 20:39:11 ppp [wan_link0] CHAP: rec'd CHALLENGE #1 len: 69
Dec 3 20:39:11 ppp [wan_link0] Name: "acc-aln2.lv-wal"
Dec 3 20:39:11 ppp [wan_link0] CHAP: Using authname "bthomehub@btinternet.com"
Dec 3 20:39:11 ppp [wan_link0] CHAP: sending RESPONSE #1 len: 45
Dec 3 20:39:11 ppp [wan_link0] CHAP: rec'd SUCCESS #1 len: 31
Dec 3 20:39:11 ppp [wan_link0] MESG: CHAP authentication success
Dec 3 20:39:11 ppp [wan_link0] LCP: authorization successful
Dec 3 20:39:11 ppp [wan_link0] Link: Matched action 'bundle "wan" ""'
Dec 3 20:39:11 ppp [wan_link0] Link: Join bundle "wan"
Dec 3 20:39:11 ppp [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
Dec 3 20:39:11 ppp [wan] IPCP: Open event
Dec 3 20:39:11 ppp [wan] IPCP: state change Initial --> Starting
Dec 3 20:39:11 ppp [wan] IPCP: LayerStart
Dec 3 20:39:11 ppp [wan] IPV6CP: Open event
Dec 3 20:39:11 ppp [wan] IPV6CP: state change Initial --> Starting
Dec 3 20:39:11 ppp [wan] IPV6CP: LayerStart
Dec 3 20:39:11 ppp [wan] IPCP: Up event
Dec 3 20:39:11 ppp [wan] IPCP: state change Starting --> Req-Sent
Dec 3 20:39:11 ppp [wan] IPCP: SendConfigReq #1
Dec 3 20:39:11 ppp [wan] IPADDR 0.0.0.0
Dec 3 20:39:11 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Dec 3 20:39:11 ppp [wan] IPV6CP: Up event
Dec 3 20:39:11 ppp [wan] IPV6CP: state change Starting --> Req-Sent
Dec 3 20:39:11 ppp [wan] IPV6CP: SendConfigReq #1
Dec 3 20:39:11 ppp [wan] IPV6CP: rec'd Configure Request #146 (Req-Sent)
Dec 3 20:39:11 ppp [wan] IPV6CP: SendConfigAck #146
Dec 3 20:39:11 ppp [wan] IPV6CP: state change Req-Sent --> Ack-Sent
Dec 3 20:39:11 ppp [wan] IPCP: rec'd Configure Request #49 (Req-Sent)
Dec 3 20:39:11 ppp [wan] IPADDR 172.16.13.94
Dec 3 20:39:11 ppp [wan] 172.16.13.94 is OK
Dec 3 20:39:11 ppp [wan] IPCP: SendConfigAck #49
Dec 3 20:39:11 ppp [wan] IPADDR 172.16.13.94
Dec 3 20:39:11 ppp [wan] IPCP: state change Req-Sent --> Ack-Sent
Dec 3 20:39:11 ppp [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
Dec 3 20:39:11 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Dec 3 20:39:11 ppp [wan] IPCP: SendConfigReq #2
Dec 3 20:39:11 ppp [wan] IPADDR 0.0.0.0
Dec 3 20:39:11 ppp [wan] IPV6CP: rec'd Configure Ack #1 (Ack-Sent)
Dec 3 20:39:11 ppp [wan] IPV6CP: state change Ack-Sent --> Opened
Dec 3 20:39:11 ppp [wan] IPV6CP: LayerUp
Dec 3 20:39:11 ppp [wan] 0215:5dff:fe00:0102 -> 22e0:9cff:fe16:c801
Dec 3 20:39:11 ppp [wan] IFACE: Up event
Dec 3 20:39:11 ppp [wan] IFACE: Rename interface ng0 to pppoe2
Dec 3 20:39:11 ppp [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
Dec 3 20:39:11 ppp [wan] IPADDR 86.178.28.70
Dec 3 20:39:11 ppp [wan] 86.178.28.70 is OK
Dec 3 20:39:11 ppp [wan] IPCP: SendConfigReq #3
Dec 3 20:39:11 ppp [wan] IPADDR 86.178.28.70
Dec 3 20:39:11 ppp [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
Dec 3 20:39:11 ppp [wan] IPADDR 86.178.28.70
Dec 3 20:39:11 ppp [wan] IPCP: state change Ack-Sent --> Opened
Dec 3 20:39:11 ppp [wan] IPCP: LayerUp
Dec 3 20:39:11 ppp [wan] 86.178.28.70 -> 172.16.13.94

Any help you can give is greatly appreciated.

Thanks

[albertj]

Hi @abyssunderground 

Thanks for posting.

Have you got another third party router to test?

I don’t have much experience with PFSense, but can’t think of something that would cause this to happen.

Have you posted on the PFSense Community as well? https://forum.netgate.com

Thanks

[licquorice]

Also worth asking on Kitz, there are a few PFsense users over there. https://forum.kitz.co.uk/

[brookheather]

I use pfSense with FTTP without any performance issues.  What hardware are you using - does it have an Intel network card with two ports (one for the ONT and one to your LAN)?

[abyssunderground · Accepted solution]

Thanks for the reply everyone. I have managed to solve this issue after a lot of head banging (months of it!).

Setting the Ethernet card to use jumbo frames with a setting of 9014 solved this.

Because I route everything through Hyper-V (PFsense is virtualised) the MTU wasn't high enough to encase the Hyper-V virtual switch as well as the PPPoE connection and whatever else was going on, so it was fragmenting badly (confirmed with some command line testing with a knowledgeable friend). Some how this was causing issues in specific instances which are now resolved by turning on jumbo frames on the Hyper-V host's network card which the BT modem was connected to.

Out of interest, both places I tried to download from also used Hyper-V so it must be that adding them into the mix was also exceeding the size of the MTU by the time it was reaching the target machine. It's definitely an odd case as I doubt this is a very common setup.

I actually found no mention of this anywhere else and had spent days researching for a solution. It was only through trial and error testing that I came to the solution.

But again, thank you for your replies in any case!

Andy