This topic

Slow(ish) Speeds on Full Fibre 900

kst — Tue, 24 Aug 2021 11:10:37 GMT

Hi,

I'm experiencing slower speeds than expected on Full FIbre 900.

FTTP was installed several weeks ago, "normal available download speed will be between 900 - 900 Mbps", and the "minimum guaranteed download speed will be 450Mbps".

Between 112.5MB/s and 112.5MB/s (very unlikely), and a minimum guaranteed speed of 56.25MB/s (likely).

To rule out the router BT provided as the issue, I've connected the computer directly to the ONT and set up a PPPOE connection.
The cable is Cat 7a, up to 10gig supported.
NIC is an Intel Ethernet Connection I219-V, 1Gbit, latest driver installed, tried auto negotiation and 1gbit full duplex, link speed shows as 1gbit on either.
The disk being wrote to is an NVME SSD PCI-E at x4, read is 2.6GB/s and write is 1.8GB/s.
Running Windows 10 64bit, fully up to date, with an i7-8700k overclocked to 4.9GH/z, 16GB of DDR4 RAM.

When running online speedtests, the speed is mostly what I would expect.
These are likely conducted fully in memory, on a nice optimised server or CDN to give the best results possible.

OVH Speedtest - UK 10Gbit Server:

Google Speedtest:

Ookla Speedtest:

Those numbers look great.

Let's repeat the same tests using actual speedtest files over HTTPS, showing real world performance:

OVH 1GB Speedtest File:

Firefox -

wget -
2021-08-24 11:05:53 (32.7 MB/s) - '1Gb.dat.1' saved [1073741824/1073741824]

Leaseweb 1GB Speedtest File:

Firefox -

wget -
2021-08-24 11:22:34 (28.6 MB/s) - '1000mb.bin.1' saved [1000000000/1000000000]

aria2, 8 threads -
079e3e|OK | 60MiB/s

NFORCE 1GB Speedtest File:

Firefox -

wget -
2021-08-24 11:28:07 (23.9 MB/s) - '1000mb.bin.2' saved [1048576000/1048576000]

aria2, 8 threads -
b682fd|OK | 47MiB/s

Hetzner 1GB Speedtest File:

Firefox -

wget -
2021-08-24 11:33:07 (21.0 MB/s) - '1GB.bin.1' saved [1073741824/1073741824]

aria2, 8 threads -
8abce4|OK | 47MiB/s

As we can see, the real world performance against multiple hosts, in multiple data centres, in multiple regions (UK, NL, DE), shows a significantly different picture.
At those speeds I'm not anywhere near the "minimum guaranteed download speed".

Previously I had Virgin Media and the 600Mbit package, which I was able to fully saturate.
Nothing else has changed, except BT, so I should expect to see speeds at least at the "minimum guaranteed download speed".

Can anyone suggest any further troubleshooting I can do to narrow down the issue?

Re: Slow(ish) Speeds on Full Fibre 900

qs123456 — Tue, 24 Aug 2021 14:50:09 GMT

Following

Re: Slow(ish) Speeds on Full Fibre 900

kst — Tue, 24 Aug 2021 17:14:47 GMT

Further testing:

I've set up a new server at OVH with 10gbit down and 1gbit up.
It's running Ubuntu 20.04, with NGINX on it only.

Speedtest ran from server using the Ookla speedtest CLI tool -
Download: 9406.84 Mbps (data used: 9.6 GB)
Upload: 1956.79 Mbps (data used: 1.1 GB)

Speed when downloading a file from the same server, over HTTP, using my BT connection:

Same situation as all other tests ran, the download speed is abysmal.
This is a private dedicated server, with mostly guaranteed speeds, and should easily be able to saturate the line.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Tue, 24 Aug 2021 18:15:15 GMT

Spoke to BT over the phone, done a factory reset of the router and tested the speeds using the BT Wholesale speedtest.

As expected, the speeds it reached are what I would expect to see, as does every other web based test.

An engineer is coming out tomorrow to have a look, will update afterwards.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 10:46:21 GMT

Openreach engineer came out, ran some speed tests over fast.com which showed expected speeds.

Followed up with the BT Fibre Team and was stonewalled.

They ran a speedtest which showed the speed was acceptable.

I was told they can see everything is fine, they don't need to do anything, there is nobody in BT they can escalate my ticket to, if I was to open a complaint, it would be open and closed instantly, and they are not trained to deal with issues like this, they have no idea what I'm talking about.

Tried a few hypotheticals to see where a ticket would go if I was to report massive packet loss, packet loss within the BT network to a core router, a streaming service that didn't work like Netflix despite the speed test showing all OK, etc. Apparently the ticket would go nowhere because they have nobody to escalate to and Openreach manage BT's internal network.

Basically, if fast.com, or speedtest.net say your connection is great, BT doesn't care, despite any real world speedtest showing otherwise.

I am expecting a call back from someone by 2:30PM today, will provide an update then.

I can understand 1st line support not being trained to handle more than the run-of-the-mill issue, that's fine, I've sat in that seat previously in my career.

What I don't understand, is that they have nobody to escalate to, and a complaint of this kind would be open and closed instantly.

I have never been stonewalled like that before, and it is beyond frustrating.
I've followed the process, but apparently it dies when BT's Fibre team who receive minimal training cannot help.

Not the experience I was expecting from BT.

Re: Slow(ish) Speeds on Full Fibre 900

Will168 — Wed, 25 Aug 2021 11:01:52 GMT

@kst I'm not as technical as you when it comes to all the speed testing etc. so sorry if this does not help, but I tried testing a large-ish file...the Windows 10 64bit ISO from Microsoft.com (not sure where their server/CDN is located) and ran a couple of speed tests and mine does seem to saturate the line, as it seems to sustain 108 to 110 MB/s downloading.

Re: Slow(ish) Speeds on Full Fibre 900

-Richie- — Wed, 25 Aug 2021 11:08:25 GMT

@kstwrote:
What I don't understand, is that they have nobody to escalate to, and a complaint of this kind would be open and closed instantly.

This is incorrect, BT have a clear escalation process for handling complaints, tell them to escalate and you don't give permission to close the complaint.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 11:49:59 GMT

Hi @Will168,

When I try to download an ISO from Microsoft I'm seeing roughly the same speeds as all other tests:

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 12:10:28 GMT

My complaint has been submitted, it has made its way to the duty manager who I've spoken to twice.

BT unfortunately do not have someone that can deal with this issue.

BT is offering a product without any technical support beyond running a speedtest and sending an Openreach engineer.

BT did have a further level of support, but that was done away with a long time ago.

There is nothing more BT can do and I will be issued a Deadlock letter to take it up with the ombudsman.

To sum up the issue:

Other than web based speedtests, I am not able to utilise my connection, to the minimum guaranteed download speed, in any real world test when downloading.
I've tried speedtest files from the EU's biggest cloud/dedicated server hosting providers, in multiple regions (including the UK).
I've set up a brand new dedicated 10gbit down/1gbit up (which is burstable to 2gbit up) server in the UK where I can mostly guarantee speeds, and demonstrate them, but even this cannot get to the minimum guaranteed download speed.
I've also tested on another dedicated gbit server located in Germany, this cannot reach the minimum guaranteed download speed either.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 12:18:11 GMT

>This is incorrect, BT have a clear escalation process for handling complaints, tell them to escalate and you don't give permission to close the complaint.

This is what really annoyed me.

The person said they would open a complaint, and personally close it immediately, because as far as they can see, everything is working great. It would include everything I've told them though....

Very bad customer service, and there is no way the person taking the complaint would be closing it.

Suffice to say, the CSAT text message I got after the call was responded to in full.

Re: Slow(ish) Speeds on Full Fibre 900

-Richie- — Wed, 25 Aug 2021 12:27:18 GMT

BT will only guarantee the speed to the hub, if you are receiving the speeds then BT are providing the contractually agreed service.

Downloading from 3rd party sources including from Europe adds other factors, peering/transit agreements, the hard drive used to download and other PC components etc all play a factor outside BT's control.
Not being able to download at full speeds does not indicate you're not getting the speeds advertised.

If you're being issued a deadlock letter, that means BT have not been able to reach agreement with you, so your complaint has been escalated as front line advisors can't send deadlock letters, this part confuses me because you say it wasn't escalated but it must of.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 12:52:34 GMT

>BT will only guarantee the speed to the hub, if you are receiving the speeds then BT are providing the contractually agreed service.

There is no point of having a 900mbit package if the only way to utilise it is on web based speedtests and speedtests which are likely internal to BT's network.

The package isn't sold as 900mbit internally, and up to 240mbit externally.

>Downloading from 3rd party sources including from Europe adds other factors, peering/transit agreement

I've tested in the UK, Germany, Netherlands, and Ireland.

OVH have public peering with LONIX and private peering with Telehouse (with lots of capacity) who BT also peers with.
I don't think AWS, Leaseweb, NFORCE, and Hetzner will be much different.

>the hard drive used

Tested, read is 2.6GB/s and write is 1.8GB/s.

>other PC components etc

All tested.

I have previously saturated a 600mbit line, with Virgin Media, the issue does not sit with my set up.

>Not being able to download at full speeds does not indicate you're not getting the speeds advertised.

Not being able to hit the minimum speed does.

Re: Slow(ish) Speeds on Full Fibre 900

Checked2013 — Wed, 25 Aug 2021 14:38:15 GMT

It would be interesting to see if you've done any tests using a different pc, as that may double check things like wrong MTU or a browser issue (Firefox has been known in the past for slow multi-threaded downloads).

You say that it was fine on VM, but what works on VM might not work on everything else.

Also, we might be getting ultra high speed connections, but you may have to wait a bit for the rest of the Internet to catch up. For the moment in many cases a 900mb connection works best if there is multiple people using the connection at a time and that might be a good test also, by trying to download on multiple devices at the same time.

One last thing, make sure you don't have jumbo frames enabled, as that can also slow downloads.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 15:03:30 GMT

>It would be interesting to see if you've done any tests using a different pc

Two computers, both worked fine over VM.

>MTU

MTU is 1492.

>browser issue

Tried Firefox and Chrome.

>You say that it was fine on VM, but what works on VM might not work on everything else.

BT has enough capacity, peering, and transit to handle hitting the minimum guaranteed download speed.

>Also, we might be getting ultra high speed connections, but you may have to wait a bit for the rest of the Internet to catch up.

Gbit, 2Gbit, and 10Gbit servers tested, in multiple regions, on multiple providers - I'm not even hitting the minimum guaranteed speed, which I would be happy with.

>works best if there is multiple people using the connection at a time and that might be a good test also, by trying to download on multiple devices at the same time.

I've tried multi-threaded downloads and hitting around the same speeds.

Re: Slow(ish) Speeds on Full Fibre 900

Checked2013 — Wed, 25 Aug 2021 15:57:40 GMT

My point about another computer, is to see if default values work and not one you've used previously, just in case there has been settings you've changed.

Usually providers of servers quote total speed (meaning that it's shared between all their servers)

I also didn't mean try multi-threaded downloads, but try doing downloads on let's say 3 to 4 devices at the same time and see what the total speeds are.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 16:39:21 GMT

>My point about another computer, is to see if default values work and not one you've used previously, just in case there has been settings you've changed.

Nothing really has been changed, they're default Windows 10 installs with nothing fancy going on.

>Usually providers of servers quote total speed (meaning that it's shared between all their servers)

iperf3 run, UK server as the client, uploading to another server running iperf3:

root@some_host:~# iperf3 -c lon.speedtest.clouvider.net -p 5200
Connecting to host lon.speedtest.clouvider.net, port 5200
[ 5] local some_ip port 43680 connected to another_ip port 5200
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 264 MBytes 2.21 Gbits/sec 2139 167 KBytes
[ 5] 1.00-2.00 sec 222 MBytes 1.86 Gbits/sec 3168 319 KBytes
[ 5] 2.00-3.00 sec 226 MBytes 1.89 Gbits/sec 3256 180 KBytes
[ 5] 3.00-4.00 sec 224 MBytes 1.88 Gbits/sec 3049 174 KBytes
[ 5] 4.00-5.00 sec 223 MBytes 1.87 Gbits/sec 3734 134 KBytes
[ 5] 5.00-6.00 sec 225 MBytes 1.89 Gbits/sec 3388 160 KBytes
[ 5] 6.00-7.00 sec 225 MBytes 1.88 Gbits/sec 3361 130 KBytes
[ 5] 7.00-8.00 sec 225 MBytes 1.88 Gbits/sec 3298 141 KBytes
[ 5] 8.00-9.00 sec 223 MBytes 1.87 Gbits/sec 3326 226 KBytes
[ 5] 9.00-10.00 sec 225 MBytes 1.89 Gbits/sec 3604 146 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.23 GBytes 1.91 Gbits/sec 32323 sender
[ 5] 0.00-10.00 sec 2.23 GBytes 1.91 Gbits/sec receiver

Ookla Speedtests from the UK server, using their CLI tool:

Download: 9399.89 Mbps (data used: 9.7 GB)
Upload: 1916.97 Mbps (data used: 2.3 GB)

Download: 8398.18 Mbps (data used: 9.3 GB)
Upload: 1938.68 Mbps (data used: 2.2 GB)

Download: 6436.37 Mbps (data used: 4.7 GB)
Upload: 1912.91 Mbps (data used: 2.3 GB)

Download: 6052.92 Mbps (data used: 5.8 GB)
Upload: 1887.84 Mbps (data used: 1.8 GB)

As we can see, the UK server is sending at around 1.91Gb/s.

I've tested speeds from the biggest cloud/dedicated server hosts in the EU region, using their UK data centres where possible, including AWS, and still cannot get anywhere near the minimum guaranteed speed.

This is a BT network issue, there's nothing more I can do.

Re: Slow(ish) Speeds on Full Fibre 900

Checked2013 — Wed, 25 Aug 2021 16:44:07 GMT

I think you seem to be missing the point and probably why you aren't getting anywhere with BT.

Never mind being able or not able to transfer from servers in data centres. But can you for instance download the latest NVIDIA drivers and can you then do it on multiple computers at the same time? The reason for this test is to determine if you have local congestion and it's the only way you would be able to get BT to listen as they would not be interested on 3rd party servers.

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 16:52:37 GMT

Over two devices, I saw a peak combined speed of around 38MB/s, 8MB/s higher than the peak I've seen of 30MB/s on previous tests, if I remember correctly.

Still below the minimum guaranteed speed advertised.

Re: Slow(ish) Speeds on Full Fibre 900

Checked2013 — Wed, 25 Aug 2021 17:52:42 GMT

You could post some screenshots of that and a ping to Google or something similar while you are doing the downloads?

Re: Slow(ish) Speeds on Full Fibre 900

kst — Wed, 25 Aug 2021 23:08:37 GMT

Hi @Checked2013,

Done some more testing tonight and narrowed it down to single thread performance.

Turns out aria2 doesn't work very well on Windows and my initial tests to rule that out, which did, can be considered worthless.

Single threaded:

This matches what I'm seeing when downloading using Firefox, Chrome, or using FTPS.

Multi-threaded:

One graph:

Speedtest files, 8 threads:

OVH -

Leaseweb -

NFORCE -

Hetzner -

Contrast those to the speeds I was seeing on my first post.
The consistency as seen in the graphs isn't great, but at least the speeds are much higher than what I was seeing on a single thread download.

So the question is, why is single threaded network performance on the BT network so bad?

I'll phone BT in the morning to have these notes added to my complaint and see if they want to kick it back to Openreach for further investigation, as they apparently run and manage BT's internal network...