Is there anyone at BT who can take a report of an issue on one of BT's own network switches?

[stephence]

Today there is major packet loss (25%) and slow throughput (150-228ms response time) on 166-49-214-198.gia.bt.net.

The BT "report a fault" team do not appear to have any way take a report of such an issue and get it to the appropriate networking team.

After about an hour on the phone today the best they could do today was to try to get me to "go to a computer shop to get a new IPS address"....

Code: [Select]

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                                  PIGLET -    0 | 4456 | 4456 |    0 |    0 |    4 |    0 |
|                           172.16.15.150 -    0 | 4456 | 4456 |    3 |    3 |   72 |    3 |
|                           31.55.185.177 -  100 |  896 |    1 |    0 |    6 |    6 |    6 |
|                           31.55.185.184 -    0 | 4456 | 4456 |    6 |    6 |   73 |    7 |
|core2-hu0-15-0-6.colindale.ukcore.bt.net -    0 | 4455 | 4455 |    5 |    6 |   74 |    6 |
|  peer8-et-0-1-4.telehouse.ukcore.bt.net -    1 | 4426 | 4418 |    6 |    9 |   97 |    7 |
|               166-49-214-168.gia.bt.net -    0 | 4456 | 4456 |    6 |    7 |   74 |    7 |
|               166-49-214-198.gia.bt.net -   25 | 2272 | 1721 |  148 |  150 |  228 |  150 |

[pippincp]

I suspect this is just a snippet of the full path you are attempting to trace and proves nothing and just shows a router treating ICMP requests as low priority which is normal.

Pinging a core router proves nothing as it concentrates on forwarding packets and can (and does) ignore ICMP requests hence the apparent problem if you don't know how to interpret the results of a trace. What was the full result of the trace to the destination IP?

[stephence]

Here's the full route:

I suspect it was treating ICMP pings as low priority because it was overwhelmed

|---------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|---------------------------------------------------|------|------|------|------|------|------|
| PIGLET - 0 | 4456 | 4456 | 0 | 0 | 4 | 0 | Bournemouth
| 172.16.15.150 - 0 | 4456 | 4456 | 3 | 3 | 72 | 3 |
| 31.55.185.177 - 100 | 896 | 1 | 0 | 6 | 6 | 6 |
| 31.55.185.184 - 0 | 4456 | 4456 | 6 | 6 | 73 | 7 |
|core2-hu0-15-0-6.colindale.ukcore.bt.net - 0 | 4455 | 4455 | 5 | 6 | 74 | 6 | Colindale (London)
| peer8-et-0-1-4.telehouse.ukcore.bt.net - 1 | 4426 | 4418 | 6 | 9 | 97 | 7 | Telehouse (London)
| 166-49-214-168.gia.bt.net - 0 | 4456 | 4456 | 6 | 7 | 74 | 7 |
| 166-49-214-198.gia.bt.net - 25 | 2272 | 1721 | 148 | 150 | 228 | 150 | London
| No response from host - 100 | 895 | 0 | 0 | 0 | 0 | 0 |
| uk-lon03a-ri2-ae-2-0.aorta.net - 24 | 2304 | 1761 | 149 | 151 | 212 | 150 |
| No response from host - 100 | 895 | 0 | 0 | 0 | 0 | 0 |
| brnt-ic-3-ae0-0.network.virginmedia.net - 25 | 2278 | 1728 | 149 | 152 | 201 | 152 | Sheffield
| No response from host - 100 | 895 | 0 | 0 | 0 | 0 | 0 |
|sotn-core-2b-ae2-0.network.virginmedia.net - 26 | 2222 | 1659 | 151 | 152 | 189 | 152 | Southampton
|sotn-metnet-3a-lag-56.network.virginmedia.net - 16 | 2770 | 2343 | 108 | 109 | 182 | 110 |
| drkn-bh1-ia1.network.virginmedia.net - 26 | 2198 | 1628 | 111 | 153 | 190 | 153 |
| miasma.rocks - 27 | 2190 | 1618 | 152 | 153 | 190 | 153 | Bournemouth
|___________________________________________________|______|______|______|______|______|______|

This is how it looked once the 25% packet loss (for UDP traffic, not ICMP) was resolved:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| PIGLET - 0 | 45 | 45 | 0 | 0 | 0 | 0 |
| 172.16.15.150 - 0 | 45 | 45 | 3 | 3 | 5 | 3 |
| No response from host - 100 | 9 | 0 | 0 | 0 | 0 | 0 |
| 31.55.185.184 - 0 | 45 | 45 | 6 | 6 | 7 | 7 |
|core2-hu0-15-0-6.colindale.ukcore.bt.net - 0 | 45 | 45 | 6 | 6 | 7 | 6 |
| core6-hu0-3-0-15.faraday.ukcore.bt.net - 3 | 42 | 41 | 6 | 7 | 8 | 7 |
| 166-49-209-194.gia.bt.net - 0 | 45 | 45 | 6 | 7 | 24 | 6 |
| lag-127.ear2.London2.Level3.net - 0 | 45 | 45 | 7 | 7 | 8 | 7 |
| No response from host - 100 | 9 | 0 | 0 | 0 | 0 | 0 |
| LibertyGlobal-level3-London2.Level3.net - 0 | 45 | 45 | 7 | 9 | 26 | 7 |
| No response from host - 100 | 9 | 0 | 0 | 0 | 0 | 0 |
| brnt-ic-3-ae0-0.network.virginmedia.net - 0 | 45 | 45 | 8 | 9 | 23 | 8 |
| No response from host - 100 | 9 | 0 | 0 | 0 | 0 | 0 |
|sotn-core-2b-ae2-0.network.virginmedia.net - 0 | 45 | 45 | 9 | 9 | 11 | 9 |
|sotn-metnet-3a-lag-56.network.virginmedia.net - 3 | 42 | 41 | 9 | 9 | 10 | 9 |
| drkn-bh1-ia1.network.virginmedia.net - 0 | 45 | 45 | 10 | 10 | 14 | 10 |
| miasma.rocks - 0 | 45 | 45 | 10 | 10 | 12 | 11 |
|________________________________________________|______|______|______|______|______|______|

[pippincp]

I think you'll find the issue was with Virgin Media, it all started with a national outage 4th April and have had intermittant issues in different areas since then.

[stephence]

While I'm aware that the switches prioritise, the 25% ICMP ping loss at 166-49-214-198.gia.bt.net corresponded exactly with the 25% UDP packet loss I was seeing end to end and the +150 ping in reponse from 166-49-214-198.gia.bt.net corresponded almost exactly with the end to end increase in end to end time I was seeing with data UDP packets between applications running on the machnines at both ends over the same period.

Is there any sensible way to ascertain source of delay and packet loss for TCP/UDP traffic along the route between two points you have control of other than using windows tracert ICMP or linux traceroute UDP packets?

Is there any team at BT that can investigate such problems if they arise and persist? The "technical" team I talked to, as I mentioned in my first post, advised me to go to a computer shop to get a new "IPS address" - and had no team they knew to refer to for anything beyond sending an engineer out to check the cabling into the premesis.

[licquorice]

BT's network is constantly monitored 24/7 and  they would be perfectly aware of any problems without having to resort to members of the public informing them of such.

[stephence]

You'd certainly hope so. However, when issues go on hour after hour and you can see 25% of your information being lost end to end it doesn't always seem like this is the case.

The "technical" teams facing clients have absolutely no idea what to do when such a continuing issue is seen (unless I really do need to go to a computer shop to get a new IPS address to fix the networking problem).

The question was two-fold.

1. Is there a way for the owner of two end-points to identify the problem point in the routing, or is this impossible? In my tracert output it still seems clear to me that the point dropping exactly the percentage of ICMP and taking pretty much exactly the increase in time to respond I was seeing in other traffic was the culpret

2. Is there a team at BT who can take reports that there is a problem where the routing doesn't get sorted out reasonably fast?