I've just been thinking about this some more...
As most people here probably know, BT use IP profiles to limit your download bandwidth, usually to slightly below your actual sync speed. Now I'm not entirely sure about this, but I think one of the main reasons for this is so that their end (i.e. the exchange/cabinet) knows how much data it can push down your line without overwhelming it. Without that (or some other way of managing it), I think downloading any file with no bandwidth limitation would consume your connection's entire download bandwidth, holding up other packets and causing the sorts of issues we're seeing here - high pings, packet loss etc.
So I now strongly believe that the actual cause of the issue (at least in my case, and anyone else who's seeing the same sort of speedtest behaviour) is that something, somewhere along the connection path (but probably at a lower level than the IP layer, so not a particular hop that would show up on a traceroute or anything like that) is dynamically limiting the available download bandwidth. Rather than just allowing the maximum possible bandwidth at all times, it seems to be reducing that to somewhere around 10-20mbit/s as soon as the connection goes idle (or when it goes below that speed I guess?). And then it's taking about a second to adjust that upwards when it sees that more bandwidth is needed, which causes the high latency and packet loss in the meantime.
When I first started having this problem, after ruling out all equipment and everything on my side (right up to the master/test socket), I suspected it might be a faulty line card in the fibre cabinet. But obviously now we know that lots of people are having the same issue, so it can't really be that - but it does seem strange to me for it to be anything much further along than that.
So perhaps it could be caused by a firmware update that was applied to the line cards? But then it seems when people have moved to other ISPs the issue has gone away, and they would still be connected to the same line card (I think?) and using the same BT Wholesale/Openreach/whatever it is hardware in the cabinet and exchange etc, so I guess not. So something further up within BT, but which is being applied to each connection individually... some sort of traffic management-type system (gone a bit wrong) I guess?
I also don't know who within BT would actually be able to do anything about it - I haven't even tried to contact them directly myself to even let them know there's an issue yet, because I know that's just going to be incredibly frustrating and not get anywhere unless I can speak to someone higher up, or it becomes a known issue that they're actually aware of.
