That's because it is unicast not multicast (1 to 1 vs 1 to many) and it doesn't care if the rest of your household suffers because you are using the majority of the connection speed.
With BT TV it is their way of trying to manage some sort of consistent service so you can watch BT Sport Ultimate through the YouView box and potentially record another multicast delivered channel and still have some sort of basic internet service.
If you were on the highest speed BT sells you wouldn't have any issues! The speeds available depend on your specific line etc so you're on the best speeds available on your line not the best speeds available from BT
It does state as follows in the terms & conditions
4K: Selected content available in 4K. Min line speed and compatible equipment required.
Basically it’s a mess one of main reasons I signed for V.I.P package was to get the 4K should of been told when I rang up and ordered