Now a few months ago this used to be manageable as this only seemed to happen when I signed in to XBL, but now this seems to be happening way more frequently and just happened to me in a middle of a game.
Essentially what happens is sometimes when I try to sign into XBL, the loading circle will be stuck in a loop and would eventually time out which would coincide with the internet dropping across the house. Sometimes the hub will stay blue but devices will complain there's no connection, other times the router would turn orange and other times the router would just completely restart.
And at times when the internet wouldn't come back by it self, disconnecting the ethernet cable from the Xbox instantly eliminated all problems.
So I swapped the smart hub for a HH4 that was kicking around and the issue still persists.
I have a 10m long ethernet cable going from the router to a network switch and then I have a short ethernet cable going from the switch to the console.
As you can imagine: I have other devices running from this switch including another Xbox and none of these devices seem to cause problems to the network.
So why is this happening? I've put a spare router in and reset the Xbox to factory defaults.