But you don't need the Openreach modem, the HH5 plugs directly into the telephone socket instead of the modem. By doing that you can get some meaningful information to help diagnose your problem. If you are not going to help us, we can't help you.
Unplug the lead that is plugged into your modem that comes from the phone socket and just plug it straight into the DSL socket on the HH5 instead. Remove the RJ45 lead between the modem and HH5.
It is not a fine, if the fault is with your equipment etc it is a charge for getting Openreach out to fix a problem that they are not responsible for.
If you are 100% certain that the fault is not anything to do with your equipment, wiring or any possible legacy wiring within your house you will not be charged.
If you haven't already done it you will need to contact BT and request an engineer.