Why would you need a modem on an FTTP connection? The ONT is the modem.
So can I plug a mesh system directing in to the ONT or does that not have to go through the router for authentication purposes first?
You are confusing your terminology, you referred to using the router as a modem, which you don't need as the ONT is the modem. You will need a router of some description, either a mesh system that incorporates a router (as many do) or a standalone router and pure wifi mesh without router capability.
I've split this off from the thread you posted on as it is not relevant to that thread.
First of all, BT hubs don't have a modem only mode, all you have (presumably) done is to turn off wifi and DHCP.
If the Deco mesh has a router mode, then yes, you can connect it directly to the ONT without a hub. You will need to enter the generic PPPoE credentials of
Username bthomehub@btbroadband.com
Password BT
There is no firewall in the ONT