Is BT hub type A compatible with TL-WA850RE? I am having problem setting up wifi range extender
So the assumption is that you have a BT Home Hub 5 type A. This is compatible with WiFi 5 (AC) and lower. The TP-Link TL-WA850RE is WiFi4 (N). So yes your home hub is compatible, though thanks to the way WiFi extenders work that N300, is more like N150 (thanks to retransmission).
Unless you've turned the feature off, then you should be able to connect the two devices INITIALLY by pressing the WPS button on the Home Hub and then doing the same on the Range extender. If that doesn't work then use the reset button on the extender to reset and try again. Should that still not work, you'll probably need to dig out the instructions on the range extender and follow them like a pedant - usually requires connecting to the extender, getting into its settings and then entering the connection settings to the router manually, but it's almost never necessary to do this!
*Don't overthink it, and don't try to take shortcuts!