It may be helpful to read this first to get a better understanding of pingplotter.
Here's 2 screenshots to my site, one using my VPN and one via my ISP. Neither show much difference in latency but the ISP one could lead someone to think there was packet loss which there wasn't.
Thanks for all the replies. Apologies about assuming the packet loss was the problem. It wasn't and it was to do with the automatic WiFi frequency. Setting to 5GHz frequency as standard fixed the issue. Cheers!