Web server performance behind Smart Hub 2

[jpstmuk]

I've ran a web server at home for many years now, from early ADSL days to FTTC and for the last year or two with FTTP. 

The Smart Hub 2 is configured with 2 forwarded ports (80/443) to my webserver and I use NOIP to register a dynamic dns name and have my real A record use it as an alias - whatever IP my router negotiates with BT, the real FQDN then resolves to within a few seconds of the IP changing (which it hardly ever does). I have the usual 'sign in to NOIP to verify your account' every month or so, but that's fine. All good.

Website is a mix of static and dynamic content - mainly personal hobby stuff (cars, scale models, programming); using Dokuwiki as a simple markup/content display. It's linked from quite a few places and has found its way into a number of search results so I'd say it was moderately popular for a home website - there are always requests ongoing in the background; right now there are ~300 open connections from clients (mixture of transfers, keep-alive and closing/time_wait) and approx. 40+ active requests ongoing - a couple hundred kbytes of bandwidth is always being used, at a minimum. It has never swamped my upload bandwidth outside of the odd crawl by an LLM.

I've recently changed from the 300mbit product which I was on for the last 2 years or so to 900mbit as I was out of contract. No problems with speed from my home devices - full bandwidth up and down (~930mbit down, ~110 up by most speed checkers). Certainly feels faster from client devices.

However performance of my web server has absolutely tanked over the last couple of months and it feels objectively worse with the 900mbit product.

From any remote device (5g on phone, access from work, friends/family locations etc) I've tried it takes ages to get through to my website - sometimes connections will timeout, sometimes you get SSL handshake errors, other times you'll get a page loaded but not all the images/css/js will be present.

Accessing the webserver from the internal (home) network is still lightning fast and shows no difference in performance - so I suspect some rate limiting or firewall overloading of the SH2.

Is there any mechanism available to view system load of the SH2, or state of the firewall? 

[Kimberlin]

@jpstmuk 

I genuinely don't know the answer to your question, but have you at all considered using a good third party router instead of the SH2?

[WSH]

Yes, you may well be right.  I’ve noticed some sort of issue with the handshake since HTTPS was implemented at the last firmware upgrade.  It seems very slow to connect but I’ve not looked into it.

[jpstmuk]

I think an alternative router may well be an option, as has been mentioned. I do have digital voice however (for as little as we use it), so that may become a blocker.

If I had to guess what was going on, it feels as those the state table of the firewall for incoming connections is getting full - sometimes you get a connection, sometimes you don't, and sometimes only some of your browser requests get fulfilled and not all of them (e.g. elements on the page).

I don't know if this is a symptom of the router not being optimised for exposing 'services', my website getting more popular, or the increased capacity of faster and faster home network connections and the general trend of crawlers to slurp up everything in their path. 

Possibly a combination of all of those factors. It would be nice to be able to confirm that was the case though.

[jpstmuk]

I just asked one of my work colleagues to check things for me and they confirmed that some of my images/css didn't load at first.

I then restarted the router and asked them to check again - within seconds of restarting the router they confirmed that they could load pretty much any page on my website instantly.

It has been a few minutes since then and they're reporting that it's virtually back to the same symptoms as before - sometimes connects, sometimes doesn't, and page content is often missing.

I think that this must confirm it - the router is getting overloaded by incoming requests, scans or SYN/ACK messages and this is choking up the incoming firewall state memory. It doesn't have an effect on outgoing connections - everything is happily downloading at fantastic speeds... but anyone wanting content from my website is having a hard time.

sigh... I guess I am on the lookout for a replacement router; something that can actually handle the load of being connected to the modern internet and its giant, hoovering LLM machinery.... 

[Kimberlin]

@jpstmuk 

There are some great third party routers out there that will be more than capable, some under 200 quid. Look at Draytek, TP-Link and Netgear.

From someone who hasn’t used a ISP router for years, once you’ve gone the third party route, you’re unlikely to go back.

[jpstmuk]

Hopefully one final test to verify my theory - disable SH2 firewall, leaving just port forwards on 80/443 in place.

No restart of the router, just a 'save' of the firewall settings.

Performance back to instant page loads, no missing page elements. As fast as you could expect on a 110mbit/s upload.

I think that this well and truly confirms it that the SH2 firewall / connection tracking state memory is being exhausted by the incoming connections.

Firewall is now back on (in 'default' mode - and performance it back to awful levels) until I have a permanent solution.

I suppose the easiest solution, if I want to retain DV, is to leave the SH2 as is, get a dedicated, ethernet-only router and fit it behind the BT device. I'd then disable wifi and firewall on the SH2 and treat it as being outside of my home network, allowing the second device behind it to do firewalling, wifi and port forwards.

[WSH]

I suspect you'd still have double NAT issues, even with the firewall disabled, but it's something else I've never really looked into in any depth.

[Kimberlin]

@jpstmuk 

Do you actually need your landline? Assuming you have smartphones in your house with WiFi calling enabled, unless you’re signed up to call abroad packages etc, or you have call alarms or are just generally in an area that has frequent power cuts, then the landline often just becomes a fairly pointless addition.

[jpstmuk]

@Kimberlin - I'm rapidly coming to the same conclusion about the landline. I'm not sure if we're quite ready to drop it yet, but I think the argument for keeping it (lots of contact details to update) is getting to be fairly weak.