I’ve been obsessing over this on and off for about a month, and I just cant seem to figure it out. The situation is as follows:
public internet ———-> VPS <====WG Tunnel====> pfSense ———-> LAN/Server
I have a Wireguard tunnel set up to route all traffic between my pfSense LAN and a VPS. This means everything on that subnet sees it’s public IP as that of the VPS, and likewise public facing services are accessible only through the VPS and not the WAN address of my pfSense. This is done via iptables rules on the VPS to masquerade and forward the appropriate traffic and ports. To my slight surprise, this works perfectly!
The whole purpose of this is to have a public IP which isn’t residential, isn’t on the Spamhaus blocklist and generally has a good reputation among email providers and search engines. I also detest the idea of Virgin Media performing deep packet inspection (innocent or not) and blocking some DNS queries.
The only bone I have to pick is throughput. The thing is… it’s OK. But ONLY OK. It’s not bad enough to point to something being obviously wrong – like MTU or MSS. It’s working as it should, but I’m not getting the performance that I could. I’m only getting about 150mbps when I should be getting a theoretical maximum of 800mbps download.
As far as I can test, network is not the bottleneck. iperf3 in every direction using every protocol on every peer shows speeds of AT LEAST 500mbps (with the exception of my WAN upload, which is only 100mbps).
As far as I can see, CPU is also not the bottleneck. htop shows low usage on both ends.
So what gives? Am I doing something wrong?
Things I’ve checked:
Iperf Results
doesn’t even matter what’s what… it’s all the same.
WG Performance Tuning Guide
Followed this guide to a T – no change with any of the settings, unfortunately. Left flow control at BBR, because why not. Great write up, definitely worth a skim.
Checking Adapter Settings
I also checked all the adapter settings using ethtool. GRO was on, and everything else was mostly fixed. I did however enable rx-udp-gro-forwarding and rx-gro-list on recommendation of this tailscale guide:
Not exactly my use case but figured it couldn’t hurt. Unfortunately it didn’t help either.
Crappy ISP? Virgin Media Modems Perhaps?
I valiantly battled a previous issue whereby the Virgin Media SuperHub 3 would shit itself if presented with more than a few UDP packets in modem mode. This was a VERY irritating problem at the time, but we have since upgraded to a SuperHub 5, where (as far as I can tell) this problem does not exist. No one seems to have had problems, and my iperf UDP tests confirm this.
MTU/MSS
Also done this to death. Tried everything between 1380 and 1420 on both ends of the tunnel with not much difference. I have set it back to 1420, since bigger can’t hurt when the difference is negligible. No problem here.
Power Savings & C States on pfSense Side
Disabled and set to high performance in BIOS. I can route at over gigabit speeds, so no problems here.
So It’s a VPS Problem?
Note the VPS has 1vCPU and 1GB of RAM – not stunning, but should be enough, no?
Well it’s clearly not enough. I rented 2 other VPS – one on Hostworld and another on IONOS (where I currently have the WG VPS). Both with 2 cores and 2GB of memory. The Hostworld 2vCPU VPS yielded only 100mbps about to my home network, the IONOS 2vCPU 142mbps avg. Between the two IONOS servers I got about 170mbps. Slightly better on average, but not groundbreaking and I’ve seen that kind of speed to my pfSense under optimal circumstances.
Ok then… how do we solve it? An extra core didn’t seem to do anything! A problem for another day or a forum somewhere. I’m off to have some dinner.