To clarify some points about my hardware:
My node runs on WSL2 under a windows host system
It has access to 8 of my 12 cores and 64gbs of my 128gbs of ram.
Its running on a PCI-e gen 4 NVME
My network is around 200down/20up node uses at most 1-2mb/s up and down from observations.
My node is not currently public.

This specific instance of performance problem only surfaced during the many transactions from those specific accounts, once they stopped feeding transactions while on low RC everything went back to normal with 20-100ms block offsets and stable peers.
This is one of the reasons why I wonder if it's an internal issue in the software or not, although it could be something related to WSL2 that I don't know about.
Perhaps it could also be a packet bottleneck with my internet, but that seems unlikely as nothing else that I'm doing suffers from any for of network latency.

It could be that I've set my node to prefur 30 peers over the default 20, but that would indicate it is a software issue of some kind as I can't see my hardware/network being the issue.