It’s interesting to note that turning off the power saving features for the common Wifi dongle chipsets used on Raspbian seems to fix not only the pauses where before it would sleep during inactivity, but also it seems to fix/improve the laggy responsiveness even when typing commands over SSH to a Pi. Links to the settings in my previous post here.