Really High Latency, Only in Game

[Ultamatum]

Getting a weird issue here can't really figure out how to sort it.

Whenever I'm going through the steam menus I get an fairly alright 20-30ms latency

As soon as I open a game it goes to 100+ms. I would normally assume that that's my gpu having issues but it's a 2070 and I'm trying to run FFVI hardly the most demanding game in the world, any ideas on how to solve or am I SOL

Streaming from PopOS to the Chromecast with Google TV at varying resolutions from 720-4k same issue in all cases

[_I_]

are host and client connected to the same router?

make sure windows network is set to private not public
all vpn disabled
steam -> settings -> remote play -> allow direct connection = all/my/auto
check router wifi settings, disable wifi isolation

any of those will force remote play to use steams internet relays, and add more delays instead of direct connection on lan/wifi