Lagging issues as host

[Hughjanus]

I play with my wife and 1 friend on rebirth mod and I'm getting horrible fps down in the 20's they are not affected but I am. I'm running the game on medium settings shadows off on a 4070 super and a 11700k. The game is only using 30 percent utilization of both the cpu and gpu and we have really stupidly fast internet. Any ideas what's causing this especially with such low utilization?