Multiplayer Performance problems

[InfiniteVR]

Heya everyone! I recently decided to play mutiplayer with friends that are new to beatsaber.
We played on the newest version (1.21.0) and i was hosting the lobby (standard beatsaber servers) accept my custom saber s and my custom notes i did not use any other mods while playing. we also didnt play any custom songs or DLC songs.

When starting up the lobby everything was fine. jumping into a song however the game started to lag giving me around 20-15 FPS.

I checked my Taskmanager for usage of CPU, GPU and RAM. The usages were fine with my CPU hanging around 30%,
GPU around 40%,
and RAM at max of 50%.

But it was also off since i didnt know why else my game would be lagging.

I also noticed that it got worse with every song that we played and it stayed like this till i restarted the game.

Doe sanyone know the issue? is there a fix to this?

My hardware Info:
Ryzen 5 3600X
Nvidia RTX 2060 6GB
16GB RAM G4 2666Mhz

My VR gear:
HTC Vive
2 1.0 Basestations
Valve Index knuckle controller