4 player glitched lobby with friends on Steam

[JoyJoyHu]

As titled, my and 3 other friends are trying to play heists together in 1 lobby, but at least one of us would either be loading forever, or one person can only see two people loaded fully but one other player's name with no character, that person would also see the same thing on their screen, or one or more people would be standing still when a heist starts even though they're moving around doing things on the map. We've also had 3 players in a lobby and when the 4th player joins it says the 3rd person disconnected and for that 3rd player it'd say the 4th player disconnected. We've all tried to take turns hosting, checked game files multiple times and reinstalled Payday 2 and restarted Steam multiple times, none of us are using mods or using other platforms, we all made sure our steams are in sync, even reset all of our routers and it still doesn't work.

It seems like most times when we just have 3 players out of 4 is okay, but the last player that joins can't seem to do anything just standing in one spot spinning.

We've all tried joining other random lobbies and start heists and it's all fine, I myself have hosted and joined countless random 4 player lobbies and it's never had a problem, I constantly play with 1 of our friends out of 3 and with just us two it's never a problem.

We've really tried everything that seemed obvious, does anyone have any suggestions or any tip on what we can do?