Severe de-sync lag (especially with DSL). Long lag solution (last update)

[Buck Fell]

First off congratulations on creating a fantastic game, you deserve the 'overwhelmingly positive' reviews you're receiving. Unfortunately, as the devs know (based on sticky post on issues) there is a severe problem with lag in coop. I'd really suggest you make a unique sticky post for the issue where players can provide feedback, it's currently buried as an aside in the existing post.

Currently this issue has the highest votes on the bug list, feel free to vote it higher.
https://valheimbugs.featureupvote.com/suggestions/158504/multiplayer-sync-issues

I'm not sure about other parts of the world but In the US in suburban areas it's not uncommon for people to have DSL with a 7-10MBit down and 500Kbit up data rate. By it's nature DSL has low upload rates. The game seems to want to sit around 50KB/s up so some players are pegging their upload rate. You can see your network usage by pressing "F2" in game.

The game uses a distributed processing model. In a solo game the client is responsible for the physics and entity (e.g. NPC) control. With two clients each playing solo but using the same seed they share a world due to the procedural generation but have their own physics and entity control. The server architecture just ties them together and acts as a hub. Which ever client first uncovers a portion of the world they are responsible for it till the player moves and that area becomes idle or is handed off to a player remaining in that area. In addition It appears there's a queue structure for updates so if one player has a slow link all players are impacted. Further it appears this is made worse by proximity and total items or complex construction.

So if there are any work arounds that players can take while you're investigating or things you're planning to do it would be good to get this into the sticky. Things like:

• Run speed test on client to make sure you're getting proper bandwidth, DSL will resync at lower bit rates when there's noise on the line, typically have to power cycle router or get ISP to reset DSLAM switch to return to proper speed
  
• have player with worst network be further away from others (no clue if that helps, it's an example),
  
• Test network from client to server with something that can track packet loss. I use pingplotter (pingplotter.com), there's a free trial but I pay for mine.
  
• limit clutter on the servers (no clue if that helps, it's an example)
  
• shift to dedicated rather than player hosted server (this can help but if one player has slow (less than 60KB/s upload bandwidth it won't)
  
• Monitor network traffic on clients and kill any apps that can generate upload traffic (dropbox, chrome, onedrive, google photos etc).
  
• Use a wired connection if at all possible (really just do this, DSL routers suck at wireless)

Possible game change suggestions

• Add trash heap object where players can delete items from the world
  
• Add more aggressive abandoned item garbage collection in server
  
• Add option to cut client update rate in half. It would add a bit of jerkiness but for folks using DSL that should be enough to drop below the inherent transmit rate of DSL
  
• Change server queuing system so it just moves on if a player hasn't sent updates, no rill reason for the other players to lag in a non-competitive coop game.

Because the client/server traffic is so asymmetric it may be beneficial to have the person with the slowest link host (assuming their computer is up to it). The traffic from the server to the client is much less than client to server. This may work for 2 player coop but not for 3 or 4 as the outbound traffic for the host scales with the number of clients.

On the off hand chance a dev sees this. I will happily test/report on any development branch you'd like.

UPDATE

Richard (dev) posted a reply here: https://steamcommunity.com/app/892970/discussions/0/4939856028569039376/?ctp=4#c3106891514117216510
They're working on it. As a work around you could play coop if the person with the lossy network always plays a trailing support role. I was personally always in the lead with my group so my DSL was overly taxed, in a support role where other players are uncovering new territory the issue should be mitigated. That's at least a work around till a broader solution is in place.

UPDATE

Depending on your DSL carrier you may well be able to switch from residential to business for the same money and better bandwidth. For instance I was able to convert a CenturyLink 15mbit down/.5mbit up residential line to a 20mbit down/1.5 mbit up business line for the same monthly price. In addition the residential does occasionally impose data caps that the business line does not suffer from.

UPDATE

There appear to possibly be 3 major causes for lag/desync, inherent network limits from carriers like DSL, packet loss from bad networks and new recently discovered server side limit.

Note that different clients and servers are reporting different limits so possibly the installer tests the network limit and sets that value accordingly. If so then editing that value will not do anything and in fact really shouldn't be edited.

Thanks to Uzi for heads up, according to this reddit thread with fix: https://www.reddit.com/r/valheim/comments/lkea78/valheim_dedicated_server_lag/
there appears to be a 64KB/s cap on dedicated server traffic. The reddit post goes into some detail how to unpack the server DLL and force change that cap to something larger. If you try that please post comments in the reddit thread, not in this thread or in steam in general.

UPDATE

There is no guarentee the below recommendation will help, it may make things worse, proceed at your own risk.

Note that different clients and servers are reporting different limits so possibly the installer tests the network limit and sets that value accordingly. It appears for bandwidth limited clients, e.g. it's setting it at the tested limit which is not a great choice. It should be *below* the carrier bandwidth limit or it just triggers congestion.

One of the behaviors I see is long periods of time with 0KB/s send and receive. I suspect this is aggravated TCP congestion control. By setting the max data per second to higher than the network can handle it simply results in dropped frames and 3 second ACK timeouts. When I set the client limit to 50KB/s, lower than the DSL, things are quite a bit smoother with no more of the long 0KB/s windows. In retrospect this makes sense. So I'd suggest measuring your max upload rate and then set the m_dataPerSec to 75 or 80% of that rate.

Follow the directions in the reddit post above but on your PC. Make a back up of the DLL before proceeding. Start dnSpy, click file-open and select the valheim_assembly.dll libary in Valheim\valheim_Data\Managed folder. Search for m_dataPerSec, double click the returned string and then edit the value. It appears to default to 60KB/s (61440/1024). This is roughly the limit of DSL so if you're using DSL it won't help. If you're using a higher speed connection it may help. No guarentees. [/strike]

Here's a screenshot of the ediit: https://i.imgur.com/QBFn9Qh.png

If you screw it up you can simply refresh your local game files through Steam.

[Valco]

Me and my friends at same city, our server on fiber 100/100 Mbps, the crazy lag and desync start 3 days ago. if u tried F5 and ping, it goes crazy around 90k ms to 300k ms. relogin around 10-20 times till ping stay around 80-110ms (on same city lol). Server on valheim not purely on players network? still need go through valheim server? Sorry for bad english.

[Buck Fell]

Originally posted by Valco:

Me and my friends at same city, our server on fiber 100/100 Mbps, the crazy lag and desync start 3 days ago. if u tried F5 and ping, it goes crazy around 90k ms to 300k ms. relogin around 10-20 times till ping stay around 80-110ms (on same city lol). Server on valheim not purely on players network? still need go through valheim server? Sorry for bad english.

Nothing changed in the game 3 (or 4) days ago so sounds like a network issue on your end.

Are you playing a hosted or dedicated server. If hosted have the host test his network connection to some external site, pingplotter is good for locating loss.

If it's a dedicated server there could be something between your network in the city (probably all the same provider) and the game hosting facility (likely a different provider). Again pingplotter is a good package for testing between you and the server.

The games' queuing system is very sensitive to dropped packets so if the ISP or any intervening hop is dropping packets it will be very noticeable in game.

[Valco]

It's hosted on my friend with the highest speed between us, we play other games too, like swbattlefront2, bfv, all good no body lagging or other issue. We tried changing server between us and still desync after login and very high ping. 2 of us with same isp and same city too. It happen this last 3 days, not a clue what happen lol. Next time maybe we will try to play on same network, wonder what will happen.

[Buck Fell]

Because of the p2p nature if you're all in the game it only takes one of you to have a bad network connection. Hosted vs dedicated won't really matter much, they react the same way to a bad client connection. You could try having one person start the game, then one person join. If they get lag have that person drop and another join. If no lag then you can add players till you get lag, that last person is having an issue.

Other games are likely not as sensitive to packet loss.

[Nausica]

I think in addition to playing with the player(s) with bad connection, a lot of objects (drops on the floor, buildings, fires, etc) and everything in close proximity worsen the lag. We had a solid 4-5 people last night (1 was building at the main base) while the rest of us were very far away exploring the map and there was no lag at all. Hopefully it stays like this cuz the Plains are dangerous and it would be bad to lag there. The lag for me/us is always noticeable and at its worse when everyone is at the community center (where we have the huge building, beds, food, everything, etc).

I used to have a HUGE azz tree farm within close proximity to my base but I chopped it all day including the stumps thinking that all those tall trees swaying back and forth would add to the lag. Seems to have helped a little bit?

Lately, it's been more manageable. A few core friends had built another base much further away while the community center (my home) is pretty much where I reside. So we only have to deal with the lag when we meet up and are crafting.

[Buck Fell]

Updated first post with a few more bits of info.

[grimoire]

They really need to go to a true dedicated server model. I'm not going to refund the game because it is kind of neat, but the p2p netcode is a deal breaker for me. There's no way that's going to work out performance-wise and it's also a security risk.

[Pythos]

Just thought I'd add my own little bump to the thread.

Others have put it perfectly. Been trying to play the game with friends, some with fibre, some with dsl, and the problems we experience are as the thread states.

I certainly hope the devs can somehow implement a fix for this.

Many thanks!

[Buck Fell]

The patch released this morning did not change the sync behavior. Peak client transmit rate continues to be in excess of 50KB/s and desync still occurs. At this point it would be nice to get some acknowledgement of the issue beyond the brief aside in the bugs & crashes sticky that some MSI based laptops might have an issue with MSI based Killer Network software but that is not the underlying issue.

If Odin's Wholf sees this, could we get a clearer statement regarding the issue. The top 3 rated issues here https://valheimbugs.featureupvote.com/ all relate to this problem and account for roughly the same number of votes as all the other issues combined.

As it stands now coop mode is unplayable for a significant fraction of the player base.

[richard]

Hello. A common misconception is that the game uses some kind of P2P network. It does not, all data is sent between client<->server and never between clients. However game logic is somewhat distributed and run on the game clients rather then on the server. If player-A is alone in an area all creatures and objects will be controlled by that player and synced to the server. If player-B walks over to player-A, player-A will still control the game logic of most objects in that area. If however player-A leaves the area, player-B will take control of the objects and so on. This system allows us to do a lot of heavy physics on the client side and without overloading the server and so on. It is however pretty sensitive to one player having a bad connection. If player-A is controlling the monsters in the area has high ping to the server then player-B will also notice this, even though player-B has a perfectly fine connection to the server. I have some ideas of how to improve the system, but its pretty complicated and not something that can be fixed easily.

[Buck Fell]

Thanks Richard. I understand the nuance but it was easier to say P2P than explain the nuance. I fully appreciate that it's complicated and appreciate the response.

I think there are two classes of users,

• those with DSL who are limited by the transmit rate. It's very close to DSL limits
  
• those with lossy networks.

The former can really only be addressed by limiting the bandwidth. short of some compression scheme the one obvious approach would be to halve the update rate. Makes for some choppy behavior but it would work. There are some other ideas but get much more complicated, e.g. balance responsibility for objects by network capacity but that risks clobbering machines with great internet and poor processing.

The latter can be addressed in small part by posting a sticky here with instructions for how to diagnose and correct most forms of loss. I've seen that done in other games and would be happy to help contribute. It has the added advantage of helping users QoL beyond the game. After that there's not a lot one can do for lossy traffic with this server model.

[Boo]

Originally posted by richard:

Hello. A common misconception is that the game uses some kind of P2P network. it does not,

thats very unfortunate. I accidentally used up more than 2 of my hours trying to get the game working with my friends, so I guess it'll sit in my library for the next year or two until its playable.

[kenji]

Bumping, this issue is currently harming the overall otherwise fantastic gaming experience quite a bit.

[Gyson]

Originally posted by richard:

Hello. A common misconception is that the game uses some kind of P2P network. It does not, all data is sent between client<->server and never between clients. However game logic is somewhat distributed and run on the game clients rather then on the server. If player-A is alone in an area all creatures and objects will be controlled by that player and synced to the server. If player-B walks over to player-A, player-A will still control the game logic of most objects in that area. If however player-A leaves the area, player-B will take control of the objects and so on. This system allows us to do a lot of heavy physics on the client side and without overloading the server and so on. It is however pretty sensitive to one player having a bad connection. If player-A is controlling the monsters in the area has high ping to the server then player-B will also notice this, even though player-B has a perfectly fine connection to the server. I have some ideas of how to improve the system, but its pretty complicated and not something that can be fixed easily.

I'm trying to understand how the setup you described is contributing to the scenarios we're observing in-game.

I've only been playing Valheim with one other player. She hosts (although we have had the same lack of luck when I host) as she has a much faster internet package. She starts the session, and I join her game once the option becomes available. I log into our base, right next to her. Our base is very small, a small hut with a single room and a small pen outside, with walls around our plot of land for protection. It takes less than three seconds to move from the door of the hut to the door of the gate at the protective walls, if that gives you an idea of the tiny scale.

Moments after first appearing in the game my client's upload rate pegs ~50KB/s. I'm on DSL and that is the maximum transfer rate for my upload speed. I can't even begin to imagine what I could possibly (and for long minutes) be uploading to anywhere at that point, especially given your explanation as I am the second one arriving on the scene. Why am I uploading megabytes of data when all I'm doing is standing in a room relatively still? We are almost immediately out of sync with each other, each of us seeing different events (movements, opening or closing the door, etc.) occurring at different times.

Maybe after a few minutes of standing still the upload transfer rate has dropped to ~4KB/s. Everything starts syncing again. We walk out of the hut together and walk over to the gate at the walls surrounding our base. Suddenly I can't get the gate to respond; we're out of sync again and my client is uploading ~50 KB/s of data once more. To my friend's client, my movements and actions are taking place 20+ seconds later.

The thing is, we didn't have any sync issues until we started manipulating the world (cutting down trees, constructing buildings). When we first began our game and it was just us running around together, fighting and exploring (and gathering rocks and branches off the ground) everything seemed perfectly fine - for hours. It all began going horribly wrong once we started changing the world, and has been like that ever since.

[Buck Fell]

@Gyson agreed there's still a bit of a disconnect between the high level explanation from Richard and the low level reality players are experiencing. A better understanding would help players come up with strategies to work around the issue. For instance I considered just working in a following/support role so that my client was responsible for less of the world but that doesn't seem to quite work.

Depending on your carrier you may have options. I just talked to CenturyLink and it turns out that I can get a business line with 20mbit down/ 1.5mbit up for exactly the same price I'm paying for my 15mbit down/ .5 mbit up DSL residential line. Switching just required 10 minutes of online chat and they're hooking it up next Wednesday. In my area that's the best we can get, broadband access is still a big issue for big chunks of the United States.