Severe de-sync lag (especially with DSL). Long lag solution (last update)

[Buck Fell]

First off congratulations on creating a fantastic game, you deserve the 'overwhelmingly positive' reviews you're receiving. Unfortunately, as the devs know (based on sticky post on issues) there is a severe problem with lag in coop. I'd really suggest you make a unique sticky post for the issue where players can provide feedback, it's currently buried as an aside in the existing post.

Currently this issue has the highest votes on the bug list, feel free to vote it higher.
https://valheimbugs.featureupvote.com/suggestions/158504/multiplayer-sync-issues

I'm not sure about other parts of the world but In the US in suburban areas it's not uncommon for people to have DSL with a 7-10MBit down and 500Kbit up data rate. By it's nature DSL has low upload rates. The game seems to want to sit around 50KB/s up so some players are pegging their upload rate. You can see your network usage by pressing "F2" in game.

The game uses a distributed processing model. In a solo game the client is responsible for the physics and entity (e.g. NPC) control. With two clients each playing solo but using the same seed they share a world due to the procedural generation but have their own physics and entity control. The server architecture just ties them together and acts as a hub. Which ever client first uncovers a portion of the world they are responsible for it till the player moves and that area becomes idle or is handed off to a player remaining in that area. In addition It appears there's a queue structure for updates so if one player has a slow link all players are impacted. Further it appears this is made worse by proximity and total items or complex construction.

So if there are any work arounds that players can take while you're investigating or things you're planning to do it would be good to get this into the sticky. Things like:

• Run speed test on client to make sure you're getting proper bandwidth, DSL will resync at lower bit rates when there's noise on the line, typically have to power cycle router or get ISP to reset DSLAM switch to return to proper speed
  
• have player with worst network be further away from others (no clue if that helps, it's an example),
  
• Test network from client to server with something that can track packet loss. I use pingplotter (pingplotter.com), there's a free trial but I pay for mine.
  
• limit clutter on the servers (no clue if that helps, it's an example)
  
• shift to dedicated rather than player hosted server (this can help but if one player has slow (less than 60KB/s upload bandwidth it won't)
  
• Monitor network traffic on clients and kill any apps that can generate upload traffic (dropbox, chrome, onedrive, google photos etc).
  
• Use a wired connection if at all possible (really just do this, DSL routers suck at wireless)

Possible game change suggestions

• Add trash heap object where players can delete items from the world
  
• Add more aggressive abandoned item garbage collection in server
  
• Add option to cut client update rate in half. It would add a bit of jerkiness but for folks using DSL that should be enough to drop below the inherent transmit rate of DSL
  
• Change server queuing system so it just moves on if a player hasn't sent updates, no rill reason for the other players to lag in a non-competitive coop game.

Because the client/server traffic is so asymmetric it may be beneficial to have the person with the slowest link host (assuming their computer is up to it). The traffic from the server to the client is much less than client to server. This may work for 2 player coop but not for 3 or 4 as the outbound traffic for the host scales with the number of clients.

On the off hand chance a dev sees this. I will happily test/report on any development branch you'd like.

UPDATE

Richard (dev) posted a reply here: https://steamcommunity.com/app/892970/discussions/0/4939856028569039376/?ctp=4#c3106891514117216510
They're working on it. As a work around you could play coop if the person with the lossy network always plays a trailing support role. I was personally always in the lead with my group so my DSL was overly taxed, in a support role where other players are uncovering new territory the issue should be mitigated. That's at least a work around till a broader solution is in place.

UPDATE

Depending on your DSL carrier you may well be able to switch from residential to business for the same money and better bandwidth. For instance I was able to convert a CenturyLink 15mbit down/.5mbit up residential line to a 20mbit down/1.5 mbit up business line for the same monthly price. In addition the residential does occasionally impose data caps that the business line does not suffer from.

UPDATE

There appear to possibly be 3 major causes for lag/desync, inherent network limits from carriers like DSL, packet loss from bad networks and new recently discovered server side limit.

Note that different clients and servers are reporting different limits so possibly the installer tests the network limit and sets that value accordingly. If so then editing that value will not do anything and in fact really shouldn't be edited.

Thanks to Uzi for heads up, according to this reddit thread with fix: https://www.reddit.com/r/valheim/comments/lkea78/valheim_dedicated_server_lag/
there appears to be a 64KB/s cap on dedicated server traffic. The reddit post goes into some detail how to unpack the server DLL and force change that cap to something larger. If you try that please post comments in the reddit thread, not in this thread or in steam in general.

UPDATE

There is no guarentee the below recommendation will help, it may make things worse, proceed at your own risk.

Note that different clients and servers are reporting different limits so possibly the installer tests the network limit and sets that value accordingly. It appears for bandwidth limited clients, e.g. it's setting it at the tested limit which is not a great choice. It should be *below* the carrier bandwidth limit or it just triggers congestion.

One of the behaviors I see is long periods of time with 0KB/s send and receive. I suspect this is aggravated TCP congestion control. By setting the max data per second to higher than the network can handle it simply results in dropped frames and 3 second ACK timeouts. When I set the client limit to 50KB/s, lower than the DSL, things are quite a bit smoother with no more of the long 0KB/s windows. In retrospect this makes sense. So I'd suggest measuring your max upload rate and then set the m_dataPerSec to 75 or 80% of that rate.

Follow the directions in the reddit post above but on your PC. Make a back up of the DLL before proceeding. Start dnSpy, click file-open and select the valheim_assembly.dll libary in Valheim\valheim_Data\Managed folder. Search for m_dataPerSec, double click the returned string and then edit the value. It appears to default to 60KB/s (61440/1024). This is roughly the limit of DSL so if you're using DSL it won't help. If you're using a higher speed connection it may help. No guarentees. [/strike]

Here's a screenshot of the ediit: https://i.imgur.com/QBFn9Qh.png

If you screw it up you can simply refresh your local game files through Steam.

[Buck Fell]

Sole bump and then slide into oblivion

[Buck Fell]

Quick update. There is some very strange network behavior with the game. What's not so strange is the total traffic up from the client to the server appears to be affected by the amount of construction nearby.

In a decently large base (pallisade, tower, 2 story 6x4 house) my client generates 30 to 60KB/s traffic up and averages around 50KB/s.

Far away from any bases my client generates 10 to 40KB/s with a steady state around 30KB/s.

For people with DSL that upload rate can be greater than they can handle. Once that happens it appears a the second problem kicks in. The server appears to have a action queuing system and if one player is failing to send updates then everything just halts for the other players.

This is particularly obvious when a player is hosting and still lags due to some other player.

[Boo]

bump. this is a very good post that people should see

[Buck Fell]

Thanks. What I'd really like to see is the devs to make a clearer sticky post acknowledging the issue with some work arounds based on facts from their understanding of the code vs my surmises.

For many users I bet they see 0 issues, I'm suspicious it only impacts DSL users who have limited upload bandwidth. For much of the world DSL is going away but for many people it's still the only real option.

[Buck Fell]

I've done some more tests. It looks a lot like the client is sending world data to the server, not just player input. Even without other players around a person with a limited upload capacity (e.g. DSL) can experience lag on a dedicated server. The area you're in will be in sync, you can open chests etc but as you move away you'll see dwarves, deer or items you can't interact with. After some time they'll respond.

This implies the client is sending world info to the server, once the server vettes the clients assertions the server sends it back and you can interact with them. If the bandwidth limit is severe enough as you walk on you'll eventually come to an empty world, no trees, no rocks, no animals no nothing just rolling hills and grass. As you keep moving your client can't send info to the server fast enough to keep up and just gets further and further behind.

[Buck Fell]

Here's an example set of screenshots. Ran through world till lag, World is now empty, no trees, rocks, NPCs, structures etc.
https://steamcommunity.com/sharedfiles/filedetails/?id=2393385875

20 seconds later some of the trees show up (haven't moved, facing same direction).
https://steamcommunity.com/sharedfiles/filedetails/?id=2393386077

Another 20 seconds or so and the full world blips in:
https://steamcommunity.com/sharedfiles/filedetails/?id=2393386250

You can see telemetry information by pressing F2. Here's the data rate shortly after those pics. Note the client transmit rate is 59KB/s, 0KB/s from the server because it's still waiting for the bandwidth limited transmit from the client. Any DSL circuit will be hard pressed to sustain 60KB/s, and it's way over the bandwidth of a normal 10 or 15Mbit DSL circuit.

https://steamcommunity.com/sharedfiles/filedetails/?id=2393386429

[BLK]

Nailed anything I would've wanted to say. Hopefully the devs get this resolved. This is an amazing game and easily already makes my top 10 in the last decade. Looking forward to the future of this game as well as future adventures with my fellow viking brethren.

[Nosterex]

"This is a well known issue, the server utilises a netcode similar to p2p which is awful for dedicated servers. Meaning that it can lead to lag when playing with more people and standing next to each other.

This is nothing that can be fixed with more CPU or Memory, this is simply because the one with the slowest Internet connection dictates the speed of all others as well and this can lead to lag / de-sync and rubberbanding.

Once you guys run in different directions of the map and do not stand next to each other anymore, you will notice the lag will get less and you can start playing again smoothly."

Hopefully Iron Gate, who have done an overall tremendous job, will rectify this and rid us of P2P.

Good gaming.

[Jayslow]

Im so glad someone has succinctly put down exactly what I have been suffering with since day one, I feel ♥♥♥♥ because I cant play with my friends and if I try I screw the server up, Im not playing this on my own so I guess I have to wait. Annoying when I can play CS, Ark or anything else no problem, but early access I guess.

[Boo]

Wow! Thanks for breaking it down even further than the initial posts. I really hope the dev(s) sees this and puts it in a sticky or resolve it quickly. Theres no reason why the client should be sending that much data.

[Kalizaar]

One of my friends is having this issue as well. Really hope Iron Gate fixes this soon!

[EddyNeave]

i'm getting same issue just teleporting and massive delay in player actions for sometimes minutes while in close proximity. Server works fine but the second you go near another player all hell breaks loose. Honestly don't think it's anything we as users can fix

[Broscers]

Bump

[Cooltrain]

It seemed like you were slowly testing towards realising the game is P2P (As someone said above). All the "Dedicated" server does is host the world/save, and keep the connection open, that's it.

All the players form a connection web where each client will send its data to every client. All the clients together maintain the world you're all playing on. If someone is having a problem then everyone has a problem. This is one of the reasons the multiplayer is capped to 10 players.

It would be great if the game moved to a more traditional server model for server/client but that's going to be a lot of work and it won't happen anytime soon if it did.

[Buck Fell]

Originally posted by Cooltrain:

It seemed like you were slowly testing towards realising the game is P2P (As someone said above). All the "Dedicated" server does is host the world/save, and keep the connection open, that's it.

Actually it was pretty apparent from how they set up hosted connections and how the dedicated server set up worked that the latter was essentially a p2p hub. Mostly I was proving it not deducing it.

The dedicated server doesn't even do any sanity check from the client input. If you're lagging and start stuffing coal in a smelter you can easily get 25 or more in rather than the 20 limit before the game notices, likewise you can get 15 or more ore in rather than the 10 limit.

I agree moving to a traditional server model isn't going to happen anytime soon but if they could cut the client upload traffic in half they'd pretty much get rid of most of the lag issues since it's so close to the DSL limit.

I imagine if there was a poll among people who experience lag the vast majority would have a DSL connection, the remainder probably have very bad internal wireless.