[BUG] Massive Co-op Desyncs in 2.16

[BlueRaja]

During the later stages, sometimes everything will be popped on my screen, but on my partner's screen there are still a ton of balloons (or vice-versa - it doesn't matter who is host).

In one game, every balloon was popped on my screen, but then we lost because on his screen a bunch of balloons got through!

We just played for the first time yesterday, so I don't know if this happened prior to 2.16.

------------------------------------------------------------------------------------------

[Edit] As a developer with lots of networking experience, I may be able to help.

The main idea is to create a Command Queue. When someone executes a command (playing a tower, hitting play/fast-foward, etc), don't execute it immediately, but instead queue it to be done at some point in the future (something like ~250ms). You wait until both players know which commands will be executed and when, then execute them on both machines simultaneously.

This is how all modern RTS's like Starcraft 1+2 work. For more information see here[www.gamasutra.com] or here[gamedev.stackexchange.com] or here[gaming.stackexchange.com].

If you are already doing this, you need to snuff out sources of non-determinism:

• Make sure you're using fixed-step engine calculations. The outcome must have nothing to do with the user's framerate.
  
• Make sure both clients use the same PRNG seed, and are guaranteed to always call it for the same things in the same order.
  
• Make sure if one client's fixed-step engine frames slow down, the other client has a way to know that and wait for him to catch up.
  
• Make sure you're using a transport-layer protocol with guaranteed delivery. Either TCP, or UDP with confirmation/resend logic.
  
• Make sure you're not using floating-point numbers in the engine code. Most developers are shocked to learn that they are non-determinisitic[gafferongames.com]. (See also[stackoverflow.com])

[Gakm4]

Desyncing has been happening in btd5 steam ever since i started playing it. A couple things that seem to make it worse are your location in the world compared to who you are playing it and the internet speed of those playing. Also if the btd5 servers are having issues at the time which is fairly rare.
Desyncing can cause a bloon to get popped on one screen but get missed on another causing the targeting on all towers to shift slightly possibly causeing you to lose lives ocasionally. I have lost a few games due to this. I have found that it happens more frequently on rounds with alot of regrows so i try to build towers that are good against them so they dont replicate a ton. Wizards, ninjas, and bloonchippers are known to make regrow waves alot worse.

[BlueRaja]

btd5 servers? I'm pretty sure the game is peer-to-peer, so there wouldn't be any servers, except the ones for daily maps, high-scores and other account information.

[76561198164174249]

Originally posted by BlueRaja:

btd5 servers? I'm pretty sure the game is peer-to-peer, so there wouldn't be any servers, except the ones for daily maps, high-scores and other account information.

I'm not so sure about that...Even as someone who has played Flash BTD5 Co-op since before it was released.

Now I am curious to hear the official answer on this from Sam at NK:

- Is Steam BTD5 Co-op a true peer-to-peer game mode?

- Or do both Co-op players still need to connect to a Steam BTD5 Co-op server somewhere.

This would be great to know for sure...

Sam?

[samninjakiwi]

Both co-op players still need to connect to our servers.

When you say "later stages", can you let me know what round this tends to start happening, please? Also, does it happen every game you play? :)

[BlueRaja]

It's not a specific stage, it seems to just be a function of how much stuff is on the screen (Edit: That turns out to be incorrect, it's just random. See below).
And yes, in the few times we've played so far, it started happening eventually in all of them.

If my assumptions in the first post are correct, this should be easy to reproduce. Set up a latency simulating proxy[stackoverflow.com] like DummyNet (which can also be installed in a VM, if you don't have a machine with two NICs available) in your testing environment, then give high values to packet loss (5%) and jitter (400ms). Then you can test two computers over a LAN and see the results side-by-side.

[76561198164174249]

Originally posted by BlueRaja:

It's not a specific stage, it seems to just be a function of how much stuff is on the screen.
And yes, of the times we've played so far, it starts happening eventually in all of them.

If my assumptions in the first post are correct, this should be easy to reproduce. Set up a {LINK REMOVED}latency simulating proxy like DummyNet (which can also be installed in a VM, if you don't have a machine with two NICs available) in your testing environment, then give high values to packet loss (5%) and jitter (400ms). Then you can test two computers over a LAN and see the results side-by-side.

Indeed, what you suggest would be a good experiment for those more technically-able folks out there.

:)

You do have a point about the correlation between lag and the amount of stuff on the screen. This happens constantly in the Flash version of BTD5 / Flash Co-op and Flash Battles.

Not a coder, but obviously in the case of these Flash games, Flash itself is likely getting overwhelmed - too many calculations, etc, etc, etc resulting in very unpleasant stuff.

As someone who has been with [and loving!] Flash BTD5 / Co-op and Flash Battles for 3 years, I can suggest some stuff from experience.

Let's take Flash Battles out of the equation though since the problems there are a bit different.

What I am suggesting here is a workaround in how you play/approach BTD5 and Co-op.

In both Co-op and solo BTD5. my own goal is always to try and get to Temple as fast as I can. This requires strong, early farming but there is a side benefit - using as few towers as possible to get through the game which also mitigates the lag issue to a noticeable degree.

In other words, just being as efficient as possible.

True, this may be a boring approach, but it certainly is effective in mitigating a great deal of in-game lag, especially in Co-op.

Most expert players in Co-op [speaking specifcally about Flash Co-op, since I have not done any Co-op under Steam BTD5 due to its earlier instability] will play maps in the following manner:

- 1 person basically does nothing but Farm and the other person does nothing but receive money and setup defenses...with the same goal though of getting to a single strong Temple as soon as possible.

It's a totally common sense approach to playing Co-op.

This approach works on basically every single Co-op map, though some both players need to do a little defense, but the idea is always the same: 1 guy farms, the other defends with towers geared toward creating a strong Temple [especially if we are playing Co-op for Hard difficulty].

Also, the types of towers you use under Co-op can also make a huge difference, especially under Flash version.

Always try and stay away from "lag inducing" towers - the most guilty parties here are usually towers that cause massive explosions/animations [Mortars and Spikes especially] and even stuff like having too many Blade Maelstroms, Glaive Lords, Tornado Wizards, Carrier Ships.

Under Flash Co-op when I would match up with noobies or those who maybe are not so experienced to go for strong Temples, as long as I can Farm and provide income for us both, I will also try and use towers to help out as needed that are NOT so lag-inducing.

My fave is usually Sniper.

With newer players, if I see they are having problems defending or using questionable towers, I will begin building Snipers to help out. Lots of benefits there - Snipers generally do not contribute to in-game lag very much, are reasonable cheap and with a good selection of targeting priorities can handle pretty much everything up to round 86 when used properly.

Either I will throw down some Snipers as needed or a Super someplace effective - of course under Village for camo rounds.

Anyway, sorry so long, but just throwing some experience at you with the game.

The very best players in BTD5, Co-op or solo also are those who are the most effective at mitigating game lag.

If that means using a more limited selection of towers, well, that's the tradeoff.

cheers man...just some food for thought...
-les

EDIT:

Sorry for the word vomit, but all of the above really is just detailing how people have had to adjust the way they play Flash games like BTD5 / Co-op - pretty much out of necessity due to the weaknesses of the platform.

Having not played much Steam BTD5 Co-op, I am surprised to hear of this kind of lag/de-syncing that happening, especially given that the code base for Steam BTD5 is NOT Flash.

From solo Steam BTD5 play, I can say without question game lag, even at the highest rounds, are a fraction of what they are under Flash. In fact, the highest legitimate rounds achievable under Flash are in the ~400-500 category MAXIMUM - and those kinds of uber-high round games take weeks and weeks to play since each round begins taking upwards of 30 minutes at the later stages.

People have already done documented 1100 round solo games in Steam BTD5. I personally have a game right now at round 875. And let me tell you I have TONS and TONS of towers placed - every pixel is filled as much as possible.

Yet Steam BTD5 lag again a tiny fraction of higher round Flash lag and overall resource usage on my system is pegged right around 200MB RAM and very little CPU time.

All of which is why I am surprised to hear of the issues you describe under Steam BTD5 Co-op, but obviously there may be lots of latency issues happening there.

But your suggestions may indeed help NK!

[BlueRaja]

We just had it happen on round 4 with only one dart monkey each. On my screen (the host) one balloon got through, and we lost on Impoppable difficulty. On his screen, we killed every balloon. We were playing at fast-speed, and I think we didn't build or upgrade anything that round.

So, it's definitely an issue of the game not being completely deterministic.

[76561198164174249]

Originally posted by BlueRaja:

We just had it happen on round 4 with only one dart monkey each. On my screen (the host) one balloon got through, and we lost on Impoppable difficulty. On his screen, we killed every balloon. We were playing at fast-speed, and I think we didn't build or upgrade anything that round.

So, it's definitely an issue of the game not being completely deterministic.

You're obviously onto something it would seem.

Again, 2 years of playing Flash BTD5 Co-op, I don't believe I have ever seen that kind of lag that early in a game with so few towers. At least not as often as you appear to be now in Steam BTD5 Co-op.

Here's an idea...

Flash BTD5 is 100% free to play. Just for a point of comparison, consider giving Flash BTD5 Co-op a whirl and see if you can feel any differences.

Just don't have any other way to try and help you out here, sorry.

Perhaps NK devs will take your suggestions to heart.

[Broker - Kappa]

+1 I got this bug too, finally I leaved it.

[samninjakiwi]

Getting this checked out just now. Thanks for the info, BlueRaja. Will update once I know more :)

[samninjakiwi]

Originally posted by BlueRaja:

As a developer with lots of networking experience, I may be able to help.

The main idea is to create a Command Queue. When someone executes a command (playing a tower, hitting play/fast-foward, etc), don't execute it immediately, but instead queue it to be done at some point in the future (something like ~250ms). You wait until both players know which commands will be executed and when, then execute them on both machines simultaneously.

This is how all modern RTS's like Starcraft 1+2 work. For more information see here[www.gamasutra.com] or here[gamedev.stackexchange.com] or here[gaming.stackexchange.com].

If you are already doing this, you need to snuff out sources of non-determinism:

• Make sure you're using fixed-step engine calculations. The outcome must have nothing to do with the user's framerate.
  
• Make sure both clients use the same PRNG seed, and are guaranteed to always call it for the same things in the same order.
  
• Make sure if one client's fixed-step engine frames slow down, the other client has a way to know that and wait for him to catch up.
  
• Make sure you're using a transport-layer protocol with guaranteed delivery. Either TCP, or UDP with confirmation/resend logic.
  
• Make sure you're not using floating-point numbers in the engine code. Most developers are shocked to learn that they are non-determinisitic[gafferongames.com]. (See also[stackoverflow.com])

So, our infra guy has confirmed that we do indeed do all of these, but he reckons there's perhaps a specific bug in-game that causes the desync. Do you notice it after placing any towers in particular?

[BlueRaja]

@samninjakiwi, it happens often in literally every game. It typically manifests more obviously in later stages (I'm assuming due to the butterfly effect), but I mentioned above a case where we noticed it on stage 4 with only two dart-monkeys out, because it cost us the game.

If you play just a normal game with someone and talk over skype about what you each are seeing, you're bound to run into it.

If you set up a latency proxy (see above) and play with two machines side-by-side, the difference should be obvious.

I will try screen-capturing a game or two this weekend with my bloons partner, and create a side-by-side comparison.

[BlueRaja]

I added a video here: https://www.youtube.com/watch?v=a4zLICpSokk I took out most of the middle, but can provide it if requested.

Some notes:

• Not a single round was synced correctly between the clients. You'll notice from the very first round, the clients run the simulation differently. Not only does the right-player run at a higher speed, but the balloons popped are different. This means Bloons is not at all deterministic or using the lockstep protocol I suggested.
  
• We noticed another bug while playing: the 'Bigger Beacons" upgrade does not sync across clients. Even though we both have the upgrade, my towers are only affected by my partner's beacon in a tiny range.
  
• At the end of the recorded game, my game said we won, even though we lost on both our screens.
  
• A few games later, the game crashed (I did not get this recorded, it happened on the second-to-last round on Impoppable difficulty). Is there a crashlog I can provide?

[76561198164174249]

^ Very interesting video. Hopefully Sam and NK will look more deeply at this desync'ing issue.

Just a straight BTD5 question for you though:

Any reason you chose not to go for a "strong" Temple?

Not saying that would in any way alleviate this issue. Just that a single, properly constructed, decently placed Temple will win you every single map you want to play, under any difficulty, solo or Co-op play.

Perhaps you could try again but do more farming early and one of you throws down a "strong" Temple.

Just in case you're not aware, there are indeed major differences in Temple construction.

Through exhaustive research over the last few years, it has been determined [and generally agreed upon in the BTD5 community] that the following recipe will yield the "strongest, perfect Temple" possible in the game:

- 1 SunGod
- 6-7 max Bombs OR 3 max Mortars
- 3 max Ice
- 3 max Glue
- 3 max Tornado Wizard
- 1 2/3 RoboPlasma
- 1 x/4 Village

Actually, many folks will also just use an abbreviated version of the above - simply use 1 each of max Ice, Glue and Wizard. The abbreviated Temple will also get you through 85 rounds with no problem at all.

Sorry for the tangent...just passing info.

Thanks again for the revealing video.

cheers,
-les

[BlueRaja]

lestastar, if you have a video of achieving that without all the starting money or double-money upgrades (which make the game trivially easy) on Co-op Impoppable difficulty on that level, I'd be interested in seeing it.