Dev Diary - Networking 1

[ttcrose]

Hello Worlds explorers/discoverers/creators!

I'm Ciaran, one of the younger programmers on the LEGO Worlds team here at TT. I've been working on the game for over 18 months now on a wide variety of features, but my most significant contributions have been the terrain editing tools and extensions to the terrain generation like caves. But for the past eight months all of my efforts have been devoted to one of our biggest and most requested features: online multiplayer. Making a game run on multiple machines over a network is a monumental task, requiring us to touch almost every element of the game in order to make it synchronise correctly and efficiently, and I'd like to share with you some of the interesting challenges that have arisen.


Getting Started

Our longer-term fans may remember that we actually released LEGO titles a few years ago that were online-enabled: LEGO Indiana Jones 2 and LEGO Harry Potter allowed you to go on bricktastic adventures with a friend via the internet. So there was already a basic framework in place in our code for networking. However it hadn't been used for five years, which is a very long time in the programming world. So like an old vacant building, it needed a substantial amount of renovation to get it up and running again. Some things worked fine, others needed patching over, and some needed ripping out and replacing. My first month on this task was just spent in a loop of connecting two instances of the game, watching them crash and tracking down the cause to fix it.

Getting the old stuff up and running was only the first step. Our traditional LEGO games feature up to two players following a scripted narrative through relatively compact levels and our networking infrastructure was designed to accommodate this. In contrast, LEGO Worlds provides a wide-open, procedurally-generated and fully-customisable world to explore and create with your friends. As such, we need to write a lot of new systems to support these ambitious features.


Data and Determinism

The most important new feature is of course the terrain itself. Taking this online is inherently a very difficult technical task because of the sheer density of the data. Many players have commented on how large their save data becomes as they play. We continue to work to find new ways of compressing this data, but ultimately there's only so small a representation we can make for the countless millions of bricks each landscape contains. So when it comes to synchronising this data, we really need to do something better than just transmitting it all over the internet to every player in the game, if at all possible.

The key is our deterministic world generation. Determinism means that when two machines do the same thing, they get the same result. As you probably know, our worlds are seeded and you can generate the same world as often as you like by inputting its original seed value. Using this fact, we can have client players generate the world themselves most of the time, only requiring the host to transmit regions that have been modified. This means that no terrain data has to be sent at all as you and your friends explore dragon-guarded peaks and skeleton-infested valleys.

Interestingly, when I implemented this, I discovered that our world generation wasn't perfectly deterministic after all! I found that there were in fact tiny differences between different generations of the same world, which prevented the games from syncing up correctly. So like an obsessive LEGO builder, I had to track down every one of these little imperfections and trace them back to their root cause in the generation algorithm. But now I promise that you really are getting the same world every time, or your money back! (NOTE FROM CHRIS; Ciaran will be paying for this himself!)


Building and Terraforming

So your party of adventurers have found a suitably epic vista and you're ready to don your hardhats and get building. One of the things that's awesome about LEGO Worlds is our large-scale terraforming tools that let you add and remove thousands of bricks at a time. We wanted that power to carry over into the multiplayer, but once again we needed a strategy to tackle the large amount of data. We did initially consider transmitting the bricks in a compressed data structure, leveraging the same code that packs chunks of terrain, but we quickly dismissed this because it would be too big a bandwidth strain and too slow to be fun. Instead, I engineered a new compact representation of terrain edits. Rather than describing each of the thousand bricks you added and removed, the information exchanged between players is simply "I flattened terrain centred on this location with this radius and shape". This description can be encoded in just a handful of bytes, and after some adaptation of the code around terrain editing, multiple machines can now apply the change in a deterministic way.

The real technical challenge comes when you introduce latency. You've undoubtably noticed that transmitting over the internet isn't instantaneous, and in multiplayer games it's normal to have several hundred ms ping - that is, hundreds of milliseconds of delay between you sending a message and receiving the reply. In terms of terraforming, this means that you could be happily modelling your landscape when a message comes through from your friend's machine saying that he added some bricks to that area in the meantime and the last dozen edits you computed were slightly wrong. Leading to you all screaming; Nooooo!

To combat this problem, I've designed a system called the Change Queue. Every edit you make to the terrain goes into the queue, where it gets sequence-numbered and recorded before being performed. Each player in the game keeps their own record of the order in which they did things, as well as the order in which the host player did them. When something goes wrong, we can undo our most recent changes - using our existing undo/redo system - and perform them again in the host's order, which we define to be the correct result.

But how do we know when something goes wrong? We can't compare the whole terrain with the host player to see if we got it right, because if we knew what the host player's terrain looked like, we wouldn't have had to compute our own in the first place! Instead we use a technique called data checksumming. After an edit, we take the resulting terrain data and add and multiply it together in a specific way to give us a single number as a result, which is called the "checksum" of that area of terrain. The host can easily send us this four-byte number, and by comparing that with our own checksum, we can tell whether our terrain is in sync or not.


And So Much More

Building isn't the only fun you can have in Worlds though. There are cowboys and minotaurs, dragons and landsharks, snowmen and bathtubs, buggies, boats and helicopters, and so many more awesome dudes and doodads that all have to make their way into a multiplayer networked scenario. There were a range of other technical hurdles which we had to overcome (which may be the subject of a future dev diary!), but we have arrived and now we can’t wait to share this long-sought-after feature with you!

[RedstoneParadox]

After having read through all of this, I was wondering what the biggest technical challenge is increasing the number of people that can play together on a server, aside from the fact that the engine wasn't designed to have more than 2 (now 4) people play together in a single instance.

[Angelenelcielo]

a few hours ago I played the beta version to play with a friend of mine but when i downloaded the version that isnt beta , i can just watch the inicial credits(PArt when apears the logos of wb and lego,does anybody know why does this happen

[Tt_Toby]

Originally posted by ezio1212audi:

a few hours ago I played the beta version to play with a friend of mine but when i downloaded the version that isnt beta , i can just watch the inicial credits(PArt when apears the logos of wb and lego,does anybody know why does this happen

If you switch back to the beta branch, it should work again. We cover the details of it in this post:
http://steamcommunity.com/app/332310/discussions/0/351659808477498569/

And great Dev Diary Ciaran! :)

[Tallens]

These dev diaries are so interesting, I'd want more only if they didn't take time away from making such an awesome game!

[Michel]

so interesting

[Demon-XII]

hurry because i love you lol

[Sizel]

Clever stuff & Good communication. Thanks.

[theguyB]

Thanks for explaining how you are solving problems, i could learn a lot :)
commendations to your PR ^^