The Future of this engine?

[capitalthree]

Deep Rock Galactic's terrain engine is amazing and glorious. I've long wanted to see a game implement modifiable terrain using deformable mesh instead of voxels, and I've long assumed that the first implementations we'd see would be quite clunky and laden with issues. But nope! At least, if DRG's approach has issues, it's hiding them well, because the terrain engine seems incredibly smooth and performant.

So the big dream for me would be a dedicated-server, persistent-world game (a la minecraft, ark, conan, starbound, etc) using this terrain engine, where I can terraform a *persistent* world in a smoother manner than I can in voxel games. I know this is probably way out of scope for DRG itself, which is already still in early access, and that's why this isn't posted in Suggestions. DRG is already a fantastic game and does not need to feel bad about what it is. But I'd love it if the devs are willing to get starry-eyed with me and talk about possible futures, or at least other fans.

Is it possible that:
- The DRG team will eventually work on another game using this engine tech that has persistent worlds?
- DRG's engine could be licensed to other game developers?
- DRG itself might get mod support once it's released, that would allow people to make custom gamemodes that can save caves?
Any of these things would be amazing. And of course, I understand that we're nowhere near the point yet where they're on the table, but I'll become an even bigger fan of Ghost Ship Games if we're on the same page on this. I know this glorious terrain engine doesn't owe me any more than the game I already bought, and the game I bought is great. But ooh, does it tease me.

Are there any known showstoppers for trying such a thing? Does this mesh deformation technique work well on short-lived matches but cause performance degradation if you keep pickaxing away day after day? That would be pretty understandable, but hopefully the devs have more insight.

[LyraBnnuy]

I fully support this, or a reply from the developer explaining any limitations. Alternatively or in addition to, if the engine could be licensed to others for making such games.

[Droolguy]

Database bloat would make this impossible for the same reason that highly modified minecraft servers lag to hell and back, that information has to be stored in a table or in RAM, eventually you run out of RAM or your modification database becomes so bloated that you can't access the information fast enough anymore.

It's why many voxel game servers run off ramdisks, because database calls become the be-all-end-all limiting factor and ramdisks have the highest possible access speed.

[capitalthree]

Well I'm not sure what you're implying, because a properly managed minecraft server that isn't starved for resources can support quite a lot of mods without lagging. I'm also not sure where you get the idea that a ramdisk is necessary... an SSD is more than sufficient. Those games don't work by constantly making "database calls" as you imply... they typically load compressed chunks from disk as-needed, keep them in ram while they're loaded, and serialize and compress them back out to disk. Voxel data for loaded chunks is just arrays.

Whether the way DRG handles terrain would have a creeping data bloat issue is basically what I'm asking, but I'm hoping someone with actual insight into the game engine can answer that. Does it really keep the original mesh and an ongoing list of modifications, like Droolguy implies? I always assumed it just keeps the entire mesh in ram and modifies it as it goes. Even if it did work that way, perhaps the game could do a compacting pass each time the server restarts and loads the map.

From what I can see of the game, if you constantly create new branching tunnels, you'd continue to increase the poly count, but a lot of other terraforming seems to result in consistently low-poly terrain. If I had to guess, I'd expect players would have to do a lot of deliberate digging of winding tunnels for a long time to give the engine serious issues.

I certainly don't see any reason for us to prematurely assume this is impossible without waiting for insight from the developers.

[Droolguy]

I was generalizing all voxel or modifable games and just using lag in Minecraft as an example, and yes it can support quite a lot of already existing mods on a static boundary map, technically infinite assuming you have the disk space. The issue becomes the massive amount of IO calls when actively changing the terrain, and then subsequently sending the new blockstates (if were sticking with minecraft) to everyone.

My question to you is this, why would you need to even ask this question? The data has to be stored somewhere, either in on a drive or in RAM, that's just how computers function. So no matter what you do, every single new polygon made as a result of modification has to be recorded somewhere which invariably will lead to bloat. It has to, there is no magic storage that gives you free data.

I suppose you could ask "how much data" does it take per modification, then it's simple math to tell you how many players it could support and for how long. The disadvantage being that unlike a game like minecraft which stores blockstates so that total chunk size stays the same and you just have to store the new state, a game using mesh deformation has to store every single modification independently.

Then it's pretty simple to figure out how much bandwidth it would take to constantly send that data to hundreds of players (A lot).

Actually, I think I know how I can test this right now. Give me a couple minutes.

[Dwarfurious]

nerds

[capitalthree]

Droolguy, can you explain why you are so confident in thinking that DRG has to save changes to the terrain incrementally? It really seems unlikely that it would work that way. Let me know how your test goes.

[Droolguy]

I was willing to bet it was just kept in RAM and I think I was right.

Ran seven games on the same map with the same settings with the same character looking at the same object to see what the resting state memory load of the game was. Then used 44 fuel to drill the whole area up and returned to the exact same position each game waiting until the game bottomed out it's memory load and hit it's resting state again. This actually took quite a while, DRG takes forever to unload stuff it isn't using anymore.

There was an average difference of ~21.4Mb, but all showed >20Mb without fail.

Also to answer your question, in a persistent game like you describe, map data would be required to be dumped from RAM to disk incrementally to facilitate server restarts and crashes without having severe rollbacks.

Since DRG isn't persistent it doesn't need to dump to drive (I think, I didn't see any major IO traffic while testing) if the game crashes and you loose everything in RAM, it just starts a new one.

Now my question is this, does DRG have true mesh deformation at all, or does it cheat and use the same method that Red Faction used in the early 2000s where it simply pastes the deformation shape as an empty area over and over additively? I have also noticed that the default state for objects is a square voxel when they glitch.

If it does then it should be really efficient because essentially it's just overlaying a ton of different static shapes over and over, and then spitting out the result of them all added together and then subtracted from the maps solid area.

[Bongwater Pizza]

Originally posted by Droolguy:

The data has to be stored somewhere, either in on a drive or in RAM, that's just how computers function. So no matter what you do, every single new polygon made as a result of modification has to be recorded somewhere which invariably will lead to bloat. It has to, there is no magic storage that gives you free data.

I honestly have no clue where you're getting this.

There is this thing called overwriting... why would you need endless space? You don't need to store all past states of a chunk...

[Droolguy]

I think you misunderstand me. This game does not use chunks, minecraft does because it’s geometry all confirms to a specific grid, and only has a certain number of states that a given voxel can be in. So they just change a 1 to a 6 or whatever corresponds with the specific block that occupies that space but all the blocks are always accounted for, even empty space.

[Bongwater Pizza]

Originally posted by Droolguy:

I think you misunderstand me. This game does not use chunks, minecraft does because it’s geometry all confirms to a specific grid, and only has a certain number of states that a given voxel can be in. So they just change a 1 to a 6 or whatever corresponds with the specific block that occupies that space but all the blocks are always accounted for, even empty space.

Right, so why would this engine need an endless amount of space? Could it not save the deformed mesh? Why would it need to continually write new info?

[capitalthree]

Droolguy, did you do a control group for your experiment, where you spend all your drill fuel without digging, and check usage the same way? Just curious.

Anyways, this certainly does nothing to prove that the mesh deformation is being tracked in some incremental way like you suggest. That's speculation, and we can all speculate until the devs weigh in.

As for the map being kept in RAM... duh... I think that was clear to everyone. However the fact that people can join a game in progress proves that the game has no problem serializing and writing out the map data. The fact that there is no lag spike on the host end when someone joins, proves that the game can periodically serialize the map data without impacting a live game. This is the same functionality needed for game saving. Did you consider the fact that this is a multiplayer game and already knows how to serialize its data?

[copygirl]

Just backing up capitalthree and Bongwater Pizza, even though they've brought up much of the same points. And probably better than me, too.

Originally posted by Droolguy:

It's why many voxel game servers run off ramdisks, because database calls become the be-all-end-all limiting factor and ramdisks have the highest possible access speed.

Originally posted by Droolguy:

The issue becomes the massive amount of IO calls when actively changing the terrain, [...]

I've dabbled enough into game development, including voxel games like Minecraft to be able to tell you that you either have a lack of understanding of how programming (specifically for games) works or you're not quite picking the right terms for things.
Newer games often build their game state using OOP instead something similar to a database. And data that is actively being used is stored in memory, not on disk. So there is no "IO calls", except when changes are periodically saved to disk, or additional data needs to be loaded (into memory). This isn't a much of a bottleneck.
Traditional databases are too slow for the kind of speeds a realtime game requires, with thousands to millions of accesses and modifications every 16 milliseconds(!) for a game running 60 FPS. So if a database or database-like approach is used to store the game state, it will be in-memory.

Originally posted by Droolguy:

The data has to be stored somewhere, either in on a drive or in RAM, that's just how computers function. So no matter what you do, every single new polygon made as a result of modification has to be recorded somewhere which invariably will lead to bloat. It has to, there is no magic storage that gives you free data. [...]

Doesn't DRG already do all that you've described? It already creates meshes made up of triangles because the graphics card needs to render them to the screen. Besides some optimizations, you may already have something that you could store persistently on disk between different game sessions.

Originally posted by Droolguy:

Now my question is this, does DRG have true mesh deformation at all, or does it cheat and [...] simply paste the deformation shape as an empty area over and over additively?

If that's the case, you could still create periodic snapshots that can be saved to disk.

[Horoai]

Originally posted by Dwarfurious:

nerds

Wait.. that was nerdy already? oof.

[Droolguy]

Originally posted by Bongwater Pizza:

Originally posted by Droolguy:

I think you misunderstand me. This game does not use chunks, minecraft does because it’s geometry all confirms to a specific grid, and only has a certain number of states that a given voxel can be in. So they just change a 1 to a 6 or whatever corresponds with the specific block that occupies that space but all the blocks are always accounted for, even empty space.

Right, so why would this engine need an endless amount of space? Could it not save the deformed mesh? Why would it need to continually write new info?

The deformed mesh has more data than the original mesh, your making more points to track. More points = more data. Hence the >20Mb from just digging up a map once.

Originally posted by capitalthree:

Droolguy, did you do a control group for your experiment, where you spend all your drill fuel without digging, and check usage the same way? Just curious.

Anyways, this certainly does nothing to prove that the mesh deformation is being tracked in some incremental way like you suggest. That's speculation, and we can all speculate until the devs weigh in.

As for the map being kept in RAM... duh... I think that was clear to everyone. However the fact that people can join a game in progress proves that the game has no problem serializing and writing out the map data. The fact that there is no lag spike on the host end when someone joins, proves that the game can periodically serialize the map data without impacting a live game. This is the same functionality needed for game saving. Did you consider the fact that this is a multiplayer game and already knows how to serialize its data?

...for the control group, yes I did, didn't perceptibly change resting state memory usage at all.

As for what you mean by "is being tracked in some incremental way like you suggest" I am going to need you to quote what I said to me, since I get the feeling we are not on the same page.

Sending the changes to a relatively small map to at max 3 other players is in no way indicative of the load that is applied with extremely large maps and very high playerbases. In other words, "It doesn't lag when one person joins" is totally irrelevant.

[Chibbity]

Yeah, I'm with Droolguy on this one, I really don't see an engine built for small scale one off maps, meant to be played for half an hour at most by 4 players ever scaling effectively to what OP describes, it'd prolly be less work to just make an entirely new engine from scratch.