what is the secret? (for the devs.)

[PyroFox]

i am super curious. how did you code this beast of a game to handle all the constant creation and deletion of items on conveyor belts all over the world being moved around constantly in the millions upon millions per second without lag? like i understand rendering only what is on screen but how did yall handle all the off screen movement of bots, objects, ext. like in any normal game this would lag the game for sure yet my huge base has no problem ever even though it pumps out enough of every science to feed a full express belt constantly.

[PunCrathod]

A lot of it is the fact that the company was owned by the people who were actually making the game instead of someone who is only in it for the money. Also they actually used modern tools like profiling and automated testing(these have been around for like 30 years but I still only occasionally see "veteran" developers use them). They also blogged a lot of the stuff they did and listened to the community and made their own about ideas on how to fix/optimize the code and tried them and tested them to see if it was actually better instead of just assuming that because they have been programming for 20 years they know the best way.

[Aimee]

The game does not visualy render places that are far away from the player, while still tracking objects and creatures.
That saves a lot of resources already.

I dont think you have to make every item into an object when its on a conveyor belt, a picture or two is enough. And just tracking how many items there are, can be done with simple numbers. You dont need to know if a conveyer belt block has max items, all you need to track is how many full conveyor belts there are and render a single one of them. Moving items would could be a single animation in a seperate layer that only moves 1 block.
Often a conveyor belt has a consistent amount of items on it, or atleast a repeating amount... so easier to track number of the same belt, as opposed to track every belt and item individualy.

Thats what i assume atleast.

[Phrice]

Involvement with the modding community %100 of the time was also a testament to their commitment to producing quality code.
No other developer compares.
When modders had issues, the devs actually listened and worked with them.

[Hurkyl]

I do recommend reading through their FFF posts if you're curious about details since they do talk about them a lot.

Regarding conveyors specifically, one of the biggest tricks they do is that when items are pressed up against each other, the conveyor belt will never change their positions -- the grouping will only change when something else disrupts the group, such as an inserter or a splitter.

Thus, what they do is that they actually group them together, and then they only need to track where the group is on the conveyor belt, rather than where the individual items are.

So a full express belt of items barely requires any more effort to update than a belt with a single item. More work will be spent at the start and end of the belts where they are not compressed.

[PunCrathod]

Originally posted by Aimee:

The game does not visualy render places that are far away from the player, while still tracking objects and creatures.
That saves a lot of resources already.

I dont think you have to make every item into an object when its on a conveyor belt, a picture or two is enough. And just tracking how many items there are, can be done with simple numbers. You dont need to know if a conveyer belt block has max items, all you need to track is how many full conveyor belts there are and render a single one of them. Moving items would could be a single animation in a seperate layer that only moves 1 block.
Often a conveyor belt has a consistent amount of items on it, or atleast a repeating amount... so easier to track number of the same belt, as opposed to track every belt and item individualy.

Thats what i assume atleast.

Ummm... no. Each and every item is a separate object and they are each rendered separately. It is actually cheaper this way than to make separate animations for each combination of items on a belt. It is really cheap to render one thing a thousand times compared to rendering a thousand things once. It is also cheaper and faster to just have individual items and render them all individually than it is to calculate what sprite you need to render based on what items should be on the belt.

Also there is no computer in the world that has enough memory to hold all the texture data needed to use prerendered animations for each different combination of items and their positions on a belt.

Edit: in case you need to explain to someone who does not understand why you can't prerender items onto belts. Lets make a simplified version where a belt can hold 8 items and they dont move. And there are 208(according to google) different items that can be placed on a belt in factorio+1 if the position is empty. Therfore you need 209^8 different sprites or 3640577568861717121*32*32*24 bits of memory to store those sprites. That is roughly 80 million petabytes. And that is for a simplified case where there are only 8 possible positions for items. If you wanted the smooth movement and arbitrary sized gaps between items you would need more memory than the entire world has to store those sprites.

[Aimee]

Originally posted by PunCrathod:

Also there is no computer in the world that has enough memory to hold all the texture data needed to use prerendered animations for each different combination of items and their positions on a belt.

If you use a still image, as is the case with every object on the conveyor belt. You can simply have a picture set, and use those instead of having to ''prerender'' As if you use a .gif image instead of an actual object.
Plenty of games do this, especialy top down games, and 2d games.

But it seems someone already answered it;

Originally posted by Hurkyl:

Thus, what they do is that they actually group them together, and then they only need to track where the group is on the conveyor belt, rather than where the individual items are.

Not that far from my assumtion of using a picture for items on the belt, instead of having them seperate. Still i was pretty far off, glad someone answered it atleast.

Originally posted by PunCrathod:

a belt can hold 8 items and they dont move. And there are 208 different items that can be placed on a belt in factorio+1 if the position is empty. Therfore you need 209^8 different sprites or 3640577568861717121*32*32*24 bits of memory to store those sprites. That is roughly 80 million petabytes.

Im pretty sure this calculation is incorrect.
You have 8 different positions for items, so thats 8^2 different outcomes for each item (64 pictures/sprites per item) times 208 items is 13.312 pictures or sprites.
If you add your 1 empty sprite it becomes: ((8^2)+1)*208 = 13520
Not sure where the rest of your calculation comes from, but im pretty sure we are talking about something different regardless.

[Vyndicu]

Originally posted by PyroFox:

i am super curious. how did you code this beast of a game to handle all the constant creation and deletion of items on conveyor belts all over the world being moved around constantly in the millions upon millions per second without lag? like i understand rendering only what is on screen but how did yall handle all the off screen movement of bots, objects, ext. like in any normal game this would lag the game for sure yet my huge base has no problem ever even though it pumps out enough of every science to feed a full express belt constantly.

The Factorio uses an optimization that divides the entire map into "several entities" when moving items on belts instead of hundreds of individual belt entities.

https://www.factorio.com/blog/post/fff-148

At the same link, you can click on Blog on the top right to read other articles talking about how they approach optimizations.

[KPaulH]

Magic don't peek behind the curtain just enjoy the magic lol

[RiO]

Originally posted by Aimee:

Im pretty sure this calculation is incorrect.
You have 8 different positions for items, so thats 8^2 different outcomes for each item (64 pictures/sprites per item) times 208 items is 13.312 pictures or sprites.
If you add your 1 empty sprite it becomes: ((8^2)+1)*208 = 13520
Not sure where the rest of your calculation comes from, but im pretty sure we are talking about something different regardless.

No. PunCrathod is correct.

It's 8 positions per tile, which would be 2^8 combinations if you are only tracking the presence of an item. It's a basic binary number system: 0 - nothing present; 1 - something present.

In the case of 8 positions, it is basically a byte storing binary values. And one byte stores 2^8 = 256 values -- namely the values 0 through 255. It does not store 8^2 = 64 values.

Now, if you are also tracking what item, then given a pool of N items you're looking at an (N+1)-ary number system: 0 - nothing present; 1 - first item present; 2 - second item present; ... ; N - last item present.

And that makes for (N+1)^8 possible combinations.



Optimization-wise, what the Factorio developers actually implemented is segmenting the contents on the belt into groups, and then sliding the groups along the belt using an offset. And then where anything takes from a group or adds to a group, that group is reevaluated together with its direct neighbors; resulting in possibly changing the split, or - where the whole thing becomes a compressed fully populated (or completely empty) segment - merging the three all together.

Fully populated - i.e. 'compressed belt' - groups that hold nothing but the same item are treated specially and are basically reduced to a simple "holds item X everywhere in the group".


Internally each non-compressed group could theoretically still track the actual placement of individual items within the group very efficiently by using bit-masks, under the assumption that each group will only hold a limited amount of types of items.

Let's say, for example, that a group has a theoretical limit of 7 different items on it.
Storing the empty space and 7 different items means 2^4 = 8 applies; i.e. it takes 4 bits to store the state of one 'slot.' A single 32-bit integer could store the full contents of a single belt segment as it has 8 slots. Moreover, you can store left and right lanes as the low 16-bits and high 16-bits and access those efficiently separately as well, if you'd need to.

(Of course; that part is just theoretical. No idea if the developers actually did that. But it would've been one heck of a nice way to compress belt data and manage it efficiently. Especially since you can also start using bitwise AND; OR; XOR, etc. operations to efficiently test for presence. E.g. for inserters that are 'looking for' certain items to pick off of a belt.)

[PunCrathod]

Originally posted by Aimee:

Im pretty sure this calculation is incorrect.
You have 8 different positions for items, so thats 8^2 different outcomes for each item (64 pictures/sprites per item) times 208 items is 13.312 pictures or sprites.
If you add your 1 empty sprite it becomes: ((8^2)+1)*208 = 13520
Not sure where the rest of your calculation comes from, but im pretty sure we are talking about something different regardless.

There is an error in my calculations. But not in the way you think. I said 80 million petabytes. When it is actually 80 million petabits aka 10 million petabytes. The math is simple. In the first slot can be 209 different things. Multiply that by what can be in the second slot and that by what can be in the third slot etc and you get 209*209*209*209*209*209*209*209 or 209^8.

Or are you thinking that each item on a belt is rendered separately on top of the belt in pre-rendered animations containing all different positions an item can have on a belt? Sure then you would only need 64 sprites per different item but that is utter waste of resources. Only a complete madman would go out of their way to complicate rendering like that. You can just have one sprite per item and tell the gpu where to draw that one sprite. It can be in any position not just in a grid of tiles and have any size. When you draw things with a modern gpu(one that has geometry/compute shaders aka anything from the last two decades) you only need to give the gpu three numbers per sprite. X coordinate as a float(usually .0 at the left border of the screen and 1.0 on the right but can be anything as long as you code your shaders etc correctly), Y coordinate as a float(same as x but top is usually .0 and bottom is 1.0) and the index as an integer on what part of the texture atlas needs to go there. And the gpu will calculate the rest as that is what it was designed to do. Most of the calculations a gpu will do don't even involve pixels. Pixels are just the last step. They are just arbitrary numbers and arbitrary formulas. The gpu doesn't even have a concept of tiles or sprites. It's just floating point math and some integer math until the very end.

Originally posted by RiO:

Originally posted by Aimee:

Im pretty sure this calculation is incorrect.
You have 8 different positions for items, so thats 8^2 different outcomes for each item (64 pictures/sprites per item) times 208 items is 13.312 pictures or sprites.
If you add your 1 empty sprite it becomes: ((8^2)+1)*208 = 13520
Not sure where the rest of your calculation comes from, but im pretty sure we are talking about something different regardless.

No. PunCrathod is correct.

It's 8 positions per tile, which would be 2^8 combinations if you are only tracking the presence of an item. It's a basic binary number system: 0 - nothing present; 1 - something present.

In the case of 8 positions, it is basically a byte storing binary values. And one byte stores 2^8 = 256 values -- namely the values 0 through 255. It does not store 8^2 = 64 values.

Now, if you are also tracking what item, then given a pool of N items you're looking at an (N+1)-ary number system: 0 - nothing present; 1 - first item present; 2 - second item present; ... ; N - last item present.

And that makes for (N+1)^8 possible combinations.



Optimization-wise, what the Factorio developers actually implemented is segmenting the contents on the belt into groups, and then sliding the groups along the belt using an offset. And then where anything takes from a group or adds to a group, that group is reevaluated together with its direct neighbors; resulting in possibly changing the split, or - where the whole thing becomes a compressed fully populated (or completely empty) segment - merging the three all together.

Fully populated - i.e. 'compressed belt' - groups that hold nothing but the same item are treated specially and are basically reduced to a simple "holds item X everywhere in the group".


Internally each non-compressed group could theoretically still track the actual placement of individual items within the group very efficiently by using bit-masks, under the assumption that each group will only hold a limited amount of types of items.

Let's say, for example, that a group has a theoretical limit of 7 different items on it.
Storing the empty space and 7 different items means 2^4 = 8 applies; i.e. it takes 4 bits to store the state of one 'slot.' A single 32-bit integer could store the full contents of a single belt segment as it has 8 slots. Moreover, you can store left and right lanes as the low 16-bits and high 16-bits and access those efficiently separately as well, if you'd need to.

(Of course; that part is just theoretical. No idea if the developers actually did that. But it would've been one heck of a nice way to compress belt data and manage it efficiently. Especially since you can also start using bitwise AND; OR; XOR, etc. operations to efficiently test for presence. E.g. for inserters that are 'looking for' certain items to pick off of a belt.)

If I remember correcty it works such that for each item or a group of touching items on a belt segment there is a pointer to the item(I think if it is an undamaged item that does not have an inventory like the powerarmor or spidetron then it is just pointing to the prototype of that item so you don't have a million copies of iron plate in memory) or an array of items if it is a group and a float offset to the item in front of it. Belt segments are continuous lengths of belts that don't have sideloading, circuited belt or splitters and it is further split into segments by the presense of inserters to make picking up items efficient. This way you only need to keep track of the first moving item on a belt segment each tick and only calculate the position of the rest if they are under an active inserter. An active inserter keeps a pointer to and an offset to the leading item it is currently trying to pick up so unless that item moves out of it's reach it only needs to calculate one float to see if the item is still within reach and if it isn't then it is really easy to just increment the pointer and get the next item on the belt.

There really isn't a point to try and reduce the memory footprint more than that. If you have a million items on a belt and there is a small gap between each item it's still just 64+32*1000000 bits or 11 megabytes. It's much more important to reduce the amount of calculations done to those items to save as many cpu cycles as possible.