Thanks for wasting my time. Steam ignores backed up data and just downloads the entire game from internet.

[doru]

I've created 1Tb of games backup using the steam backup option.

The idea was that if I want to play a game in the future I don't need to re download the game again. Steam starts the restore operation then ignores 80Gb of backup data and starts downloading the entire game again from internet.

I'm certain there will be no solution to this thread.

Anyway, thanks on wasting my time with this backup feature

[Thermal Lance]

The point of those backups is to move the games from one computer to another when changing hardware. It won't magically void any updates the game has recieved over the years.

It's mean as a solution to save some bandwith during a hardware change, not to store game for later down the line.

[Satoru]

That depends on when the backups were taken. If a backup was taken a long time ago and the game has updated significantly, steam will have to discard most of the backup data as its no longer relevant.

[doru]

both replies are not correct. here is why.

the point of a backup is to be a backup, not some specific scenario of moving data between computers. this is not mentioned anywhere, you are either inventing this now yourself or you are wrongly informed

if steam would had restored the game first from the backup files and after that check for updates, then this issue would not be an issue anymore.

Once the old game is back on disk, installed as it was before the backup, steam then can just download the update and patch the game.

this would mean downloading less data to patch the game, instead of downloading the entire game again

they did the lazy thing, if the game version from the backup is different that the one on steam servers, steam will just ignore the backup. Is simple as that and this shows the amount of respect they have for people time. Not only that but they have included this backup option just to troll people.

I don't see any other explanation. Fell free to change my mind

[Thermal Lance]

Originally posted by doru:

both replies are not correct. here is why.

the point of a backup is to be a backup, not some specific scenario of moving data between computers. this is not mentioned anywhere, you are either inventing this now yourself or you are wrongly informed

if steam would had restored the game first from the backup files and after that check for updates, then this issue would not be an issue anymore.

Once the old game is back on disk, installed as it was before the backup, steam then can just download the update and patch the game.

this would mean downloading less data to patch the game, instead of downloading the entire game again

they did the lazy thing, if the game version from the backup is different that the one on steam servers, steam will just ignore the backup. Is simple as that and this shows the amount of respect they have for people time. Not only that but they have included this backup option just to troll people.

I don't see any other explanation. Fell free to change my mind

The very definition of the word backup dosen't matter. This is exactly why the feature is here. As for the rest, have a good downloading session.

[Crazy Tiger]

Depending on the game it can actually be quicker to download the new version of the game instead of restoring the backup, downloading the update and patching the game. Especially with larger games.

[doru]

I don't see how it can be faster for me to wait 20 hours to download a 50 to 80 Gb game all over again.

[Dr.Shadowds 🐉]

Backups are only good as long there no future updates for said game, or too far out of date from said game changes, which end of the day you're gonna update / download game. I think this need to be rework.

Now I would suggest, just make another steam library via Steam settings add on your portable storage, or extra storage, use the Steam move, to move games you want to store, and there you go, that about it, and when next time launch Steam, just ensure storage is read ready, so whenever want to play those games it always there.

Now if happen to launch Steam without portable storage plug in, or extra storage wasn't ready to read, can try restarting Steam, if doesn't appear, do this, visit the storage steam library folder is on, rename that folder just add -old at the end of, now launch steam, add a new steam library folder to that storage it not able to detect, now shutdown steam, delete that new folder made, and rename back the folder you added -old at the end to it original name, and launch steam, and should fix the problem.

[Crazy Tiger]

Originally posted by doru:

I don't see how it can be faster for me to wait 20 hours to download a 50 to 80 Gb game all over again.

Might depend on the internet speed one has. But certainly with larger games and verification it can double the patching process, which for me means that a fresh download can be quicker.

[aiusepsi]

Originally posted by doru:

Once the old game is back on disk, installed as it was before the backup, steam then can just download the update and patch the game.

this would mean downloading less data to patch the game, instead of downloading the entire game again

I'm going to have to explain how Steam's patching system works to explain why this isn't accurate.

Steam breaks a game up into lots of small chunks. When a developer wants to upload a patch to Steam, a tool analyses the new version of the game, and works out where the new version contains chunks from the old version. It creates new chunks whenever there's new or modified data which doesn't correspond to any old chunks. It then uploads only the new chunks, along with a manifest of how to assemble all the chunks into the new versions of the game's files.

When Steam is updating a game on your computer, it compares the manifest of the old version of the game and the manifest of the new version, and downloads any chunks which exist in the new version but not the old version. Any chunks which exist in the both the new version and the old version are just copied over from old to new.

This method has lots to recommend it; it does a pretty good job of only making you download the differences between the two versions, and it supports updates between arbitrary versions in a single step.

When you do a backup, Steam saves all the chunks from the current version of the game.

When you install from a backup, Steam basically does the same thing as doing a patch. It looks at the manifest for the game version, and makes a list of all the chunks it needs. Any chunks it needs that are in the backup it will use, any chunks it needs that aren't, it'll download.

So, if it did just install the game as it was from the backup and did the update, you'd get the same result. The list of chunks making up the old backed-up version wouldn't change by installing the game, so doing the update would be just as extensive in the amount it would have to download.

If the backup is really old and there's been a lot of extensive updates, the odds are going to diminish that the backup and the most recent version share many chunks in common.

Theoretically you could do a better job by calculating a bespoke update between two versions using something like the rsync algorithm, but that doesn't scale well along multiple dimensions.

[doru]

Thank You for explaining how steam works, makes sense, looks like a good system to have.

But, today is not the first time I've got this issue I've explained in this thread. I've figured out this problem this summer so from then I've used a different method of backup games.

I just copy the game folder and the app manifest files manually. When I want to reinstall the game I can just copy back this folder and the file.

I have a couple of games saved with this method and when moving back one of this games, steam detects an update and downloads a smaller patch. not the entire game.

let me rephrase this. when I manually backup the game folder steam will work as explained by you. when using the steam backup functionality instead it use the dumb method of downloading the entire game by default

unfortunately I have only a small fraction of the games saved with this method. the rest are with the "official" method. which will require further time lost on my side. And the reason for this thread

[aiusepsi]

Originally posted by doru:

Let me rephrase this. when I manually backup the game folder steam will work as explained by you. when using the steam backup functionality instead it use the dumb method of downloading the entire game by default

Huh. It's not supposed to do that. Could be a bug? Or I'm wrong about how backups work.

I kind of want to investigate it; with the .csm files from a backup of a game I own it should be (theoretically) possible to work out how much it ought to be able to recover from the backup, although I would have to write some code. This is on the verge of nerd sniping[xkcd.com] me.

[doru]

it may just be a bug because I recall one or two years back the backups actually worked as you said, it used a mix of installing from disk then going online then installing from disk. it switched back and forth.

I think now once it hits a backup file that is different it doesn't switches back from the download from internet mode.

Is why I have 1 TB of these useless files now. It worked till maybe this spring

[aiusepsi]

This absolutely nerd-sniped me. I did a bit of Googling and found some Python scripts which someone wrote to read the .csm files, which confirmed (most of) the format of them.

The .csm has a list of chunks, and for each chunk it has a SHA1 hash and an offset and length. The offset and length are where the compressed data for each chunk is in the corresponding .csd file. That ought to be enough for Steam to use any of the chunks because the chunks are addressed by SHA1 hash.

Steam does some logging when it's installing, to the file Steam/logs/content_log.txt, that might be somewhat illuminating. I just did a backup and then immediately installed from that backup, and got the following in content_log.txt:

[2022-12-13 00:43:15] AppID 1313140 update started : download 0/0, store 0/914708832, reuse 0/0, delta 0/0, stage 0/1298698935 
[2022-12-13 00:43:15] AppID 1313140 state changed : Update Required,Update Queued,Update Running,Update Started,
[2022-12-13 00:43:15] AppID 1313140 update changed : Running Update,Staging,
[2022-12-13 00:43:15] Using 1490/1490 chunks from Path/Cult of the Lamb/Disk_1/1313142_depotcache_1.csd
[2022-12-13 00:43:18] YieldingStageFromChunkStores stats (Reads, Unpacks, Writes):
[2022-12-13 00:43:18] 6 Total, 0 Pending, 987963335 Bytes, 0.680611s Busy, 2.686553s Idle, 1001us Waited, 480326us Running
[2022-12-13 00:43:18] 1490 Total, 0 Pending, 1282841113 Bytes, 2.139383s Busy, 1.227775s Idle, 243282679us Waited, 7652267us Running
[2022-12-13 00:43:18] 1511 Total, 0 Pending, 1298698935 Bytes, 2.094956s Busy, 1.272204s Idle, 8675724us Waited, 726864us Running

I've not seen the "YieldingStageFromChunkStores" bit before, so maybe something has changed with this recently. Anyway, this logging might reveal what's going on? I wish I had an older backup to hand to test.

Unrelatedly and interestingly, it seems the sku.sis file has the manifest ID for each depot, if it's possible to get hold of that manifest version it would be theoretically possible to reconstruct the original files.

[aiusepsi]

Turns out I did have an older backup to hand, from May 2018, for Sega Mega Drive & Genesis classics. Some selected log lines of interest:

[2022-12-13 00:55:06] AppID 34270 update started : download 0/177656096, store 0/450592096, reuse 0/0, delta 0/0, stage 0/1441891593

[2022-12-13 00:55:08] Using 1256/1367 chunks from /Users/andy/Desktop/SEGA Mega Drive & Genesis Classics/Disk_1/71121_depotcache_1.csd

So, it only partially used the backup of depot 71121. But, looking at the manifest ID in sku.sis, and looking at the depot history on SteamDB, there's been two updates since the backup was made. So, maybe the download of 177ish MB is right. The whole backup is 584 MB and Steam used 450 MB of it, about 77%. Not sure what to draw from that.

[Satoru]

Originally posted by aiusepsi:

Originally posted by doru:

Let me rephrase this. when I manually backup the game folder steam will work as explained by you. when using the steam backup functionality instead it use the dumb method of downloading the entire game by default

Huh. It's not supposed to do that. Could be a bug? Or I'm wrong about how backups work.

I kind of want to investigate it; with the .csm files from a backup of a game I own it should be (theoretically) possible to work out how much it ought to be able to recover from the backup, although I would have to write some code. This is on the verge of nerd sniping[xkcd.com] me.

Note I think people misunderstand the UI elements and assume steam is “downloading” everything again when it’s not actually