LZMA2 Compression (7Zip)

[Tea Spirit]

My idea is instead of Valve using their current Algorithm for compressing their games on the servers, why not use 7Zip LZMA/LZMA2 compression? It's got faster compression times, Faster decompression times and the highest compression results for non lossy compression in the world. I noticed the other day when I was compressing my games with everything set to max on 7Zip in a .7Zip archive, the file size was shaved off considerably more than what I downloaded in the first place. Skyrim has a download on Steam of around 5.8gb, and when I compressed it, it came out as 5.3. That's 18 minutes or so to someone who has slow connection. I belive it would be very benificial to start compressing and decompressing the files nativley to the steam client. 7Zip is a opensource project therefore it can be used free of charge, even to big organisations like steam for server compressing. Having done some alterations to the Steam Client myself in the past, I know how relitivley easy it is to change the clients native extractor.

[Satoru]

Steam breaks games into chunks then compresses them

It does this so it can better scale out the back end side since they use pure HTTP to transfer data to users. That scalabilty is more important for them than the front end compression.

[Tea Spirit]

Originally posted by Satoru:

Steam breaks games into 1MB chunks then compresses them

It does this so it can better scale out the back end side since they use pure HTTP to transfer data to users. That scalabilty is more important for them than the front end compression.

Oh that's intresting. Thanks for that.

[Charlie]

With game sizes ever increasing (Titanfalls 36GB of AUDIO FILES!!!) and Internet speeds not soo much I'm in agreement, for instance I can take Assassis's Creed Syndcate (which is a 41GB game) and compress it into a 20GB archive. If given the option I would much rather download a compressed archive, uncompress it and manually place the game files where they need to go (pretty much just \steam\common and the .acf file)

It takes me around 30 minutes to uncompress that archive from 20GB to 41GB... Whereas on my connection (2.5MB/s) it would take around an extra 2 hours for the extra 21GB... It would also enormously help those with data caps.

[wuddih]

Originally posted by GhostMotley:

With game sizes ever increasing (Titanfalls 36GB of AUDIO FILES!!!) and Internet speeds not soo much I'm in agreement, for instance I can take Assassis's Creed Syndcate (which is a 41GB game) and compress it into a 20GB archive. If given the option I would much rather download a compressed archive, uncompress it and manually place the game files where they need to go (pretty much just \steam\common and the .acf file)

It takes me around 30 minutes to uncompress that archive from 20GB to 41GB... Whereas on my connection (2.5MB/s) it would take around an extra 2 hours for the extra 21GB... It would also enormously help those with data caps.

and how long did you need to compress the 41gigs into 20?

you are already downloading compressed data.

skullgirls:
1.7gb steam download
1.4gb lzma2 ultra (64-64-4096) - 12minutes on 4core
1.7gb lzma2 fast (1-32-128) - 2:30minutes on 1core
4.0gb on disk

there is no reason to compress files with resource intensive algorithm settings to save a few hundred megs. steam uses an on the fly data compression which is comparable to lzma2 fast

and if the gamefiles are in some form encrypted or are already compressed in some way, you are shooting with peanuts against a bunker.

[Charlie]

Originally posted by wuddih:

and how long did you need to compress the 41gigs into 20?

Around the same time as it took to decompress it, probably around 5-10 minutes more. It certainly doesn't take long though ~again on my system~

you are already downloading compressed data.

And often times with certain games it's such a small amount of compression it may as well not be there. On some games you literally download a few MBs less than it is on disk.

skullgirls:
1.7gb steam download
1.4gb lzma2 ultra (64-64-4096) - 12minutes on 4core
1.7gb lzma2 fast (1-32-128) - 2:30minutes on 1core
4.0gb on disk

Again it varies with the game, some compress better than others.

there is no reason to compress files with resource intensive algorithm settings to save a few hundred megs. steam uses an on the fly data compression which is comparable to lzma2 fast

It can certainly be more than a few hundred megs, 20GB is a huge saving on the download. Again 20GB - 41GB on my machine ~30 minutes~ or another 2 hours downloading the extra 20GB, I sure know which one I'd rather pick.

and if the gamefiles are in some form encrypted or are already compressed in some way, you are shooting with peanuts against a bunker.

As I've said prior depends on the game, some games you can literally half the file size, sometimes a few GBs and sometimes a few MBs if lucky. All I'm saying is I'm in agreement with the OP, it would be nice if Valve considered an option similar to this, sure it wouldn't be applicable for every game but at the same time it is applicable for a lot of games. It could be upto the publisher to offer a more compressed version for those with

A) Data Caps
B) Slower Internet Speeds but with PCs that can decompress archives quickly - Again 30 minutes vs 2 hours.

[Dusk of Oolacile]

Problem is, uncompressing today's 20-30-50 GB games would take hours on anything but very high-end CPUs. Valve support would be flooded with complaints about Steam halting the computer for hours.

[shs]

Originally posted by Dusk of Oolacile:

Problem is, uncompressing today's 20-30-50 GB games would take hours on anything but very high-end CPUs. Valve support would be flooded with complaints about Steam halting the computer for hours.

Nope, an i5 @ 3 ghz uncompresses 7zip @ 1.5 gb per minute. Which is about 120 times the download speed I get from Steam.

[Tea Spirit]

Bottom line, they can easily implement a better compression algorithm. The problem is whether they would or not. Also, any computer that couldn't decompress the game files in a decent time probably couldn't run the game efficiently anyways. Therefore it would be beneficial for 'Higher end' computers e.g. gaming PCs.

[Satoru]

Originally posted by StarSparkleNova:

Bottom line, they can easily implement a better compression algorithm. The problem is whether they would or not. Also, any computer that couldn't decompress the game files in a decent time probably couldn't run the game efficiently anyways. Therefore it would be beneficial for 'Higher end' computers e.g. gaming PCs.

Again its not really about the 'algorithm' its simply about how the back end delivers data. It doesnt deliver 'files' it delivers 'chunks'. The only way 'compression' would help is if it delivered entire files to you. Steam no longer does that. If it did, then every Payday2 update would be 20 GB in size because it would have to send the entire game to you every single time. Rome2/Shogun2 would have to deliver 2-4GB files instead of 100MB patches.

Devs have tried to 'compress' data when submitting to steam like Starbound. That made things worse because it meant Steam couldnt deliver small patches because the zipped file looked totally different every time. They eventually rearranged their data structures and then patches became orders of magnitude smaller.

Steam isn't going to compress entire files because doing so essentially breaks how Steam Pipe works to begin with. It also makes the infrastructure harder to scale.

Also zipping files requires 3 stages to actually deploy, which wastes even more disk space locally.

[The Muppet Surgery Special]

Perhaps they should look at what CCP does with Eve Online for patching. Compressed and on the fly content downloading, along with delta patches.

[Satoru]

Originally posted by The Muppet Surgery Special:

Perhaps they should look at what CCP does with Eve Online for patching. Compressed and on the fly content downloading, along with delta patches.

Steam already does that with the chunks of data it sends

How much comp;ression depends on the game, file structure, etc.

[Tea Spirit]

Originally posted by Satoru:

Originally posted by StarSparkleNova:

Bottom line, they can easily implement a better compression algorithm. The problem is whether they would or not. Also, any computer that couldn't decompress the game files in a decent time probably couldn't run the game efficiently anyways. Therefore it would be beneficial for 'Higher end' computers e.g. gaming PCs.

Again its not really about the 'algorithm' its simply about how the back end delivers data. It doesn't deliver 'files' it delivers 'chunks'. The only way 'compression' would help is if it delivered entire files to you. Steam no longer does that. If it did, then every Payday2 update would be 20 GB in size because it would have to send the entire game to you every single time. Rome2/Shogun2 would have to deliver 2-4GB files instead of 100MB patches.

Devs have tried to 'compress' data when submitting to steam like Starbound. That made things worse because it meant Steam couldn't deliver small patches because the zipped file looked totally different every time. They eventually rearranged their data structures and then patches became orders of magnitude smaller.

Steam isn't going to compress entire files because doing so essentially breaks how Steam Pipe works to begin with. It also makes the infrastructure harder to scale.

Also zipping files requires 3 stages to actually deploy, which wastes even more disk space locally.

I see. Although you can "Chunk" the files in 7zip too. I believe you can do it right down to 1Mb each.

So for the sake of simplicity can you give us a run down of how exactly Steam sends data from their servers to the consumer? I'm slightly confused about all this "back end" stuff.