Game compression

[Acetyl]

I use 7-zip to compress game files (quite often 50 - 70% size reduction, some cases of 90+%), add them to a "compressed" list, and then disable automatic updates for those games. It would be ideal if this process could be automated and the steam client would track the compression state along with the other aspects indicating whether a game is installed and usable, so it won't try to update, automatically decompresses if you try to launch it, etc. "A "Compress game" or "Archive" option could be added to the client. This would do all of the aforementioned and automatically return the game to its uncompressed state if you try to launch.

Edit for clarification. The game will not be stored compressed by default, nor will it be decompressed and recompressed with any file changes every time you launch it. The idea is if you go into the properties page and click Archive, it will be compressed. If you launch it while compressed, it will be decompressed until you decide to compress it again.

Steam already has an archival system, but it uses poor compression and is set to split the file into CD or DVD size pieces. This could be modernized in several ways.

A list of compression results will be maintained here

[Spawn of Totoro]

You can use this.

https://www.reddit.com/r/Windows10/comments/76hj26/i_tested_25_games_against_the_windows_compact/?st=J8THCDU8&sh=b22efea6

Seems many have has success with it.

Some things should be up to the user, especially as there may be issues when messing with files in such a way.

Seems even Window's build in function can do it too, and even has special feature for programs.

The Compact functions introduced in Windows 10 (which the program hooks into) have flags that are designed for programs; specifically, the "/EXE" flag, which uses algorithms that compress folders, but are also less taxing to decompress at runtime.

It would cause a lot of HDD/SSD activity to keep compressing and uncompressing it and that may shorted the life of the drives, so something that I am sure Valve wouldn't want to be found responsible for.

[Acetyl]

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

I think the utility is fairly obvious. It's not all that complicated either.

"Things users can do without the client if they choose to do so" does not mean Steam needs to implement every whim of every user out there. The most basic fact is if you want something badly enough, and there's tools available; you use them & move on, not ask for something with little benefit to community especially since most people will complain it takes so long if they utilize it.

I'm not really sure what you're trying to say here. To me it looks like you're just repeatedly stating the obvious, if I'm wrong please clarify. Not much else to say here. As far as it taking a long time, people have multithreaded high performance CPUs these days. I don't think it would be too big of an issue as long as you provide a progress bar, you don't see people complaining about eg how long it takes to validate files.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

1) No one will be forcing you to compress your games.
2) Compression doesn't take that long.
3) Filling up your case with hard drives to store games you could reduce in size by 30 - 70% is not sensible.
4) Repeatedly transmitting the same information over the information is not efficient

High grade compression takes much longer than quick compression, and quick compression is pointless towards your suggestion. It would be faster to download and install through the client. We have 6TB drives for next to nothing, you really have zero excuse especially if you have a larger library. How many games do you own in Steam? How many of them are AAA? How many of them are 5GB or less?

1) In many cases even quick compression would achieve a net benefit.
2) I used high compression settings. See below.
3) Not everyone lives in the city and has 2 MB/s internet. Where I am I max out ~830 KB/s. Back in the day it was more like 360 KB/s. In fact, I downloaded all of the Orange Box over dial up back then. Dial up is not common anymore, but these are not uncommon arrangements. Even if I had T3 internet blasting into my house, eh, I would likely still archive some things. You can't rely on "the cloud", having a local copy has its uses.
4) It doesn't matter what a hard drive goes for these days, far as I'm concerned with modern multicore CPUs, memory, and mature highly optimized compression algorithms, you have no excuse wasting storage space and saturating the net with unneccesary traffic. Do you save all of your images in BMP? Do you keep your PNGs uncompressed? Why bother wasting time compressing a png? Storage is so cheap! The answer is no, you don't do any of that, no one does. You have it all in jpg and png like everyone else. There is a reason for this.
5) I've had my account for 15 years and have 273 games, don't know how many are AAA or the median install size.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

Not so. I'm running some crap single core AM3 opteron and just compressed Dead Space from 7.8 Gigs down to 3.5, took around an hour and I just left it on idle priority in the background. Ram usage was ~4.5 gigs. Settings were:
-mx=9 -myx=9 -ms=5g -mqs=on -mf=on -slp -m0=LZMA2:d=415m:fb=273:mc=1000

Again, we're talking best compression vs quick compression. Older CPUs will struggle to pack something down to the smallest size possible, and older AMD CPUs will basically red line due to how the old architecture is.

Those are the highest settings for 7-zip. I have an older AMD and just did it.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

You're kidding me right. You have people that have their OS installed on an SSD, that's hundreds to thousands of registry and system file reads and writes every second.

Tiny files vs GBs of data each pack, unpack, temporary storage use etc.
That's just an immensely bad faith deflection on your part of two incomparable things for pointing out a serious flaw in this self-serving suggestion.

It's comaprable. Most modern SSD controlelrs and OS's buffer data and batch write to cells. The way 7z flushes its output to disk could be tailored if this is a major concern (though it really isn't). Temporary files aren't generated. The work is done in memory and written to disk.

It seems like something isn't clicking here. The game will not be stored compressed by default, nor will it be decompressed and compressed every single time you launch it. If you go into the properties page and click compress, it will be compressed. If you launch it while compressed, it will be decompressed. That's it. You might as well be complaining about temp files generated while downloading patches.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

Not really sure what your point is here, it's really reaching.

Compress and decompress 25, 50, 75 and 100+GB games. Repeatedly. All day. Demonstrate that there is no such side effect, as I'd rather not chunk a drive similar to when people run performance tests. HDDs would be far better suited than SSDs for this purpose, and new drives are very fast compared to 2003.

I have no idea how you read my OP and subsequent posts and came to that conclusion. Read above. Might have to edit the OP if this keeps happening.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

This is not what I said. I said (in most cases) you cannot fully know without actually implementing something, and you certainly can't model demand.

"Implement my suggestion so we can figure it out after" is not a good suggestion. This is Valve we're talking about, not amateur hour, meaning that they can easily know how something would function, be utilized, etc. This isn't a one-man operation of just throwing something at the dart board hoping it ends up a good idea; Valve has the competence to know that this is not whatsoever worth even entertaining, and would be a waste of programmers time.

This is very strange. Unless you work at Valve, I'm just going to move on from this line of speculation.

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

Are you familiar with the Steam codebase? If not, you don't really have much basis to speculate on what it would take to implement a given feature. I assume it wouldn't be difficult and iirc several LZMA implementations exist with viable licenses, but again, pointless speculation.

I very much doubt you are, and that's a very poor deflection from the response. Just because Valve can, does not mean they will or should. You're making far more speculation, which is just hypocritical a just another OP that can't stand the immense flaws in a suggestion.

No comment necessary. You're clearly seeing something else through this exchange. Not my problem.

Originally posted by Mr. Gentlebot:

It's also beyond silly to download compressed packages only to later compress them again, which users may freely do themselves if they want to back it up for a much later use. After that, all they have to do is uninstall the game so it wont queue, and unpackage when they're ready to update & play.

This doesn't work reliablty. I don't know if it's looking at file modifed timestamps or something, but often after decompressing it will want to update or redownload certain files, despite no update coming in the meantime.

That's an interesting point though. A game could start out in the compressed state initially until first launch. Though using solid compression or larger blocks (don't know how valve transmits things) could achieve much better compression.

Originally posted by Mr. Gentlebot:

Nothing is stopping you from manually accomplishing this, the issue with most suggestions is people want Steam to be an everything-client, which mostly comes from users not wanting to put effort into things they want which Steam does not offer, which if they entertained each suggestion the client would be a giant bloated mess rather than essentially a "thin" client with the functionalities desired for an online store & game library client.

Despite talking about Valve's competence above, you seem weirdly protective of the Steam client. Don't really know what the deal is here. I'm well aware I can manually accomplish it, I've been doing it for months and outlined it in the OP. I mean what?

[Acetyl]

Originally posted by Spawn of Totoro:

You can use this.

https://www.reddit.com/r/Windows10/comments/76hj26/i_tested_25_games_against_the_windows_compact/?st=J8THCDU8&sh=b22efea6

Seems many have has success with it.

Some things should be up to the user, especially as there may be issues when messing with files in such a way.

Seems even Window's build in function can do it too, and even has special feature for programs.

Interesting. I'm still on Win 7 and didn't know about this feature. Seems like an evolution of the old compress rarely used files setting.

Originally posted by Spawn of Totoro:

It would cause a lot of HDD/SSD activity to keep compressing and uncompressing it and that may shorted the life of the drives, so something that I am sure Valve wouldn't want to be found responsible for.

I edited the OP for clarity. Took a bit to realize what people thought I was suggesting.

[Acetyl]

I suppose a halfway approach would be having a compression script also modify the corresponding appmanifest to turn off autoupdate.

[Quint the Alligator Snapper]

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

I think the utility is fairly obvious. It's not all that complicated either.

"Things users can do without the client if they choose to do so" does not mean Steam needs to implement every whim of every user out there. The most basic fact is if you want something badly enough, and there's tools available; you use them & move on, not ask for something with little benefit to community especially since most people will complain it takes so long if they utilize it.

There's no harm in OP asking. (Since when were you the judge of what's "benefit to community"?)

Also, as OP described, it's an option. It's not like Steam goes in and compresses everything without your asking it to. (And it should of course default to off, if such an option is implemented.)

Originally posted by Mr. Gentlebot:

Originally posted by Acetyl:

1) No one will be forcing you to compress your games.
2) Compression doesn't take that long.
3) Filling up your case with hard drives to store games you could reduce in size by 30 - 70% is not sensible.
4) Repeatedly transmitting the same information over the information is not efficient

High grade compression takes much longer than quick compression, and quick compression is pointless towards your suggestion. It would be faster to download and install through the client. We have 6TB drives for next to nothing, you really have zero excuse especially if you have a larger library. How many games do you own in Steam? How many of them are AAA? How many of them are 5GB or less?

Originally posted by Acetyl:

Not so. I'm running some crap single core AM3 opteron and just compressed Dead Space from 7.8 Gigs down to 3.5, took around an hour and I just left it on idle priority in the background. Ram usage was ~4.5 gigs. Settings were:
-mx=9 -myx=9 -ms=5g -mqs=on -mf=on -slp -m0=LZMA2:d=415m:fb=273:mc=1000

Again, we're talking best compression vs quick compression. Older CPUs will struggle to pack something down to the smallest size possible, and older AMD CPUs will basically red line due to how the old architecture is.

This all depends on people's internet connection speeds vs. their processor speeds. Both of which vary quite a bit.

Either could be faster.

So if Steam gives people the option, they can play around to see which one works better for them.

Originally posted by Mr. Gentlebot:

Nothing is stopping you from manually accomplishing this, the issue with most suggestions is people want Steam to be an everything-client, which mostly comes from users not wanting to put effort into things they want which Steam does not offer, which if they entertained each suggestion the client would be a giant bloated mess rather than essentially a "thin" client with the functionalities desired for an online store & game library client.

The Steam client is already not a "thin" client anyway, as I've explained elsewhere. It's already quite bloated. A truly "thin" client would just do its game launching and DRM checks on the command line, not even needing a UI.

Also, why are you taking it upon yourself to try to gatekeep what other people suggest? That ought to be Valve's job.

[Acetyl]

Compression results:
03/22/22 Don't Starve (without soundtrack): 4421 MB -> 1949 MB (44%).

[Cathulhu]

I can already see users complaining that their games heavily lag all the time because Steam hogs all the CPU power just to decompress data so the game can actually load it.
Especially with weaker systems.

[Acetyl]

Originally posted by Cathulhu:

I can already see users complaining that their games heavily lag all the time because Steam hogs all the CPU power just to decompress data so the game can actually load it.
Especially with weaker systems.

Did you read the OP?

[Acetyl]

Resurrecting my thread. The first time around no real discussion was had, new demographics may have a different take.

[fluxtorrent]

unlikely, time didnt make it a better idea

[Acetyl]

Originally posted by fluxtorrent:

unlikely, time didnt make it a better idea

There is nothing wrong with the idea.

[Washell]

Valve could make setting the folder to compressed and the game to "Only updated when launched" with a single click in the client. Given that in the 8 months you haven't found a single person who cares about it, and the thread dies within a few hours, I don't think they're going to bother. It may be time to let it go.

[Psymon²]

Originally posted by Acetyl:

Resurrecting my thread. The first time around no real discussion was had, new demographics may have a different take.

bad news
those ones who did everything in their power to derail and bait you into overreacting have been endlessly driving away the "new demographic" you were hoping for with the same tactics.

apologies for not having an opinion on the compression, as i uninstall games when I'm done with them and have a comfortably quick internet connection should i want it back.

[Brian9824]

Originally posted by Acetyl:

Resurrecting my thread. The first time around no real discussion was had, new demographics may have a different take.

Waiting 6 months doesn't really change any of the previous answers. Sure Valve COULD do it, but seems like a lot of work for a highly specific scenario that is not the norm, nor is it best practice.

As others have stated there are far better solutions then what you describe that make far more sense IMO.

[Acetyl]

Originally posted by Psymon²:

Originally posted by Acetyl:

Resurrecting my thread. The first time around no real discussion was had, new demographics may have a different take.

bad news
those ones who did everything in their power to derail and bait you into overreacting have been endlessly driving away the "new demographic" you were hoping for with the same tactics.

Yes, I've noticed.

Originally posted by brian9824:

Originally posted by Acetyl:

Resurrecting my thread. The first time around no real discussion was had, new demographics may have a different take.

Waiting 6 months doesn't really change any of the previous answers. Sure Valve COULD do it, but seems like a lot of work for a highly specific scenario that is not the norm, nor is it best practice.

As others have stated there are far better solutions then what you describe that make far more sense IMO.

There's a lot of junk to wade through ITT, but I covered the reasons why it should done by the client. Namely updates, and though I haven't tested I imagine issues with file integrity could arise if steam relies on file modified timestamps.