Can we get Steam-managed file system compression or saved post-update commands?

[Tyrian Mollusk]

I use a tool called "Compactor" on Windows, which is a front end to the Windows 10+ tool compact.exe, to compress some of my games using Windows transparent filesystem compression, which leaves the data in a file compressed and automatically decompresses it on the fly whenever the system reads it, saving space while potentially slowing initial reads some (potentially, because systems are so fast now you can run decent decompression in the time it takes for the data to move out of storage, and have so much RAM that caching the decompressed data while the game is running may not even impact anything).

Many game developers have frankly become terrible about taking care of their deliverables, and I get many games wasting half or more of their install storage, some even upwards of 80% wasted space. My disk space is mine to waste, and not devs' to squander, but this problem only seems to be worsening, with most devs I bring it up to ignoring it or admitting they don't even know how to handle compressed data files.

Compactor makes for a reasonable compromise, but the way Windows transparent compression works, you must manually compress the file, and any change to it will automatically decompress it and it will stay decompressed. So, every time a game updates, I have to manually re-compact it to get the space back, which means I have to manually update all my games, and I keep a separate Steam folder for the games needing compacting, to make finding and managing them easier, which Steam still fights against because somewhere along the way Valve decided we shouldn't have more than one Steam install per drive, even though that was just a dumb thing to assert. Also, the current Steam client beta makes it harder to catch games wasting space, because the installation messages have changed from the non-beta client and no longer helpfully display both the download size and install size at the same time. Obviously, the download archive size is not the same as a runnable compressed install size, but it gives a very clear sign that a game is wasting space and will gain useful benefit from compacting, so it's valuable to have those two sizes readily displayed--even if you do need to start the install to see them, which is later than would be ideal.

Anyway, we really should not need to pay attention to all of this. That's exactly the kind of thing our software is for in the first place. Steam could do the simple things Compactor does to compress and flag the files for Windows (and Compactor is basically a nice blueprint on doing it right, eg skipping files that don't save "enough" space and recording them to save time/work on future scans), or Steam could let us designate post-update commands, where we can set things like this to run normal housekeeping when game files change.

Obviously for this specific task, Steam managing it would be far superior and extremely handy, since it could then correctly report disk space used and saved by our games (currently, Steam ignores the compressed status and reports the disk usage based on the decompressed size), but post-update commands method is more trivial to implement and surely comes with other benefits, like auto-updating manual mod installs or whatnot.

No doubt there's something you can do on Linux systems to get a similar transparent compression effect, so this kind of thing would probably eventually help Deck (and SteamOS) users as well, who have even more reasons to not like developers blowing off taking 2-3+ times as much space as their games actually require to run.

This would all be wholly positive, straightforward, and easy, with all the real work already handled at the OS level, and just some managerial details exposed that Compactor has already nicely worked out. I know I'd really appreciate these features, as I like to keep a lot of games installed and ready to play. We like helping users buy more games, right Valve?

Microsoft compact.exe reference:
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/compact

Compactor GUI front-end to compact.exe reference:
https://github.com/Freaky/Compactor

Example using Under-Night In-Birth II Sys:Celes:
Initial install: 16.25GB
Compacted install: 5.16GB (32%)
Savings: 11.1GB, 68%
Example games from my installed recent library that can all fit together in that chunk of saved space even without compacting them in turn: Deep Rock Galactic, Drox Operative 2, Nova Drift, Combat Complex, Asterogues, Nidus, Devil Blade Reboot, Marble It Up Ultra, Devil Engine, Drainus, Elsie, Shape Shifter Formations -- twelve good games in the extra space from one ain't nothing.

[Holmes]

" because systems are so fast now you can run decent decompression in the time it takes for the data to move out of storage, and have so much RAM that caching the decompressed data while the game is running may not even impact anything)."

I have an i7 3770s and 8gb of ram, this would ruin my Steam experience. Not everyone has a super duper computer my friend.

[Tyrian Mollusk]

Originally posted by Holmes:

I have an i7 3770s and 8gb of ram, this would ruin my Steam experience. Not everyone has a super duper computer my friend.

A) Even so, the compression wouldn't ruin your experience, on at least some games. Especially if you also don't have an SSD. That "systems are so fast" comment is not some superlative. Chips have been fast enough to run this kind of special fast-decompression alongside the relatively slower data stream coming from disk for a lot longer than you might think.

B) I obviously never said to forcibly apply it to everything. I asked for the option to have Steam manage it for games where we turn it on, or to let us apply post-install commands which would have to be specific to individual games. Either way, if you didn't want to use it, you just wouldn't use it.

It would have zero effect on anything if you don't use it.

[Gwarsbane]

Originally posted by Tyrian Mollusk:

Originally posted by Holmes:

I have an i7 3770s and 8gb of ram, this would ruin my Steam experience. Not everyone has a super duper computer my friend.

A) Even so, the compression wouldn't ruin your experience, on at least some games. Especially if you also don't have an SSD. That "systems are so fast" comment is not some superlative. Chips have been fast enough to run this kind of special fast-decompression alongside the relatively slower data stream coming from disk for a lot longer than you might think.

B) I obviously never said to forcibly apply it to everything. I asked for the option to have Steam manage it for games where we turn it on, or to let us apply post-install commands which would have to be specific to individual games. Either way, if you didn't want to use it, you just wouldn't use it.

It would have zero effect on anything if you don't use it.

So to play a game, I have to wait 20 or 30 minutes while my game decompresses... while also needing the room for the game to decompress... ya no thanks to this.

If you want games to take up less space, ask the game devs to compress their games more... but as mentioned before you're still going to need space to install updates because the games need to be decompressed to have the updates put in them and then recompessed again.

Payday 2 did this... people complained about the size (it was 80ish gigs at the time). So they compressed the game (taking it down to around 35ish), then people started complaining about the time it takes to update the game.... because it has to do what I mentioned above. Now the game is back up to 85ish gigs.... while compressed and updates still take a while.


Also the more compression and decompression that is done on SSDs the more it gets used and the higher the chance its going to cause issues with the SSDs.

It should not be up to Steam to compress a game for you. Also pretty sure this would require the game to be able to handle the files being compressed to play.

[Tyrian Mollusk]

Originally posted by Gwarsbane:

So to play a game, I have to wait 20 or 30 minutes while my game decompresses... while also needing the room for the game to decompress... ya no thanks to this.

No, the system decompresses the compressed files on the fly while reading them to memory from disk. This is not like when you zip files. You don't get max compression, but the system can decompress it much faster. Your CPU does a lot of idling waiting for data from disk. This happens in that time. Sometimes you wait some seconds, if the game uses a large single file, but many times you won't even notice.

And if you do, you just don't use it. It's obviously not a mandatory thing you'd turn on for every game. You don't need to be hostile to options that improve things for people.

If you want games to take up less space, ask the game devs to compress their games more...

Like I said, changing and educating every dev just isn't realistic, and it's not going to be useful, because there are always more devs. Having Steam manage the system compression on the other hand is completely trivial, and improves many space wasting cases.

but as mentioned before you're still going to need space to install updates because the games need to be decompressed to have the updates put in them and then recompessed again.

Yeah, you have to keep that much space free, obviously, but it's not like you keep that space for every game at the same time, The savings is still massive across games, and if you'd rather lose the extra space for some reason, you just don't use it on that game.

Payday 2 did this... people complained about the size (it was 80ish gigs at the time). So they compressed the game (taking it down to around 35ish), then people started complaining about the time it takes to update the game.... because it has to do what I mentioned above. Now the game is back up to 85ish gigs.... while compressed and updates still take a while.

Great anecdote about even pro devs doing a bad job with their updates and compression and why monolithic files are problematic, but I'd still rather updates took a while than a game eats an extra 85GB. Games don't need to update that often for me to burn huge amounts of disk space so it will be slightly faster. This is also something that hopefully would rarely get used on games that big, since devs making those kinds of games ought to have people figuring compression out, and that large of a game might lead to too much time wasted on caching re-decompressed data.

Also the more compression and decompression that is done on SSDs the more it gets used and the higher the chance its going to cause issues with the SSDs.

The decompression is done while reading to RAM, not on disk, so there is no extra disk use going on as you run the game. There's actually less disk use in play, because the system reads less data. The game updates, then changed files get recompressed, and aren't touched again unless they update again. It's not going to change your SSD use.

It should not be up to Steam to compress a game for you.

Steam manages the game installs, and this is a significant OS feature that's easy to add to managing those installs. Steam knows when the files get updated, so it's clearly the best point to do this housekeeping on updates.

Also pretty sure this would require the game to be able to handle the files being compressed to play.

Yeah, you obviously didn't read pretty much everything about how it works. It's called transparent for a reason. The game never even knows the files are stored compressed. It just sees normal files as it reads them. Programs don't look at files on disk in modern systems. They look at RAM copies of files on disk. The trick is just decompressing as the system makes those RAM copies.