Disable disk space preallocation for SSD write-back caches

[Vertex]

I would like an option to disable the preallocation of disk space when installing steam games. I use PrimoCache to create an SSD write-back cache for my HDD storage. Everytime I install a large steam game, the steam client floods and empties my cache.

If there's other concurrent write operations happening my system, it slow them down drastically. Unnecessary write operations like this also kill the nand flash cell on my SSD. When the level 1 ram cache is flooded, it immediately goes to the disk cache.

This feature is very outdated and is not relevant anymore for all use cases. Some users such as myself are writing to a 3TB drive and can feel confident there will almost always be space availible. Furthermore, most users do not have a cache on HDDs and it takes a ridiculously long time for large steam games to preallocate the space.

In all honestly, I'd rather the steam client throw an error if the write operation fails. Currently, it's breaking my disk configuration. This preallocation feature doesn't even work properly on my Windows 10 configuration. The task completes before any space is preallocated. My cache swallows the operation before it has a chance to contact the disk! The steam client doesn't even know if there's enough space to install the game.

Lastly, there is absoutely no performance gain for preallcoation on a SSD write-back cache. Overall there is actually a net loss becuase the procressing is handled by the caching algorithm. It takes anywhere from 30-90s longer to install the game. I have no loss of speed during a download. There is always sufficient resources for procressing.

[MonkehMaster]

100% agree, they need to give an option to disable preallocation.

[KuramaKitsune]

Pre-Allocating path of exile has taken over 30 minutes for a 5GB patch....
this is ridiculous

[[SK] Kaldaien]

Messaggio originale di RiO:

Messaggio originale di Plex:

there is a big myth that pre allocation does wear out SSD nand cells, but I don't think so.
All it does it create 0 byte files, its not actually writing big files.
the activity is just faked to make you think its actually writing big files full of zeroes but it does not do that.

If you use the proper APIs then pre-allocation essentially just sets aside space and is instant. But as pre-allocation is in fact not instant (and some users even get stuck with weird ~20m wait periods while a game pre-allocates), we can deduce that Valve is doing it in -- not the proper way.

That's not entirely the case. If you allow the OS to allocate storage for something on the disk (or even RAM), but neglect to immediately write something a desktop kernel likes to lazily do no actual work.

You just delay figuring out where to stuff things until some later point, and the entire purpose of pre-allocation is to reduce fragmentation so you want to generate page faults across the entire allocated span of memory. If you just rely on the kernel's behavior of zero- or one-filling reserved pages of memory, this whole exercise is pointless :)

[Katze]

Messaggio originale di SK Kaldaien:

Messaggio originale di RiO:

If you use the proper APIs then pre-allocation essentially just sets aside space and is instant. But as pre-allocation is in fact not instant (and some users even get stuck with weird ~20m wait periods while a game pre-allocates), we can deduce that Valve is doing it in -- not the proper way.

That's not entirely the case. If you allow the OS to allocate storage for something on the disk (or even RAM), but neglect to immediately write something a desktop kernel likes to lazily do no actual work.

You just delay figuring out where to stuff things until some later point, and the entire purpose of pre-allocation is to reduce fragmentation so you want to generate page faults across the entire allocated span of memory. If you just rely on the kernel's behavior of zero- or one-filling reserved pages of memory, this whole exercise is pointless :)

Welcome to 2020, in this decade (and actually for the past one too) computers are smarter than you think.
With btrfs, zfs, and other filesystems (which even run on Windows if you try hard enough) you *cannot* rely on any of that behaviour.
Even writing data to disk doesn't mean you have allocated the exact amount of space needed thank to deduplication, compression (an all time favourite even supported in NTFS), and other such nifty caveats.
If you create snapshots on these filesystems regularly you might even encounter the phenomena that deleting files actually *costs* you memory instead of giving it back, making for some super confusing errors.

So we should all come to terms that the very best we can do is simply disable preallocation, check the reported free diskspace against what we expect to consume and make that a skippable *warning* (not a hard error, since we can't rely on it in either positive or negative ways) and implement proper error handling in case the diskspace does run out while we're at it.

Just don't assume anything about a computer. Computers are too complex too assume.

[[SK] Kaldaien]

That's precisely the difference between an RTOS and Desktop/Server OS. RT can never conclude the problem is too complex, it must have finite bounds. Everything else just puts stuff off and hopes the worst-case never happens.

It is good to try and walk a middle ground between these two schools of thought, right up to the point of diminishing returns. If you at least gave 1 second's thought to the problem, you are doing way better than a lot of game engines these days :)

Unity's C# garbage collection is a great example, who the hell knows how frequently it is going to trigger or if the developer responsible for the game even understands what happens during GC. When stuff goes wrong in Unity, it goes _really_ wrong.

[No0b]

"I would like an option to disable the preallocation of disk space when installing steam games."

This is simply a user request that would not be difficult to enact. Steam used to have the ability to not preallocate, if I am remembering correctly.

Either way, the merits discussing if it is necessary do not apply. The argument is sound and the only user to become harmed by it, would be the user who knowingly selected to "not" preallocate disk space. The punitive measure there, would be failed install or download, again to the very same user. I do not believe this should be a default configuration as shipped.

I just do not see the need for the discussion, on the merits or necessity.

[[SK] Kaldaien]

I'd also like an option to reserve the correct amount of storage for pre-loads. They require 3x as much storage or will fail to decrypt. I have gigabit Internet and small SSDs, so I've stopped using pre-loads but it's a very real problem for people who need them.

[Vertex]

Messaggio originale di SK Kaldaien:

I'd also like an option to reserve the correct amount of storage for pre-loads. They require 3x as much storage or will fail to decrypt. I have gigabit Internet and small SSDs, so I've stopped using pre-loads but it's a very real problem for people who need them.

You should create a new thread for that. A solution might be to use your default steam library folder to install game updates. This way you don't need extra SSD space to extract content.

[disconnect]

Should be disabled completely! Who uses HDD for games these days? Very few i think. For SSD pre allocation is totally useless, moreover it does double more wear on drive!
And of course it makes download take much longer time!

[Azreil24]

Was looking for a topic like this as recently I moved exclusively to M2 SSDs and was watching how my writes on the SSD was piling up and basically doubling the writes for each game. Bought the SSD for a week and already on 1TB of writes, mainly due to Steam's stupid pre allocation. I know SSDs got cheaper, but still, no point in wearing them due to a stupid function. OK, for the inexperienced leave it as an option, but before enabling it the app should show a clear message that it causes increased SSD wear.

[Vaupell]

2020 - Still a damn issue.

with fast internet and what not, it now takes LONGER for pre-allocation than the actual download.

Let us disable it, we dont really need it, i mean most people have several TB spare storage these days.

[Jessie]

Messaggio originale di Vaupell:

with fast internet and what not, it now takes LONGER for pre-allocation than the actual download.

how ? It takes a few seconds at most to complete pre-allocation of a few GB on my drive, and a few minutes to download with a gigabit connection...

even at full speed of 100MB/sec it will take about 10 sec for 1GB to download.. I can't imagine pre-allocation being slower than that other than with a slow drive in which case it should take an equal amount of time

[Azreil24]

Messaggio originale di Jessie:

Messaggio originale di Vaupell:

with fast internet and what not, it now takes LONGER for pre-allocation than the actual download.

how ? It takes a few seconds at most to complete pre-allocation of a few GB on my drive, and a few minutes to download with a gigabit connection...

even at full speed of 100MB/sec it will take about 10 sec for 1GB to download.. I can't imagine pre-allocation being slower than that other than with a slow drive in which case it should take an equal amount of time

Happens to me sometimes too, especially on HDD, it takes a damn long time to pre-allocate, for some reason. Can't explain it. For some games it's done usually fast, but for some it goes really slow.

[zzzephire]

This trick should save your SSD lifetime. Steam's preallocation is useless for SSD because it works in another way than HDD. Please note that preallocation is a write operation.
For this trick you'll need 2 physical storage devices. First device is where you install games and second device will be used for preallocations and downloading. In this way you save SSD lifecycle (if it first device).
Our goal is move ...\Steam\steamapps\downloading to second device by using symbolic link[en.wikipedia.org]. To do this open console (cmd or powershell).
Please note to create symbolic link you need to have administrative rights. Or console is opened with administrative rights. Directory 'downloading' must be deleted, 'steam-downloading' must be created
Cmd example:

mklink /d "C:\Program Files (x86)\Steam\steamapps\downloading" "D:\steam-downloading"

PowerShell example:

New-Item -ItemType 'SymbolicLink' -path 'C:\Program Files (x86)\Steam\steamapps\downloading' -value 'D:\steam-downloading'

We did a pointer 'downloading' to directory placed in second device. Steam interpret this pointer (symbolic link) like a directory and works with it as usual

[KuramaKitsune]

i have internet download speeds of 1250 Megabit.
on a 9900k overclocked on water.
with an NVME SSD that has more bandwidth than a fat guys belt.

for the love of god.
PLEASE STOP PRE-ALLOCATING
153MB update taking about an HOUR...