An option to disable space preallocation

[TorMazila]

Hi!
I have large part of my libraries served over iSCSI (it's like a physical drive but accessible over network; actually it's RAIDZ and RAIDZ2 sets of several drives - so I can painlessly live with loosing 1 or 2 drives) - and that means I'm stuck at sub-gigabit speeds. Which is almost ok for everyday usage but installing anything large becomes a huge waste of time for doing nothing but "allocating space" like it'll help anything.

Disabling this option will allow starting the downloads right away and may also save write cycles for SSD drives.

Edit: March 19, 2020. The Steam BETA has stopped preallocating disk space and just downloads the games "the normal way"
April, 2020: Seems not so: it definitely preallocates space prior to patching. World of warships update on Steam has benchmarked my SSD :\.

[JPMcMillen]

The point of 'allocating space' is to make empty files that will be filled with data as it is installed. This is to make absolutely sure that not only is there enough room before the installation starts, but that nothing that happens during the installation can suddenly take space that the installer thought it was going to have access too.

Also, given that very few people are trying to install games on a network drive, I doubt Valve will spend the time to change how downloads/installs work.

[Washell]

With a niche setup come niche problems. You fix them yourself or you live with them. It's unreasonable to expect a company to accommodate you. In a regular setup the pros of doing this outweigh the cons, as JPMcMillen explained.

[chrcoluk]

it seems preallocating causes memory usage to skyrocket when downloading, using the normal installer gets round this by not downloading until the allocation is done but when using download_Depot, the download is done whilst preallocating, and the downloaded content is stored in the steam internal cache until the allocation is done, if it exceeds 3.5gig then stea will silently crash after hitting the 32 bit memory limit.

So yes there is a valid reason to disable pre allocate.

[TorMazila]

Originally posted by JPMcMillen:

The point of 'allocating space' is to make empty files that will be filled with data as it is installed. This is to make absolutely sure that not only is there enough room before the installation starts, but that nothing that happens during the installation can suddenly take space that the installer thought it was going to have access too.

Also, given that very few people are trying to install games on a network drive, I doubt Valve will spend the time to change how downloads/installs work.

Creating empty files isn't going to "absolutely" guarantee anything but wear cycles for SSD and delay prior to start of the download. I guess you're aware of NTFS compression - and there are even more complications with that - you can write more than you had free space or, sometimes, you can actually write less because the weird way NTFS compression works (write uncompressed, write compressed, delete uncompressed data).

I believe with NAS becoming more and more common thing at home (yep, costs a fortune to have 4-6 HDDs) and growing network speeds - we will see people increasingly having similar setups. E.g. it's VERY convinient to just share a snapshot-based clone of your library to your kids PC - so you have "all the games" and don't waste space for dups.

Originally posted by chrcoluk:

it seems preallocating causes memory usage to skyrocket when downloading, using the normal installer gets round this by not downloading until the allocation is done but when using download_Depot, the download is done whilst preallocating, and the downloaded content is stored in the steam internal cache until the allocation is done, if it exceeds 3.5gig then stea will silently crash after hitting the 32 bit memory limit.

So yes there is a valid reason to disable pre allocate.

Originally posted by chrcoluk:

it seems preallocating causes memory usage to skyrocket when downloading, using the normal installer gets round this by not downloading until the allocation is done but when using download_Depot, the download is done whilst preallocating, and the downloaded content is stored in the steam internal cache until the allocation is done, if it exceeds 3.5gig then stea will silently crash after hitting the 32 bit memory limit.

So yes there is a valid reason to disable pre allocate.

Your case seems your HDD is dying. Preallocation just means you get things written as fast as they can, contrary to slow-speed download.

[chrcoluk]

hdd is not dieing, since I made that post I did some more work on diagnosing the specific issue I have/had and a bug report has been escalated to steam developers.

Bear in mind I was preallocating a 35 GIG file. That takes time on a spindle. I also was downloading at 9.3mB (megabytes) a second. Steam can only use less than 4 gig of memory due to been a 32bit process, in my case I filled up its memory limit with downloaded data before the preallocate completed, I got round it by capping my download speed, and then when the preallocate was completed and steam had written out the cached downloaded data I uncapped the download speed.

But seems the issue is bigger then I thought as some other people replied to my bug report getting issues downloading via the UI, as when they downloaded, it didnt wait for preallocation to finish and they couldnt launch their games as the downloads had yet to be written to disk.

Steam either need to disable preallocate or make the app wait for preallocate to finish before downloading.

Preallocating shouldnt really speed up a download at all, as writes still need to be carried out, for a SSD its bad as obviously it causes twice as many writes to occur increasing wear, for a hdd its neutral, doesnt speed things up, but does no real harm either (assuming the downloaded data gets written on the same sectors as the preallocated). I would think the only benefit to steam is making sure there is enough reserved space allocated to complete the download incase something else on the system tries to use the space whilst downloading.

Also to add my hdd is 80% full, so probably not much defragged free space and likely al writes on the slowest part of the spindle. hdd's slow down a lot as they fill up.

if you curious the bug report is here

https://steamcommunity.com/groups/SteamClientBeta/discussions/0/1621724915784376270/

[qd3v]

Yeah, more and more people have SSD now, and pre-allocating - is not a healthy operation. I dont' care about download speed (as far I as I understand it works like torrents here, they too pre-allocate space on HD)
At least I want to chose to keep it on or off.
This is especially important then downloading games like DOOM/GTA V with 70+GB (and more games like this are already in the market). Writing 140 GB instead of 70 - not good, not good.

[Snapjak]

SSDs failure over time is GROSSLY overestimated. All modern SSDs have write lifetimes that would optimally surpass YOUR life even with above average usage.

[Satoru]

Originally posted by vanburg:

Yeah, more and more people have SSD now, and pre-allocating - is not a healthy operation. I dont' care about download speed (as far I as I understand it works like torrents here, they too pre-allocate space on HD)
At least I want to chose to keep it on or off.
This is especially important then downloading games like DOOM/GTA V with 70+GB (and more games like this are already in the market). Writing 140 GB instead of 70 - not good, not good.

Unless you're Carbonite or Google there is no way a simple preallocation of files is going to do anything to your SSD. Writing to your SSD is normal. And steam's preallocation of files is again normal.

The idea that its magically wearing down an SSD is like saying taking a picture on your phone is wearing out your storage so you should stop taking photos.

[qd3v]

Can I make myself clear: don't touch my SSD more than it needs to! It's not your's it's mine.
P.S. And yeah, we change phones nowadays MUCH more often than SSDs, can I save my dollar at SSD here, please?

[Snapjak]

Your "habits" should not impede other user's normal behaviour at all.

[qd3v]

Originally posted by Snapjak:

Your "habits" should not impede other user's normal behaviour at all.

I pointed above: let me choose would I want to record twice amount of data, or not. I don't care about others too, man

[Start_Running]

Originally posted by TorMazila:

Originally posted by JPMcMillen:

The point of 'allocating space' is to make empty files that will be filled with data as it is installed. This is to make absolutely sure that not only is there enough room before the installation starts, but that nothing that happens during the installation can suddenly take space that the installer thought it was going to have access too.

Also, given that very few people are trying to install games on a network drive, I doubt Valve will spend the time to change how downloads/installs work.

Creating empty files isn't going to "absolutely" guarantee anything but wear cycles for SSD and delay prior to start of the download. I guess you're aware of NTFS compression - and there are even more complications with that - you can write more than you had free space or, sometimes, you can actually write less because the weird way NTFS compression works (write uncompressed, write compressed, delete uncompressed data).

Me thinks you should look into how pre allocation works. The files are empty in that no actual data is in them but they do take up space. I.e the clusters and blocks are alloicated for the file. It's the opposite of on demand allocationb which while faster can cause a lot more problems.

Also it has no effect on the wear cycles of your ssd . Since no data is actually writted during preallocation, it does no wear the drve.

[Snapjak]

Originally posted by vanburg:

Originally posted by Snapjak:

Your "habits" should not impede other user's normal behaviour at all.

I pointed above: let me choose would I want to record twice amount of data, or not. I don't care about others too, man

Alright, if you want to be able to do that it's fine. Just remember that the whole SSD lifetime write thing is blown way out. Unless of course you're still using a really old SSD. That's a totally different story.

[Satoru]

Originally posted by vanburg:

Can I make myself clear: don't touch my SSD more than it needs to! It's not your's it's mine.
P.S. And yeah, we change phones nowadays MUCH more often than SSDs, can I save my dollar at SSD here, please?

That's just silly. Steam writes files to your hard drive. That is what the hard drive is for.

Steam is not doing anything weird or excessive. Its simply preallocating files.

To say its 'your' hard drive is pointless. It also shows the sort of vast ignorance as to what you're even complaining about.

Steam is nto doing anything a normal application should not do. steam is not going to 'not touch' your SSD because you have some gross misconception as to what is or is not appropriate.

You might as well tell Steam to slow down its downloads because its "wearing out your network card" because that is just as ludicrous.

Youc an make yourself 'clear'

And others are free to point out that is as 'clear as mud' and is a gross misunderstanding of literally everything imagineable.

[qd3v]

Originally posted by Start_Running:

Also it has no effect on the wear cycles of your ssd . Since no data is actually writted during preallocation, it does no wear the drve.

I really doubt about it, marking sectors/blocks as written (allocated) should took .1 of the second then. But pre-allocating really writes, even if zeroes.

Originally posted by Snapjak:

Originally posted by vanburg:

I pointed above: let me choose would I want to record twice amount of data, or not. I don't care about others too, man

Alright, if you want to be able to do that it's fine. Just remember that the whole SSD lifetime write thing is blown way out. Unless of course you're still using a really old SSD. That's a totally different story.

Of course I know that, and yes, it's 3 years old already.