Please stop the preallocation nonsense

[ba0701]

TLDR:

Bottom line, some people, myself included, are suffering a nagging issue with extreme latency during the "pre-allocation" process that occurs during the Steam installation of games and their updates. Some who are experiencing the issue believe it to be related to the Steam client, while others believe it is related to individual PCs and their configurations. As one of those suffering the issue, I have begun a series of tests, and am documenting the outcome, in an attempt to try and identify the root cause.

OP:

My rig:
i9-7940X
32GB ram
Asus ROG Rampage VI Extreme
RTX2080TI x 2
C Drive - 2 x 1TB M.2 SSDs RAID 0
D Drive (where Steam is installed, and Steam games are stored) - 5 x 10TB Seagate Barracuda Pro 7200RPM 6GBS drives in RAID 5
E Drive (random storage) - 1 x 1TB M.2 SSD
Monitor - Asus ROG PG27UQ
Internet speed is Comcast Business 150Mbs, soon to be upgraded to 1Gbs

So, my Steam library contains 1493 games, and given the wealth of storage I have available, which was the purpose of having so many large drives in a RAID 5, I have every single one installed. My problem comes with any time I begin installing my Steam games, and Steam is the only game service out of 5 actively operating on my system (battle.net, Origin, GOG, and Windows 10 Store being the others) that I have any problems with. The particular one that lead me to post this was a 59MB update file, that literally took 27 minutes to preallocate, and an actual 18 seconds to download. I have read, with tremendous frustration mind you, all of the comments to others complaining about precisely the exact same thing I am complaining about now, oinly to be met with rather pointed responses, responses justifying Steam's use of this unneccessary, and clearly broken, process of preallocation. I have read the comments "It's there to make sure you have the proper amount of space for the game, before downloading" "It locks the required free space, to keep any other programs from taking it while the download is happening" ♥♥♥♥♥♥♥t to all of that.

For starters, if you need Steam to police your free space on your HDD, and you attempt to download a game that you do not have adequate resources to store, then you either need to go read Computers for Dummys, take a course to teach you how to properly use the equipment you own, or when all else fails RTFM. However, it is not Steams responsibility to verify that you have the required amount of space available, and to then lock said space away, in order to download a game. Send that game down the pipe, and let the person who hit the download button deal with the fallout if they did not have enough space. I can assure you, if they have to clean up a HDD that has filled up entirely, and as a result do so on an incredibly slow and stuttering PC, they will not make the same mistake twice.

But, the biggest issue here isn't Steam's use of this process that is the problem, it is their implementation of said process that is the problem. However they are performing the steps neccessary to block off the required amount of space, whatever they have written to accomplish this, is horribly coded. One game of 72GB in size may literally take me 30 seconds to preallocate today, and the same game can take hours tomorrow, while the very next update file of 59MB might take 27 minutes (in some cases a few hours for some games). This is horrendous, and entirely uneccessary.

Firstly, this has pushed me away from Steam. Having been on Steam since it began, back when HL2 launched and required you to install it. Back then, I hated Steam, as I was deployed to Iraq and in the beginning Steam forced updates, locking the game out until the update had completed, and our Internet was terrible, which meant in some cases waiting hours to play a game, even for the smallest of updates. Over the years they worked out the kinks, and I have learned to love Steam, being proud of having been a member for so long. It was my favorite of all of the download services, and having the fastest downloads of them all didn't hurt in that regard either. But, then this preallocation crap began, and even the smallest of downloads began taking very extended amounts of time to complete. Where as I used to always look to steam first for all of my game purchases only buying from the other guys when it involved games that Steam did not have. This can be seen in the sheer number of games in my Steam library, compared to the other services, my Steam library has several times more games than all of the other services combined. But recently, because their downloads are much faster, I have begun purchasing games from Origin and GOG when I can, looking only to Steam for games that I cannot get anywhere else. I hate that my library is becoming much less consolidated as a result, but unless I am happy spending hours to download a game, that I can download elsewhwere in mere minutes, and obviously I am not accepting of this option.

I believe that Steam, if they feel the need to offer such a service as preallocation of space, then by all means allow those of us who know how to properly use our PC to opt out of it, at least give us that option. However, given that none of the other services are doing anything like this, personally I do not think such a service is neccessary to begin with.

Please Steam, fix this nonsense!!!

/RANT

[Richard Upton Pickman]

It always bothered me how much time this feature would consume. If they dont change it there is Epic too now. They seem to be launching another service. That being said its never certain that the others will be able to maintain these faster speeds if a big number of people switch to gog, epic and origin.

[Start_Running]

OP You do realize preallocation isn't just about checking free space right?
Secondly Unless something is off with your system Prellocation should never take more than a couple minutes tops. So if its taking anywhere longer than 10 mins , something's off with your Drive, Filesystem or you got a third party app confounding things.

[ba0701]

Originally posted by Richard Upton Pickman:

It always bothered me how much time this feature would consume. If they dont change it there is Epic too now. They seem to be launching another service. That being said its never certain that the others will be able to maintain these faster speeds if a big number of people switch to gog, epic and origin.

Agree entirely, the sheer amount of time this process takes is exceedingly out of sorts with what is trying to be accomplished by it. I also agree that the other services might not continue to be faster, if a lot more people begin using them, and that is very concerning. I have no doubt Steam has the biggest infrastructure, probably several times the size of all of the other combined, and thanks to this process, they are no longer anywhere near being the faster service.

Originally posted by Start_Running:

OP You do realize preallocation isn't just about checking free space right?
Secondly Unless something is off with your system Prellocation should never take more than a couple minutes tops. So if its taking anywhere longer than 10 mins , something's off with your Drive, Filesystem or you got a third party app confounding things.

Yes, I do understand this, as my post clearly states. however, if ytou have knowledge of a reason for such a process, as Steam's Preallocation, that does not involve free space verification, as well as to block said space from being utilized by other programs during the actual download, both of which I clearly stated in my OP, then by all means fill us in.

Given this PC is literally just a couple of weeks from having been powered up for the first time, it is a brand new build, as well as the fact that no other services are having any of these issues, and all of them are running from the exact same drive, the problem has nothing to do with any of the points you mentioned. Also add to it, I am a Network Engineer, was a Systems Engineer before that (several MCSEs), I can assure you the system was properly baselined, as I do with all of my systems when they are first powered up. Every single drive in the array, as well as the Array itself are all picture perfect.

However, I might even be inclined to look at saome of the things you listed, if it weren't for the fact that this exact same issue, along with some other annoying, yet less problematic, issues, are happening on every single system I own, and as this system proves (although not the first to prove this), the problem even occurs with a brand new built PC, with a fresh install of Windows. No, I don't believe NTFS is at fault here either.

It could have to do with, perhaps, the sheer number of games I have simultaneously installed, that might be. However, I have seen nothing from Steam that says we should limit the number of games we have installed simultaneously. Perhaps if I had 10 games installed, instead of 1410, perhaps the problem would not exist. That may very well be the case, but if it is, then that is an issue with the code, not with my system, the size of my library, or anything else. I also have over 600 games installed on both my Xbox One X and my PS4 Pro, and they both run, and download everything perfectly. As does all of the other services I have running on this same computer we are discussing. Downloaded Battlefield V in less than an hour in Origin, then takes me longer than that in Steam to download a 500MB game. Steam is broken, and I am not the only one who has this issue. They need to fix their code, instead of trying to blame the users.

[Activated Rumble]

right now i have 17mb update preallocating for 10 minutes

[ba0701]

Yeah, well, right now I have a 361MB update that has been downloading for 40 mins and counting, after an 8GB game took 50m mins to preallocate. I believe the stop and start of the downloads, as well as the preallocation crap are all realated. I also find it literally impossible to believe, given that this issue has been happening for months, across several different systems, that Steam/Valve is fully aware of the situation, and either it has been done intentionally, which I cannot see a valid reason for this, or they simply are unable to, or don't care enough to, fix the problem. It's really sad, and the worst part is they can get away with treating us like garbage, simply because of the monopoly they have created in the PC gaming realm. We don't even have anywhere else to go, unless we wish to just turn to our consoles.

[ANDO]

The less fragmented your drive is, the faster this process will be finished. That's how it is in my experience, at least. Big games allocate immediately on a freshly defragged drive.

[Baraz]

Steam could begin by quickly looking at the available space. *If* there is plenty, bypass the allocation process. If there is just enough, then impose the allocation process.

[Richard Upton Pickman]

Originally posted by AUTONOMOUS:

The less fragmented your drive is, the faster this process will be finished. That's how it is in my experience, at least. Big games allocate immediately on a freshly defragged drive.

Something you shouldnt do and that wouldnt help on a SSD. You gain nothing. I got a 1TB SSD and while its fine for most titles some do take rather long.

[ANDO]

SSD is a lot less volatile than it used to be, but how many of them are like that now, I wouldn't know, but yes, you still shouldn't defrag an ssd. You should trim that thing, though. That might make allocation faster on those. I never had to wait for allocation on mine, but I've only installed a few games, one at a time on it. So I can't speak to that, but as far as hard drives go, defrag will speed up allocation a lot for steam.

FWIW, auslogics disk defrag has a special SSD optimization algo. I don't typically use it, just on the mech drives, but it's there.

[Richard Upton Pickman]

Again it shouldnt really make a difference and you want to avoid any additional writing and reading on them. Thinking about it i think it was bethesda titles doing that. On the old harddrive stuff like rage.

Though whats said above would be nice. An option to not use that function. Also i did start using gog and other services a while ago and most of them dont take as long. Maybe blizzard still takes some time after certain updates but the others dont tend to take much time.

[ba0701]

Originally posted by Baraz:

Steam could begin by quickly looking at the available space. *If* there is plenty, bypass the allocation process. If there is just enough, then impose the allocation process.

This, exactly! Had an update just today take nearly an hour, for an 11MB update. I assure you it isn't my system, it is 100% Steam, and Steam only. My system is literally only about 1 month old.

[ba0701]

Originally posted by ODINSON:

The less fragmented your drive is, the faster this process will be finished. That's how it is in my experience, at least. Big games allocate immediately on a freshly defragged drive.

Diskeeper 18 is running against my RAID 5 array, but is def not running on the 3 M.2 drives

[Darren]

Originally posted by ba0701:

Originally posted by Baraz:

Steam could begin by quickly looking at the available space. *If* there is plenty, bypass the allocation process. If there is just enough, then impose the allocation process.

This, exactly! Had an update just today take nearly an hour, for an 11MB update. I assure you it isn't my system, it is 100% Steam, and Steam only. My system is literally only about 1 month old.

Why do you think it being less than a month old would mean it's not defective.

I had to return *3* gaming laptops in a row until I got one that worked properly, and this was within days of them being purchased. All were hard drive issues.

Have you checked the cabling on your hard drives to make sure they are in securely?
Have you run a deep scan on your drives to ensure they have no bad sectors?
Have you disabled your virus scanner?
Have you tried disabling Diskeeper 18?
Have you tried disabling all other software other than Steam, and your Operating System and tried to install a game then?

How pre-allocation works it physically should not take longer than 10 minutes, it just marks the disk blocks as in use, this is like a handful of bytes per block. It doesn't actually write any data to the blocks during this.

If it's taking an hour something is getting in the way, a virus scanner, or some disk utility.

[ba0701]

Originally posted by Darren:

Originally posted by ba0701:

This, exactly! Had an update just today take nearly an hour, for an 11MB update. I assure you it isn't my system, it is 100% Steam, and Steam only. My system is literally only about 1 month old.

Why do you think it being less than a month old would mean it's not defective.

I had to return *3* gaming laptops in a row until I got one that worked properly, and this was within days of them being purchased. All were hard drive issues.

Have you checked the cabling on your hard drives to make sure they are in securely?
Have you run a deep scan on your drives to ensure they have no bad sectors?
Have you disabled your virus scanner?
Have you tried disabling Diskeeper 18?
Have you tried disabling all other software other than Steam, and your Operating System and tried to install a game then?

How pre-allocation works it physically should not take longer than 10 minutes, it just marks the disk blocks as in use, this is like a handful of bytes per block. It doesn't actually write any data to the blocks during this.

If it's taking an hour something is getting in the way, a virus scanner, or some disk utility.

Because, if you took the time to read the thread, you would see that this issue is only happening with Steam. I have GOG, Origin, Battle.Net, and Windows Store games, and not a single one of them has any issues, in fact they download and install lightning fast, which would be the purpose behind a new PC. You can also put a check on the fact that perhaps I don't understand what I am doing. I have had 3 MCSE's, first one was NT4.0 and I got it in 1998, 2 CCNPs, and 4 CCNAs, been a Systems/Network Engineer since 1997, so I can assure you, people actually pay me money to fix and configure their systems. All of my new build, as well as current system baseline data, which likely exceeds anything you or any other normal person might do with their new system, is perfectly fine, and shows very impressive numbers across all matrixs.

By the way, all of your recommendations are tier 1 suggestions, I am way past that at this point, but thanks for trying.

[MonkehMaster]

yup, i agree with the OP.

we dont need this "preallocation" process, its super annoying and takes way to much time randomly.

also people saying "its your system" or something like that are false, then you have people saying "other programs" are to blame, thats false aswell, you cant say oh have you turned your antivirus off and tried... what if i dont have an antivirus, so whats the next program to blame.

------------------------------------------------------------------------------------------------------------------

here is an example of the blame game, i had an issue (about a year or so ago) that was a widely known issue with steam for a year or more, steam was constantly showing me as logged out and when i tried doing anything profile related or other certain things that required you to be logged in, i would be met with "you need to login" or "this profile is private", only way to get around it was to constantly logout and back in or spam click on the profile picture that used to be in the bottom right corner of the steam window, i came to steam discussion several times and steam support all getting generic copy paste nonsense and very frequently a "list" of programs steam didnt like, later on during that issue i built my own desktop pc and had none of those programs on that "list" or an antivirus and still had that issue for months, well guess what fixed the issue.... its was steam with an update and the problem was solved.

tdlr: stop playing blame games... pointing fingers at everything other than steam (or games for that matter) and trying to sell the idea that "its on your end", thats almost always false, and in the case that it is actually on "their end" its usually an easy fix.

------------------------------------------------------------------------------------------------------------------

this "preallocation" stuff is an issue across all types of hardware, new and old and seem to me like some sort of process to slow down or limit steam users to help with bandwidth and server issues on steams end.

no reason to "preallocate" something on a persons pc, when 90% of them know they have enough space and even if they didnt, whats the harm in a download not finishing, you simply free up space and redownload.

also this "preallocation" puts triple the wear and tear on our expensive ssd's for no good reason other than to save someone from having to redownload again because they ran out of space, i certainly dont need, a "preallocation" being written, then a download being written, then a install being written on my ssd for a single update/patch for every single update from all of my games.

that is way to much wear and tear if you add that all up according to your games and said games update process.

another thing that is an issue is steam updates constantly dropping out going from (in my case) 25MB (i have 200mb internet speed) to literally 0 and jumping up and back down (rinse and repeat), this doesnt happen consistantly and i havent tested if it happens during certain hours yet, but i do think this is connected somehow to these recent issues being posted with "preallocation" and "download constantly dropping" as these have been flooding in since it started.

ill repeat this tho, its not my hardware or my internet, its not programs on my pc, its certainly not driver related, everything is up to date, reinstalling doesnt fix it and certainly isnt a worthy option given the issues mentioned, so whats left to blame?

steam....