Please stop the preallocation nonsense

[ba0701]

TLDR:

Bottom line, some people, myself included, are suffering a nagging issue with extreme latency during the "pre-allocation" process that occurs during the Steam installation of games and their updates. Some who are experiencing the issue believe it to be related to the Steam client, while others believe it is related to individual PCs and their configurations. As one of those suffering the issue, I have begun a series of tests, and am documenting the outcome, in an attempt to try and identify the root cause.

OP:

My rig:
i9-7940X
32GB ram
Asus ROG Rampage VI Extreme
RTX2080TI x 2
C Drive - 2 x 1TB M.2 SSDs RAID 0
D Drive (where Steam is installed, and Steam games are stored) - 5 x 10TB Seagate Barracuda Pro 7200RPM 6GBS drives in RAID 5
E Drive (random storage) - 1 x 1TB M.2 SSD
Monitor - Asus ROG PG27UQ
Internet speed is Comcast Business 150Mbs, soon to be upgraded to 1Gbs

So, my Steam library contains 1493 games, and given the wealth of storage I have available, which was the purpose of having so many large drives in a RAID 5, I have every single one installed. My problem comes with any time I begin installing my Steam games, and Steam is the only game service out of 5 actively operating on my system (battle.net, Origin, GOG, and Windows 10 Store being the others) that I have any problems with. The particular one that lead me to post this was a 59MB update file, that literally took 27 minutes to preallocate, and an actual 18 seconds to download. I have read, with tremendous frustration mind you, all of the comments to others complaining about precisely the exact same thing I am complaining about now, oinly to be met with rather pointed responses, responses justifying Steam's use of this unneccessary, and clearly broken, process of preallocation. I have read the comments "It's there to make sure you have the proper amount of space for the game, before downloading" "It locks the required free space, to keep any other programs from taking it while the download is happening" ♥♥♥♥♥♥♥t to all of that.

For starters, if you need Steam to police your free space on your HDD, and you attempt to download a game that you do not have adequate resources to store, then you either need to go read Computers for Dummys, take a course to teach you how to properly use the equipment you own, or when all else fails RTFM. However, it is not Steams responsibility to verify that you have the required amount of space available, and to then lock said space away, in order to download a game. Send that game down the pipe, and let the person who hit the download button deal with the fallout if they did not have enough space. I can assure you, if they have to clean up a HDD that has filled up entirely, and as a result do so on an incredibly slow and stuttering PC, they will not make the same mistake twice.

But, the biggest issue here isn't Steam's use of this process that is the problem, it is their implementation of said process that is the problem. However they are performing the steps neccessary to block off the required amount of space, whatever they have written to accomplish this, is horribly coded. One game of 72GB in size may literally take me 30 seconds to preallocate today, and the same game can take hours tomorrow, while the very next update file of 59MB might take 27 minutes (in some cases a few hours for some games). This is horrendous, and entirely uneccessary.

Firstly, this has pushed me away from Steam. Having been on Steam since it began, back when HL2 launched and required you to install it. Back then, I hated Steam, as I was deployed to Iraq and in the beginning Steam forced updates, locking the game out until the update had completed, and our Internet was terrible, which meant in some cases waiting hours to play a game, even for the smallest of updates. Over the years they worked out the kinks, and I have learned to love Steam, being proud of having been a member for so long. It was my favorite of all of the download services, and having the fastest downloads of them all didn't hurt in that regard either. But, then this preallocation crap began, and even the smallest of downloads began taking very extended amounts of time to complete. Where as I used to always look to steam first for all of my game purchases only buying from the other guys when it involved games that Steam did not have. This can be seen in the sheer number of games in my Steam library, compared to the other services, my Steam library has several times more games than all of the other services combined. But recently, because their downloads are much faster, I have begun purchasing games from Origin and GOG when I can, looking only to Steam for games that I cannot get anywhere else. I hate that my library is becoming much less consolidated as a result, but unless I am happy spending hours to download a game, that I can download elsewhwere in mere minutes, and obviously I am not accepting of this option.

I believe that Steam, if they feel the need to offer such a service as preallocation of space, then by all means allow those of us who know how to properly use our PC to opt out of it, at least give us that option. However, given that none of the other services are doing anything like this, personally I do not think such a service is neccessary to begin with.

Please Steam, fix this nonsense!!!

/RANT

[Start_Running]

MonkehMaster の投稿を引用：

in many games it clearly shown the load times are far beyond a "few seconds"

Different setups, different results.

not only that it helps the gpu get gfx stuff quicker thus helps with fps (not much, but a couple fps is better than none) and stuff the gpu needs to render, also put less work on the whole system when needing information that it would usually have to wait for when using an hdd.

I highly doubt that. Your GPU doesn'ty have any interaction with the drives... textures and ops are read into system ram and then copied into VRAM. And again this is done in fixed regular intervals like when you enter a new map area. The only reason your Drive would affect your FPS is if its constantly paging data which basically means your system needs more RAM.

this isnt even including games and/or software in general that uses paging file (some wont even run without a paging file), games like cities skyline for example will heavily use the paging file and on an hdd that cause sever stutter and fps drops when its on an hdd, also the OS uses the paging file as well, so when a game and the OS are using a paging file its cause system slow downs among other issues.

HAve't play skyline but I've yet to encounter a game that made such an oddly ass backwards use of Datastreaming.

[quote[hdd's are for storage, nuff said...[/quote]
And games require a ♥♥♥♥♥♥♥ of storage space :).

[McGillicutti]

I am going to apologize to the OP for reposting post #166 but it seems with someone asking if I know something like they don't, it is necessary to re-post this in hopes a few folks go to the links (with a typo or 2 I just noticed fixed hahaha), and some additional information at the end that I hope is helpful:

McGillicutti の投稿を引用：

To the OP more specifically and hopefully for some helpful insights for others as well I'm sharing some info I ran across.


Essentially, like many terms and some assumptions of context and meaning therefrom preallocation is being treated like some sort of crazy idea. It seems a good point to mention that Windows adopted Linux/Unix use of preallocation to minimize fragmentation and it also aids in security. I believe the following information may help explain why too.

So please enjoy these links unless they're removed, in which case I've made a commentary to hopefully convey the gist of the links I am unable to share with you:

-- Regarding Steam specifically, https://gaming.stackexchange.com/questions/333991/what-does-it-mean-when-a-steam-download-says-preallocating. << This means windows or some 3rd party program you use doesn't have to optimize the file later, if you're picky about that sort of thing and do optimize yourself.

--https://www.reddit.com/r/DataHoarder/comments/b2xxnn/preallocate_for_torrenting_does_it_help_hdd_health/ << As you can see the debate over its use in regard to hard drive wear rages, but the agreement is that preallocation, allocating the file locations at the time of download means there is little chance of the files getting fragmented, this can maybe make updates easier too.

-- https://www.steamgifts.com/discussion/026cN/allocating-disk-space-is-taking-ages-lately << one poster pipes up about the fact their drive is only 1% fragmented ever due to Windows and one of the other posters explains that preallocation is about the number of files and not the amount of data.

Many here who are older may remember since Windows 98 one of the easiest speedups anyone can do is to go in and allocate the specific 1.5x of RAM in Virtual Memory size.... Well that was all of us locking the sectors on our hard drive, protecting them from ever being overwritten, and assuring the hard drive space was always there. That was preallocation then, and the way it is used today includes indexing the locations to specific files so that the download writes the files in a way that optimizes them. The more files there are in the actual download package or decrypted file, the longer the preallocation will take. So while it may seem it's a Steam specific issue, as someone after reading the last thread I posted above posts:

"So does anyone have an idea what's causing this a year later?
My allocating times are not on the level the OP's are but it still takes easily an hour+ for games like Hitman that's 70GB and Total war war titles that tend to be 50GB+.
In my case it isn't the drive either, it's a rather new SSD drive with enough free disk space, my torrent client is set to pre-allocate as well and does this is seconds unlike steam even when downloading large files.

"I'm wondering if this is a steam related problem or a win10 settings problem, too bad I deleted my secondary OS (win7) or I could have tested if it was as slow on both operating systems."

You can see they don't understand that preallocation has to do with the file package itself, its indexes, and the number of files that comprise the whole download, meaning that a Dev who made their particular program require a rather large file index is the one that is responsible for what appears to be unreasonable preallocation delay.

As I don't know personally I'll say that it may be true buying the product somewhere else and dowloading it from them may not result in the same preallocation and its delays, however the game itself may have more lag when played. Again I am not sure but mentioning because the preallocation is probably a standard the Developer is well aware of with Steam but where others don't use it the Developer probably isn't going to set the download file to follow a preallocation of where it will be (with exception of Linux that always preallocates from what I understand in reading up on this).

It is also worth noting that lack of preallocation can lead to the operating system optimizing the file. If your system is a bit taxed when multi-tasking, which most gaming rigs aren't, the preallocation can save you the multitasking of the OS while playing the game imposed by the operating system in optimizing in the background.


I do hope you can see the links but understand why they might get removed (Note to Steam Community Moderators if any of the links I posted is taboo on Steam please absolutely do remove them and let me know so I don't make that mistake again, thank you) and hope you'll look into the point of preallocatoin to get a better idea what causes it to delay but how that can be a big timesaver (and lagsaver) in the long run. This doesn't mean Steam may not have an issue of some sort making it worse so everyone's diligence to look at Steam is definitely wise, just saying that there may be more to it than at least some of us realized and figured sharing that might be helpful to some folks.

I wish you all well and know of no other way to hopefully help folks learn about what has been in Linux since the beginning and is a big factor in its speed and not needing optimizations.

In my days coding we had sequential and random access text files. Sequential had to load into RAM and there wasn't much in virtual memory on micro computers at that time, so if they file exceeded RAM, you couldn't load it.

Random Access text files required the coder to set the number of fields, the type of field, string, integer, floating point, etc., and also the length of the record itself. If you exceeded the record size, oh well you can't write the record to the database, and this also meant there were a lot of empty spaces left where information wasn't filled in.

Sequential files would have no index (unless the coder made one) and often due to the size of the database itself, you had little room to execute the search, so often the records took more time to bring up. If the file fit in memory, however, a shell sort would be almost instant.

With Random Access files finding a record was almost instantaneous no matter how big the database became because the record sizes were all uniform, and any index was specific to a field of fixed length etc. vs the code, essentially, "reading" the file looking for a particularly searched string.

Preallocation is the equivalent of a Random Access system with each file being a record so there is no fixed length, but setting specific allocation based on the DEVELOPERS view that these locations will be optimal.

Essentially the great uniformity that helped Random Access find records faster is what's being implemented, with an allocation system that assures no need to optimize the files.

Where it gets unwieldy is when the number of files is huge, essentially making the file index a significant sized database all its own, and that only exacerbates delay when coupled with the verification of each file's integrity prior to each preallocation, again this is pursuant to the Developer's specifications in how they structured the files for their product and the optimal access to those files.

This is the best I could do to try to explain what knowledge I am operating from.

Good luck to the OP and anyone else with is issue and I do hope this helps better appreciate and understand where to focus your complaints about it, to the Developer.

[pasa]

Start_Running の投稿を引用：

pasa の投稿を引用：

From the reports it is quite evident that the issue has nothing to do with
"preallocation" that is stuck in the message. As long as folks keep bashing that bush and create theories derived from that, we will see no progress to figure out what is relevant, what is the problem so maybe find some fix too. just ranting and blaming will hardly create progress.

Yeah but likewise.,.. how do you prove its simply stuck at the message.. granted that is a possibility and likely is the case in some scenarios.

I suggested plenty of methods to start investigating upstream. This far zero reports on any of them even attempted. Others can be figured too.

The small amount of actual info we have suggests no disk activity in the "stuck" phase, that implies preallocation is already over. procmon timestamps could provide hard proof.

[SeriousCCIE]

Does the observed, lengthy like the OP cited, preallocation phase affect the same people when receiving an update of any size?

I just received an update for total warhammer II, a 6.5GB update, and experienced nothing out of the ordinary about it. That doesn't mean something is preallocated at the start of steam's launch when updates are queried based on library contents.

It could be that the update process itself is prepared ahead of the actual start of the download, though. But I have no idea, so I thought I'd offer my anecdotal experience to see if it matches others experiences.

[MonkehMaster]

Crazy Tiger の投稿を引用：

MonkehMaster の投稿を引用：

yet modders and devs told me what i mentioned.... yes please make up more stories.

You believe what you want, I don't care. Like I said, I know what modders and devs told me on the Paradox forum, which is what actually matters as they do know what they're talking about.

Your posts are mostly filled with nonsense which deflects from actually wanting to solve the issue outlined in the thread. It does appear you rather argue and throw blame than actually solve this thing. Which is sad, actually.

yet modders and devs told me what i said, which is what matters as they know more than you do.

your comments are nonsense, stop trying to derail the thread.

[MonkehMaster]

Start_Running の投稿を引用：

MonkehMaster の投稿を引用：

in many games it clearly shown the load times are far beyond a "few seconds"

Different setups, different results.

not only that it helps the gpu get gfx stuff quicker thus helps with fps (not much, but a couple fps is better than none) and stuff the gpu needs to render, also put less work on the whole system when needing information that it would usually have to wait for when using an hdd.

I highly doubt that. Your GPU doesn'ty have any interaction with the drives... textures and ops are read into system ram and then copied into VRAM. And again this is done in fixed regular intervals like when you enter a new map area. The only reason your Drive would affect your FPS is if its constantly paging data which basically means your system needs more RAM.

this isnt even including games and/or software in general that uses paging file (some wont even run without a paging file), games like cities skyline for example will heavily use the paging file and on an hdd that cause sever stutter and fps drops when its on an hdd, also the OS uses the paging file as well, so when a game and the OS are using a paging file its cause system slow downs among other issues.

HAve't play skyline but I've yet to encounter a game that made such an oddly ass backwards use of Datastreaming.

[quote[hdd's are for storage, nuff said...[/quote]
And games require a ♥♥♥♥♥♥♥ of storage space :).

where do you think textures ect.. come from, they come from the files in the game installed on your drive,which are then read when needed, so yes the gpu does interact with the drive, it has a huge factor in how quick the gpu gets its info, thats why you can easily see a difference in render times and fps on low end systems going from hdd to a ssd (heck even an msata).

tbh, unity engine is quite bad, for some reason the devs using unity seem to think forcing use of paging file is good, they also use this awful process called garbage collection which causes stuttering and fps drops and when the stutter is to long it will cause rubberbanding / desync, which is a huge problem in rust and everyone complains about it.

they may require a good deal of storage space but ssd's have gotten much bigger, no reason to be clinging to old hardware such as hdd's, unless ofc your using it strictly for storage (ie.. pictures, movies/vids, recordings, any sort of text files, or backups), gaming on them is just not standard now a days and rightfully so and the only people i see using them for gaming is budget builders or people still clinging to old outdated pc's, i guess i can say data servers as well, but we are talking gaming here.

[voidCaster]

SeriousCCIE の投稿を引用：

Does the observed, lengthy like the OP cited, preallocation phase affect the same people when receiving an update of any size?

I just received an update for total warhammer II, a 6.5GB update, and experienced nothing out of the ordinary about it. That doesn't mean something is preallocated at the start of steam's launch when updates are queried based on library contents.

It could be that the update process itself is prepared ahead of the actual start of the download, though. But I have no idea, so I thought I'd offer my anecdotal experience to see if it matches others experiences.

After an initial pre-allocation, there's not much you can do to keep files from fragmenting if your update actually expands the size of the file on disk. You have to assume that your update comes days or weeks later, and that the user has added new files to disk in that period, so you can expand the file via immediate/unfragmented preallocation but short of running defrag on the whole disk first you can't guarantee that the new data isn't fragmented from the original.

On the upside, unless you're doing a major overhaul with added features and content (or you're just a really bad developer) most of your patches and updates should fit into the address space that's already there. You're fixing little things, changing a +1 to a +2 for example to fix a bug or make the "Easy" gameplay level even easier, so while the file changes its length does not.

[^èa†èr^]

if you dont defrag it is a pain in the but for steam. or if you are full on the disc.

[MonkehMaster]

^ea†er^ の投稿を引用：

if you dont defrag it is a pain in the but for steam. or if you are full on the disc.

lol

[ba0701]

TLDR:

Having rebuilt the entire system from the ground up, there is a huge improvement in performance. However, while occurring much less, there is still some latency in the Pre-allocation process from the Steam platform. Other platforms do not experience such latency.



Start of comment:

I have finally finished my restore, whew! You really have no idea everything that has happened during all of this time, but I finally finished what we have set out to test. I will list what I did, along with my results.

Steps taken:

I began by deleting the RAID 5 array, separating the drives into 5 singular 10TB drives, and have spread my Steam library equally across 4 of them. But after having deleted the drive array, I began having issues with Windows itself after a large Microsoft update that occurred at the same time. So, instead of trying to figure out the Windows issue, or even uninstalling the update, it dawned on me that perhaps a fresh install of the OS might also aid in our testing, so at that point the C drive was wiped clean, and a fresh Windows install was performed, and in keeping with the test, I did not install any AV or any other programs for that matter. I then installed Steam. My C drive remains a RAID 0 across two M.2 SSDs, as I did not have issues with it, however, should the primary system drive respond poorly again, to another Windows update or in general, I will remove that array as well, and run the OS on a singular drive as well.

I began restoring my games library from the backups I had created. Although this takes up a large amount of space, and in my case I had to buy 3 USB 8TB, I highly recommend this for folks with large libraries. If I had downloaded my entire library, it would have taken me at least two weeks to complete, as it stands the restore took me less than 4 days.

On the very first batch of restores, I experienced high latency during the Pre-allocation process for SOME games. I also found that the issue appears to happen with those same games repeatedly, ie: the aforementioned Age of Fear.

HOWEVER, I see huge improvements across the board in several areas. For starters the Pre-allocation process, for the games that do still experience the problem, the latency is in fact improved. Such as during the first batch of restores, it included all of the As (restored as I backed up, alphabetically), which also included all of the Assassin's Creed games. The pre-allocation for some of those particular games was far more than some other games of like size, between 20-30 minutes. On severe offender is Remnant From The Ashes, no matter what is happening in that game the Pre-allocation process if vastly exaggerated as compared to other games of like size. For instance, a 340MB update for that game might take upwards of 30 minutes to complete.

Secondly, and this is a big one, I am no longer experiencing the broken up downloads, that is the stopping and starting up of retrieving data from the Steam servers. This issue alone had, at times, added hours of time to my downloading of games and updates. While it happened nearly every time something was downloaded, to what extent fluctuated greatly. Now, that is not to say that this issue shall not return, but I certainly expected to see it during the restore process, and I did not observe it happening one single time.

Third overall system wide performance appears to be greatly improved. Since Steam is literally the only thing installed, outside of Windows and the required drivers, it is the only thing I have to base my observations on. But, the disk activity to the drives that were once members of the array, from inside of the Steam app, has improved by no less than 4x. Whereas, under the array I would see Disk Activity measured at 54MB/s and below, Steam now lists this same activity at sustained between 180-220MB/s, with bursts to over 300MB/s.

My take away:

While the pre-allocation process latency is still occurring, it is much less exaggerated than previously experienced. This latency appears to be relegated to particular games now, where previously it affected those same particular games as well as countless others. However, I still am not experiencing a like issue from any of the other gaming download sights, this issue still appears to only affect Steam downloads. I know someone had asked if I had the same game to test across two separate platforms, and in fact I do, I have several, some through the Steam-GOG cross platform offerings, and others simply by mistake or intentionally purchased for other reasons, and I tested a few. None of these games have any issue when downloading/updating from other platforms, such as Singularity on GOG, but does experience some pre-allocation latency in Steam. Using the on-board RAID option appears to impact this issue greatly. Given that every previous system I experienced this issue on had an array of some sort, it appears that may well have been the source of many of my frustrations. It would seem that something Steam does inside of the Pre-allocation process differs from what other like platforms are doing, and is much more sensitive to the environment in which it operates.

Other areas of the system performance have vastly improved, and overall I am much happier with the performance of my system. To date I have had very few updates (previously I experienced near daily updates that exceeded 20 games a day), and I will report back over time as to if this remains the status quo. I will be happy to run any particular benchmark/analytics anyone might wish to see, just tell me what it is you would like to see specifically, and I can provide it.

So, there you have it, things are greatly improved across almost all areas, but the pre-allocation latency does still exist in some games, however to a much lesser extent than previously experienced. I do not experience a 30 minute wait before any game download begins inside of other platforms, but I do experience this, at times now, during the "Pre-allocation" process of Steam. However, even that 30 minutes is a vast improvement from the previous experience, where some games would take literal hours. I am much happier with my system's overall performance, as well as that of Steam, than I was when I started this thread!!

[McGillicutti]

ba0701 の投稿を引用：

TLDR:

Having rebuilt the entire system from the ground up, there is a huge improvement in performance. However, while occurring much less, there is still some latency in the Pre-allocation process from the Steam platform. Other platforms do not experience such latency.



Start of comment:

I have finally finished my restore, whew! You really have no idea everything that has happened during all of this time, but I finally finished what we have set out to test. I will list what I did, along with my results.

Steps taken:

I began by deleting the RAID 5 array, separating the drives into 5 singular 10TB drives, and have spread my Steam library equally across 4 of them. But after having deleted the drive array, I began having issues with Windows itself after a large Microsoft update that occurred at the same time. So, instead of trying to figure out the Windows issue, or even uninstalling the update, it dawned on me that perhaps a fresh install of the OS might also aid in our testing, so at that point the C drive was wiped clean, and a fresh Windows install was performed, and in keeping with the test, I did not install any AV or any other programs for that matter. I then installed Steam. My C drive remains a RAID 0 across two M.2 SSDs, as I did not have issues with it, however, should the primary system drive respond poorly again, to another Windows update or in general, I will remove that array as well, and run the OS on a singular drive as well.

I began restoring my games library from the backups I had created. Although this takes up a large amount of space, and in my case I had to buy 3 USB 8TB, I highly recommend this for folks with large libraries. If I had downloaded my entire library, it would have taken me at least two weeks to complete, as it stands the restore took me less than 4 days.

On the very first batch of restores, I experienced high latency during the Pre-allocation process for SOME games. I also found that the issue appears to happen with those same games repeatedly, ie: the aforementioned Age of Fear.

HOWEVER, I see huge improvements across the board in several areas. For starters the Pre-allocation process, for the games that do still experience the problem, the latency is in fact improved. Such as during the first batch of restores, it included all of the As (restored as I backed up, alphabetically), which also included all of the Assassin's Creed games. The pre-allocation for some of those particular games was far more than some other games of like size, between 20-30 minutes. On severe offender is Remnant From The Ashes, no matter what is happening in that game the Pre-allocation process if vastly exaggerated as compared to other games of like size. For instance, a 340MB update for that game might take upwards of 30 minutes to complete.

Secondly, and this is a big one, I am no longer experiencing the broken up downloads, that is the stopping and starting up of retrieving data from the Steam servers. This issue alone had, at times, added hours of time to my downloading of games and updates. While it happened nearly every time something was downloaded, to what extent fluctuated greatly. Now, that is not to say that this issue shall not return, but I certainly expected to see it during the restore process, and I did not observe it happening one single time.

Third overall system wide performance appears to be greatly improved. Since Steam is literally the only thing installed, outside of Windows and the required drivers, it is the only thing I have to base my observations on. But, the disk activity to the drives that were once members of the array, from inside of the Steam app, has improved by no less than 4x. Whereas, under the array I would see Disk Activity measured at 54MB/s and below, Steam now lists this same activity at sustained between 180-220MB/s, with bursts to over 300MB/s.

My take away:

While the pre-allocation process latency is still occurring, it is much less exaggerated than previously experienced. This latency appears to be relegated to particular games now, where previously it affected those same particular games as well as countless others. However, I still am not experiencing a like issue from any of the other gaming download sights, this issue still appears to only affect Steam downloads. I know someone had asked if I had the same game to test across two separate platforms, and in fact I do, I have several, some through the Steam-GOG cross platform offerings, and others simply by mistake or intentionally purchased for other reasons, and I tested a few. None of these games have any issue when downloading/updating from other platforms, such as Singularity on GOG, but does experience some pre-allocation latency in Steam. Using the on-board RAID option appears to impact this issue greatly. Given that every previous system I experienced this issue on had an array of some sort, it appears that may well have been the source of many of my frustrations. It would seem that something Steam does inside of the Pre-allocation process differs from what other like platforms are doing, and is much more sensitive to the environment in which it operates.

Other areas of the system performance have vastly improved, and overall I am much happier with the performance of my system. To date I have had very few updates (previously I experienced near daily updates that exceeded 20 games a day), and I will report back over time as to if this remains the status quo. I will be happy to run any particular benchmark/analytics anyone might wish to see, just tell me what it is you would like to see specifically, and I can provide it.

So, there you have it, things are greatly improved across almost all areas, but the pre-allocation latency does still exist in some games, however to a much lesser extent than previously experienced. I do not experience a 30 minute wait before any game download begins inside of other platforms, but I do experience this, at times now, during the "Pre-allocation" process of Steam. However, even that 30 minutes is a vast improvement from the previous experience, where some games would take literal hours. I am much happier with my system's overall performance, as well as that of Steam, than I was when I started this thread!!

What I am reading is: "I had some things that didn't install or take properly when I first installed them, not sure what but in doing a full restore and making some significant changes I apparently fixed those things and maybe by some updates also went around some bottlenecks I didn't know I had before. Everything is better but I still see issues with Steam, but not all of Steam but particular games."

That's what I got out of it and pointing to this because it's very possible that the high number of files of one developer and the pre-allocation of those files to avoid optimization will take a lot longer than a game that has many less files. So this is the expected result to me and I am glad things are better, I am sorry they aren't perfect, and I strongly suggest taking up the issue with the Developer of each of the games with long pre-allocation as they need to know their method is creating a problem with their users and that it may even translate into lag and other issues during game play, ways they can do a better job, pick a better scheme or flowchart per se ("method" is a coding term and often gets confused if used in normal conversation).

So glad you got it restored and things are better and hope these Devs will hear you out on what they need to do to make their games pre-allocate better :)

[SeriousCCIE]

I concur; I think the RAID5 write penalty coupled with the pre-allocation tactics of a few vendors were causing the most observable issues.

And the RAID5 write penalty for much of the system as a general rule. It may be smoothed over with disk caching (meaning--3rd party software or something like IRST ssd caching) but won't solve problems caused by externalities--like strange tactics for the pre-allocation.

The fact that the system was rebuilt and many issues went away helps demonstrate the problem isn't entirely the fault of programs experiencing the worst performance hits.

It is an old tactic to blow away data on a disk array (such as a RAID) and restore from tape or other backup--it's then restored in a contiguous file sequence all at once, and is written to the disk in a mostly unfragmented state, unless efforts were made to start a bunch of different copies at once in an effort to use cpu threads or something to 'speed the restore up'.

In any event, it is good to hear the OP with similar hardware to mine is now performing a I'd have otherwise seen for myself--I don't have the RAID 5 (I made a RAID 10 out of 10TB drives instead; an upgrade from 1TB drives that had been holding a steam library for over ten years) and it only got faster as a result of the upgrade. Everything else in the OP's hardware list generally is similar. Here is hoping both my and the OP's RAID do not slow down over time! (And if it does, that the fix is easy...)

[Start_Running]

So we can take away that it was at least in part due the RAID. Now quick question, and I don't know if you already answered this.. Whats the rpm and cache on your drives like?

Also clarify what you mean by 'restore' in reference to those games.

Otherwise you are to be commended for your exhaustive efforts. If nothing else this thread may become the basis of a KB article.

[ba0701]

Start_Running の投稿を引用：

So we can take away that it was at least in part due the RAID. Now quick question, and I don't know if you already answered this.. Whats the rpm and cache on your drives like?

Also clarify what you mean by 'restore' in reference to those games.

Otherwise you are to be commended for your exhaustive efforts. If nothing else this thread may become the basis of a KB article.

You are correct Sir, the onboard RAID contributed to the issue, at least in part. My drives are Seagate Barracuda Pro 10TB 7200rpm 6g/bs ith 256MB of cache.

The restore I was referring to s the process of restoring games that were previously backed p sing the Steam utility.

Hopefully his may help someone, that might have a similar setup, and perhaps save someone a tremendously long time of frustration. I know I could have used this quite awhile ago.

Thank you for your help!

[Start_Running]

Small thought, un install the restored games and straight up reinstall them from steam . That just might cure a few things.