Undesirable preallocate behaviour when downloading steamapps using download_depot

[chrcoluk]

Ok so earlier I noticed a repeatable problem with steam closing itself during a download, the download was launched via the download_depot command in the console, not using the usual download in the GUI.

After using various tools and analysing behaviour I have found the exact cause of the problem, and the circumstances to make it happen.

This will affect very little people as these are the requirements for this to occur.

1 - The individual file (not total game size) been downloaded has to be very large, probably in 10s of gigs or higher. But if you have a extremely fast (gigabit) internet connection then 4 gig might be large enough.
2 - The storage it is downloaded to is not very fast e.g. a spindle hdd thats almost full.
3 - Downloading using download_depot.
4 - Fastish internet connection, hard to say a minimum speed but my connection is 74mbit/sec excluding TCP overheads. Thats 9.3MB/sec on steam speeds.

So a lot of things need to be happening for this to occur.

The basis of what happens is this.

Normally when downloading a game from steam, it will preallocate the space first, and then only when this is done it will start downloading data, data will be buffered in small chunks and written to disk. So the amount of data in the steam cache at any given time is no more than a few 10s of meg, based on my observation. Any data in the cache shows up in the steam.exe process a 32bit binary.

When downloading using download_depot, the behaviour changes, it will preallocate "and" download at the same time, any data downloaded before preallocation is completed is dumped in the steam memory cache, and increases the memory utilisation of steam.exe accordingly. Once the preallocation is complete the data will be flushed from the cache and written to disk, so the memory usage will plummet.

Now what was happening is the first 0000.bin file which is 13gig in size, was taking long enough to preallocate that about 2.5gig of data was downloaded to steam before it completed, so the steam memory usage increased to 2.5gig, then plummeted back to about 80 meg, until that file finished.
On the second 0001.bin file which is 35 gig the preallocation never finished before the memory limit for steam.exe got saturated, steam was downloading until it hit around 3.5gig memory usage and then simply closed itself as it hit 32bit memory sized limits.The process limit is actually 4 gig, but I wasnt looking at all types of memory allocation in task manager.

After it clicked in my head what was going on, I deliberately capped the download speed on the second file whilst preallocate was running, preallocate did eventually finish, and then I let it download at full speed until the download finished without problems.

My hdd was preallocating at about 50mB/sec write speed.

For my problem to not occur I calculated if my download speed was 21mbit/sec or slower I would just about have been ok. I actually capped it much lower to about 12mbit/sec which had the steam.exe using 1.5gig when preallocation finished. Although it was initially full speed for a minute or so before I applied the cap.

If I was downloading to a fast SSD I expect I would have been ok also.

To prevent steam.exe from crashing in this scenario any of these solutions could be implemented by steam.

1 - use a 64bit binary, this would prevent the crash but not prevent steam from using high amounts of memory whilst preallocating large files.
2 - have an algorithm that monitor memory usage, if it hits a critical level, take some kind of action such as pausing the download until its flushed to disk. like #1 this would not prevent a few gig of memory usage.
3 - Adjust the behaviour of download_depot to match UI initiated downloads so the download does not start until preallocation is completed. This is likely the most user friendly and compatible solution.

I already posted about this in the feedback section and a steam moderator has escalated it as well.

That thread is here, I apologise the early posts were not productive, but the later posts are useful.

https://steamcommunity.com/discussions/forum/10/1621724915783336855/?tscn=1514156725

If relevant

My OS is win 8.1 64bit
16 gig ram
Nvidia GTX 1070 GPU
Intel 4670k CPU
HDD been downloaded to is a western digital RED drive 2TB about 80% full.
Latest non beta steam version.

Obviously normally I download via the UI, I was using download_depot as I wanted an older version of star ocean 4 to diagnose issues I am having with that game, so I dont download this way normally.

[Groove]

Hey, I've been having trouble with a download and was pointed this direction in hopes you can help. My original post is here >> http://steamcommunity.com/discussions/forum/1/1621724915784704249/

Short version: Bought Sims 3 + 3 DLC in the sale, when I try to download it the "allocating disc space for Sims 3" box never finishes, even after downloading the full 10GB game. I thought the download shouldnt start until it's finished allocating space?

Anyway, if I close the allocating disc space box and allow the download to complete.. I try to run the game and get a "#SteamUI_JoinDialog_KeyRequestFailed_text" error.

Hope you can help, I want to play Sims!

[Nas00]

We all have the same problem so it seems. All of us bought the bundle.

[Groove]

Yeah, I read on the Sims 3 forums that it may actually be a case of theyve sold more games than expected and didnt have enough keys.

[SweatingTV]

I'm having the same problem to. Did it get fixed for anyone?

[chrcoluk]

That is a very odd problem, normally in steam when you download using the UI, it will wait for preallocating to finish first.

But if it finishes downloading, reports success, and steam doesnt crash, then it should mean it is still downloaded correctly.

You can check this using the verify files function.

[Groove]

Not expecting any sort of "fix" until tomorrow at least...

[Scoot]

My SSD runs at 6GB a second (16GB RAM, i7-8700K Coffee Lake Processor, 11GB 1080 GTX Ti Graphics (basically WAY above what this game needs in all aspects), so the speed of your hard drive has nothing to do with it - I'm having the same problem as pretty much everyone else who bought this bundle in the sale :(

[Toddybad]

Originally posted by Nas00:

We all have the same problem so it seems. All of us bought the bundle.

Eaxctly the same problem here too with SIMs 3 bundle. Do the keys get automatically allocted at some point when supplied by EA?

[Scoot]

Originally posted by Toddybad:

Originally posted by Nas00:

We all have the same problem so it seems. All of us bought the bundle.

Eaxctly the same problem here too with SIMs 3 bundle. Do the keys get automatically allocted at some point when supplied by EA?

From what I've found around Google, this happens every time Sims 3 (and similar larger games) go on sale. Valve don't have the keys and EA haven't prepared any to send out. They have literally sold us nothing, at this point. We don't have a game, we have it in our menu but we don't own it until Valve send us all a key.

Which could literally take a couple of weeks.

[Punkabe]

Just letting y'all know, keys are restocked now.

[chrcoluk]

Originally posted by Scout:

My SSD runs at 6GB a second (16GB RAM, i7-8700K Coffee Lake Processor, 11GB 1080 GTX Ti Graphics (basically WAY above what this game needs in all aspects), so the speed of your hard drive has nothing to do with it - I'm having the same problem as pretty much everyone else who bought this bundle in the sale :(

guys bear in mind my bug is different, you guys didnt download via download_depot which my bug report is specific to, so if you have proven its different (fast ssd not delaying writes), then its best to make a new thread so as to avoid confusion.

On my specific issue, the problem I had one of the factors is the storage not able to preallocate fast enough, so on my bug report the storage speed is relevant.

To verify your ssd is writing at that speed you need to monitor it with something like resource monitor, just running a bench doesnt mean preallocate is at that speed.