Separate the downloader from the decompressor

[shak59]

When downloading a game the downloader will stop or slow down while the files are decompressing.

Instead, separate them.

So the download always moves at full speed and the decompression is queued but unlike now does not slow or get in the way of the downloader.

So there can come a point for larger games where the files are all downloaded and the decompressor is still working for a few minutes, which is better because then you can do other things with the internet connection like playing a multiplayer game or watch a movie while you wait for the decompressor to do it's thing and finish it's queue.

[nullable]

It won't change the amount of work being done, or the time it takes for a game to be installed and playable. It's a superficial change.

You can already do things while downloading. At least I do. There's options to allow downloading while playing a game, there's options to limit bandwidth so you can do other things while downloading.

If you want to run your connection at full speed and not do anything else on your connection while downloading that's a choice you make. Your requested change doesn't remedy or change that, at least not how I'm reading it.

Although I admit I always liked Blizzard's system where you could start playing a game while still downloading. That seemed like the future to me. Also to be clear I don't think Valve's system is the best. So while it could use some work, I don't think shuffling around the existing process has as much benefit as you think. If your idea was so good why do you suppose Valve made the choices they did?

[aiusepsi]

One thing that's important to note is that the bytes which have been downloaded but haven't yet gone through decompression and onto disk have to go somewhere.

At the moment, the buffer between the download process and the decompression process is fairly small, so if the decompression isn't fast enough to keep up, the buffer fills up, it exerts back-pressure on the downloader, and the downloader slows down.

You could just make that buffer bigger, of course, but that has downsides. If the downloader gets a long way ahead of the decompression, all those bytes will be in memory, which will increase Steam's memory usage. If the potential buffer size is unbounded, it'd be possible for there to exist circumstances in which Steam uses all available memory and puts the system into a high memory pressure state where lots of stuff gets paged to disk, which is ruinous for performance.

If the buffer size is bounded, then you get the same effect of the downloader being forced to slow down, it'll just happen less often. And a downside of the buffer just being larger is that it's volatile; if, for example, Steam is forced to close on system shutdown, everything in the buffer is lost.

The other alternative is the compressed (not yet decompressed) data being written to disk if the buffer is full, but that's not ideal because that increases the requirement for temporary storage space, and increases the contention on disk I/O, which might actually make the writing to disk of decompressed data slower, which just makes everything worse.

Which is to say: there's an argument for the potential size of the buffer to be larger by default, but it's not a 100% win.

İlk olarak nullable tarafından gönderildi:

Although I admit I always liked Blizzard's system where you could start playing a game while still downloading. That seemed like the future to me.

Fun fact: you could do this with Steam when it first launched in beta in 2002. That was the purposed of packing all game data into GCF files in the first place. The fundamental issue is that it adds a lot of complexity, and complexity is bad for reliability (It's a leading reason why Offline Mode was so flaky in the early days, I reckon). Basically no third parties ever adopted it, except one or two games which used Source. Valve subsequently removed it from their own games too.

To make this kind of thing work at all, you need the content distribution system and game to be working in the kind of perfect harmony that only exists when it's the same organisation developing both.

[cSg|mc-Hotsauce]

İlk olarak aiusepsi tarafından gönderildi:

İlk olarak nullable tarafından gönderildi:

Although I admit I always liked Blizzard's system where you could start playing a game while still downloading. That seemed like the future to me.

Fun fact: you could do this with Steam when it first launched in beta in 2002. The fundamental issue is that it adds a lot of complexity, and complexity is bad for reliability (It's a leading reason why Offline Mode was so flaky in the early days, I reckon). Basically no third parties ever adopted it, except one or two games which used Source.

Didn't Mortal Kombat X do that? I heard it was terrible.

[aiusepsi]

İlk olarak cSg|mc-Hotsauce tarafından gönderildi:

Didn't Mortal Kombat X do that? I heard it was terrible.

Sort of. It's unrelated to the capability Steam launched with. Steam's original system was a kind of virtual filesystem, which games would access via a Steam API rather than the OS filesystem API. If a game accessed a file, Steam could tell the game via that API on a file-by-file basis if that file was available yet or not, what the download progress was, etc.

What Mortal Kombat X did was partition the game's content into a bunch of nominally optional DLCs, and games could tell Steam to install each DLC at runtime. Much clunkier. And the DLC model is not really a good fit; using a model which treats an optional 4K texture pack DLC the same way as a chunk of the base game's content just works badly.