Steam writting to hard drive really slow

[Geeknificent]

So I have a 300mbps internet connection which equates to about 37MB/s. and I'm starting to have the issue with steam where it downloads the entire patch and then starts writing the data to the hard drive agonizingly slow.
Steam claims it is writing the data to disk at a rate of 41 MB/s but the patch for one of my recent games (path of exile) was literally only 3.5 MB which it downloaded in a split second.
and yes im reading the measurements right the patch is 3.5 Mega-Bytes. but steam says its going to take 10 minutes to write that data to disk. that is NOT a rate of 41MB/s
What the hell is going on with steam? I can transfer 100 gigs between my drives in that time.
It shouldn't have to hold that small amount of data in memory that long.

[ReBoot]

Цитата допису flaburgan:

>"days" is an estimate based on an imperfect algorithm. it won't take days.

It does, I launched it yesterday à 19h40. It is 11h10 now in France

>Quite often some security software messes with Steam reducing performance.

There is a software problem somewhere for sure. The drive is a mechanical one, containing only my games and pictures. The OS is on the SSD. I'm running Linux Mint with the last 5.7.14 linux kernel, but the disk containing the game has an NTFS file system. It never has been plugged to a windows OS however, and is almost full, so I am wondering if there is a fragmentation problem coming from the usage of a windows filesystem only from Linux for more than two years now. The weird thing is, I don't have those low speed problems when using my disk outside of Steam. Also, I'm running Steam Beta, maybe they changed something recently, I don't know...

Well, you just named a couple things yourself.
1. Defragment the disk
2. Quit the Steam beta
3. You can quit NTFS and use ext, Steam can run games off ext volumes under Linux just fine

[flaburgan]

Цитата допису ReBoot:

Well, you just named a couple things yourself.
1. Defragment the disk
2. Quit the Steam beta
3. You can quit NTFS and use ext, Steam can run games off ext volumes under Linux just fine

1 Yes I need to search how I can defragment a NTFS disk from Linux. I found utilities to fix NTFS problems but it doesn't seem to have any, the disk is probably just too full and too fragmented

2 That was partialy my reason of writing here: I wanted to know if I am the only Beta user impacted or if there was a regression

3 Probably a sane long term goal. I'm not even sure why I created this partition as NTFS on the first place as I am Linux only user for more than ten years now... I will need to copy the data to another disk, then format the current one and then copy the data back to do that I guess.

Thank you all for your help anyway.

[ReBoot]

The beta forum is the right place to ask for beta-specific problems, you may want to go there.

[flaburgan]

For future readers, I defragmented my NTFS partition with udefrag from Linux Mint and the problem is now solved, Steam is able to write at 8MB/s on the disk when it was stuck at 50KB/s.

[Eddy Bahringer II]

Цитата допису ReBoot:

It doesn't write 3,5 MB. It unpacks a game file, applies the 3,5 MB patch and repacks the file. That results in way more megabytes written hence the longer duration. And before you ask, the reason is simple: For the vast majority of players (I even dare to say, everyone), internet bandwidth is way more limited, than storage bandwidth. And speaking of, not few people have limited traffic. The system I just described allows for smaller downloads at the cost of more storage activity. Which is a useful tradeoff as it trades heavy usage of a very limited resource for heavy usage of a less-limited resource.

[ReBoot]

Цитата допису flaburgan:

For future readers, I defragmented my NTFS partition with udefrag from Linux Mint and the problem is now solved, Steam is able to write at 8MB/s on the disk when it was stuck at 50KB/s.

Can udefrag also analyze NTFS volumes before defragmenting? If yes, you could schedule a weekly task.

[Death's Hands]

Цитата допису ReBoot:

Цитата допису Geeknificent:

so it basically downloaded a hyper-compressed file is what you are saying. I didnt know that you could compress that big of a patch in that small of a file

Sort of. You downloaded a patch file. While it certainly IS compressed, it's not that heavy compressed. What's causing all the I/O (Satoru is damn right, your IOPS, I/O operations per second is what's causing the long times) is the actual patching. Game files tend to come in archives. Meaning that not every single texture is a separate file. Rather, all those small files get packed into huge package files. Those may be compressed, but they're typically rather huge. Then what Steam does is to unpack the file you already have on your HDD, apply the patch and repack the now-patched file. Without those system, you'd have to download 15 GB (bold guess, I've seen games wiht 35 GB archive files) because of a 3,5 MB patch. As Satoru's right and it's more about IOPS than pure bandwidth, an SSD would help a huge lot (if the long patch time's bothering you, that is).

I am on an SSD, a Samsung 970Evo M.2, and a 1.4GB update takes like 30 minutes now, beofre when it was done in 5 minutes it was done, now i have ot wait forever. they changed something with the download structure and it has definitely mad eit a lot slower to update games, and it all started once they release the new steam UI. while i like the new UI, it isnt worth waiting an extra 15 minutes for a game to download. it never took this long before, it shouldnt take this long now. im not download a whole game, and even if i were, i have GIGABIT internet, and downloading a 60GB and installing it takes me like 10 minutes, tops. so why the hell does a 1.5GB update(my current problem) take 3 times longer than that. Its almost like i would be better off uninstalling then reinstalling the game, rather than update it every damn time, waiting 3x longer than i need to

[ReBoot]

Цитата допису Death's Hands:

Цитата допису ReBoot:

Sort of. You downloaded a patch file. While it certainly IS compressed, it's not that heavy compressed. What's causing all the I/O (Satoru is damn right, your IOPS, I/O operations per second is what's causing the long times) is the actual patching. Game files tend to come in archives. Meaning that not every single texture is a separate file. Rather, all those small files get packed into huge package files. Those may be compressed, but they're typically rather huge. Then what Steam does is to unpack the file you already have on your HDD, apply the patch and repack the now-patched file. Without those system, you'd have to download 15 GB (bold guess, I've seen games wiht 35 GB archive files) because of a 3,5 MB patch. As Satoru's right and it's more about IOPS than pure bandwidth, an SSD would help a huge lot (if the long patch time's bothering you, that is).

I am on an SSD, a Samsung 970Evo M.2, and a 1.4GB update takes like 30 minutes now, beofre when it was done in 5 minutes it was done, now i have ot wait forever. they changed something with the download structure and it has definitely mad eit a lot slower to update games, and it all started once they release the new steam UI. while i like the new UI, it isnt worth waiting an extra 15 minutes for a game to download. it never took this long before, it shouldnt take this long now. im not download a whole game, and even if i were, i have GIGABIT internet, and downloading a 60GB and installing it takes me like 10 minutes, tops. so why the hell does a 1.5GB update(my current problem) take 3 times longer than that. Its almost like i would be better off uninstalling then reinstalling the game, rather than update it every damn time, waiting 3x longer than i need to

By freely intermixing disk & network IO, you're really putting me off from trying to help your situation.

[Death's Hands]

Цитата допису ReBoot:

Цитата допису Death's Hands:

I am on an SSD, a Samsung 970Evo M.2, and a 1.4GB update takes like 30 minutes now, beofre when it was done in 5 minutes it was done, now i have ot wait forever. they changed something with the download structure and it has definitely mad eit a lot slower to update games, and it all started once they release the new steam UI. while i like the new UI, it isnt worth waiting an extra 15 minutes for a game to download. it never took this long before, it shouldnt take this long now. im not download a whole game, and even if i were, i have GIGABIT internet, and downloading a 60GB and installing it takes me like 10 minutes, tops. so why the hell does a 1.5GB update(my current problem) take 3 times longer than that. Its almost like i would be better off uninstalling then reinstalling the game, rather than update it every damn time, waiting 3x longer than i need to

By freely intermixing disk & network IO, you're really putting me off from trying to help your situation.

You are right, my bad still kinda waking up. So my M.2 SSD reads/write specs are below. so please explain why the patching process takes me so long
Random Write (4KB, QD32): Up to 450,000 IOPS Random Write
Random Write (4KB, QD1): Up to 50,000 IOPS Random Write
and the Sequential Write speed is up to 2300MB/s

Also, to clarify, the download process doesnt take long, especially for 1.5GB, but its the patching process that takes forever even though i am on a really good M.2 SSD