An option to disable space preallocation

[TorMazila]

Hi!
I have large part of my libraries served over iSCSI (it's like a physical drive but accessible over network; actually it's RAIDZ and RAIDZ2 sets of several drives - so I can painlessly live with loosing 1 or 2 drives) - and that means I'm stuck at sub-gigabit speeds. Which is almost ok for everyday usage but installing anything large becomes a huge waste of time for doing nothing but "allocating space" like it'll help anything.

Disabling this option will allow starting the downloads right away and may also save write cycles for SSD drives.

Edit: March 19, 2020. The Steam BETA has stopped preallocating disk space and just downloads the games "the normal way"
April, 2020: Seems not so: it definitely preallocates space prior to patching. World of warships update on Steam has benchmarked my SSD :\.

[ShiroKuroh]

I've been running a SAN as a DAS since 2008. IT wasn't an issue, Now 11 years later the family wants in on all this insane drive space I have. We have 4 different steam accounts, and everyone finally has a decent gamer rig. But our bigger issue is now, I pre-allocate like wildfire. However, expanding this out, and even with 1 TB local NVME's they're running out of room. SO whether I do iSCSI from ZFS2 (raid6 o/raid, Attach a VHDX across the network, even at 10 GB/sec its super slow. NowI can transfer files across this at 120~!30 seconds on 10 Gbit, and 21 minutes of GBit, so I know its not the setup. Even d say doing backups of the game form one pc and restore to another, i.e.; SotTR the pre-allocate took 14 minutes to allocate. The game installed in 12 minutes. Its not a niche problem anymore these days. But it would be helpful to disable this.

[TorMazila]

Publicado originalmente por Shiro_Kuroh:

I've been running a SAN as a DAS since 2008. IT wasn't an issue, Now 11 years later the family wants in on all this insane drive space I have. We have 4 different steam accounts, and everyone finally has a decent gamer rig. But our bigger issue is now, I pre-allocate like wildfire. However, expanding this out, and even with 1 TB local NVME's they're running out of room. SO whether I do iSCSI from ZFS

Raidz2 /raidz3 is a pain when writing to it, though random reads may work faster due to a bit more spindles for reading that have "the same" information (depends on how it's made in software).

I have zvol's, a PC with "main" library, do a snapshot and then serve that snapshot clones to other PC's, main zvol is served to "main" libraru PC.

service ctld stop
sleep 4
#destroy clones
zfs destroy P2/MMO256CL09
zfs destroy P2/MMO256CL12
#destroy old snapshot and make the current 
zfs destroy P2/MMO256@snap
zfs snapshot P2/MMO256@snap
sleep 1
zfs clone P2/MMO256@snap PZP2/MMO256CL09
zfs clone P2/MMO256@snap PZP2/MMO256CL12
#wait some time - 25s is not enough - for clients to disconnect and unmount drives
sleep 25
service ctld start

If things go bad (e.g. the P2 pool runs out of space) - I destroy clones and do a rollback to snapshot
service ctld stop
zfs rollback -rRf P2/MMO256@snap

[ShiroKuroh]

Thanks, its amazingly helpful for another issue I have, unrelated. for me atm, to add a game takes forever, but after the download starts it moves as normal. its just the pre-allocation of space.

[TorMazila]

Publicado originalmente por Shiro_Kuroh:

Thanks, its amazingly helpful for another issue I have, unrelated. for me atm, to add a game takes forever, but after the download starts it moves as normal. its just the pre-allocation of space.

Well, with modern 60GB titles - it's tens of minutes of waiting (for NAS-based storage) instead of downloading. Not to mention things like SSD wearout for normal desktop use.

[ShiroKuroh]

Its not all the much work and might make life easier, just getting the housemates to agree.

[lucym202]

Putting aside technical concerns... how is "it takes a long time if your storage isn't fast and it's not necessary if you already know you have more than sufficient space" not sufficient justification?

[ShiroKuroh]

From the technical I found 1 error on my side. my l2arc log started showing signs of wear with SMART on zfs. I stepped up from evo 960 500 GB m.2's to the Adata XPG 8600 1TB. I also did an upgrade to the NAS. Even after replacing a "possible" bad l2arc drive one with something insanely fast I had the same issue. Work is picking up for us and we're at 2 TB of data daily. I invested in 2 24 bay norcos some Xeons and asrock rack boards with two vdevs at 24x 4TB, raidz2 (raid6), 82.0 TB, w=434MB/s , rw=189MB/s , r=1063MB/s and I'm within 0.06% of the theoretical claim for speed. Note, I moved away from Freenas and Unraid and I'm doing a pure Arch install at this point. I also stepped up to 10 GBIT for my side of things. I decided to start fresh. I thought the problem disappeared. I'm not sure why but after game number 11, and I started over with smaller stuff just to see if size was a constraint, but at game #11 this pre-allocation delay came back. A decent size game 50 GB for example still takes 40 minutes time to preallocate. I'm moving a TB size of data locally at just under 15 minutes. I'm not giving up on steam, but I may just have to find other ways in the future as this extreme slow installation is just nonsense dedicated to those who happen to have a Windows (any version,) and a NAS with multiple games. Note all test ran with no one on the network. I love making content, but this is a buzz kill if one of my two in house content creators decide they want to use steam to DL and install a game mid stream.

[TorMazila]

Publicado originalmente por Shiro_Kuroh:

From the technical I found 1 error on my side. my l2arc log started showing signs of wear with SMART on zfs. I stepped up from evo 960 500 GB m.2's to the Adata XPG 8600 1TB. I also did an upgrade to the NAS. Even after replacing a "possible" bad l2arc drive one with something insanely fast I had the same issue. Work is

Those shiny desktop drives are not the best choice for l2arc, especially if they are TLC. Once I tried using famous and very robust Intel G2 drives as a journal for FS (~6MB/s writes, practically linear I believe) - the "signs of wear" in a form of remaps started appearing right away in a day or so, which didn't happen with any loads I threw on them before and after for years (like huge databases with stupidly huge temp tables and "normal desktop use" later, till this day.

Publicado originalmente por Shiro_Kuroh:

picking up for us and we're at 2 TB of data daily. I invested in 2 24 bay norcos some Xeons and asrock rack boards with two vdevs at 24x 4TB, raidz2 (raid6), 82.0 TB, w=434MB/s , rw=189MB/s , r=1063MB/s and I'm within 0.06% of the theoretical claim for speed. Note, I moved away from Freenas and Unraid and I'm doing a pure Arch install at this point. I also

Impressive, but I'd check out how cpu's are utilized, and where are all those drives attached to - looks like 4xPCI-E :) ?

raidz2 may have just a bit of extra random reads performance compared to raidz but in writes and linear reads it'll fall behind.

What are the blocksize and recordsize in your zpool/FS (or if you serve zvol's you need to patch libzfs to allow zvol creation with volblocksize>128KB)? Larger recordsize is good for compression and linear read speeds, but absolutely trashes performance in small random reads/writes. In NTFS you end up with 4KB (default) or up to 64KB clusters so the performance will suffer. I had to go (down) to 256KB volblocksize (+I have compression) or my games were taking next to forever to load with 1MB volblocksize.

Having zpool of complete hard drives and not of partitions (=you'll get many pools) will degrade performance a lot - it causes more seeks (which means speed drops down to several MB/s or even under 1MB/s per drive) and hard drive speed is like twice lower at its end. Replacing a drive in a pool will be horrific experience - writing 4TB takes... long. And remember, ZFS is copy-on-write = to change 1 byte in a file on 1MB recordsize pool means to read 1MB (checksum check+ optional decompression), do stuff to get some data to upper layers, change that byte, pass down, optionally compress, calculate checksum, write that 1MB. "read and write amplification", that is.

24 hdd's (or better say 22+2) in raidz2 pool is not the best idea either - with e.g. 4kb blocksize and "default" 128KB recordsize you can't evenly split the load (128kb/22 drives=some float point) and it'll be like 4kb random reads/writes in the end. I'd go with 16+2=18 drives (or raidz3 with 16+3=19 drives). Raidz2 for 24 drives means higher risk of failure - 3 of 24 drives going bad is way more probable than 4 of 24 or 3 of 8 or 16.

Jumping from FreeBSD (FreeNAS) to Linux (Unraid,Arch) kernel is really a significant environment change and I can't predict if it's better or worse at the moment (it's a bit holywarish thing and highly dependent on hardware drivers as well).

In FreeBSD I'd use gstat to monitor drives load to see if the performance is pinned by drives themselves. Check out ARC usage, try turning off ARC compression.

Publicado originalmente por Shiro_Kuroh:

stepped up to 10 GBIT for my side of things. I decided to start fresh. I thought the problem

Oh-my :) Wish I could try that. But just a small note - to fill up gigabit read speed (100+MB/s) it was enough 800MHz Ivy Bridge Celeron (2 cores), strangely read speed got lower with higher frequency :). Writes need more CPU power as I use compression.

Publicado originalmente por Shiro_Kuroh:

disappeared. I'm not sure why but after game number 11, and I started over with smaller stuff just to see if size was a constraint, but at game #11 this pre-allocation delay came back. A decent size game 50 GB for example still takes 40 minutes time to preallocate.

That is over the net? factor in NIC drivers/performance as well.

Publicado originalmente por Shiro_Kuroh:

I love making content, but this is a buzz kill if one of my two in house content creators decide they want to use steam to DL and install a game mid stream.

Those are 2 different load types - games=random i/o, streams/video editing=mostly linear i/o, where 1MB recordsize or volblocksize will do good.

So, you just should separate some drives for games and some for videos. Not much point in keeping temporary material on raidz2 as well (unless you make $$$ on it and loss will cause financial issues)

Steam supports only like 10 library folders - you should keep that in mind.

And finally - 24 3.5" drives alone will constantly eat over 100W power (peaking at some 300W if they start all at once), it can be more profitable to have local drives than a huge always-on NAS :).

[ShiroKuroh]

Wen I have more than a lunch break I'll reply, I appreciate the criticism and feedback. Make me grow. For my background I've have had over 1000 servers total over 3 geographic locations back in 2005. We went cloud at the day job, but being in this trio the only one with a day job, budget to 2 newer, and doing well content creators gets a little tough. I'm the last one with a day job; and moving them to the new machine and they get back 3 hours of there day; well we still have room for improvement. Either way, they get very particular about the looks of there rig and that gets very limiting on space. I've tried the whole give me an m.2 slot and a sata port with the SSD and primocache local just to make it feel faster and have access to the game library overall, but that's OK if you open everything 3-6 times before you start the stream, not while you're on it. But I digress we got away from Original topic. When something new hits and they want to click DL and chat..... I mean, the preallocation is so slow, Its only ZFS and its only iscsi targets. After we get past that point it flies. I just cant get over that Bump. I took the old drives, set 3 up with a 12gps 3008-8i internally. setup an internally on a Manjaro steam client. The issue doesn't exist. Trying to keep it easy from their point of view so the only local worries are capture cards.

[Brabebhin]

Preallocation needs to go. Downloading and patching games takes hours even for the smallest of patches which should be done in under 2 minutes on fast SSDs (like other software similar to Steam do). If you are not using SSDs, the process of installing 80GB+ games can take several hours just because Steam needs to write zeros all over the place.

[Start_Running]

Publicado originalmente por Brabebhin:

Preallocation needs to go. Downloading and patching games takes hours even for the smallest of patches which should be done in under 2 minutes on fast SSDs (like other software similar to Steam do). If you are not using SSDs, the process of installing 80GB+ games can take several hours just because Steam needs to write zeros all over the place.

If yopu rpre-allocation is takking hours. Then something is not right with your drive or file system... or perhaps its you AV program mucking things up. Try telling yyou AV to ignore/exclude you steam directories from automatic detections.

[TorMazila]

Publicado originalmente por Start_Running:

Try telling yyou AV to ignore/exclude you steam directories from automatic detections.

Worst idea ever. While Steam does scan things and typical malware must pass that scans to get to you (it's likely it actually will pass your AV as well) - as the time goes by and you have downloaded product on your drive it becomes only your AV responsibility to detect malware...

[Start_Running]

Publicado originalmente por TorMazila:

Publicado originalmente por Start_Running:

Try telling yyou AV to ignore/exclude you steam directories from automatic detections.

Worst idea ever. While Steam does scan things and typical malware must pass that scans to get to you (it's likely it actually will pass your AV as well) - as the time goes by and you have downloaded product on your drive it becomes only your AV responsibility to detect malware...

I can guarantee you. The ANti viurus software steam employs is a league or two what you're running. Plusd if you cactually do incur damages from a virus that came form a gamedistributed through steam, you can sue steam for the damages and then some.

Point is, AV in autoprotect tends to delay all read and write operations. Hence why I recommended creating an exception rule. That will in all liklihood remove those lengthy preallocation cycles.

[Satoru]

Publicado originalmente por TorMazila:

Publicado originalmente por Start_Running:

Try telling yyou AV to ignore/exclude you steam directories from automatic detections.

Worst idea ever. While Steam does scan things and typical malware must pass that scans to get to you (it's likely it actually will pass your AV as well) - as the time goes by and you have downloaded product on your drive it becomes only your AV responsibility to detect malware...

You somehow think your junky consumer anti virus is going to detect something that sneaks by CloudFire, Akamai, and Steam

Yeah there is a zero % chance of that happening

[TorMazila]

Publicado originalmente por Satoru:

You somehow think your junky consumer anti virus is going to detect something that sneaks by CloudFire, Akamai, and Steam

Thing is that your "my junky av" will detect that virus TOMORROW (or moths later) when it gets the updates, while none of the AV products detect that threat TODAY, when we download and install the app.

Publicado originalmente por Start_Running:

I can guarantee you.

Dear Mr. God (cause He is the only one who can guarantee anything),

Publicado originalmente por Start_Running:

The ANti viurus software steam employs is a league or two what you're running.

Noone has a "silver bullet", they can run dozen AV products with all heuristics and such - but as long as they don't get the source code for the apps (and analyze it) - they can only guarantee one thing, their best effort as of "TODAY". Even if they do serious analysis in a sandbox - who says that the trojan or some malware will activate right away as you launch a game? I'm a bit into programming (from BASIC to C and Assembly, and modding FreeBSD/Linux/Solaris kernels) since the end of 1980's - so I clearly see the possibilities that the devs get by installing their software on your HDD. Really endless unless they break something and make the right people really look into the issue. And your gaming PC typically is not "purely for gaming" but contains lots of easily readable information.

Just a couple weeks ago we've seen a "second PUBG" emerging from what was a "Measurement Problem" - a game written in 2016. The devs account was likely hacked and hijacked (no details if the hackers got the game source code or the additional software that gets installed before you run the game for the first time was modified) - resulting in a massive keys giveaway several months ago (we all like giveaways with cards and such - don't we? And we never ever suspect anything wrong in this action) and a strange, meaningless, news article in Russian appearing just that couple of weeks ago with the game changing its title to PUBG (it actually RESEMBLED PUBG due to using similar-scripted letters from the other alphabet). With only the image containing a sentence like "a thousand eyes are watching for you and creating a digital dossier that can be purchased".

Publicado originalmente por Start_Running:

Plus if you cactually do incur damages from a virus that came form a game distributed through steam, you can sue steam for the damages and then some.

The US population is roughly 5% of the world population - that is it'll be from hard&useless to impossible to go to court with such case for most people :). And that's not accounting for the fact you may not be able to even guess and prove the Steam being involved in your real life issues - like theft of documents, identity and alike causing consequences - even if the "consequences" case gets completely investigated it'll end up in dead end like "I purchased this person information somewhere on the net - don't remember where"

Publicado originalmente por Start_Running:

Point is, AV in autoprotect tends to delay all read and write operations. Hence why I recommended creating an exception rule. That will in all liklihood remove those lengthy preallocation cycles.

A preallocation is technically writing of zero bytes - with RAM speeds at several GB/s AV checks are unlikely to affect that - simply because they stop pretty early as there's not much to compare to zero bytes. Nevertheless, writing to HDD some 60GBytes at a rate of "almost 100MB/s" takes ~10 minutes and doesn't really ensure you'll not run out of space. With trend of using network storage for your games - you definitely will not jump over 125MBytes/s (a gigabit LAN limit, as 10G-100G is not really usable at home so far) and that is rarely needed, in fact you often face random i/o when running games.