Local 10Gbit/s Steam Cache, client installation speed problems

[SpeedProg]

GB=Gigabytes
MB=Megabytes

So I am running a local steam cache server at home, because I don't want to have all my games installed on my main PC and just install them really fast if I wanna play them.

So I recently upgraded the connection between the machine running the steam cache and my main pc to 10gbit/s. But now when I install a game it only loads with about 115MB/s, and 8 of my 16 cores are in full use, on pc installing the game. Server CPU is pretty much idle.

Also the server was sending the game, I installed for testing, completely from a tmpfs (RAM). So the server disk should not have bottle necked, and the NVMe Storage it was installed on did not look like it was bottle necking either.


So does this mean Steam is running decompression on the fly while loading the files?
And if so, is there some known way to increase the number of cores used? (If that is even possible with the used compression)


If the decompression is not ran on the fly, anyone has an idea what else could limit the speed? (I already tested with iperf that the actual connection it self goes at ~1GB/s)


^_^ If anyone has any experience with this and is open to share some info with me, that would be great ^_^

I mean it's already more then twice as fast as with the 1gbit Ethernet but, yeah if I can get it faster somehow, I take it

[Phantom]

Steam downloads are highly compressed, unlike other gaming clients, and most hard-drives and CPU's can't keep up with the file-checking.

Your gigabit connection basically becomes moot when your CPU / HDD can't do much

Steam has the ability to saturate gigabit connections without any effort.

[cSg|mc-Hotsauce]

Originally posted by SpeedProg:

GB=Gigabytes
MB=Megabytes

...it only loads with about 115MB/s...

8 bits = 1 BYTE

Steam defaults to MB/s.

115 MB/s = 920 mb/s

Sounds good to me.

Steam > Settings > Downloads >>> Display download rates in bits per second

[SpeedProg]

Originally posted by cSg|mc-Hotsauce:

Originally posted by SpeedProg:

GB=Gigabytes
MB=Megabytes

...it only loads with about 115MB/s...

8 bits = 1 BYTE

Steam defaults to MB/s.

115 MB/s = 920 mb/s

Sounds good to me.

Steam > Settings > Downloads >>> Display download rates in bits per second

:qr:

Except you ignored the part where I said the connection is 10Gigabits ~ 10000 mega bits :)

[cSg|mc-Hotsauce]

https://arstechnica.com/gaming/2017/01/building-a-local-steam-caching-server-to-ease-the-bandwidth-blues/

[wuddih]

Originally posted by SpeedProg:

So does this mean Steam is running decompression on the fly while loading the files?

yup

Originally posted by SpeedProg:

And if so, is there some known way to increase the number of cores used? (If that is even possible with the used compression)

nope, Steam doesn't use all cores on purpose. if you have less cores it will also use less cores.

Originally posted by SpeedProg:

If the decompression is not ran on the fly, anyone has an idea what else could limit the speed? (I already tested with iperf that the actual connection it self goes at ~1GB/s)

besides how Steam handles downloads already being a culprit for limitations, the actual culprit is the compression itself, which can be wildly differ between products.
so the speed depends on what you actually download.

that is considering your storage device is not the bottleneck.

if you use hdds, changing to ssds might make it faster, the most you can expect from a 7200rpm hdd is about 150mbyte/s and that is raw write speed, not including all the shindigs Steam does.

overclocking cpu can also improve speeds.

other then this ... you have not much more options. maybe one day Steam will change that cpu thing. everything else would require changes on Steams backend. f.e. not serving compressed data to begin with because what you cache is the compressed data which Steam still has to manhandle to get something installed .. and that is not really likely to happen. they like their compression.

i guess you know this video already:
https://www.youtube.com/watch?v=gk1eKPRLaJA

[Komrade]

Originally posted by SpeedProg:

Originally posted by cSg|mc-Hotsauce:

8 bits = 1 BYTE

Steam defaults to MB/s.

115 MB/s = 920 mb/s

Sounds good to me.

Steam > Settings > Downloads >>> Display download rates in bits per second

:qr:

Except you ignored the part where I said the connection is 10Gigabits ~ 10000 mega bits :)

You sure you have 10gigabits? Run a speedtest.net scan

[SpeedProg]

Originally posted by wuddih:

yup

nope, Steam doesn't use all cores on purpose. if you have less cores it will also use less cores.

Thought so, just wasn't sure. Guess I gonna wait till the next LAN with friends where we use the steam cache since a while, with only 1gbit connection before.
There is some with higher core count processors there normally, and I will see if it uses more then 8 on their machines. If it really does do, I might have some chance making it use all 16 on mine by intercepting some kernel calls or sth for determining the core count :)

Originally posted by wuddih:

besides how Steam handles downloads already being a culprit for limitations, the actual culprit is the compression itself, which can be wildly differ between products.
so the speed depends on what you actually download.

that is considering your storage device is not the bottleneck.

if you use hdds, changing to ssds might make it faster, the most you can expect from a 7200rpm hdd is about 150mbyte/s and that is raw write speed, not including all the shindigs Steam does.

The server for testing purpose is currently just using tmpfs (RAM disk) and the steam client is installing on a NVMe m.2 SSD, so yeah I might wanna try just installing into a RAM disk tbf.

Originally posted by wuddih:

overclocking cpu can also improve speeds.

Well since the compression seems to be limiting it, yeah it might. Not by too much through since overclocking on a normal PC is fairly limited :).

Originally posted by wuddih:

other then this ... you have not much more options. maybe one day Steam will change that cpu thing. everything else would require changes on Steams backend. f.e. not serving compressed data to begin with because what you cache is the compressed data which Steam still has to manhandle to get something installed .. and that is not really likely to happen. they like their compression.

Yeah, I mean I guess I could backup installed games without compression or one that can use more cores onto my server disk and just have a custom client for downloading/copying them :)

But in the end this server is supposed to also be deployed at the bi-yearly private lan, and ease of deployment there would increase way to much, like make everyone install a custom game downloader :P

Currently I just have a router intercept all port 53 UDP and TCP traffic and forward it to a customized DNS server so people get the cache ip for steam(and some other platforms).
So people on lan don't need to do anything at all and use it even if they have a manual DNS server set in their OS.

Originally posted by wuddih:

i guess you know this video already:
https://www.youtube.com/watch?v=gk1eKPRLaJA

Nope, seem to have been created way after I setup the cache for the fist time :).
Watched it now through, but the info in there is basically none existent.

;) I use a similar setup through, apart from no docker and such. Basefiles are nowadays basically from opensourcelan repo, with some modifications.
Video doesn't really contain much info through, except stuff you would know after simply reading the readme of any of the public steam caching project :)

I don't own the game where they show the download on the video, but apparently the one time they show CPU usage it is also only using about 8 cores. (Even through they are pretending it is only using like 2 xDD)

First answer that seemed to actually have read my post in full, before answering thank you a lot for that :)

Edit:

So tried some other games now and on some (e.g. Ys II) I got 380-420MB/s (would probably be slightly higher if they'd take more then 2s-4s to load).
So yeah seems to be entirely on the decompression.

Thanks to everyone that helped so far :)