NAS Support for Steam Client as Backup-Station / Local Library-Server

[RaEyE]

Hello,

I'd like to propose the idea of implementing the feature to use a NAS as some kind of lokal Steam-Library server.
What I imagine is that a slimed down version of the steam client runs on a NAS (e.g. Synology, Thecus or FreeNAS) and manages downloads regarding games in your library or updates for those.

I personaly got 3 desktops in my household, which more or less run the same games. Whenever there is an update (especially the big ones with 2Gb+) each of these system need to download that update seperately. Not only does this block my internet connection it also needs time.

Instead a NAS could load these updates and distribute them localy through LAN.
Therefore the update would be downloaded only once, and the local distribution would be much faster (100 MB/s or even more). Also the NAS could hold the installationfiles to all games present in your library, regardless if installed on a discrete system or not. If you choose to have a go at some oldie you haven't played for ages, but would need to download, you could easily get the game from your NAS via LAN, without burdening your bandwith.

In the end, the NAS would act as a local and private steam-server limited to distributing the products you own, to steam-clients registered to the same account.

Especially for users living in regions with rahter limited bandwith that would be a great help.

I know not everyone owns a NAS, and not everyone would be willing to buy one, therfore this idea will probably never be popular, but you never know, if you don't ask.

Greets,
RaEyE

[Rainwatcher]

Apparantly we think alike. Here's a quote from Try Downloading from Local Source First.

Originally posted by Rainwatcher:

Honestly, a steam cache for a local network should be very easy to implement, Family Sharing can recognise linked accounts, and In-home Streaming can move gameplay from one system to another.

One possible workaround, though it may not be suitable for a laptop that is used for gaming on the go, would be to point all your steam library folders for all your systems to a shared network drive on a single computer. Although I don't know enough about how steam validates its downloads based on the user downloading a particuliar game to know if this would work or break things in a spectaculiar manner for multiple accounts using the same shared library folder.

However a NAS would be an even better solution if Steam can make use of a library stored in a network resource. If you don't mind doing a bit of tinkering, and possibaly re-downloading your library a couple times, try making a library folder in your network drive and adding it to Settings-->Downloads-->Steam Library Folders.

[RaEyE]

Yes an no.
While downloading your library content to a mapped network drive would technicaly work, probably, it wouldn't be a usefull solution, that could be used by other systems in your network as well. If you know what you do and are wiling to tinker a lot ... well maybe it would work, but it's not a solution for everyone. Maybe not even for each game.

What I think of, is a real library, not an installationpath located on an external drive or a network path.

Idealy your local steam client should connect to your NAS / Server as it would connect to a steam server, from which u usualy load your content.
Diference is ofcourse, your NAS is much smaller than Valves Steam Servers and holds only the data you got acces to, according to your account-information.

Think of it as a local cache of steams server, personalised for your individual needs.
Within your steam client you could choose the option to use another "Download-region" (Settings->Downloads), which would point your client to your own NAS.


Sure, from this solution would aries the necessity to download a to be installed ttle from your own server and install it, instead of just running it from a networkpath. Running a software from a networkpath on the other hand, does not suplly you with e.g. entries within a registry, or doesn't allow the software to be run safe and stable by multiple users (not necissarily at the same time).

[Rainwatcher]

True, putting programs in a network drive can be kinda weird. My theory on why it might work is the library backup and validation system. Since Steam can install a game from an external drive or internal drive from a previous sytem to reinstall (I did this with my 1T internal drive where my library was saved, just drop it in the new system, point steam at it, and boom, library re-validated), I see no reason it shouldn't be able to do the same thing from a network drive simply by pointing multiple PC's to a single network location. As far as Steam would be concerned, it's just validating a previous download from in a non C: location.

[RaEyE]

Yes, it should work, if all users refrain from modifying any files located at the network drive, meaning all users only get read-access.

For a single private person, it might be a solution, but not for a household.

Also, just imagin, the desktop whose files are located at the network-drive receives an update for one of it's library entries. The newly updated title had to be pushed completely to the network-drive, and all recepients would need to draw that whole title again.

A local steam server could maintain a copy of the original installationfiles and each version of the update, as a lot of developers / publishers already do on the original steam servers.
A local server in the style of the steam servers could work like that as well.

By the way, have you ever tried to run hugely complex software from a networkdrive?
You may google terms like latency and bandwidth. Another point is, what happens when multiple users wich to use the ressource. All possible, but I#m not the least willing to setup a windows or linux server handling such requests and supplying temporary shadow copies, VM-solutions or whatever. That definetily would be way too much for a simple task, as supplying a steam-compatible library.

Just remember, I don't wish to implement a "steam-library", I would like to see a database which stores and archives, the steam content of with this database associated client-accounts.

[Darren]

What stops you downloading the game on one machine. Then using the backup game option to back it up to a directory on your NAS and instead of doing install select the restore from backup on the other machines?

I think SteamCMD supports both installing and backing up so you might even be able to script the NAS part of the work.

[Rainwatcher]

Originally posted by RaEyE:

By the way, have you ever tried to run hugely complex software from a networkdrive?

Network drive? No. But maybe half of my steam library is currently on a 1T USB 2.0 external drive from my last backup.

Wikipedia : USB 2.0 protocall.[en.wikipedia.org]
USB 2.0 was released in April 2000, adding a higher maximum signaling rate of 480 Mbit/s (High Speed or High Bandwidth), in addition to the USB 1.x Full Speed signaling rate of 12 Mbit/s. Due to bus access constraints, the effective throughput of the High Speed signaling rate is limited to 280 Mbit/s or 35 MB/s.

The standard ethernet running through your house, assuming wired networking which is standard for gaming PC's, consoles, and steam machines, to reduce latancy is perfectly caipable of gigabit speeds, assuming that all devices in the chain between storage NAS and target PC can handle it. Most routers do now, unless they're older which most provided by ISP's are. Even assuming half that speed for synchronious connections (which is not how ethernet works, but for the sake of making a point...) It still runs at double the speed of USB 2.0's theoretical maximum, which most devices can't actually do.

Wikipedia : Gigabit_Ethernet[en.wikipedia.org]

Technically at that point your transfer rate will be limited by the PCIe 3.0 bus speed of 985 MB/s per lane.

WIkipedia : PCIe 3.0[en.wikipedia.org]

Originally posted by RaEyE:

Just remember, I don't wish to implement a "steam-library", I would like to see a database which stores and archives, the steam content of with this database associated client-accounts.

But you're also right, I'm off topic.

Point is, the technology for this already partially exists within Steam. So implementing it shouldn't be a big deal on Valves end. Really, it could be as simple as installing Steam on a single PC and telling it to go into 'local cache mode' at which point it tells other instances of Steam on the local network that it is the download location for all updates. Then it forwards all requests to the actual steam servers, grabs them itself then redistributes them. Basically you'd set one instance of Steam to MITM attack the others. At that point, it could actually be account agnostic, or more likely you'd need to provide a login token to the caching server to say that you are account X and actually hae a license for this game.

Originally posted by Darren:

What stops you downloading the game on one machine. *snip*

Stopping steam from downloading updates. There's no option in the library to stop games from downloading updates outright. You might be able to rig up something using download restrictions (download only for 1 minute daily with a bandwidth cap of 16kb/s), but that's janky at best, and will probably cause issues when you try to install from your newly updated backup.

[RaEyE]

Originally posted by Rainwatcher:

[...] it could be as simple as installing Steam on a single PC and telling it to go into 'local cache mode' at which point it tells other instances of Steam on the local network that it is the download location for all updates. Then it forwards all requests to the actual steam servers, grabs them itself then redistributes them. [...]

That exactly what I'd like to see implemented, just with an added caching feature.
Instead or relaying a request for an update, the server would do this automaticaly, without any request from a client on the local network.

An example for such behaviour could be the Wikipedia - WSUS[en.wikipedia.org]

This service provides a local copy for select Microsoft Products on a Server wihin you Network environment.
Windows machines may register the server running this servies as it's source for updates, instead of the original MS-Servers.

I'd wish for a system similar to this, but for Steam-Content.
I don't believe it would be that hard to implement a local path into Steam-Clients to prioritze a local server before offical steam servers.
The server itsel shouldn't be that different, since the software basically already exists.


Greetings,
RaEyE

[TorMazila]

I've worked out the following idea:
create ZFS volume, serve via iscsi to windows, format/install games, make snapshot and clones from it, serve them via iscsi. But updating is like "destroy extra clones, update, make new snapshot,nuke old snapshot, create clones and so on.
At home I run FreeBSD-10 in virtualbox (hyper-v is way better but its support gets broken every now and then), with 6x1TB 2,5" drives (low power/noise) configured into RAIDZ/RAIDZ2 with some ZFS source tweaks (notably recordsize>128KB support for volumes). Space is served to Win10 as NFS shares (troublesome thing when it comes to filenames, need to investigate NFSv4+Win10 interaction) and iSCSI volumes. If you're not into getting gigabit speeds out of virtualbox - things work pretty solid.
Some games can be run off NFS share, some can use only iSCSI. Some get broken if you have "blocksize 4096" in your ctl.conf - Saints Row 3, Just Cause 2, Red Faction - that are some of such games.

[RaEyE]

@TorMazila

Well, thats a rather crafy config of yours.
If it works for you, tahts fine, but it is a rather frickle solution that definetily needs a lot of work.

But in the first place, it is NOT about running games from an external drive be it USB / SAMBA / ZFS or whatever. My idea is about integrationg a local server which serves as a middle man between your local clients and the steam servers.

Instead of directly loading content from the steam servers with your client, each client in your network registers at your local server, which then delivers the content (e.g. games & their updates / software / movies). This way, content will only be downloaded and stored once to your local server and then distributet from there within your local network.

Any method, that does not involve a seperate copy at a local client is bound to have issues regarding compromised files or in case of multiple clients RAW errors or access rights.

[Rainwatcher]

Originally posted by RaEyE:

An example for such behaviour could be the Wikipedia - WSUS[en.wikipedia.org] *snip*

Oh god, WSUS. Nothing but a pain in the that one. Letting WSUS update and push automatically has never worked quite right for me, or when it does ends up breaking the rest of the environment, or grabbing things that I never asked for (GTFO win 10 upgrade, no such foul demon shal exist on my workstations).

Of course since Steam doesn't let you keep games/software on older versions there would have to be some built in automation for grabbing the updates on request, but letting a cache update fully on its own has caused me no end of headache with other, similar, systems.

[RaEyE]

Originally posted by Rainwatcher:

[...]Oh god, WSUS. Nothing but a pain in the :cyberdeck: that one. Letting WSUS update and push automatically has never worked quite right for me, or when it does ends up breaking the rest of the environment, or grabbing things that I never asked for (GTFO win 10 upgrade, no such foul demon shal exist on my workstations).
[...]

Well you need to configure WSUS the right way. Easies solution, only download the updates for products you approved and only push out the updates you explicitly approved of. Then there shouldn't be aproblem.

But the general behaviour of WSUS is exactly what I#d wish for a NAS / Server solution for Steam.

Btw. You may keep older versions of games, if you choose not to update an installed version and you may revert to older versions (if supported by the developers / the publisher) by choosing to install a 'beta' Version of the game.

e.g. Stellaris is such an example, where the developers keep track of older versions via 'beta' releases of their game. Ofcourse, this is in general not the case for most games out there, but there are still some, especialy those which heavily rely on modding.