steam cloud sync failure over ipv6

[Chris Solomon]

I use a RD6 IPV6 tunnel in order to get IPV6 internet. I noticed that whenever steam tries to sync factorio's save it ends up failing. most of the time one might think it is the IPv6 sefvice being slow. gowever in mycase, I have what the FCC calls basic internet service which max out at 10mbps. the ipv6 tunnel is as fast as my ipv4 connections. which means either A. the ipv6 requirements for steam cloud sync for facrotrio need a faster speed, or B. the timeout settings for the IPv6 need adjusting. either way it is something that i can not do anything about. it is something either valve, or the factorio devs need to fix

[PunCrathod]

Factorio devs can't do anything to fix it. It's a steam problem that happens on almost all games to some people. And valve has been asked to fix cloud sync for years now. People complain that cloud sync either doesn't upload or download or worst case it deleted their saves. They have made a lot of improvements over the years but the system is still breaks all the time.

[Chris Solomon]

well, all i know is factorio is the one game that i have that has issues syncing

[ShutEye_DK]

Originally posted by Chris Solomon:

well, all i know is factorio is the one game that i have that has issues syncing

Factorio's saves can get rather large. Also the autosaves are uploaded as well.
Transfers over 1GB is not uncommon.

How large is your save?

Edit: Those 10 Mbps is your downstream speed, I guess. How fast is the upstream?

[Chris Solomon]

Originally posted by ShutEye_DK:

Originally posted by Chris Solomon:

well, all i know is factorio is the one game that i have that has issues syncing

Factorio's saves can get rather large. Also the autosaves are uploaded as well.
Transfers over 1GB is not uncommon.

How large is your save?

Edit: Those 10 Mbps is your downstream speed, I guess. How fast is the upstream?

derp... your right about the up/down speeds... i only have 1mbps upload.

according to steam i am using 136.57 Mb out of 4.52Gb. I only have one two saves. the tutorial map, and the current save i use.

[Chris Solomon]

most irritatingly, i keep getting stuck behind this dialog at times when i got to launch the game that says waiting for game to synchronize. and it just sits there, and sits there, and sits there. only option really is to abort the launch and wait for steam to give up on the sync

[Khan Boyzitbig of Mercia]

With an upload that low you should disable cloud sync for factorio, It will be timing out anyway before the upload can complete most likely and certainly once you get a decent sized base on a map.

[PunCrathod]

Yeah. Disable the cloud sync for factorio and do backups manually. https://wiki.factorio.com/Application_directory tells you how to find the save files. Also if you want you can complain to valve to implement a system where if the cloud sync times out it can resume instead of starting over. Maybe move it to a background task limited to 50KB/s that keeps trying until it succeeds or the user tries to launch the game.
Maybe if enough people ask they finally implement it.

[RiO]

Originally posted by PunCrathod:

Also if you want you can complain to valve to implement a system where if the cloud sync times out it can resume instead of starting over.

I think the cloud sync is (depending on your region) storing either in AWS or Google Cloud. And in case of Google Cloud, goes directly into storage buckets; which iirc don't support resuming uploads. A file upload either succeeds; and ends up stored and replicated; or it fails. Possibly, they have some kind of service in front of that, which proxies the upload to the bucket in a streamed fashion. (Stands to reason they do, actually - because of authentication.)

Either way: buckets were never designed to directly support user uploads. They were meant as an in-cloud means of file-persistence between multiple services, and as a (very, very) bare bones means to serve out files for download. In-cloud connection to bucket storage from a cloud-hosted service has basically GBit connection speeds, so the default timeout is kept intentionally low to comply with the fail-fast paradigm that is central to cloud-based development.

What Valve should be doing is extending the timeout value from its default.
In case of Google's solution, they could literally put in a timeout of zero. Which means 'no timeout' - i.e. the connection will only ever be timed out by the client.

Which is exactly what you want in this type of setup with public allowed uploads.

[Chris Solomon]

Originally posted by RiO:

Originally posted by PunCrathod:

Also if you want you can complain to valve to implement a system where if the cloud sync times out it can resume instead of starting over.

I think the cloud sync is (depending on your region) storing either in AWS or Google Cloud. And in case of Google Cloud, goes directly into storage buckets; which iirc don't support resuming uploads. A file upload either succeeds; and ends up stored and replicated; or it fails. Possibly, they have some kind of service in front of that, which proxies the upload to the bucket in a streamed fashion. (Stands to reason they do, actually - because of authentication.)

Either way: buckets were never designed to directly support user uploads. They were meant as an in-cloud means of file-persistence between multiple services, and as a (very, very) bare bones means to serve out files for download. In-cloud connection to bucket storage from a cloud-hosted service has basically GBit connection speeds, so the default timeout is kept intentionally low to comply with the fail-fast paradigm that is central to cloud-based development.

What Valve should be doing is extending the timeout value from its default.
In case of Google's solution, they could literally put in a timeout of zero. Which means 'no timeout' - i.e. the connection will only ever be timed out by the client.

Which is exactly what you want in this type of setup with public allowed uploads.

it looks like from the log from steam, it is google. i completely agree on the extending of the timeout for the connection.

[Chris Solomon]

not sure what is triggering it, but some of my game saves for other steam titles are being backed up (and restored as necessary) by Microsoft OneDrive.

I have my system setup to use my Microsoft accoutn and onedrive. and some for some of my games, steam is triggering onedrive to backup/restore saves. I am not certain if this is in addition to steam cloud saves or instead of. but there seems to be no issue with the uploading/downloading to the onedrive location. I am not sure why only certain games are doing it.

update:

seems it is the games which use microsoft's proposed preferred game save location that steam is triggering onedrive with (c:\users\%username%\Documents\My Games\ or c:\users\%username%\Documents\).

Maybe if the factorio devs either changed the game save location to either of those locations instead of 'c:\users\%username%\appdata\Roaming\Factorio', it would likely be a good around to the issues with the cloud sync. the save still would not be uploaded to the steam cloud, but for those who have configured their systems to use onedrive, the saves would be backed up to their onedrive.

[RiO]

Originally posted by Chris Solomon:

seems it is the games which use microsoft's proposed preferred game save location that steam is triggering onedrive with (c:\users\%username%\Documents\My Games\ or c:\users\%username%\Documents\).

Maybe if the factorio devs either changed the game save location to either of those locations instead of 'c:\users\%username%\appdata\Roaming\Factorio', it would likely be a good around to the issues with the cloud sync. the save still would not be uploaded to the steam cloud, but for those who have configured their systems to use onedrive, the saves would be backed up to their onedrive.

OneDrive has a few settings to quick-enable synchronization and 'backup' for a few of the special folders on your system. The Documents and Pictures folders are among those that it will suggest to enable automatically on first setup.

That said: neither \%userprofile%\Documents nor \%userprofile%\Documents\My Games are actually the preferred locations for save data for video games. The preferred location for that is \%userprofile%\Saved Games and if you do it by the book the folder hierarchy inside that folder must obey the pattern \{publisher-name}\{game-title}\.

\%userprofile%\AppData\ is actually the least correct of the bunch, because AppData is meant for temporary storage of easily recreateable data such as user-specific configuration files; cache; scratch-space; undo/redo backups; and temporary saved files as part of a suspend&resume feature.


The amount of modern games that actually use this correctly is actually staggeringly few. It's the embarrassment of the industry: not even the largest triple-A titans can manage to do it right.

[Chris Solomon]

Well, it's the place that Microsoft wishes that all the would games use and a fair number do. this goes back to the days of XP or maybe Vista. before that the saves were usually saved in the program's directory (back before the days of steam) Microsoft wanted to game devs to use these locations, but adoption has been beyond spotty. the wanting of microsoft wanting to use those locations peaked with windows 7, with the special game folder (a hold over from vista i think)

among the steam games i own that DO use the My Document and the My Games or C:\users\%username%\Save games directories are the following:

Star Wars: Th Force Unleashed
BioShock 2 HD
GRID
Evochron Legacy
Ghostbusters
Do not Starve
Do Not Starve Together
Star Wars Squadrons
Just Cause 2
Just Cause 3
Just Cause 4
The Crew 2
(oddly enough) Steam VR
Timberborn
The Witcher
Batman Arkham Asylum
Batman Arkham City
Batman Arkham Knight
Teenage Mutant Ninja Turtles Revenge of the Shredder
Transformers Devastation
H.A.W.X.

In the ..\documents\My Games\
i found folders for the following steam games
Bioshock Infinite
Borderlands 2
Borderlands the presequel
Crossout
Crysis
Dishonored
Satisfactory
Fallout NewVegas
far cry 2
faster than light
GRID 2
hacknet

I am of the opinion that a standard place be desginated and used, similar to the way MS introduced the documents folder concept starting with win95, and like they tried to do with the My Games and Game Saves in the documents directory starting with Win Vista/7. No one likes having to travel all over their drive to locate what should be similar kind if files (game saves). having them grouped together makes managing and backing them up much easier.

as far as i know, most backup software, the type that comes with an external rives only backs up the same locations as OneDrive. only way to catch save files that reside in places like ..\appdata\ is to either have software that is capable of doing a full system image, or have software that will allow you to backup custom locations. (my backup software actually can do both)

[Dagguh]

Same here, Factorio uploads over IPv6 time out, but work perfectly fine over IPv4 over the same LAN+ISP.

[Chindraba]

Originally posted by Chris Solomon:

not sure what is triggering it, but some of my game saves for other steam titles are being backed up (and restored as necessary) by Microsoft OneDrive.

I have my system setup to use my Microsoft accoutn and onedrive. and some for some of my games, steam is triggering onedrive to backup/restore saves. I am not certain if this is in addition to steam cloud saves or instead of. but there seems to be no issue with the uploading/downloading to the onedrive location. I am not sure why only certain games are doing it.

update:

seems it is the games which use microsoft's proposed preferred game save location that steam is triggering onedrive with (c:\users\%username%\Documents\My Games\ or c:\users\%username%\Documents\).

Maybe if the factorio devs either changed the game save location to either of those locations instead of 'c:\users\%username%\appdata\Roaming\Factorio', it would likely be a good around to the issues with the cloud sync. the save still would not be uploaded to the steam cloud, but for those who have configured their systems to use onedrive, the saves would be backed up to their onedrive.

There's no need to wait on anyone to fix anything. You could tell Steam where to put it. Take that how you will, but in this case it's not the unfriendly version. Steam allows you to decide where things will be placed, having multiple locations available if you make them. It will even attempt to move files if you choose to. It's not that good at it, honestly, but they do try.

There is also the option to tell Factorio where to store it's data. You have to at least reconfigure it, and maybe uninstall and reinstall it. It's possible, however to pick either of two preset places: the 'preferred' location, which appears to be the roaming profile, or next to the executable's directory. Other than those two places you can 'hard-code' literally any directory to which you have read/write permissions on your computer. If you happen to choose one of the auto-watch directories from OneDrive you will get the double insurance for this game which you get for the others in Microsoft's "preferred" locations.

[Chris Solomon]

it seems that the ipv6 tunnel i was forced to use for ipv6 acess was at fault, several other programs also seemed to have trouble while the 1pv6 tunnel was enabled. mainly it was the nvidia app and steelseries engine app. in bo0th programs, while the tunnel was setup, i could not login to the app. also, attempting to 'google' anyhing would cause google.com to complain about suspicous activity or make me prove i wasnt a bot. disabling the tunnel solved all that, but left me with only a ipv4 connection.