Confused as to how patching works..

[WarNerve]

Is it normal for a patch that is only a few MB to require a scan( or whatever it's doing)of the entire 56GB game file?
I may be wrong. In fact, I most likely am wrong, but I don't think my other games run through the entirety of their files to patch in a few MB of data.
It's not a huge deal. I'm not posting this to complain. I'm just curious if this is normal procedure or not. It just struck me as a bit odd.

[Rocket]

It is a very frustrating issue for us. Note that we have zero control over how patches are applied on steam. The way delta patching occurs on steam causes this issue, especially on highly optimized binarized data. This means steam reserves a whole bunch of new sectors, copies the data across, and cross references.

We get many complaints about this, but the only thing we could realistically do would be uncompress the data completely, which is what games like Ark do. They don't pack the data and leave it all unpacked, and uncompressed.

If we tried to do this with Icarus, it would very dramatically increase both the load times and RAM used by assets during loading, as well as filesize very dramatically.

Unfortunately it seems following the best practice guidelines for packing data for Unreal, which we did, triggers big problems for steam downloads. I raised the issue with valve who told me to get customers to raise tickets, which I passed on. However, valve then just closed the tickets. So sadly, I think we all just have sit with how it is until Valve decide to change that.

This is a reason why we limit the project to only one update a week.

[Grand Forks]

love that response @Rocket "highly optimized binarized data" I had to giggle a bit.

[Rocket]

Originally posted by Grand Forks:

love that response @Rocket "highly optimized binarized data" I had to giggle a bit.

Sorry I don't quite understand what you mean?

Perhaps you misunderstood me; but the phrase "highly optimized binarized data" is just a descriptive way of saying "packing" something. It's just a factual statement describing how the files are packed.

Take for example if you took your whole computer hard drive and "zipped" it up. By saying its binarized and highly optimized, I wouldn't be necessarily saying its the best way to work on those files, but it is the most efficient way of storing it. Additionally, if when zipping it, the packing process looked at what files reference what others, and in particular, what files you typically work on after others - and group them together - this is what the engine packing does. Extending the analogy further, by zipping your whole harddrive up in this manor, if you updated *one* file, then depending on how it gets distributed to customers, the software used to download it might use novel ways to copy, apply the delta state change, and then unpack it.

[ホロ]

Originally posted by Rocket:

Originally posted by Grand Forks:

love that response @Rocket "highly optimized binarized data" I had to giggle a bit.

Sorry I don't quite understand what you mean?

Perhaps you misunderstood me; but the phrase "highly optimized binarized data" is just a descriptive way of saying "packing" something. It's just a factual statement describing how the files are packed.

Take for example if you took your whole computer hard drive and "zipped" it up. By saying its binarized and highly optimized, I wouldn't be necessarily saying its the best way to work on those files, but it is the most efficient way of storing it. Additionally, if when zipping it, the packing process looked at what files reference what others, and in particular, what files you typically work on after others - and group them together - this is what the engine packing does. Extending the analogy further, by zipping your whole harddrive up in this manor, if you updated *one* file, then depending on how it gets distributed to customers, the software used to download it might use novel ways to copy, apply the delta state change, and then unpack it.

hi kinda off topic how much will the upcoming dlc cost?

[WarNerve]

Originally posted by Rocket:

It is a very frustrating issue for us. Note that we have zero control over how patches are applied on steam. The way delta patching occurs on steam causes this issue, especially on highly optimized binarized data. This means steam reserves a whole bunch of new sectors, copies the data across, and cross references.

We get many complaints about this, but the only thing we could realistically do would be uncompress the data completely, which is what games like Ark do. They don't pack the data and leave it all unpacked, and uncompressed.

If we tried to do this with Icarus, it would very dramatically increase both the load times and RAM used by assets during loading, as well as filesize very dramatically.

Unfortunately it seems following the best practice guidelines for packing data for Unreal, which we did, triggers big problems for steam downloads. I raised the issue with valve who told me to get customers to raise tickets, which I passed on. However, valve then just closed the tickets. So sadly, I think we all just have sit with how it is until Valve decide to change that.

This is a reason why we limit the project to only one update a week.

Thank you very much for this very informative, and interesting, response. Please don't see my post as a complaint. I was only curious about how things worked is all. I really appreciate you taking the time to give us all such a detailed explanation and insight into how the mechanics of patching works. I find it very interesting.
You guys are doing an excellent job with the updates, and I love the game. Please keep up the good work! =D

[Rocket]

Originally posted by WarNerve:

Please don't see my post as a complaint.

Did not come across like that at all, and many have this issue - occasionally even myself. So discussing it is good. While some might get a bit annoyed as it has been discussed before, it can be very hard to know if information is current. So discussing it and continuing to look at options to deal with it are well worth it.

Originally posted by WarNerve:

I really appreciate you taking the time to give us all such a detailed explanation and insight into how the mechanics of patching works

Community engagement is how we have gotten the game to the much better position it is in now, and part of that is having good "robust" discussions and liaison with the community. We hope we keep delivering good updates and a better game!

[КрендельФигов]

I don't know what you're talking about here, but apparently when talking about
"highly optimized binarized data"
it was about overcompressed type of archive -> solid archive
Wiki (if anyone wants to read more about what it is) -> https://en.wikipedia.org/wiki/Solid_compression
And there, file layout schemes for compression can really be extremely sophisticated. For example: I think everyone knows that there are official games, and there is a 'torrent edition' (I think it's clear what I'm talking about).
So, some repackers through 7z archiving manage to almost halve the size of the game installer, taking into account the fact that initially all the game resources were already in the archives, so this is really a very serious thing (solid archive).

[Cole Proto]

For people that get frustrated by long update times. Check if your system can be upgraded with a NVMe hard drive. Last update took about 2mins on one.

[WarNerve]

Originally posted by Cole Proto:

For people that get frustrated by long update times. Check if your system can be upgraded with a NVMe hard drive. Last update took about 2mins on one.

Very true. I've got a Corsair MP600 2TB NVMe and it's orders of magnitude faster than a 7200 rpm HDD. I know a lot of people just have a small SSD boot drive to run their OS off of, but I put everything on my NVMe. Load times almost completely disappear in some games.

[Cole Proto]

To add. This isn't the only game suffering from long update times on Steam. People who have been playing Ark Survival Evolved like me know what a 400GB game including all DLCs update. Long story short, that beast surprised us with updates suddenly (not like Icarus that updates weekly on the same time) and it could completely take away the chance of playing a hour after work when an update dropped. Seriously, been there, done that, upgraded to NVMe since SSD prices are really low these days.

Also, you don't run Icarus on HDD. Just don't. You will get a bunch of "out of VRAM", stutters etc. just cause your drive can't handle it. Max speed SSD will solve a lot of problems especially for this game. Leave the HDD for museums. That tech is no longer acceptable for gaming.

[WarNerve]

Originally posted by Cole Proto:

To add. This isn't the only game suffering from long update times on Steam. People who have been playing Ark Survival Evolved like me know what a 400GB game including all DLCs update. Long story short, that beast surprised us with updates suddenly (not like Icarus that updates weekly on the same time) and it could completely take away the chance of playing a hour after work when an update dropped. Seriously, been there, done that, upgraded to NVMe since SSD prices are really low these days.

Also, you don't run Icarus on HDD. Just don't. You will get a bunch of "out of VRAM", stutters etc. just cause your drive can't handle it. Max speed SSD will solve a lot of problems especially for this game. Leave the HDD for museums. That tech is no longer acceptable for gaming.

I agree completely. The funny thing is, there are people out there that think a game requiring an SSD is "ridiculous", and that's it's "too soon", for games to be requiring SSDs. SSDs have been on the market for over 15 years at this point. And, like you said, they are very affordable. If gaming is your main hobby, or source of recreation at least, there's no reason to not have an SSD. The increase in speed is astronomical.

[ZombieHunter]

The patching is frustrating but also be aware that Steam has a tendency to fail to patch the game. I often get 'Corrupt Update Files' after it downloads the patch. Then if you let it do it again it fixes itself. I've also had Steam update the game but mess up 10 to 11 files which can be fixed by validation.

Steam use to have a problem where games that had not been played for awhile or had just been patched would somehow become corrupted. I thought this was fixed but I've had issues with RDR 2, GTA V, Icarus and other games on other drives that have had issues with this. Personally I think there is a bug somewhere in Steam's update system that is causing failed patches. I cannot 100% verify this but I have scanned all my drives (a mixture of SSDs and hard disk) and all are fine yet my games continually crash b/c they become corrupted over time even if they never had a CTD during my gaming sessions. But after a patch or after not playing for a bit they CTD.

So after every update I recommend you first verify files. Make sure that what Rocketwerkz put out there is what your system downloaded. B/c I know for sure this is not true in all cases.

I've often had RDR 2 go into super low res mode and fail to load in a new area. Then when I verify files Steam downloads some common Steamworks redist file and all is well. But after time the same thing happens no matter how often or how infrequent I play. I have no idea why Steam update has been so odd as of the past year or so. All my drives cannot be bad and no non Steam applications have any issue. MSVS starts up all the time every time and that is far more complicated. So either it just so happens all my drives are corrupt only where my Steam games are (highly unlikely) or Steam has introduced an issue in their update system that has not yet been fixed and/or found.

[WarNerve]

Originally posted by ZombieHunter:

The patching is frustrating but also be aware that Steam has a tendency to fail to patch the game. I often get 'Corrupt Update Files' after it downloads the patch. Then if you let it do it again it fixes itself. I've also had Steam update the game but mess up 10 to 11 files which can be fixed by validation.

Steam use to have a problem where games that had not been played for awhile or had just been patched would somehow become corrupted. I thought this was fixed but I've had issues with RDR 2, GTA V, Icarus and other games on other drives that have had issues with this. Personally I think there is a bug somewhere in Steam's update system that is causing failed patches. I cannot 100% verify this but I have scanned all my drives (a mixture of SSDs and hard disk) and all are fine yet my games continually crash b/c they become corrupted over time even if they never had a CTD during my gaming sessions. But after a patch or after not playing for a bit they CTD.

So after every update I recommend you first verify files. Make sure that what Rocketwerkz put out there is what your system downloaded. B/c I know for sure this is not true in all cases.

I've often had RDR 2 go into super low res mode and fail to load in a new area. Then when I verify files Steam downloads some common Steamworks redist file and all is well. But after time the same thing happens no matter how often or how infrequent I play. I have no idea why Steam update has been so odd as of the past year or so. All my drives cannot be bad and no non Steam applications have any issue. MSVS starts up all the time every time and that is far more complicated. So either it just so happens all my drives are corrupt only where my Steam games are (highly unlikely) or Steam has introduced an issue in their update system that has not yet been fixed and/or found.

Just out of curiosity, do you install your games to an SSD or to a HDD? I only ask because you said that you, quite often, encounter file corruption when downloading/installing games or patches,etc. This made me wonder if,perhaps you were using a HDD and it is malfunctioning in someway. In over ten years and hundreds of games downloaded,installed, and patched on Steam, I have never encountered any sort of file corruption, or errors such as the ones you are reporting.

[ZombieHunter]

Originally posted by WarNerve:

Originally posted by ZombieHunter:

The patching is frustrating but also be aware that Steam has a tendency to fail to patch the game. I often get 'Corrupt Update Files' after it downloads the patch. Then if you let it do it again it fixes itself. I've also had Steam update the game but mess up 10 to 11 files which can be fixed by validation.

Steam use to have a problem where games that had not been played for awhile or had just been patched would somehow become corrupted. I thought this was fixed but I've had issues with RDR 2, GTA V, Icarus and other games on other drives that have had issues with this. Personally I think there is a bug somewhere in Steam's update system that is causing failed patches. I cannot 100% verify this but I have scanned all my drives (a mixture of SSDs and hard disk) and all are fine yet my games continually crash b/c they become corrupted over time even if they never had a CTD during my gaming sessions. But after a patch or after not playing for a bit they CTD.

So after every update I recommend you first verify files. Make sure that what Rocketwerkz put out there is what your system downloaded. B/c I know for sure this is not true in all cases.

I've often had RDR 2 go into super low res mode and fail to load in a new area. Then when I verify files Steam downloads some common Steamworks redist file and all is well. But after time the same thing happens no matter how often or how infrequent I play. I have no idea why Steam update has been so odd as of the past year or so. All my drives cannot be bad and no non Steam applications have any issue. MSVS starts up all the time every time and that is far more complicated. So either it just so happens all my drives are corrupt only where my Steam games are (highly unlikely) or Steam has introduced an issue in their update system that has not yet been fixed and/or found.

Just out of curiosity, do you install your games to an SSD or to a HDD? I only ask because you said that you, quite often, encounter file corruption when downloading/installing games or patches,etc. This made me wonder if,perhaps you were using a HDD and it is malfunctioning in someway. In over ten years and hundreds of games downloaded,installed, and patched on Steam, I have never encountered any sort of file corruption, or errors such as the ones you are reporting.

SSD. But even then I said all drives have been scanned. They are fine. I also said that all other applications on said drives work all the time every time. If this did not occur or patching was perfect then why does this game have a sticky about making sure you verify files after every update? Why on 9 out of 10 updates does it fail on Icarus with Corrupt update files? But then when you click it and let it download again it works? I've been on Steam for a long time and this has been an issue on Steam for a long time across multiple systems, multiple drives, multiple configurations and so on. The Steam cache often has issues so you must clear it periodically.