Evil West- Vulkan shader cache download does nothing?

[Flappy Pannus]

Due to being an UE4 game without a precompiling step, this game has some notorious shader stutter on Windows, of a level that not even gplasync has any effect on, which is a first. Async is never a silver bullet for JIT shader compiling stutter mind you, but there usually is at least _some_ reduction in stutter using it - not here. Tried it from Windows using just drop in gplasync dll's and from Lutris forcing gplasync, no change.

So, I returned my GoG version and got it on Steam, hoping that at least the downloaded Vulkan cache would alleviate it. It downloads just under 300mb of vulkan caches, which when processed consist of around 500mb on disk. But...no change. Not even on the first level, same exact stutters in the opening as with no cache. Steam compiles the shaders for about ~1 minute on bootup, but there is no reduction in shader stutter at all.

Can wipe the GlCache in the Steam shader cache folder and notice no difference in several levels. Only way to get rid of them is by playing through the game myself and compiling them through gameplay. So what the heck is that 500mb of cache doing?

I mean sure it's not a huge hit of a game, but it's from late 2022 so I would expect there would be enough shared cache distributed by now to cover a good chunk of it, but it seems to have no effect. Other shared caches for other games seem to work.

Tried several Proton versions with no change.

Anyone else?

Rig: RTX 3060, latest stable Nvidia release
I5-12400, 32GB DDR4
OS: System 76 POP! OS

[XLR8]

RTX 3060 no need pre-compiled shaders, turn it off, its usually for low-level GPUs

[Flappy Pannus]

Originally posted by Ringo:

Obligatory "if you're running Mesa 23.1+ you can probably just disable Shader Prechaching" comment

Obligatory "Graphics Pipeline Library does absolutely nothing for games that don't do any precompiling". As evidenced by the very title I'm talking about. I mean if no shader download was necessary, then I wouldn't have ever had shader stuttering with the GoG version either, and this thread wouldn't have been posted.

I see this constantly whenever someone dare mentions dxvk async, "You don't need it anymore!", so many people fundamentally don't understand the issue with Unreal Engine 4 games and shader processing and what GPL actually is addressing.

GPL does nothing for these games as just like Windows, the driver has no idea of the shader until the draw call happens, because the developer did not make any attempt to precompile the shaders up front. For games like Batman: Arkham Knight, Days Gone, or the majority of other Windows games that do in fact, have a precompiling stage on bootup, with GPL you will indeed have little to no shader stutter without Steam's downloaded caches. It works great there.

For UE4 games that don't bother with that step though, it does nothing. Turn off shader downloading for example and fire up Borderlands 3. Massive, crippling stutter. You need that *6 GB* of shader cache download as that's the only way to have those precompiled. GPL does nothing here.

[Flappy Pannus]

Originally posted by XLR8:

RTX 3060 no need pre-compiled shaders, turn it off, its usually for low-level GPUs

Shader compiling is a CPU bottleneck, not a GPU. The power of your GPU means nothing wrt how long a shader compiling stage takes. You would get the exact same length of shader stutter on a 4090 vs. a 3050 if both cards were installed in a system with the same CPU.

And obviously it needs them, or I wouldn't have had stutters. jfc.

[Der tüddelige Fußgänger]

Originally posted by Ringo:

Obligatory "if you're running Mesa 23.1+ you can probably just disable Shader Prechaching" comment

Thanks for the reminder that newer mesa versions fixed shader caching.
I had over 20GBs of shaders before.

[Der tüddelige Fußgänger]

Originally posted by Flappy Pannus:

Originally posted by Ringo:

Obligatory "if you're running Mesa 23.1+ you can probably just disable Shader Prechaching" comment

Obligatory "Graphics Pipeline Library does absolutely nothing for games that don't do any precompiling". As evidenced by the very title I'm talking about. I mean if no shader download was necessary, then I wouldn't have ever had shader stuttering with the GoG version either, and this thread wouldn't have been posted.

I see this constantly whenever someone dare mentions dxvk async, "You don't need it anymore!", so many people fundamentally don't understand the issue with Unreal Engine 4 games and shader processing and what GPL actually is addressing.

GPL does nothing for these games as just like Windows, the driver has no idea of the shader until the draw call happens, because the developer did not make any attempt to precompile the shaders up front. For games like Batman: Arkham Knight, Days Gone, or the majority of other Windows games that do in fact, have a precompiling stage on bootup, with GPL you will indeed have little to no shader stutter without Steam's downloaded caches. It works great there.

For UE4 games that don't bother with that step though, it does nothing. Turn off shader downloading for example and fire up Borderlands 3. Massive, crippling stutter. You need that *6 GB* of shader cache download as that's the only way to have those precompiled. GPL does nothing here.

aayyyy oh wait…
I suppose I unhecked that box in Steam's settings a bit too early.

[Flappy Pannus]

Originally posted by Felix:

[
aayyyy oh wait…
I suppose I unhecked that box in Steam's settings a bit too early.

You'll figure it out soon enough if you play games that don't have any precompiling step, they'll stutter without any downloaded caches. Most Windows games do some form of precompiling however.

Steam could really use a feature to enable downloadable shader caches on a game by game basis.

[Marlock]

Originally posted by Flappy Pannus:

Steam could really use a feature to enable downloadable shader caches on a game by game basis.

THIS, so much THIS!

Shader pre-caching (downloading from steam servers) is a nuissance if on by default for all games, especially if a user has a big locally installed library and/or slow/metered internet and/or limited drive space

On the other hand the driver improvements haven't made the feature completely obsolete yet, it's a useful fix for some games

Ergo: let us use it as a fix only for games that need it

ps: to add to the confusion, valve distributes re-encoded game videos (to avoid issues on linux with lacking patent-encumbeted codecs in some setups) as if they were part of the shader pre-cachong feature, instead of exposing it as a separate option like it should be

so a lot of the size of shader pre-cache downloads sometimes is actually recoded videos... and disabling shader precaching can break in-game full motion videos, which doesn't make any sense unless you know the mess valve did

[XLR8]

Originally posted by Flappy Pannus:

Originally posted by XLR8:

RTX 3060 no need pre-compiled shaders, turn it off, its usually for low-level GPUs

Shader compiling is a CPU bottleneck, not a GPU. The power of your GPU means nothing wrt how long a shader compiling stage takes. You would get the exact same length of shader stutter on a 4090 vs. a 3050 if both cards were installed in a system with the same CPU.

And obviously it needs them, or I wouldn't have had stutters. jfc.

In Apex Legends I had caches with about 13GB one year ago, so just kill pre-cached shaders by Steam option. Mesa do it much better and dont need to update every day about 10 GB of cache from Steam servers, so its question not in CPU or GPU computing, but in your time wasting when you have installed very intensive pre-cached shaders games

[Flappy Pannus]

Originally posted by XLR8:

In Apex Legends I had caches with about 13GB one year ago, so just kill pre-cached shaders by Steam option. Mesa do it much better and dont need to update every day about 10 GB of cache from Steam servers, so its question not in CPU or GPU computing, but in your time wasting when you have installed very intensive pre-cached shaders games

That has nothing to do with my reply. You don't understand what this thread is about.

[Marlock]

fossilize is an awesome piece of tech that can guess which shaders a game might need without actually running the game...

...but that does generate a bunch of precompiled shaders that the game won't really ever use, which is part of the larger storage space occupied by the pre-cached shaders distributed by Steam

the other, probably larger, part of shader pre-caching storage space that exceeds storage space occupied via on-the-fly shader compilation is the re-encoded videos which i mentioned earlier

all that being said, each user will have a different reason to like or dislike shader precaching, given factors that were also mentioned earlier

[Flappy Pannus]

Originally posted by Marlock:

fossilize is an awesome piece of tech that can guess which shaders a game might need without actually running the game...

...but that does generate a bunch of precompiled shaders that the game won't really ever use, which is part of the larger storage space occupied by the pre-cached shaders distributed by Steam

Thanks, this is where I'm confused though - where fossilize gets this data from, and what this data actually consists of. The most common 'theory' I've read online is that at least the DXVK state cache data is uploaded from other users who have played the game in question and generated it, albeit the downloaded shader size (videos excluded) are far larger than your typical state cache, so that doesn't really fit. Is the data rather Vulkan pipelines that are not GPU Driver specific, then the binary shader blob (\GLCache) is generated from your driver based on that data?

Does Valve generate this pipeline data themselves or is it actually uploaded from gamers automatically? I understand Valve likely handles it for the Steamdeck at least, but I was under the impression for just regular Steam Linux PC, this data is shared from the community. This not the case?

Any site which goes into this process in detail? The fossilize github doesn't really explain it in these terms, or at least ways that I can understand.

[Marlock]

Fossilize is a precompilation software that can run on a user machine
https://github.com/ValveSoftware/Fossilize

AFAIK Steam runs fossilize as a background task and can grab the precompiled shaders from some users and redistributes them across the other users with similar setups (same gpu model + os version + gpu driver version + steam version + proton version if windows game + game version) if they have pre-caching enabled... but maybe there are other ways to generate the precompiled shaders without this workload being distributed between end-users...

IIRC those things were thoroughly explained when Valve first released fossilize and the pre-caching feature... Gaming-on-Linux most likely published something about this at the time, and it's likely Phoronix did too

edit:
https://www.gamingonlinux.com/2020/05/steam-beta-adds-vulkan-shader-processing/
https://www.phoronix.com/news/Valve-Fossilize-Vulkan