I found a way to bypass the UI in Reshade.

[Hambango]

Hi!
I did a little bit of tinkering in the last few days and stumbled upon something I'd like to share with you:
The ones of you with experience with post process injectors like Reshade probably know that there has always been one problem: due to the shader effects being applied to the finished picture there's no way for the injector to distinguish between the game's UI and the 3D scene, which means the UI gets affected as well. The results can be quite ugly and most effects can only be used for fancy screenshots but rarely while actually playing the game.

I found a way to bypass TW:WH2's UI for Reshade.

I made this primarily for my own private use and due to time constraints and because I'm usually not active on Steam ( first post, yay! ), I don't intend to publish it on the Steam Workshop.
Anyone who wants to publish it or make his own changes to it, is very welcome to do so. No credit to me needs to be given.
I did the same thing years ago with my original Spectator / AI General mod on the official Total War forums ( https://forums.totalwar.com/discussion/190065/spectator-mod/p1 ). That mod was eventually brought to the WS by my stalwart modding companion Decomposed and recently much improved by the amazing work of Paperpancake.
This one is far less complex and should require next to no maintenance.

The mod consists of a .pack file for TW:WH2 and a shader file for Reshade.

The shader basically creates a mask for the UI on the fly by looking for pixels with an alpha component of less than 1.0.

The ingame mod checks for UI components and reduces their opacity by a tiny, unnoticeable amount, which means that they are getting added to the UI mask.

It works for both battles and the campaign map. I haven't found a good method to implement it for the game's main menu and options frontend, but I'd consider those of low priority.

It has a bit of a problem with UI images that have a slightly translucent frame, which can lead to a small halo for some effects. This can probably only be prevented by modding the actual image.
While the game is paused during a battle, new UI components won't get the opacity treatment by the mod. This happens because the repeating callback will only trigger again once the game is resumed.

But otherwise it seems to work pretty well.

Here are some examples of the 'mod' in action:

Vanilla without Reshade:
https://steamcommunity.com/sharedfiles/filedetails/?id=2070202232

With Reshade but without the mod. You can clearly see how the UI is affected:
https://steamcommunity.com/sharedfiles/filedetails/?id=2070202130

With Reshade and with the mod:
https://steamcommunity.com/sharedfiles/filedetails/?id=2070202185

Vanilla campaign map:
https://steamcommunity.com/sharedfiles/filedetails/?id=2070202333

Campaign map with Reshade and mod:
https://steamcommunity.com/sharedfiles/filedetails/?id=2070202284

Installation:

Get the Reshade package from here: https://reshade.me/ and install it into your TW:WH2 folder. The installer also lets you pick which Reshade shaders you want to install.

Download my shader file: www.mediafire .com/file/h4yu7ow4l5kbeot/TWUIBypass.fx/file and put it into your ..\Total War WARHAMMER II\reshade-shaders\Shaders directory.

Download my mod file: www.mediafire .com/file/kfrs2ambd6w7csl/ui_bypass.pack/file and put it into your ..\Total War WARHAMMER II\data directory. Make sure to activate it in your game launcher.

No idea why Steam censors those links. Remove the blanks before .com.

Ingame on the shader list in the Reshade options menu the function 'UI_Bypass' has to go all the way to the top. Every shader between the functions 'UI_Bypass' and 'UI_Bypass_Stop' on this list will not affect the UI. Both functions have to be active.

IMPORTANT: Please make sure that the sharpening effect is turned on in your game's graphics options, otherwise the shader might mask some objects it's not supposed to.
A big thank you goes to Weirding Way for discovering this!

That's it.

I hope this will be of use for some of you.


Have a good one!


EDIT: made some necessary changes to the instructions.

[Vidmeister]

Hi, Hambango. I tried to use your shader file as suggested and also followed the steps you gave. It shows up in my ReShade list and I have also got the UI_detect and the before and after activated. The pack file is also enabled in KMM. I just cannot for the life of me get it to actually work, so I was wondering if you have any idea if KMM is messing with the file or where it should be placed in the load order (if the pack file is indeed the problem)?

[Hambango]

Originally posted by Benji Vida:

Hi, Hambango. I tried to use your shader file as suggested and also followed the steps you gave. It shows up in my ReShade list and I have also got the UI_detect and the before and after activated. The pack file is also enabled in KMM. I just cannot for the life of me get it to actually work, so I was wondering if you have any idea if KMM is messing with the file or where it should be placed in the load order (if the pack file is indeed the problem)?

Hi!
Sounds like you're using the wrong shader. UI_detect is a shader that comes with Reshade's shader package. It's used to deactivate all effects when a certain predefined pixel configuration is found on-screen.
The effects of my shader are called 'UI_Bypass', UI_Bypass_Start' and 'UI_Bypass_Stop', as described in my instructions.

I've got no experience with KMM. Never used it, sorry.

[Vidmeister]

I knew I was being a dumbo. Thanks for the heads-up. It kinda works now. Sadly the UI
_Bypass_Start does not seem to change anything no matter where I put it. Just to clarify: the order they have to be in is UI_Bypass then UI_Bypass_Start then UI_Bypass_Stop? (Just the general order without taking into account shaders inbetween). AO shades seem to mess with it alot and I also have the problem with fog and clouds being added to the UI. Same with AO effects on torches and such.

[Hambango]

Originally posted by Benji Vida:

I knew I was being a dumbo. Thanks for the heads-up. It kinda works now. Sadly the UI
_Bypass_Start does not seem to change anything no matter where I put it. Just to clarify: the order they have to be in is UI_Bypass then UI_Bypass_Start then UI_Bypass_Stop? (Just the general order without taking into account shaders inbetween). AO shades seem to mess with it alot and I also have the problem with fog and clouds being added to the UI. Same with AO effects on torches and such.

Yes, UI_Bypass has to be at the very top of the list, otherwise it won't work properly. All shaders that are placed between UI_Bypass_Start and UI_Bypass_Stop should not affect the UI.
It's very strange that some people are having problems with clouds/fog while for others it seems to work perfectly. I've since chatted with three people on Steam about this mod, two had no issues and the third had exactly the same problem.
Sadly, I still haven't been able to replicate the issue myself. Maybe it's caused by some graphical mod or a specific shader? Which kind of AO shaders are you using? I'm using Quint's MXAO myself without any problems.

[Brokensword]

Hey Hambango, really nice to see somebody trying to create something to bypass the issue of our pesky UI ruining all of our nice effects!
Unfortunately I'm running into an issue that is likely due to my very limited experience with ReShade so here goes:

There seems to be some sort of large oval shaped area on the screen which separates where the added effects (DoF etc.) are applied. Effects are properly blocked from all UI elements and still apply correctly to the rest of the game...except towards the edges of the screen where this large 'oval' seems to end. Am I missing something very simple or have a created my own graphical vortex? :P

Thanks for the good work!

[Vidmeister]

Originally posted by Hambango:

Originally posted by Benji Vida:

I knew I was being a dumbo. Thanks for the heads-up. It kinda works now. Sadly the UI
_Bypass_Start does not seem to change anything no matter where I put it. Just to clarify: the order they have to be in is UI_Bypass then UI_Bypass_Start then UI_Bypass_Stop? (Just the general order without taking into account shaders inbetween). AO shades seem to mess with it alot and I also have the problem with fog and clouds being added to the UI. Same with AO effects on torches and such.

Yes, UI_Bypass has to be at the very top of the list, otherwise it won't work properly. All shaders that are placed between UI_Bypass_Start and UI_Bypass_Stop should not affect the UI.
It's very strange that some people are having problems with clouds/fog while for others it seems to work perfectly. I've since chatted with three people on Steam about this mod, two had no issues and the third had exactly the same problem.
Sadly, I still haven't been able to replicate the issue myself. Maybe it's caused by some graphical mod or a specific shader? Which kind of AO shaders are you using? I'm using Quint's MXAO myself without any problems.

So the thing is, your shader works completly without UI_Bypass_Start for me. That part does not do anything. Only moving the End file changes what is applied, and what is not. I don't know why this happens. Say I move the Start under AO (also using Quints btw), the AO ist still not applied to the UI. Same with every other effect. When I move the End it actually does what I expect it to do.

Another weird thing is the fact, even if I turn off all the shaders in ReShade, the Statistics still shows UI_Bypass picking up the clouds around the vortex, smoke/fog and the little fires in cities as UI elements. It just makes no sense to me.

Another thing to note: earlier you talked about line 15 in the code. For me that only has the line without the "&&" part you mentioned earlier. I will try copy and pasting it in, to check the results.

[Hambango]

@ Benji Vida:
Yeah, you're right about the UI_Bypass_Start function doing nothing. I changed the shader code shortly after posting it here to prevent some black framing around the UI which I just noticed at the time. These changes also made the Start function completely obsolete, which I haven't noticed until now. My bad, lol. Thanks for pointing it out! I'll update the file in the OP and the instructons accordingly. With these changes I also changed line 15 as tightening the alpha constraints didn't really lead to better results, which is why the line differs from the one in my comment.

I'm still baffled why some graphic elements are picked up by my shader for some people.
Do you run any graphics mods?
What GPU are you using? Maybe different GPUs / drivers use different methods to display the clouds/fog?
I'm currently using a RTX 2080ti with the newest drivers. I just tested it with the three pcs in our house. I works flawlessly for all of them. On all three Reshade Statistics shows that it's only picking up the UI, as it should. Our kids' pc is a clone of mine and uses an identical GPU, my wife's system has a GTX 1070. Sadly, I haven't been able to test it on an AMD GPU yet.


@Brokensword:
I'm glad you're finding my little 'mod' useful. :)
I'm not entirely sure what the oval shape you're describing might be. My first instinct was that it might have something to do with the game's vignette effect. I activated it in my game, but sadly I still don't have this issue on my system. You could still give it a try and deactivate the vignette effect in your game's graphics options, though. Maybe it'll do the trick for you.

[Vidmeister]

Originally posted by Hambango:

@ Benji Vida:
Yeah, you're right about the UI_Bypass_Start function doing nothing. I changed the shader code shortly after posting it here to prevent some black framing around the UI which I just noticed at the time. These changes also made the Start function completely obsolete, which I haven't noticed until now. My bad, lol. Thanks for pointing it out! I'll update the file in the OP and the instructons accordingly. With these changes I also changed line 15 as tightening the alpha constraints didn't really lead to better results, which is why the line differs from the one in my comment.

I'm still baffled why some graphic elements are picked up by my shader for some people.
Do you run any graphics mods?
What GPU are you using? Maybe different GPUs / drivers use different methods to display the clouds/fog?
I'm currently using a RTX 2080ti with the newest drivers. I just tested it with the three pcs in our house. I works flawlessly for all of them. On all three Reshade Statistics shows that it's only picking up the UI, as it should. Our kids' pc is a clone of mine and uses an identical GPU, my wife's system has a GTX 1070. Sadly, I haven't been able to test it on an AMD GPU yet.

I am running Unit reskin mods and a campaign map fog remover. I tried disabling that, but it made no difference sadly. I do indeed run with a RX 480 AMD card, so that might actually be something to consider. Would adding the line back into the code change anything at all?

[Brokensword]

Well, you were right! It was literally just the game's vignette option; how I didn't think to test that myself is mind-boggling but regardless, all sorted.

Thanks again, I hope this shader gets some good traction :)

[Velovar]

Small (or big) update, if anyone is having trouble with clouds or particles bypassing the mod make sure you have Sharpening turned on in graphics options (that fixed it for me)!

Once again Hambango thank you, I will update my Reshade mod soon and include a link to this thread.
You are amazing.

[Hambango]

Originally posted by Weirding Way:

Small (or big) update, if anyone is having trouble with clouds or particles bypassing the mod make sure you have Sharpening turned on in graphics options (that fixed it for me)!

Once again Hambango thank you, I will update my Reshade mod soon and include a link to this thread.
You are amazing.

Oh wow, thank you so much for this! That is indeed the solution to our little riddle.
I've always got sharpening turned on and it never even crossed my mind that it could have any impact on the alpha value of some objects.
When sharpening is off, I too am having the issue of some clouds and the vignette being picked up by my shader.
I'll update the OP with this new information.

Again, thanks a lot! Looking forward to your updated Reshade mod! :)

[enggraphice]

not working for me , tried everything followed your steps with / without velovar preset and nothing change . is there something i missed , can you clarify what did you mean with (transparency effect) . i wish you to upload your preset it'll save a lot

[NGH]

Hey man! This is absolutely excellent. I have a few questions:

1) Have you ever tried this type of method with any other UI heavy games? I am assuming because of the .pack its strictly a fix for TW:Warhammer 2 only but I am curious if this potential method would be possible with other games where you could modify the ui in a similar fashion.

2) There seems to be some flickering with UI elements, I assume there's no way around that right? It's mostly noticeable with the diplomacy screen.

Overall though, this is amazing. It fixes one of the biggest issues using ReShade in this game. Now I can use MXAO and Pascal Glitcher's RTGI and not have my UI all messed up.

Sincerely, you get a big thanks from me :)

[AbortedMan]

How does one uninstall this if they want to revert the UI back to original? I've removed the .pack and shaders and my UI is still transparent.

Derp, never mind. I'm seeing ray tracing through the UI from the RTGI shader.

[AbortedMan]

When using this with Malus Darkblade the purple "T'zarkan" haze that's around the UI creates a ring effect very similar to what it looks like when vignette is enabled in the graphics options. Is there any possible workaround for this?