Masking Keysight for semi-transparency

This guide will cover how to use the "mask mode" toggle found under the advanced Scene menu to achieve perfect semi-transparency in OBS for Keysight, with no chroma-key artifacting or loss of colour range!

What is "Mask Mode"?

[Guide written using Keysight version 1.6.1]

It is a common desire to make Keysight semi-transparent, ie to show the notes and particles and stuff without actually having a background, so that you can overlay it onto another scene. Well, back in Ye Olde Days of Keysight you'd have to do this with janky chroma-keying, but no longer!

Enter mask mode



This can be activated by setting the menu mode to Advanced, and then heading into Scene. Alternatively, the M key or right thumbstick click on a controller will toggle Mask Mode.



Theory behind this: Keysight first multiplies the source blue channel by 0.495. Then, any pixels which are flagged as "geometry" that need to have 100% alpha have 0.5 adding to the blue channel on that pixel. An OBS shader then detects whether or not a pixel is at or above 0.4975 in the blue channel, and splits out a binary mask image and corrects the original blue'd image back to the intended colours. The shader then calculates a variant of the correct image with all pixel brightnesses set to 100%. Then for any mask pixels, it shows the corrected image with 100% alpha; and for non-mask pixels it shows the max-brightness image with an alpha that is the corrected image's pixel brightness. Convoluted, but works like a charm!

Requirements

You cannot use Streamlabs OBS for this, I'm sorry! This is due to requiring the following:

- WINDOWS: This Shaderfilter plugin[github.com] for OBS. Simply copy the /obs-studio folder and paste it into wherever you installed OBS so it merges with the /obs-studio folder found there (typically C:/Program Files/ )

- MACOS: This Shaderfilter plugin installer[github.com] for OBS. Simply double click the .PKG file to install it!

- This specially crafted little shader.[drive.google.com] If you're slightly suspicious of random Google drive links, the source code is available at the bottom of this guide, and can be saved as a .txt shader file from any text editor like Notepad

These files can also be found included with the Keysight installation itself! Simply right click on Keysight in your Steam library > Browse local files > /Keysight/Extra resources/Mask mode to access them.

Oh, and: please, if you're a SLOBS user who wants to take advantage of semi-transparency with Keysight, making the switch over to regular OBS is not as painful as you might imagine and I recommend doing it for a whole host of reasons!

Keysight configuration

After turning on Mask Mode by using the menu toggle / the M key / right thumbstick click on a controller, there are some extra toggles under the Semi-transparency tab under Advanced > Scene



Toggling these will force the respective component to always be opaque as part of Mask Mode, but it's only necessary to switch away from defaults if going for a very niche, highly custom Keysight implementation

OBS configuration (basic)

There are two ways to do mask mode in OBS. This method is faster and does not require more OBS version 28 and above, but potentially introduces dark fringes around colours if you overlay Keysight onto bright scenes.

First off, if you haven't already, use a game capture pointed at Keysight to add Keysight as a source. (Or window capture, if on MacOS and do not have "Game Capture" as an option)



Next up: add a User-defined filter filter by right clicking on your Game Capture source with Keysight, heading into Filters, clicking the little plus button and selecting User-defined shader




Finally, browse to wherever you saved that .txt shader and simply select it. Done!



And now, enjoy semi-transparency without any of the limitations of traditional chroma-keying! Simply import the same Game Capture anywhere you want to see Keysight

OBS configuration (advanced)

There are two ways to do mask mode in OBS. This method is slower and requires OBS version 28 or above, but is guaranteed to look perfect against any backdrop.

We're going to need to nest some scenes for this. First off, create two scenes: "Keysight Geometry" and "Keysight Glow", and add the same game capture pointed at Keysight to both of these scenes.



On the "Keysight Geometry" scene (not the game capture!) add a User-defined filter, set it to text file mode, point it at the mask mode shader file, and then set the bloom alpha to 0.00



Next, on the "Keysight Glow" scene, do the same thing but this time set note alpha to 0.00 and set bloom alpha to 1.00



What this has done for us is give two layers of Keysight: one containing the stuff we definitely want to see all the time, and then the other containing just the semi-transparent elements. (The reason this is important is because we want to perform normal alpha overlaying on the geometry portion, but additive blending on the semi-transparency, to prevent OBS from blending towards a darker colour when overlaying semi-transparent colours onto a bright background.)

Any place you need Keysight, add both Keysight Geometry and Keysight Glow as scene sources to your desired scene. Please note than you cannot nest the Geometry and Glow scenes, they always need to be added individually.



Finally, right click on the Keysight Glow scene source, head to Blending mode, and select "additive". Depending on your backdrop and Keysight preset, you may see a huge or a minor change, but trust me, it's important!



Just to demonstrate the difference this can make, here is a "worst case scenario" comparison. Notice the grey fringes around the octave lines, for example.

Using "Glow" scene with normal blending


Using "Glow" scene with additive blending

FAQ

"I have blue fringes around things in my OBS scene. What is going on?"

Blue fringes are caused by not having a 1:1 pixel-to-pixel mapping within OBS. For example, if your OBS canvas is 1280x720, but Keysight is running in 1920x1080, you will see blue fringes thanks to the shader-based mask mode being applied *after* OBS's downsampling from 1080p > 720p. Please note that you will always see blue fringes in the source preview, since this preview is always scaled down before it applies your chosen filters, but this doesn't necessarily mean your actual OBS video will have blue fringes (you can see these if you zoom in on the above screen of OBS).

The fix is to run Keysight at exactly your canvas resolution within OBS. If you want Keysight to not fill the entire canvas in OBS, you will also need to "nest" the Keysight capture within one scene, apply the filter to the Keysight source, and then import the Keysight nested scene anywhere you want Keysight before resizing it. This means you have 1:1 resolution going from Keysight > OBS, *then* the shader is applied giving you semi-transparency, *then* the source is resampled to your desired size.

"The edges of things look jagged and ugly. How can I fix this?"

Any switch to full opacity (such as on note objects) is occurring as a binary "on/off" switch *per-pixel*. This means angled edges become aliased.

Either use a Keysight implementation where the edges of things are straight, or consider using brightly glowing objects. This aliasing is most commonly seen on note objects, and having note object borders glow brightly nicely masks the aliasing with some extra softness around the objects caused by lighting bloom.

"Can I use Mask Mode in a video editor?"

I don't think so, but there's no hard reason why it shouldn't. The issue is that I've not found a video editor which can execute shader code on a video source, but I don't see any reason this should be impossible. If anyone has any ideas on this, I would be eager to hear them so I can update this portion of the guide!

TL;DR

- Press M to enable mask mode in Keysight
- Download and install this plugin for OBS[github.com] if on Windows, or this plugin on MacOS[github.com] and this shader[drive.google.com]
- Add a User-defined shader filter to your Keysight game capture
- Set the filter to text file mode, and browse to the shader file in that filter and apply it

Shader source code

uniform float note_alpha = 1.0;
uniform float bloom_alpha = 1.0;
uniform float opacity_range = 1.0;

float4 mainImage( VertData v_in ) : TARGET
{
float2 uv = v_in.uv;
float blue_multiplier = 0.495;

float2 baseUV = float2(uv.x, uv.y);

float4 base = image.Sample(textureSampler, baseUV);

float note_mask;
if(base.b >= (0.5-((0.5 - blue_multiplier)*0.5)))
note_mask = 1.0;
else
note_mask = 0.0;

float3 corrected = float3(base.r, base.g, (base.b-(0.5*note_mask))*(1/blue_multiplier));

float pixel_multiplier = 1.0/(max(corrected.r, max(corrected.g, corrected.b)));
float bloom_mask = (clamp(((max(corrected.r, max(corrected.g, corrected.b)))*bloom_alpha), 0.0, 1.0))-note_mask*(1-note_alpha);
bloom_mask = (bloom_mask - (1-opacity_range)) * (1/opacity_range);
bloom_mask = (clamp(bloom_mask, 0.0, 1.0));

float3 bloom_layer = float3(corrected.r*pixel_multiplier, corrected.g*pixel_multiplier, corrected.b*pixel_multiplier);
float3 mixed_layer = float3((lerp(bloom_layer.r, corrected.r, note_mask)), (lerp(bloom_layer.g, corrected.g, note_mask)), (lerp(bloom_layer.b, corrected.b, note_mask)));

return float4(mixed_layer.rgb, (note_mask*note_alpha)+bloom_mask);
}

[Egglyberts]

Ah, apologies for the slow response! And also the unhelpful one: unfortunately, mask mode is incompatible with Streamlabs OBS. To the best of my knowledge and a quick google, there's no way of getting shaders into SLOBS :( As the guide says though, if this is important to you, I promise the transition to normal OBS is straightforward and you can mirror or improve any existing functionality over there!

[Petty Fox]

Is there a way to do this with Streamlabs? Having trouble with getting this going with Streamlabs

[bhh1988]

Save that code to a text file, and load it using the User-defined filter, and you should see the keysight screen captured and filtered! (other caveat is that I'm on a Macbook pro M1, and I'm not sure how the hardware might affect what keywords are recognized in the shader code). If you're not seeing the screen appear, you gotta go to the log files (Help menu at the top) and debug.

Finally, for the sync'ing of the midi playback with video playback, it helped to play-back both in 0.25 speed (I used VLC for the video-playback, which goes down to 0.25 speed easily), so that when I Alt-tab between the apps to try to quickly playback both at the same time, the error in timing effectively gets reduced by 4x. I still needed to tweak the start-offset of the midi-playback and it still took me several tries, but overall it worked out! I apologize for the wall of text, and thank you so much egglyberts!

[bhh1988]

```
float bloom_mask = (clamp(((max(corrected.r, max(corrected.g, corrected.b)))*bloom_alpha), 0.0, 1.0))-note_mask*(1-note_alpha);
bloom_mask = (bloom_mask - (1-opacity_range)) * (1/opacity_range);
bloom_mask = (clamp(bloom_mask, 0.0, 1.0));

float3 bloom_layer = float3(corrected.r*pixel_multiplier, corrected.g*pixel_multiplier, corrected.b*pixel_multiplier);
float3 mixed_layer = float3((lerp(bloom_layer.r, corrected.r, note_mask)), (lerp(bloom_layer.g, corrected.g, note_mask)), (lerp(bloom_layer.b, corrected.b, note_mask)));

return float4(mixed_layer.rgb, (note_mask*note_alpha)+bloom_mask);
}
```

(remove the back-ticks, I hoped they'd give me nice code-formatting).

[bhh1988]

The code doesn't all fit in 1000 characters, so I'll split it in half:

```
uniform float note_alpha = 1.0;
uniform float bloom_alpha = 1.0;
uniform float opacity_range = 1.0;

float4 mainImage( VertData v_in ) : TARGET
{
float2 uv = v_in.uv;
float blue_multiplier = 0.495;

float2 baseUV = float2(uv.x, uv.y);

float4 base = image.Sample(textureSampler, baseUV);

float note_mask;
if(base.b >= (0.5-((0.5 - blue_multiplier)*0.5)))
note_mask = 1.0;
else
note_mask = 0.0;

float3 corrected = float3(base.r, base.g, (base.b-(0.5*note_mask))*(1/blue_multiplier));

float pixel_multiplier = 1.0/(max(corrected.r, max(corrected.g, corrected.b)));

```
(continued) in next comment

[bhh1988]

Ok, so I successfully got this working and just wanted to report back here with what I did. I'm on a Mac, so I could NOT use the ShaderPlus OBS plugin. Instead I dug up this reddit thread with a mac-build of obs-shaderfilter: https://www.reddit.com/r/obs/comments/kzg8x1/comment/gjozpf6/?utm_source=share&utm_medium=web2x&context=3 .

Shady link? Perhaps, but it's open-source and shady links fit the Shader theme anyway :P

If you install that, the filter you want is "User-defined Shader". And then Egglybert's HLSL shader source code ALMOST works, except there's a few syntax changes to be made. In summary, I had to remove the pragmas, rename the main function, and change the name of the texturesampler variable. See the following comment for the code (character-limit).

[Egglyberts]

The other option would be attempting to sync the midi playback with video playback in OBS. If you have a little bit of allowable error (like a quarter of a second) in your desired sync, this COULD work if you did something like: build a video with a clear countdown to when you need to hit play on your midi file, use the "delay playback" option in Keysight to give yourself time to close the Keysight menu (or run the file into Keysight via a DAW), have the video in the background of OBS, and then trim the recorded video in editing to remove the sync time at the start.

I'll admit that both options are really undesirable, but honestly if you (or anyone reading this) has actual solutions to alpha video that I can bake into Keysight I'd love to hear them, as I've spent a very long time now frustrated at this limitation!

[Egglyberts]

You're correct, mask mode is very much aimed at live-streaming, or at least folks using OBS. It's weird to me that transparency channels in video is STILL such a big problem, but it really is just something with very little support.

Theoretically, if your video editor could use HLSL shaders, you could apply mask-mode "offline", but doing some quick searching around I couldn't really find this as something anyone was trying to do. Due to the more program-y nature of it, perhaps Blender video editing could somehow support it?

In terms of what you can do right now regardless of tools or research, there are two potential things: sacrificing all semi-transparent effects (such as particles and pulses), removing the background, setting void colour to green and just chroma-keying it. This would leave you with just note objects and any lighting on them, but if that's all you need it would be the simpler solution perhaps?

[bhh1988]

Correct me if I'm wrong, but this seems to pertain to live-streams, yes? So you would have the masked game-capture source, and underneath that source you would put in a video background source, for example, and stream that out in OBS?

The problem I'm trying to solve right now is creating a YouTube video, and I want to synchronize things just right so that certain parts of the background video show up at certain times of the piece. I guess I could prep the background video in advance and basically try to press play on my static MIDI file in Keysight and press play on the background video at the same time, so they're sync'ed...but I'm wondering if there's a better way for this...

The ideal thing would be if I could output a video with transparency from OBS, and then work on layering the background video underneath in something like After Effects. But I'm pretty sure OBS can't do that.

[social]

Just got it working in Piano Zen stream for the first time tonight! -eb