AR-style 3D camera alignment

Want to overlay Keysight visual effects onto a camera angle to blow people away with augmented-reality notes? This guide will cover the process of aligning the Keysight 3D camera to an IRL camera angle!

Requirements

Guide written using version 1.3.1 and is quite outdated. The Wiki version of this guide is far more up to date[egglyberts.live], but I have yet to replicate the guide here!


So! AR visuals. Exciting. This is one of the most unique things you can achieve with Keysight, and also the one that takes the most effort. You will need:

- A stable IRL camera angle
- The ability to mask Keysight to achieve semi-transparency (via mask mode, using this guide)
- An image editing program
- [Optional] A game controller
- Patience


Jonathanong[www.twitch.tv] is the most notable streamer to be using this.

Using reference alignment images

Keysight can import and display an image across the entire screen with a chosen opacity, to aid in aligning Keysight's camera to an IRL camera. You're going to need a clean still image of your camera angle as you intend to use it with the correct aspect ratio. This image is going to be stretched across Keysight's screen, so if it's the wrong aspect, it'll be stretched incorrectly and be an invalid alignment image. If you have a 1080p webcam but can only take a 720p screenshot of the preview in OBS or something, that's totally fine. Just as long as you don't take a 1800x1080 screenshot or something!

Make sure Advanced menu mode is enabled, and navigate to Core > Viewport, scroll down, and enable "alignment image overlay"



Click "open folder", drop your image into the file browser directory that opens, head back into Keysight, and it'll show the file in the dropdown selection



No, I don't have a midi-capable grand piano. This is an image from jonathanong's[www.twitch.tv] stream that I am using for purposes of demonstration

Keysight configuration

Next up, we need to disable all the Keysight elements that we don't need and make the backdrop nice and visible for the purposes of aligning. Head into Scene and disable everything, including frame and piano under "Frame and piano". You may also wish to disable light bars



To make the backdrop more visible, head into Scene > Backdrop material and enable emissive colour.



If you wish to have lighting from notes spill out beyond the A0 and C8 keys, you will also need to toggle "extend backdrop bounds horizontally". This makes the backdrop super wide giving plenty of space for light reflections, but I would toggle this option last since it makes alignment harder.

Efficiently using the fly-cam

Now comes the fiddly bit. We're going to use Keysight's first person 3D fly camera to manipulate the Keysight camera to where we want it to be. This has controller support, which I highly recommend using, as it just feels a whole lot more controllable and natural with analogue sticks to do these fine adjustments.

First off though, we need to set up the coordinate space and field of view that the camera is going to use! We'll start with field of view.

Field of view is something that can potentially be derived from your camera lens, if using a true camera, or from the specs of your webcam. A c922, for example, has a field of view of 78 degrees. A logitech brio has a FoV of 65, 78 or 90. Zooming in will tighten this FoV further though. However, quite often I just end up eyeballing it when working with real camera, which is what we're going to do here. From experience, my guess is that the FoV is around 58 degrees, so that's what I'm gonna use for this example angle. You can get the FoV a little wrong and still have great alignment, but the better your guess is at the FoV, the less headaches down the line. Hit "reset to perspective" and type in your field of view.



Now for coordinate space: by this, I mean telling Keysight which way is "up". Under default conditions, Keysight's active portion behaves relative to the camera as if it is "on a table". You can flip the coordinate system to have Keysight "on a wall", which massively helps lining up a camera for doing vertical effects. To keep Keysight "on a table", just keep camera tilt at -90 and local roll at 90. To have Keysight "on a wall", set both tilt and local roll to 0.



OPTIONAL STEP: if you have a measuring tape to hand, you can actually try to calculate where the Keysight camera should be. Keysight's coordinate origin point is the point between middle E/F right at the top of the key, so if you can measure your camera's rough distance in the three axes from this point you can give yourself a huge camera alignment headstart. You'll need to measure like this, and the unit used in Keysight is cm.



Time to actually match the camera! Grab your controller, and exit the menu. Press the B/Circle to enter fly-cam mode. L1 lowers the camera, R1 raises it. L2 slows the camera and movement speed, R2 speeds it up.

This is a bit of a fine art. You want to do your best to align the big white square of active area to the "vanishing points" of your alignment image, and this takes a lot of iteration. The best way I can explain my workflow is to imagine where the camera is relative to the Keysight keyboard, and whether that "feels the same" as the camera relative to the keyboard in your alignment image. For example, this is super bad and wrong, the lines are all off even though the match between the target note origin line is perfect:



If we hit Y/triangle to enable automated note testing mode to see the keypresses (we can disable these later), it helps understand why. We're way too high up and angled down:



So if we try to get somewhere more accurate to the shown angle, we get something like this, which is much much closer.

Extra tricks

What if "close enough" isn't good enough?

Well, this is where you can start poking at "camera rotate" values. It's really common for your IRL camera to have 0.0-2.0 of unintentional tilt, and this will cause the Keysight angle to always be off by a tiny bit. You can try to detect this by seeing if things in your camera view are actually level if they're supposed to be, but I tend to just eyeball it. In the case of this angle, I needed just -1.0 in the camera tilt to make the notes feel a lot more parallel to the piano, plus some extra camera work to re-match it with the new tilt.

I used the randomizer but a bunch of things turned themselves back on!

If you want to have a functional randomizer while doing these spicy angles with disabled elements, you'll also need to disable randomization on those elements you want to stay hidden. Mostly commonly here is that keypress indicators will turn themselves back on, as they have a 100% chance to be enabled by default. Head into the randomizer, and either toggle off entire sections or enter each section and set the chance for that object to be enabled to zero.

I want the keypress indicators but they're in the wrong place and I can see them through each other!

Ok so this is MEGA fiddly, but you can actually disconnect the virtual keyboard from it's default spot and move it around! Turn the piano back on under Scene > Component visibilities, and disable the inner wood material. This allows keypress indicators to be "occluded" by notes.



We can then set the piano materials to be jet black so you don't actually see them, you just get the keypress occlusion. Head into Scene > White/black key material subheadings, and set the material to be "basic" with a black colour, no emissiveness, metalness or specularity, and 1.00 roughness.




From here, head into Core > Viewport and enable offset keybed position and play with the values (maybe set them all to 0.00 first to have a blank slate). You're kinda on your own here though, I've never had to actually match this to a camera and it seems like it makes things 10x more delicate in terms of having a perfect match, hence the terrible example (in which I re-enabled white key emissiveness so I could see the white keys while doing this)

OBS configuration

Now, while we're still in Keysight-mode, go turn off backdrop emissiveness, possibly keypress indicators, and enable masking mode by pressing M (assuming that you are using mask mode to achieve semi-transparency)

Now for OBS configuration: simply import masked Keysight over the top of your camera! Maybe do this as a nested scene, so you can import and scale both the camera and Keysight at the same time.

You may also want to create a black and white alpha mask for Keysight. In the instance of Ong's angle here, if we crop out the microphones and various bars we can really embed Keysight into the scene and make it convincing. You'll want to import that camera image into your image editor of choice, and then make a new layer and trace around the stuff you want to cut. Any area you want to keep as white, and the rest stays transparent. Remove the image layer, and save the image.




Apply this image as an image mask/blend in OBS on top of your Keysight source, using alpha channel as the blend mode. Ta daa! Makes it FAR more convincing at a glance.



As a reminder, if you need more frame height to have notes not cut off too fast, you can use i+v at the same time to enable overdrive, and while in overdrive the slider will have a max value of 100 rather than 10 for frame height. You can also make the backdrop expand outwards beyond A0 and C8 by toggling "cull rendering of objects exiting left/right edge of active area" under Core > Viewport, visual effects subheading.

You may also want to create some kind of reference alignment trace so if your camera DOES get nudged, you can perfectly nudge it back without having to re-align Keysight and re-mask it. Simply add some nice bright green lines in an image editor around obvious things that you can line back up later. Having more straight lines across the entire screen and having them be as precise as possible helps with usefulness. But err, ideally don't nudge your camera, nudging it back perfectly is hard.

TL;DR

Err, this is too complex for a TL;DR really, but here goes:

- Brace for a lot of fiddling around, be patient
- Use a camera alignment image overlay
- Disable visibility of non-required Keysight elements
- Set backdrop to be self-emissive
- Try to use measurements of your camera's displacement from top middle of E/F keys as a starting point
- Use fly-cam to match camera angle (controller works great), maybe adding a tiny amount of roll in options if needed to account for IRL camera tilt
- Disable backdrop emissiveness
- Apply masking mode
- Make a reference outline of your angle so you can re-match the IRL camera if it gets nudged or moved
- Apply an alpha mask to Keysight to cut out any foreground bits of your camera scene to seat the Keysight effects more "in the scene"

If you're a TL;DR reader... good luck with this. It's a difficult process!

[pandiaonline]

There is no TILT option in CORE menu.