Mixed Reality Videos

[aaron.leiby]

I was asked by a dev to provide some more details around SteamVR_ExternalCamera, so I figured I'd start a thread here to go over this, and also to discuss the topic more generally.

The new Unity plugin (v1.0.8) will automatically create this for you given two conditions:
1) Add a file called externalcamera.cfg in the root of your project.
2) Attach a third controller to your system.

By root of your project, I mean next to Assets or your executable.

Look at the config struct in SteamVR_ExternalCamera.cs for the full set of values you can set. At a minimum, you needs to set the near and far clip distances since we currently treat everything explicitly (as opposed to overrides).

Example:

x=0
y=0
z=0
rx=0
ry=0
rz=0
fov=60
near=0.1
far=100
//m=-0.999059,0.015577,-0.040472,-0.0127,-0.016016,-0.999816,0.010544,0.1799,-0.040301,0.011183,0.999125,-0.0846

sceneResolutionScale=0.5

The idea is that you attach the third controller rigidly to a camera that you can use to record the person playing on a greenscreen. Then you set the xyz offset and rotations in the config file to match the virtual camera in game. We have a tool for automatically solving for this, but it's not ready to release publicly yet, so you'll have to eye ball it for now by adjusting in the editor looking at other tracked objects until they line up close enough.

This script changes the output of the companion window on your main screen. It outputs four quadrants: foreground, foreground alpha (for compositing), background and normal game view.

You'll need to use another application to perform the compositing (e.g. OBS) and can use that to stream via twitch, etc. or you can save out the video feed and put it together in post.

Since this is a pretty intensive operation, and the primary computer is already overloaded with rendering the scene multiple times, you are better off setting up a second computer to take the video feed from the camera and the game, to composite in real-time for preview (very useful for the social aspect - e.g. set up a group of people on a couch to watch) and also take the brunt of writing the files to disk.

For creating trailers at 1080p, you'll want to set your output to 4k so each quadrant is full res.

The Vive Pre headsets only support pairing two controllers, but you can take an original Vive dev kit controller and plug its dongle into the primary machine, or use the usb cable with adapter to avoid dealing with batteries.

[isotropy]

If anyone is curious, I did a 45m talk on the making of the Fantastic Contraption and Job Simulator Mixed Reality trailers at VRLA a few weeks ago. Enjoy! https://youtu.be/WIlpX7P0-74

[Co-op Sisters VR]

Originally posted by 梦伯爵X:

Co-op.Do you notice grey between foreground and background.
Try use difference color space to remove this issue.

And Did anyone notice the video provided by virzoom.(second one 0:17)
There is one pixel width black line between foreground and background.

I try many game that support MR and did not notice the one pixel width black line while i do compositing on OBS.

I really need a good solution for this.
Please valve team please.....

Hi Do you know how to change the background colour on the quadrants? from black to blue or any other colour?
Thanks buddy

[Matt (CoroWare)]

I've been trying to set up MR with the quad views in Unity for a while without any success.
I just see 3 solid black quadrants, and the HMD view in the bottom right.
I confirmed I have the externalcamera.cfg file in the root project dir, and I think everything else is hooked up correctly. I'm using a 3rd controller for the external camera. Everything is updated to the most recent version and firmware.

After troubleshooting. It seems like Unity is not reading the externalcamera.cfg file for some reason, because I can hit Run and manually change the FOV and near/far values (in the camera object under the 3rd controller) and I can see the 3 quads appear briefly, but they're not refreshing in real time.
Any suggestions?

[greengiant83]

I have seen a few people asking about changing the color of the background. If you are in control of the source for the application you can change one line in the a shader to make it output a different color.

In the file SteamVR_ClearAll.shader which is located at /Assets/SteamVR/Resources about halfway down change

float4 frag(v2f i) : COLOR {
return float4(0, 0, 0, 0);
}

to this

float4 frag(v2f i) : COLOR {
return float4(0, 1, 0, 0);
}


and it will make the background color in the top left corner green

Full code:

Shader "Custom/SteamVR_ClearAll" {
Properties { _MainTex ("Base (RGB)", 2D) = "white" {} }

CGINCLUDE

#include "UnityCG.cginc"

sampler2D _MainTex;

struct v2f {
float4 pos : SV_POSITION;
float2 tex : TEXCOORD0;
};

v2f vert(appdata_base v) {
v2f o;
o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
o.tex = v.texcoord;
return o;
}

float4 frag(v2f i) : COLOR {
return float4(0, 0, 0, 0); <---------- Change this to (0, 1, 0, 0)
}

ENDCG

SubShader {
Tags{ "Queue" = "Background" }
Pass {
ZTest Always Cull Off ZWrite On
Fog { Mode Off }

CGPROGRAM
#pragma vertex vert
#pragma fragment frag
ENDCG
}
}
}

[tom]

Im knee deep in this now, 90% of my kit has arrived.

trying to set up something like what jashan mentioned, having the MR visible on a camera's field monitor so the camera person gets a feel for the MR action

a picture tells a thousand words, and several thousand pounds!
I couldnt see how to load images here so I loaded a diagram of my proposed setup on the parrallel OBS forum that is running
https://obsproject.com/forum/threads/using-one-video-source-as-alpha-for-another.44891/

if anyone has any feedback or help I'd love to hear it!

[tom]

Im having the same foreground alpha problem as everyone else, SPT showing just white, tiltbrush is working fine, but OBS wont handle it. my plan is to get a 'good enough' one from OBS for the camera person to use, then to use that to select a cut which gets properly comped in nuke/AE etc.

[jashan]

Thought I'd share what we did at Gamescom 2016: We had one machine do the VR rendering, output the game view via HDMI to another machine, only used the background (for simplicity's sake and because we simply didn't have enough time). We put the mixed reality stream on two screens at the booth and also recorded each session (we had two "normal" stations without mixed reality, and one with mixed reality). After each session, we immediately uploaded it to YouTube (and a few ones also to Vimeo), so when players left our booth, they could watch their video within a few minutes.

Here's a few example videos of how we did it:

• Holodance Gamescom Mixed Reality "Making of" (kind of ;-) )
  
• Holodance Gamescom Mixed Reality "Making of" (kind of ;-) ) 2

The full stream (about one video every 7 minutes while we were giving demos - in the end, we had a little more than 300 videos of the event, you may need to scroll down a bit; August 21st was the last day): HolodanceVR Twitter Feed with all Sessions

And a selection of videos on Vimeo: Holodance Mixed Reality Videos on Vimeo

I guess the most annoying part was that the lighting wasn't that great so the greenscreen mask was far from ideal. Also, having a full "green room" (with ceiling and carpet) would have helped a lot. But I think we pretty much did the best we could with what we've had available (including very limited time ;-) ).

[BOLL]

Wow, that's really quite awesome :) To have it live and put it online for people to see share :D I guess that was a major selling point on your end ;) So nuts that this is a concept employed at expos now, it's brilliant for selling what VR is all about :o

[TreeFortressGames]

One of the biggest hassles with creating MR is setting up the ExternalCamera.cfg properly.

What makes this especially cumbersome is the fact that the application needs to be relaunched each time you change the cfg.

If you add the following code to SteamVR_ExternalCamera, the cfg will hotswap everytime it's edited, leading to much quicker calibration sessions :)

    bool configHasChanged = true;
    void Start() {
        var root = Directory.GetParent(Application.dataPath).FullName;
        FileSystemWatcher watcher = new FileSystemWatcher(root, "externalcamera.cfg");
        watcher.NotifyFilter = NotifyFilters.LastWrite;
        watcher.EnableRaisingEvents = true;
        watcher.Changed += (a, b) => { configHasChanged = true; };
    }
    void Update() {
        //Need to do this in Update, as camera properties can only be changed in Unity's main thread. 
        if (configHasChanged) {
            ReadConfig();
            cam.fieldOfView = config.fov;
            var offset = cam.transform;
            offset.localPosition = new Vector3(config.x, config.y, config.z);
            offset.localRotation = Quaternion.Euler(config.rx, config.ry, config.rz);
            configHasChanged = false;
        }
    }

[tom]

Tree fortress, you are talking about a unity file right? Not something for a .exe steam game like tiltbrush? I might try using that in unity to calibrate then copy the settings over to games I need a calibrated camera for

I checked tiltbrush again today and there is something dodgy about its alpha channel, if you look at your controllers in that view the parts that are 100% opaque in other views are rendered a grey colour for alpha , the stripes on the controller are white for alpha and it looks more like a 'self illumination' channel or something

Anyone using 2 capture cards? I only have one so having to do everything on one pc at the moment and I think that might be why tiltbrush started crashing today

[TreeFortressGames]

Yes, this is a small addition to the SteamVR SDK that will allow externalcamera.cfg to be hot reloaded when changed, but it needs to be added to the src code of the game within unity.

[tom]

thanks treeFortress I'll be sure to try that, hopefully tonight as it sounds totally painful doing it the 'reboot the game' way

[tom]

hey treeFortress im getting an error about namespaces etc.
a namespace can only contain types and namespace declarations

I havent done Csharp before, but I guess I have to learn a little of this.

Im guessing this bit cant go at the bottom?
bool configHasChanged = true;

cheers
Tom

[tom]

Ok, so I put your code at the bottom, but above the final } symbol as I guess that is the main function loop thing (Im an artist who codes only occasionally, Javascript etc.)
Now Im gettng this, which I think might be a missing function you are calling or something?

The type or namespace name `FileSystemWatcher' could not be found. Are you missing a using directive or an assembly reference?

[tom]

so Ive been smashing it all night tonight and found a much better way to do it for any game.

use fantastic contraptions (FC) UI to calibrate the camera, its much easier using the keys etc. (i.e. follow northway games guide)

there is a button to view the text file in the menu in FC, you can find the camera data in here

copy and paste the camera data from FC into the cameraSettings text file, mind to make the translation numbers x10 smaller as units appear to be different, i.e. 1.0 in FC units is 0.1 in the cameraSettings.

I have a perfect alignment now! respect to Northway games.