SteamVR Developer Hardware

SteamVR Developer Hardware

View Stats:
 This topic has been pinned, so it's probably important
aaron.leiby Mar 8, 2016 @ 7:13pm
Mixed Reality Videos
I was asked by a dev to provide some more details around SteamVR_ExternalCamera, so I figured I'd start a thread here to go over this, and also to discuss the topic more generally.

The new Unity plugin (v1.0.8) will automatically create this for you given two conditions:
1) Add a file called externalcamera.cfg in the root of your project.
2) Attach a third controller to your system.

By root of your project, I mean next to Assets or your executable.

Look at the config struct in SteamVR_ExternalCamera.cs for the full set of values you can set. At a minimum, you needs to set the near and far clip distances since we currently treat everything explicitly (as opposed to overrides).

Example:
x=0 y=0 z=0 rx=0 ry=0 rz=0 fov=60 near=0.1 far=100 //m=-0.999059,0.015577,-0.040472,-0.0127,-0.016016,-0.999816,0.010544,0.1799,-0.040301,0.011183,0.999125,-0.0846 sceneResolutionScale=0.5

The idea is that you attach the third controller rigidly to a camera that you can use to record the person playing on a greenscreen. Then you set the xyz offset and rotations in the config file to match the virtual camera in game. We have a tool for automatically solving for this, but it's not ready to release publicly yet, so you'll have to eye ball it for now by adjusting in the editor looking at other tracked objects until they line up close enough.

This script changes the output of the companion window on your main screen. It outputs four quadrants: foreground, foreground alpha (for compositing), background and normal game view.

You'll need to use another application to perform the compositing (e.g. OBS) and can use that to stream via twitch, etc. or you can save out the video feed and put it together in post.

Since this is a pretty intensive operation, and the primary computer is already overloaded with rendering the scene multiple times, you are better off setting up a second computer to take the video feed from the camera and the game, to composite in real-time for preview (very useful for the social aspect - e.g. set up a group of people on a couch to watch) and also take the brunt of writing the files to disk.

For creating trailers at 1080p, you'll want to set your output to 4k so each quadrant is full res.

The Vive Pre headsets only support pairing two controllers, but you can take an original Vive dev kit controller and plug its dongle into the primary machine, or use the usb cable with adapter to avoid dealing with batteries.
< >
Showing 16-30 of 256 comments
eljaydub Mar 9, 2016 @ 6:11pm 
This is great and I can't wait to try it out for my live streams and courses! Vince was going to send me a third controller and if I get my hands on it before early next week (otherwise I'm out of town for a bit) I'll try and put out some tutorial videos about how to get it all set up.
hardlab Mar 9, 2016 @ 8:28pm 
Originally posted by Shadow:
Originally posted by hardlab:
Just wondering for the consumer versions, how will we be able to accomplish this if we don't have one of the legacy controllers to act as the camera? Will a Pre gen controller work as a wired controller?
Any controller will work. We have experimented using old ones from the Pre, the dev-kit editions and older. One thing we did notice is with really bright lights the old controllers (think the ones from GDC last year) have a hard time tracking, while the Pre ones work great.

Are there plans to sell controllers by themselves? We luckily have one of the legacy Mr.Hat controllers to use, but I don't want to have to take it from different teams so that I can have a third one to use as the camera.
AR@Trion Mar 10, 2016 @ 1:42am 
Originally posted by aaron.leiby:
The idea is that you attach the third controller rigidly to a camera that you can use to record the person playing on a greenscreen. Then you set the xyz offset and rotations in the config file to match the virtual camera in game. We have a tool for automatically solving for this, but it's not ready to release publicly yet, so you'll have to eye ball it for now by adjusting in the editor looking at other tracked objects until they line up close enough.

A tool for this would be amazing. :) I spent a few hours trying this out tonight, and I could never get a configuration I was happy with. :(

Beyond that, it worked great.

I'd love to see a walkthrough from someone who knows how to best setup OBS. I got ok results with it, but suspect I could do better...
Bangerman Mar 10, 2016 @ 11:52pm 
Is there any way to make this work if you only have the two controllers with the Pre?
If we use a fixed tripod camera then we should be able to register its position once and then just use those settings rather than actively track its position.

Can we "hard code" the offset to the external camera in the [CameraRig] and then continue as normal.
Miss Stabby Mar 11, 2016 @ 4:16am 
yeah for tripods it would be cool to just "place tracking ring around camera lense and press trigger to confirm" for a basic calibration
Kevin Mar 11, 2016 @ 4:38am 
Just a bit of a tip for anyone having transparency in your foreground. You might have to manually account for the premultiply in the foreground layer by dividing the foreground color with the matte layer before you luma matte it. It'll look crazy at first, but your colors and brightness will look right in the end.
jilt Mar 11, 2016 @ 7:19am 
It looks like I'm missing something. Whatever I try I only get the lower right output, not the other 3 quadrants. A step by step guide or even a barebones unity project would be super helpful.
aaron.leiby  [developer] Mar 11, 2016 @ 3:43pm 
Originally posted by jilt:
It looks like I'm missing something. Whatever I try I only get the lower right output, not the other 3 quadrants. A step by step guide or even a barebones unity project would be super helpful.
This is usually due to having not set your near/far settings set properly in the config file.
Raticus79 Mar 13, 2016 @ 12:01am 
Great stuff! Looking forward to seeing this style of demo video for other games.
jilt Mar 13, 2016 @ 4:44am 
Originally posted by aaron.leiby:
Originally posted by jilt:
It looks like I'm missing something. Whatever I try I only get the lower right output, not the other 3 quadrants. A step by step guide or even a barebones unity project would be super helpful.
This is usually due to having not set your near/far settings set properly in the config file.

That could be, I just copied your example verbatim to start testing, should I use any other values?
jashan Mar 13, 2016 @ 9:37am 
I have a rather basic setup with a Webcam running now ... but ... does OBS actually support compositing the way it's needed? I've just installed the most current version of OBS Studio, and the closest thing I can find to what I need is the filter "Alpha Mask". But that only lets me select an image path.

What I think I'd need is that same filter but with the ability to select another Source as mask. That way, I could make the game view 3 sources: Background (bottom left quadrant), Foreground (top left) and Alpha Mask (top right). Then I'd put my Webcam between background and Foreground, apply the alpha mask to foreground and be all happy.

Or am I taking the wrong approach?
AR@Trion Mar 13, 2016 @ 11:14am 
Originally posted by jashan:
I have a rather basic setup with a Webcam running now ... but ... does OBS actually support compositing the way it's needed?

Or am I taking the wrong approach?

I reached the same point as you, jashan. As far as I can tell, there's no way in OBS Studio to use the foreground mask. However, as long as your foreground has no transparency, you can just ignore it and do a simple mask based on black instead. Then make your layers foreground, webcam, background and get some good results.
jashan Mar 13, 2016 @ 12:52pm 
Ah, cool, thank you - using black as mask sounds good. I might have to pick another color as background color because I do have some black in the game but that should be easy enough to implement. It's a shame if OBS really doesn't support feeding a video source as mask channel ... I've asked in their forum - maybe there is some way but if not, I guess I'll go with the color mask approach for streaming and only use the alpha mask for offline compositing.
AR@Trion Mar 14, 2016 @ 12:19am 
http://jerrith.com/images/MixedRealityCamera.jpg

Just thought I'd share the configuration / setup I've been using... A Logitech c920 webcam attached to an old controller with rubber bands. This attachment actually works fairly well. :)
jashan Mar 14, 2016 @ 12:43am 
Ah, that's cool, thanks for sharing! I still have a really old Logitech Webcam and it has terrible latency (like, about half a second or so). Did some research yesterday and it seems the Logitech C920 is the best low-cost solution. I wasn't sure I should be that, yet, but I guess I will now ;-)

Did you have any issues with Webcam latency? I've seen one thread where a few people using OBS said they had about a 300ms delay, even with the C920, while others said they didn't have any issues. One potential thing that may help (but apparently didn't solve it for everyone) was to disable all the automatic feature (autofocus, auto white balance and so on). Here's the full thread (this was still about OBS Classic, so it might not apply to OBS Studio): Webcam Latency Question - Logitech C910 & C920[obsproject.com].

My thread regarding using a video stream as alpha mask did not get any replies, yet: Using one Video Source as Alpha for another?[obsproject.com]

EDIT: And here's some info on the FOV of the C920 (and also C910): Logitech c920 and c910 fields of view for RGBDtoolkit [therandomlab.blogspot.de];

For the c920, with 16:9 sensor:
for 16:9 resolutions: DFOV=78º, HFOV=70.42º and VFOV=43.30º

I think for Mixed Reality capturing, a lower field of view usually should be better because the lower the field of view, the easier it is to set up the green screen (very large fov means you need to cover and light a lot of green ...)
Last edited by jashan; Mar 14, 2016 @ 1:01am
< >
Showing 16-30 of 256 comments
Per page: 15 30 50