Mixed Reality

[Co-op Sisters VR]

Hi, i produce mixed reality videos, but at the moment mixed reality only works with unity games.
So is this a unity game?
Thank you.

[[ct]agoran]

Hi, I think you'll be happy to hear we support mixed reality in our own engine.
It is pretty straightforward to use as we mix in the chroma keyed player into the game, with lighting and all. Take a look at those few shots at the end of the launch trailer to see how this looks like, there are also a few examples of this on our twitter feed https://twitter.com/Croteam.

I have yet to write a comprehensive guide for mixed reality, currently there are mixed reality options in the menus that cover some of the options (most importantly the camera capture settings and camera calibration mode). Other options are available from the console, the most important being the: vr_iMixedRealityOutput = 2

You can easily find all the options through our super user friendly console (invoked with tilde or F1 key). Just write mixedreality and hit tab for all MR related options.

One option that is important for video capture and is not immediately visible is the

vr_fForceVideoFPS

(set to 60 for our particular configuration)

I should really write a more comprehensive guide now that this is released :)

[Co-op Sisters VR]

Thanks so much for the reply and tips, This is awesome news!!! i will be picking this up tonight :) and play about in mixed reality :D

[[ct]agoran]

Just curious, what type of capture card and camera are you using? We've tried it one one older black magic card, aver media live gamer extreme and elgato (don't remember which one).

I'm just working on a wizard that should help with the setup but it will take one or two more updates.

In the mean time if you need any help, just ask here and I'll try to answer asap.

[[ct]agoran]

Sorry, could you please rename this thread into "Mixed reality" so it is easier for other interested in the topic to find it.
Thanks

[Co-op Sisters VR]

Of course :)

[Co-op Sisters VR]

Postat inițial de ctagoran:

Just curious, what type of capture card and camera are you using? We've tried it one one older black magic card, aver media live gamer extreme and elgato (don't remember which one).

I'm just working on a wizard that should help with the setup but it will take one or two more updates.

In the mean time if you need any help, just ask here and I'll try to answer asap.

HI i'm using logitech c920, but im playing around with different lenses, i dont have a capture card, i use Xsplit to compile the game quadrants and webcam, and then use screen capture(Geforce Nvidia)

I'm test your game now in mixed reality, im in the menu, but a little confused at the moment.
is there a way to display the game quadrants?
Thank you.

[[ct]agoran]

There's no need for quadrants (in fact after the initial implementation we have ditched them entirely). We mix everything in game, you just need to record the game screen with the player inside. We support both the moving camera (attached to the 3rd controller) and the static camera configurations.

Just do the camera calibration (accessible from the menus), enable the mixed reality output and you'll get the feed. There is some need to tweak the options, here's a non-comprehensive guide to the console variables we have used internally (in the absence of the more comprehensive guide that is still WIP):

-- chroma key settings (similarity is the most important one), see other options (all prefixed with "ckey_")
-- These options are an exact match from chroma key options in OBS, so you can adjust them in OBS for a particular scene and simply copy them into the game
ckey_ulSimilarity = 400
-- disable laser from weapons and controllers in mixed reality (looks stupid as it is delayed)
vr_bSkipMixedRealityLaserSightRendering = 1


-- enable controller lag compensation in mixed reality for both hand and camera controllers. Useful (essential) in matching the controller/weapon model in game and from video.
vr_iEnableMixedRealityControllerPredictionTimeOffset = 2
-- both hand and camera controller will be 70 ms in the past (for this particular setup):
vr_fMixedRealityControllerPredictionTimeOffset = -0.07

-- rendering resolution should match the desired output (horizontal resolution from aspect ratio)
vr_iMixedRealityCameraVerticalRenderingResolution = 1080


-- format and device need to be properly set (can be set up from the menu)
vr_iMixedRealityVideoCaptureDeviceIndex = 1
vr_iMixedRealityVideoCaptureFormatIndex = 4
-- often needed so we match the camera fps
vr_fForceVideoFPS = 60

-- this controls which controller gets to be used for the camera (set up in calibration screen, just follow the instructions there and be sure to identify the controllers there)
vr_strForceAux0ControllerSN

-- These variables allow you to change the player size (in meters): the size of the scene around the player (the headset) that
-- is used. This way it is possible to have only a part of the camera's view covered with green screen without the other part
-- of the view appearing in game.
vr_fMixedRealityPlayerImposterWidth = 1.8
vr_fMixedRealityPlayerImposterHeight = 2.5

-- This variable is used to brighten or darken the player in the scene (tweak it between 0 and 1 and see what looks best)
vr_fMixedRealityPlayerImposterBrightness = 0.5



-- camera offset and fov, adjusted automatically through the calibration screen, but often necessary to be further tweaked
vr_fMixedRealityCameraOffset_Banking = -2.8726922
vr_fMixedRealityCameraOffset_Heading = 152.09056
vr_fMixedRealityCameraOffset_Pitch = 4.256974
vr_fMixedRealityCameraOffset_X = -0.079996376
vr_fMixedRealityCameraOffset_Y = -0.10904464
vr_fMixedRealityCameraOffset_Z = 0.052694712
vr_fMixedRealityCameraVerticalFOV = 38.958584


-- enable output with (1 covers half a screen, 2 covers the entire screen)
vr_iMixedRealityOutput = 2

-- setup for first person rendering in mixed reality output:
-- enable/disable
vr_bRenderFirstPersonInMixedReality = 1
-- which corner to render in (0 - 3)
vr_iFirstPersonInMixedRealityCorner
-- size (vertical percentage of the entire screen)
vr_fFirstPersonInMixedRealitySizeRatio = 0.3
-- aspect ratio for first person view
vr_fFirstPersonInMixedRealityAspectRatio = 4/3

-- it is often useful to rotate the playspace while in game (so you can easily adjust the angle to the player and to the battlefield). This is acieved using the cvar (set to angle in degrees)
vr_fPlaySpaceRotation = 90

-- for recording, it is good to be in simulated fullscreen mode (just a borderless window) as this eliminates the screen tearing
gfx_bSimulatedFullScreen = 1
gfxRestart()

[Co-op Sisters VR]

Hi, i just would like to ask, do i need a video capture card for mixed reality to work?
thank you.

[AlenL]

You can use a video capture card, but you can also use a simple webcam. As the cheapest makeshift solution, you can use an Android phone with DroidCam app (or similar for iPhone or whatever). But you need some way to bring the video of the player into the PC. :)

[Co-op Sisters VR]

Thank you for your response, i have a webcam, i use OBS or Xsplit to capture my webcam and gameplay, im just not understanding the way you set it up, i think this is to complicated for me. i did have a good go but im just not getting it :(

[AlenL]

For the next update (one already went out in the mean time, so the next one I guess), agoran (commented above) is already working on a nice wizard-like interface that walks you through setting up camera input, calibrating the camera and adjusting green screen settings. It is much simpler than using OBS, you'll see.
Btw, just for reference, which other VR games did you use this for already, so we know what kind of things you know?

[Layseylo]

How to I get my camera to add me into the sceen? I see my controllers but i am not standing there.

[Co-op Sisters VR]

Postat inițial de AlenL:

For the next update (one already went out in the mean time, so the next one I guess), agoran (commented above) is already working on a nice wizard-like interface that walks you through setting up camera input, calibrating the camera and adjusting green screen settings. It is much simpler than using OBS, you'll see.
Btw, just for reference, which other VR games did you use this for already, so we know what kind of things you know?

Hi, i really look forward to the update :) i have a youtube channel that focues on VR mixed reality only. https://www.youtube.com/c/coopsisters
Thank you for all your help,
I love mixed reality and its great that your spending time on this, thanks again :)

[[ct]agoran]

Postat inițial de Layseyla:

How to I get my camera to add me into the sceen? I see my controllers but i am not standing there.

Are you wearing the headset on your head? It is also enough to be positioned between the camera and the headset (I usually put it on a chair so I'm between the headset and the camera while working the settings).

Also are you getting a feed from the camera? You can check this by setting vr_iRenderMixedRealityVideo = 2 (or 3) in the console.

Hope this helps.

[Layseylo]

yes I get the video feed from the game too the Xsplit. So in order to do this do I need to move my x split camera feed to the spot I am standing and adjust the settings larger or smaller to scale with the world? this would appear to be 2d video image of me in a Rd world right? is there a way to not need x split and see me standing in the mirrored image that the game renders?