Inside-out tracking? External tracking? Why not both?

[turtle]

I recently purchased a WMR and while I am in love with the headset tracking, I am disappointed with the controller tracking. For almost all use cases in SteamVR games the controller tracking just doesn't cut it.

How does WMR currently work?[github.com]

"Can I bring my own tracking technology to Windows Mixed Reality?
This is not currently supported."

We need to get around this.

My proposition is a combined external tracking solution for when WMR controller tracking falls out of "High Accuracy" (refer to controller tracking state table).

Tracking state	SourceLossRisk	PositionAccuracy
High accuracy	< 1.0	High
High accuracy (at risk of losing)	== 1.0	High
Approximate accuracy	== 1.0	Approximate
No position	== 1.0	Approximate

Source[docs.microsoft.com]

Solution:
Have external tracking technology "take over" when WMR loses "high accuracy" tracking state.

How can we accomplish this?

External Cameras:
Use more low-light cameras and adapt the pre-existing tech for WMR for position outside of visible headset space. This would require cooperation from Microsoft to open-source the current drivers.

Kinect:
Use driver4VR to infer better position of controllers in playspace.
Relevant thread.[www.driver4vr.com]

Wiimotes:
There are multiple papers written on using wiimotes the inverse way to track an ir light in a 3D space. The wiimote's IR camera is used to track the passive ir led. There was also work done by Johnny Lee several years ago to accomplish this same task.
https://www.youtube.com/watch?v=Jd3-eiid-Uw
This would require four wiimotes with two ir leds attached to each of the WMR controllers.
Research Paper[esc.fnwi.uva.nl]


Blockers:

Microsoft will need to give some accesible API to hook into various calls and events within the positional tracking system for WMR. If anyone has any other ideas in how to accomplish this please contribute. It's possible that through some combination of OpenVR[github.com] and OpenVR-Input-Emulator[github.com] this might be circumvented but it probably also means losing the excellent tracking WMR provides when tracking state is in "High Accuracy".

EDIT:
Just wanted to update. If I'm understanding correctly as long as we can keep using the gyro/accelerometer inside the controller and override the positional tracking using Open VR's tracking overrides[github.com] I'm fairly certain this is possible within SteamVR. Not the best solution as we will lose the inside-out tracking but for testing purposes this will suffice for now.

[BluesyMoo]

I think they can just release the exact set of sensors (a pair of IR cams) that exists on the headset, as standalone controller trackers that you can mount somewhere facing the user. It'll look much like the Rift's camera you put on top of the monitor.

[turtle]

Originally posted by MarleyMoo:

I think they can just release the exact set of sensors (a pair of IR cams) that exists on the headset, as standalone controller trackers that you can mount somewhere facing the user. It'll look much like the Rift's camera you put on top of the monitor.

If these cameras can detect IR light and these controllers are already immiting IR light then the Wiimote solution is probably the best bet for the short-term. That being said this doesn't seem to be the case.

I wouldn't bet on Microsoft to give us a product like that when everything in their VR brand is based around the "gimmick" of inside-out tracking. Don't get me wrong, I'd like like them to do this but I'm not holding my breath.

I know there are a few Microsoft dev's who frequent this sub so I am hoping they can throw some insight into this thread.

[DmAnd]

This would certanly make wmr the top headset because tracking is the only thing holding it back.
Well that and low FoV for 1st gen

[BluesyMoo]

Inside-out tracking is not a gimmick. It is the direction Oculus, HTC, and Microsoft are all going. Oculus Quest, HTC Vive Focus, and of course Microsoft WMR and Hololens are all inside-out tracking.

[turtle]

Originally posted by MarleyMoo:

Inside-out tracking is not a gimmick. It is the direction Oculus, HTC, and Microsoft are all going. Oculus Quest, HTC Vive Focus, and of course Microsoft WMR and Hololens are all inside-out tracking.

Of course its not a gimmick, thats why i put in quotations. With the current headsets its obviously not adequate though for all experiences. In the future, with more cameras like on cosmos im sure it will be better.

[Absolut1l]

Even something like the Xbox Kinect camera could drastically improve tracking in 360 degrees. People have done all sorts of amazing stuff with those cameras. I wonder if one could be setup BEHIND the player and take over tracking for anything out of LOS of the headset. Or even just put a camera or two on the rear of the headset...

[Ragefighter]

(Agreed with last part of Absolut1l)

They (OEM/Microsoft) should just add another sensor to the back and/or side of the headset and you will have full ~360 degree coverage without needing to set up an entire room to use it... The ultimate fix will be from the headset builders and Microsoft. When they do another model year of WMR they really need to consider the full range camera/sensor coverage.

How is that never an option when people bring this up. It's always "The Oculus and HTC are awesome and everything should be the same as them!!1!"

[Andrew_WOT1]

Look at Vive Cosmos, it has cameras on sides already, with good almost 360 degrees coverage who needs external sensors.

[turtle]

I've decided to use the existing lights with a camera based system for tracking using opencv. I will report back with my findings when I have something tangible.

[Girth Control]

Thinking about it, additional cameras on the headset really aren't necessary. A webcam could track the controllers and triangulate their position by comparing itself to the HMD's tracking. If the controllers are in view of the HMD, then the webcam knows exactly where the controller is and that data can be saved. Then when those controllers are in the same place but not in view of the HMD, they can still be tracked because we saved the positioning data from earlier. This style of tracking would get better and better with time as you gather more data, but if the camera moves or play space changes, the calibration would be lost. Another problem would be the polling or framerate of the webcam, if it were low it would cause jittery tracking.

People seem to forget that if they added a camera elsewhere on the HMD it would most likely have to be identical to the ones in the front, which means it would make the device bulkier, heavier, and more expensive. Allowing users to use their own webcams for tracking keeps the prices low while still giving people the opportunity to improve their tracking, or even opt for a higher quality webcam for even better tracking.

[Zim]

how well did you go ?

Originally posted by turtleneckthomas:

I've decided to use the existing lights with a camera based system for tracking using opencv. I will report back with my findings when I have something tangible.

[turtle]

There are many complexities involved with writing this. It includes first creating a wrapper to read different inputs from the controllers. If you are a developer and have interest in pursuing this work feel free to pm me. As it stands I'm no longer working on the project as there doesn't seem to be much of an ask for this and future headsets are coming out that will improve tracking.

[Xerberus]

Been following this and similar posts on Reddit for a while now. Sad to say nothing new going on. If anyone from 2020 beyond is reading this, I've also been looking into DriverVR/Kinect combos, but so far, it's still not possible.
You'll just have to start considering getting outside in tracking HMD's on your next VR purchase, sad to say. That or start figuring out how to use vive/index wands with your wmr.

[DAN-di-WARhol]

You can connect your index knuckes to WMR quite easily with some extra expenditure.

Also, head tracking isn't perfect. And the stupid Mixed Reality platform is a huge headache with crashes and things going wrong all of the time. I lose count of the amount of restarts I need to do just to play a single game.

It recently downgraded my headset to 60hz too. No explanation. HP don't know either. They sent me a new headset, and it is still doing it. Nobody can explain it. Just have to endure it until Valve or HTC release a high res headset and I can do away with the Mixed reality rubbish.