Advanced Knuckles Interaction System

How to download and use my Demo Scene and physics-based interaction system for the knuckles controllers in Unity. This system lets you interact with everything on a per-finger basis, pick things up and throw them, open doors, pull levers, etc.

The Demo Scene!

Download the demo from here![github.com]

This is a demo scene that lets you play with the features of knuckles tracking and see their potential in games and other applications. You can see the analog and impressively stable finger tracking in bits like these:




The demo room looks like this:


The table at the bottom left has some mugs, a plate and a laptop. The coffee table at the top has apples and another mug. There's a globe on the small table in the upper left. On the lower right, there is an antigravity machine, whose area of influence is shown by the blue sphere. You can adjust the range of the machine by pulling the lever on it.


You'll start out by doing a short tutorial on how to calibrate your controllers, use the interaction system and teleport (Use the trackpad to teleport around in the standard manner). Once in the main room, there are basically two things you can do. You can play with all the physics objects scattered around, and you can poke at the UI on the TV.

PLAYING WITH PHYSICS OBJECTS


Each of your fingers has it's own collider at the tip, and there's another collider on your palm. At the moment, things can pass through the middle portions of your fingers.

To grab something, pinch your index finger and thumb together. You don't have to click the trigger or touchpad, or even touch the controller. When both fingers contact the object you're trying to pick up, you'll grab it. You can also grab things using your other fingers against your palm, but this is difficult unless the object is cylindrical.

There are still bugs to be smoothed out but overall it should be fairly smooth. This demo shows how the interaction system can be used to connect you to a realistic and immersive setting.

POKING AT UI


You have to literally poke the UI with your index finger to interact with it. You can change your hand skin on the left to suit your comfort (I also recommend trying all of them, as it creates a surprisingly varied experience). You can show the finger colliders using the toggle on the bottom, to get a better idea of how the interaction system works. Finally, you can reset the scene and exit the app with the buttons on the right.

This UI screen is meant to show that it's possible to make an advanced and polished menu even with the current early stage of the UI system.


There's an easter egg in the scene as well, so try to find it! Feel free to share videos of your experience or post critiques and suggestions. If anybody wants, I could share the source for the demo scene, I only hadn't because it's enourmous and messy.

Enjoy!

Download the Source

Go to https://github.com/zulubo/knuckles to download the interaction system. If you use github, you can clone it, if not, just download it as a zip.

There are two directories: Demo Scene and Interaction Plugin. The Demo Scene has an executable of the scene shown off above. The interaction system folder contains a Unity project and a Unity Package of the interaction system.

To try it out, you can start by opening the project in Unity (version 5.6 or newer).

Add to a New Project

To add the Interaction system to a new project, all you need to do is make sure the SteamVR plugin has already been added. Then you can import Knuckles.unitypackage and you're ready to go!

How to Use It

The system has just a couple of scripts that you need to add to get it working. There's a prefab CameraRig with everything added already, but I recommend you read through here and try to get an understanding of how it all works.


There's a single script (KnucklesHandControl.cs) that takes the raw axis input from SteamVR, applies smoothing to the finger motion, groups all the finger pose data into a single class called KnucklesHandPose, and finally animates the hands.

You have to set which hand it is, give it a SteamVR_TrackedObject to get the input from, and set an animator for the hand. You can also emulate Knuckles using the trigger axis on regular Vive controllers, just remember to turn this off before shipping. I added the emulation feature one night when I really didn't feel like pairing my knuckles.

KnucklesHandPoses contain the following data:

public float middle_curl;
public float ring_curl;
public float pinky_curl;
public float index_curl;
public Vector2 thumbPos;
public float thumb_lift;
public float squeeze;

Most of these are self explanatory. The thumb_lift value is currently driven by whether the thumb is touching the trackpad or not, but I've heard there will be more sensors eventually that can get the actual height of the thumb.

Each hand has one of the KnuckleHandControl scripts attached to it, and you can access that hand's pose from anywhere by getting KnuckleHandControl.handPose.

I'm using only one hand model, so the left hand has to be a child of something with an x scale of -1. There are prefabs where this is all set up.


How to do Interaction

KnucklesPhysicalInteraction.cs is where things get fun. This script allows realistic physics driven interaction with anything you want, done for each individual finger.

NOTE: the palm is now officially a finger

(not really but I treat it as one)

You have to point it to a bunch of fingertip transforms to use (and the palm). These have to be in the animated armature so they move with the fingers. I'm just using the fingertip bones, plus a new empty transform for the palm.


To see how it works you can check the debug box. This script creates a bunch of little colliders that follow the fingers around, but they are all only moved with physics. This means there's a tiny delay, and there's also no guarantee they'll get to the right spot. They do most of the time though.

You can get some really convincing interaction, especially with subtle finger motions.

Note that the sizes for all of these colliders are currently hard coded in, but I have it all commented so you should be able to find it easily and change it if needed.

The other part of this is that if two opposing fingers (index + thumb, middle/ring/pinky + palm) get close to each other and there's something in between them, they'll grab onto that thing until you release your grip.

Because of how cool/flexible/physics driven this system is, plus the awesome finger tracking on Knuckles, you can interact with everything, pick stuff up, throw it, flick it, pinch it, etc. all without actually pressing any buttons. I think this is significant because it frees up buttons for other uses in the app, and it's just way more immersive overall.



The other piece of the interaction system is the KnucklesInteractable.cs script. This needs to be attached to every object you want the hands to be able to touch (every rigidbody, not each individual collider).

The type dropdown dictates what kind of interactable it is. Something small and light should be Grabbable, meaning you can pick it up with one hand. Something heavy or attached to something else (movable but not grippable) should be Non Grabbable. This is also the type for UI elements, but we'll get to that later. Snapping Grabbable Translate and Snapping Grabbable Rotate are special. They're for things like levers and knobs, where instead of the object attaching to the hand the hand should attach to the object when you grab on. The grabbable object will still make an effort to follow the hand (an effort constrained by its own joints and whatnot), but the hand will always be connected to it.

OnTouched invokes all its functions whenever you touch this KnucklesInteractable with any part of your hand. OnPoked is invoked when you deliberately poke it with your index finger. OnGrabbed is invoked when you pick it up. OnDropped is for when you let go.
KnucklesInteractable doesn't really do much except keep track of what's going on, and tag that object as interactable.

How to do UI

UI is based off the physics interaction system at the moment. It requires you actually go up and poke the UI elements to click on them, there's no laser pointer.

Right now, it supports buttons and toggles, but I'll be adding support for sliders and other more complex stuff in the future.

To make UI work with Knuckles, just add the KnucklesUIInteractable.cs script to each UI element you want to be able to interact with.


This script will work magic for you when you press play. It automatically adds a collider at the right size, adds a rigidbody, and adds a non-grabbable knucklesInteractable script to itself. Then it adds the UI element's OnClick function to the KnucklesInteractable's OnPoked event list (this doesn't show up in the inspector, but it still works).


To reiterate, you don't need to do any of this yourself. It does it automatically. All you need to do is add KnucklesUIInteractible to your UI elements.

I'm so excited to see what everyone makes! Don't be scared to expand on this interaction system, Experiment and innovate and make crazy VR games!

Email me at [e-mail] with any questions

[SpaceHorse]

hey guys / i have troubles calibrating the controllers
how can i force them to calibrate?
i see some finger tracking in steam home and its bit clunky
but in this unity project they dont track at all. and its written that everything is calibrated

[Caesar]

anyone find it very hard to trigger the teleport point to the door at very beginning (demo scene)?

[CircuitLord]

Looks pretty cool, but I can't help but notice that they're basically making a slightly better oculus touch.

[BeanaeB]

What do the controllers do for haptic response as of now? Is it just varying vibrations? Or is the controller capable of stopping you from closing your hand all the way when you grab a virtual object?

[createthis]

So great to see another poke based UI. Hope we can add support to CVRUI soon.

[dead bard]

SteamVR & Knuckles!

If the squeeze value thing is what I think it is I want squeezable toys in Steam Home, those squeaky things that have eyes that pop out.

[nadir-seen-fire]

Any more information on this `squeeze` float?

[Draclan]

This looks awesome. Can't wait to see support for UE4.

[nova]

Oh, wow!

[Jovian]

What! Let me buy some Knuckles controllers Valve!