Introducing the Skeletal Tracking Model

Version 2.0 of the Leap Motion API introduces a new skeletal tracking model that provides additional information about hands and fingers and also improves overall tracking data.

By modeling a human hand, the Leap Motion software can better predict the positions of fingers and hands that are not clearly in view. Five fingers are always present for a hand and hands can often cross over each other and still be tracked. Of course, the controller still needs to be able to see a finger or hand in order to accurately report its position. Keep this in mind when designing the interactions used by your application. Avoid requiring complex hand “poses” or subtle motions, especially those involving non-extended fingers.

https://di4564baj7skl.cloudfront.net/documentation/images/Visualizer_Hand_Model.png

Explore the behavior of the new hand model in the Diagnostic Visualizer

The API changes are not as dramatic as you might expect and existing API classes still work mostly the same whether or not you use the new features.

The additions include:

  • Reporting of a confidence rating based on the correlation between the internal hand model and the observed data
  • Identification of right or left handedness
  • Identification of digits
  • Reporting of the position and orientation of each finger bone
  • Reporting of grip factors indicating whether a user is pinching or grasping
  • Reporting of five fingers for each hand
  • Reporting whether a finger is extended or not

Perhaps the most significant change for existing applications is the improved persistence of hands and fingers. This should improve the usability of most Leap Motion-enabled apps. However, if you relied on fingers disappearing when curled into the hand or when they touched, then you may have to make some changes. For example, if you count visible fingers or otherwise distinguish between fingers extended outward and those retracted toward the hand, you will have to use the new isExtended finger properties to know which fingers are extended or not. If, on the other hand, you relied on fingers disappearing when they touched each other, then you will have to establish a minimum distance threshold between the finger tips instead, since the fingers will no longer disappear.

Note: Finger.jointPosition() will be removed (in favor of the Bone class) in an upcoming SDK release.

The new tracking API in JavaScript

The API changes in the new tracking model include:

Hand class

Hand.confidence, a value between 0 and 1, rates how well the internal hand model fits the observed data. A low value indicates that there are significant discrepancies; finger positions, even hand identification could be incorrect. The significance of the confidence value to your application can vary with context. Gestures and motions can be valid even without a high confidence in the hand data. On the other hand, you should make sure the confidence value is high before triggering significant application events based on postures or hand properties.

Hand.type identifies whether a hand is a right or a left hand. However, the Leap Motion controller needs to see enough of the hand to make an accurate classification. If, for example, you put your closed fist into the controller field of view, the software has very little to go on and may come to the wrong conclusion. If the software realizes it has misclassified a hand, the current hand is replaced with a Hand object having a new ID value (and corrected finger identification).

The Hand.grabStrength and pinchStrength properties are values between 0 and 1, indicating whether your hand is in a pinching or grabbing posture. A strength value is close to zero for a hand with flat, extended fingers. For a grab, the strength increases toward 1.0 as you curl your fingers into a fist. For a pinch, the strength increases as you curl the thumb and any finger toward each other. You can use these strength properties to trigger grabs or selections in your application when the value exceeds a certain threshold.

The Hand.basis property is a two dimensional array that contains the three basis vectors defining the hand’s orientation. Hand.palmWidth provides the average width of the hand (not including fingers or thumb).

New JavaScript Finger class

The new Finger class extends Pointable and provides the name of a finger and access to its various bones.

Intermediate phalanges Proximal phalanges Distal phalanges Metacarpals btipPosition dipPosition pipPosition mcpPosition carpPosition 0-length thumb metacarpal

New JavaScript Bone class

The Bone class describes be position and orientation of each of the bones of a finger as well as the metacarpal bones, which connect the fingers to the base of the hand.

Note: The model for the thumb does not quite match the standard anatomical naming system. A real thumb has one less bone than the other fingers. However, for ease of programming, the Leap Motion thumb model includes a zero-length metacarpal bone so that the thumb has the same number of bones at the same indexes as the other fingers. As a result the thumb’s anatomical metacarpal bone is labeled as a proximal phalanx and the anatomical proximal phalanx is labeled as the intermediate phalanx in the Leap Motion finger model.

JSON data changes

The new tracking model is also reflected in the tracking data supplied via the Leap Motion WebSocket interface. As with the native API, the biggest difference you are likely to notice in existing applications is that hands will always have 5 fingers now.

To access the new API features, use the v6.json endpoint (ws://127.0.0.1:6437/v6.json).

Each hand in the frame hands array has the following new properties:

  • armWidth – the average width of the forearm.
  • confidence – indicates how well the skeleton model and the observed data fit.
  • elbow – the position of the elbow of the arm to which the hand is attached
  • grabStrength – a value between 0 and 1.
  • pinchStrength – a value between 0 and 1.
  • type – a string indicating whether the hand is a left or a right hand.
  • wrist – the position of the wrist

Each finger in the frame pointables array has the following new properties:

  • type – an integer code for the finger name.
  • bases – the basis vectors for each bone, in index order (wrist to tip).
  • btipPosition – position of the extreme end of the distal phalanx as an array of 3 floating point numbers.
  • carpPosition – position of the base of metacarpal bone as an array of 3 floating point numbers.
  • dipPosition – position of the base of the distal phalanx as an array of 3 floating point numbers.
  • pipPosition – a position vector as an array of 3 floating point numbers
  • mcpPosition – a position vector as an array of 3 floating point numbers
  • extended – a boolean indicating whether the finger is pointing or not.