Frames

The Leap Motion API presents motion tracking data to your application as a series of snapshots called frames. Each frame of tracking data contains the measured positions and other information about each entity detected in that snapshot. This article discusses the details of gettting Frame objects from the Leap Motion API.

Each Frame object contains an instantaneous snapshot of the scene recorded by the Leap Motion controller. The Leap Motion WebSocket server sends frames to your application as they are created. These frame messages are consumed by the Leap Motion JavaScript API (leap.js) and are available from the Controller object as Frame objects.

The Leap Motion API provides a few ways to connect to the controller and get frame data. The simplest method is to use the Leap.loop() function, but you can also create your own Controller object, which gives you a few more options.

Getting Frames with Leap.loop()

The Leap.loop() function automatically sets up an update loop for your application, creates a Controller object, and connects to the Leap Motion WebSocket server. At each update interval, the loop() logic calls your update callback function, passing it the most current frame.

var controller = Leap.loop({enableGestures:true}, function(frame){
    var currentFrame = frame;
    var previousFrame = controller.frame(1);
    var tenFramesBack = controller.frame(10);
});

By default, the Leap.loop() function uses the following options:

{enableGestures: false,
frameEventName: "animationFrame"}

You can change these settings by passing in an object defining your desired settings.

Getting Frames with a Controller Object

When create your own Controller object, you can either set up an event loop using the Controller‘s internal event mechanism, or poll the Controller for frames at whatever interval is appropriate for your application.

Choosing a Controller Loop Mechanism

Since the Leap Motion tracking data is presented to your application as a series of frames, it makes sense to use a looping mechanism to process the data and update your application as the frames are received. The Leap Motion JavaScript API provides a built-in looping mechanism. You can also implement your own application loop, if you prefer more control.

The built-in looping mechanism gives you two options:

  • browser animation loop – Provided by requestAnimationFrame, the browser animation loop typically runs at 60 fps.
  • device loop – Runs at the Leap Motion controller frame rate.

Use the browser animation loop (animationFrame) if your application uses the Leap Motion data primarily for control and does a lot of drawing. Use the Leap Motion device loop (deviceFrame) outside the browser or if your application needs the highest data fidelity (and doesn’t perform too much drawing or other processing each frame).

Set the type of built-in loop to use when you create your Controller object by passing an options object to the constructor function specifying either “animationFrame” to use the browser loop or “deviceFrame” to use the Leap Motion controller loop. If you don’t set a value, animationFrame is used by default.

var browserLoopController = new Leap.Controller({frameEventName: 'animationFrame'});

The browser animation loop can skip frames of Leap Motion data with this method, since the controller frame rate and the browser frame rate do not match. Often skipped frames do not matter since you could not update your application in between loop iterations anyway. (If it does matter for your application, see Getting Data from Skipped Frames.)

var deviceLoopController = new Leap.Controller({frameEventName: 'deviceFrame'});

When using the device loop, the frame rate can exceed 200 fps, depending on the user’s settings and the available computing power. While a higher frame rate is often a good thing, too high an update rate can starve other portions of your application and bog everything down.

Using Frame Event Callbacks

When you create your own Controller object, you can use the on() method to specify one or more callback functions to be invoked at each update interval.

var controller = new Leap.Controller();
controller.connect();

controller.on('frame', onFrame);
function onFrame(frame)
{
    console.log("Frame event for frame " + frame.id);
}

The Controller object passes the most current frame to the callback function. If your update loop is running faster than the current Leap Motion controller frame rate, then the same Frame object can be passed to the callback function for more than one update. Likewise, if the controller frame rate exceeds your application update rate, then the Frame object passed to your callback can be be a few frames beyond the frame received in the previous update. All frames are temporarily stored in the history cache; see Getting Data from Skipped Frames if you need to access data in these in-between frames.

Polling for Frames

You can call the Controller frame() function at any time to get the latest frame data received from the Leap Motion. If no frames of data are available – perhaps the Controller has not finished connecting to the Leap Motion WebSocket – then you get an invalid frame, which contains no data. Otherwise, you get the last frame produced. Note that you can get the same frame more than once if you are polling more often than the Leap Motion controller is producing frames.

The following example uses a timer interval to poll for frames every half second:

<div id="frameID"></div>

var controller = new Leap.Controller();
var frameDisplay = document.getElementById('frameID');

this.controller.on('connect', function(){
   setInterval(function(){
      var frame = controller.frame();
      frameDisplay.innerHTML = '<p>Frame: ' + frame.id + ' is ' + (frame.valid ? 'valid.</p>' : 'invalid.</p>');
   }, 500);
});

controller.connect();

Getting Data from a Frame

The Frame class defines several functions that provide access to the data in the frame. For example, the following code illustrates how to get the basic objects tracked by the Leap Motion system:

<p>Framerate: <span id="leapFPS">FPS</span></p>
<p>Number of hands: <span id="handCount">Hands</span></p>
<p>Number of pointables: <span id="pointableCount">Pointables</span></p>
<p>Number of tools: <span id="toolCount">Tools</span></p>
<p>Number of fingers: <span id="fingerCount">Fingers</span></p>
<p>Number of gestures: <span id="gestureCount">Gestures</span></p>

<script>
var controller = new Leap.Controller({
                         enableGestures: true,
                         frameEventName: 'animationFrame'
                         });

var fpsDisplay = document.getElementById('leapFPS');
var handCountDisplay = document.getElementById('handCount');
var pointableCountDisplay = document.getElementById('pointableCount');
var fingerCountDisplay = document.getElementById('fingerCount');
var toolCountDisplay = document.getElementById('toolCount');
var gestureCountDisplay = document.getElementById('gestureCount');

controller.on('frame', function(frame){
  fpsDisplay.innerText = frame.currentFrameRate;
  handCountDisplay.innerText = frame.hands.length;
  pointableCountDisplay.innerText = frame.pointables.length;
  fingerCountDisplay.innerText = frame.fingers.length;
  toolCountDisplay.innerText = frame.tools.length;
  gestureCountDisplay.innerText = frame.gestures.length;
  
  });
controller.connect();
</script>

If you want to access data from past frames, you can pass a history parameter to the frame() function. See Getting Data from Skipped Frames.

Using IDs to track entities across frames

If you have an ID of an entity from a different frame, you can get the object representing that entity in the current frame. Pass the ID to the Frame function of the appropriate type. The following example finds a Pointable object in past frames to average the tip position.

<p>Average position: <span id="avgPosition">position</span></p>

<script>
var controller = new Leap.Controller({
                         enableGestures: true,
                         frameEventName: 'animationFrame'
                         });

var avgDisplay = document.getElementById('avgPosition');

controller.on('frame', function(frame){
    if(frame.valid && frame.fingers.length > 0)
    {
        var count = 0;
        var average = Leap.vec3.create();
        var fingerToAverage = frame.fingers[0];
        for( i = 0; i < 10; i++ )
        {
            var fingerFromFrame = controller.frame(i).finger(fingerToAverage.id);
            if( fingerFromFrame.valid )
            {
                Leap.vec3.add(average, average, fingerFromFrame.tipPosition);
                count++;
            }
        }
        if(count > 0)
        {
            Leap.vec3.scale(average, average, 1/count);
            avgDisplay.innerText = average[0] + ", " + average[1] + ", " + average[2];
        }
    }
});
controller.connect();

</script>

If an object with the same ID cannot be found – perhaps a hand or finger moved out of the Leap field of view – then a special, invalid object is returned instead. You can check for validity with the boolean valid attribute.

Getting Data from Skipped Frames

If the Leap Motion controller frame rate is running faster than your application update loop, more than one new frame of tracking data could be available by the time your update function is executed. The Leap.js library caches Frame objects as they are received. You can pass a history value to the Controller frame() function to access these stored frames. The history value counts back from the most recent frame.

The following example samples the frame history cache once per second and counts the total number of Gesture objects received since the last update:

<p>Gesture count: <span id="gestures">Gesture count</span></p>

<script>
var gestureCountDisplay = document.getElementById('gestures');

var controller = new Leap.Controller({enableGestures:true});
controller.connect();

var lastID = 0;

var update = function(){
    var gestureCount = 0;
    var frameCount = controller.frame().id - lastID;
    for(f = 0; f < frameCount; f++)
    {
        gestureCount += controller.frame(f).gestures.length;
    }
    gestureCountDisplay.innerText = gestureCount + " gesture objects counted in " + frameCount + " frames.";
    lastID = controller.frame().id;
}

setInterval(update, 1000);
</script>

The length of the history cache is an implementation detail; you should not rely on a particular length. The length is currently 200 frames for the JavaScript API.

Note that if a computer is bogged down with too many other computing tasks, frames can be abandoned by the Leap Motion software and not sent to your application. In this case, there will be a gap between consecutive frame ID values.