Frames¶

Overview¶

Each LeapFrame object contains an instantaneous snapshot of the scene recorded by the Leap Motion controller. Hands, fingers, and tools are the basic physical entities tracked by the Leap Motion system.

Get a LeapFrame object containing tracking data from a connected LeapController object. You can get a frame whenever your application is ready to process it using the frame() method of the LeapController class:

if( controller.isConnected) //controller is a Controller object
{
    LeapFrame *frame = [controller frame:0]; //The latest frame
    LeapFrame *previous = [controller frame:1]; //The previous frame
}

The frame() function takes a history parameter that indicates how many frames back to retrieve. The last 60 frames are maintained in the history buffer (but don’t rely on this exact value, it could change in the future).

Note: Ordinarily consecutive frames have consecutively increasing ID values. However, in some cases, frame IDs are skipped. On resource-constrained computers, the Leap Motion software can drop frames. In addition, when the software enters robust mode to compensate for bright IR lighting conditions, two sensor frames are analyzed for each LeapFrame object produced and the IDs for consecutive Frame objects increase by two. Finally, if you are using a LeapListener-p callback to get frames, the callback function is not invoked a second time until the first invocation returns. Thus, your application will miss frames if the callback function takes too long to process a frame. In this case, the missed frame is inserted in a history buffer.

Getting Data from a Frame¶

The LeapFrame class defines several functions that provide access to the data in the frame. For example, the following code illustrates how to get the basic objects tracked by the Leap Motion system:

LeapController *controller = [[LeapController alloc] init];
// wait until Controller.isConnected() evaluates to true
//...

LeapFrame *frame = [controller frame:0];
NSArray *hands = frame.hands;
NSArray *pointables = frame.pointables;
NSArray *fingers = frame.fingers;
NSArray *tools = frame.tools;

The objects returned by the LeapFrame object are all read-only. You can safely store them and use them in the future. They are thread-safe. Internally, the objects use the C++ Boost library shared pointer class.

Getting Frames by Polling¶

Polling the LeapController object for frames is the simplest and often best strategy when your application has a natural frame rate. You just call the LeapController frame() function when your application is ready to process a frame of data.

When you use polling, there is a chance that you will get the same frame twice in a row (if the application frame rate exceeds the Leap frame rate) or skip a frame (if the Leap frame rate exceeds the application frame rate). In many cases, missed or duplicated frames are not important. For example, if you are moving an object on the screen in response to hand movement, the movement should still be smooth (assuming the overall frame rate of your application is high enough). In those cases where it does matter, you can use the history parameter of the frame() function.

To detect whether you have already processed a frame, save the ID value assigned to the last frame processed and compare it to the current frame:

int64_t lastFrameID = 0;

- (void) test_processFrame:(LeapFrame*) frame
{
    if( frame.id == lastFrameID ) return;
    //...
    lastFrameID = frame.id;
}

If your application has skipped frames, use the history parameter of the frame() function to access the skipped frames (as long as the LeapFrame object is still in the history buffer):

int64_t lastProcessedFrameID = 0;

- (void) test_nextFrame:(LeapController*) controller
{
    int64_t currentID = [controller frame:0].id;
    for( int history = 0; history < currentID - lastProcessedFrameID; history++)
    {
    processFrame:[controller frame:history];
    }
    lastProcessedFrameID = currentID;
}

- (void) test_processNextFrame:(LeapFrame*) frame
{
    if(frame.isValid)
    {
        //...
    }
}

Getting Frames with Callbacks¶

Alternatively, you can use a LeapListener-p object to get frames at the Leap Motion controller’s frame rate. The LeapController object calls the listener’s onFrame() function when a new frame is available. In the onFrame handler, you can call the LeapController frame() function to get the LeapFrame object itself.

Using listener callbacks is more complex because the callbacks are multi-threaded; each callback is invoked on an independent thread. You must ensure that any application data accessed by multiple threads is handled in a thread-safe manner. Even though the tracking data objects you get from the API are thread safe, other parts of your application may not be. A common problem is updating objects owned by a GUI layer that can only be modified by a specific thread. In such cases you must arrange to update the non-thread safe portions of your application from the appropriate thread rather than from the callback thread.

The following example defines a minimal Listener subclass that handles new frames of data:

@interface FrameListener : NSObject
- (void) onFrame:(LeapController*) controller;
@end

@implementation FrameListener

- (void) onFrame:(LeapController*) controller
{
    LeapFrame *frame = [controller frame:0]; //The latest frame
    LeapFrame *previous = [controller frame:1]; //The previous frame
    //...
}
@end

As you can see, getting the tracking data through a LeapListener-p object is otherwise the same as polling the controller.

Note that it is possible to skip a frame even when using LeapListener-p callbacks. If your onFrame callback function takes too long to complete, then the next frame is added to the history, but the onFrame callback is skipped. Less commonly, if the Leap software itself cannot finish processing a frame in time, that frame can be abandoned and not added to the history. This problem can occur when a computer is bogged down with too many other computing tasks.

Following entities across frames¶

If you have an ID of an entity from a different frame, you can get the object representing that entity in the current frame. Pass the ID to the LeapFrame object function of the appropriate type:

LeapFinger *finger = [[controller frame:n] finger:fingerID];

If an object with the same ID cannot be found – perhaps a hand moved out of the field of view – then a special, invalid object is returned instead. Invalid objects are instances of the appropriate class, but all their members return 0 values, zero vectors, or other invalid objects. This technique makes it more convenient to chain method calls together. For example, the following code snippet averages finger tip positions over several frames:

//Average a finger position for the last 10 frames
int count = 0;
LeapVector *average = [[LeapVector alloc] init];
LeapFinger *fingerToAverage = [[frame fingers] frontmost];
for( int i = 0; i < 10; i++ )
{
    LeapFinger *fingerFromFrame = [[controller frame:i] finger:fingerToAverage.id];
    if( fingerFromFrame.isValid )
    {
        average = [average plus:fingerFromFrame.tipPosition];
        count++;
    }
}
if(count > 1) average = [average times:1/count];

Without invalid objects, this code would have to check each LeapFrame object before checking the returned Finger objects. Invalid objects reduce the amount of null checking you have to do when accessing Leap Motion tracking data.

Serialization¶

The LeapFrame class provides serialize and deserialize: functions that allow you to store the data from a frame and later reconstruct it as a valid LeapFrame object. See Serializing Tracking Data