Frames¶

Overview¶

Each Frame object contains an instantaneous snapshot of the scene recorded by the Leap Motion controller. Hands, fingers, and tools are the basic physical entities tracked by the Leap Motion system.

Get a Frame object containing tracking data from a connected Controller object. You can get a frame whenever your application is ready to process it using the frame() method of the Controller class:

if(controller.is_connected): #controller is a Leap.Controller object
    frame = controller.frame() #The latest frame
    previous = controller.frame(1) #The previous frame

The frame() function takes a history parameter that indicates how many frames back to retrieve. The last 60 frames are maintained in the history buffer (but don’t rely on this exact value, it could change in the future).

Note: Ordinarily consecutive frames have consecutively increasing ID values. However, in some cases, frame IDs are skipped. On resource-constrained computers, the Leap Motion software can drop frames. In addition, when the software enters robust mode to compensate for bright IR lighting conditions, two sensor frames are analyzed for each Frame object produced and the IDs for consecutive Frame objects increase by two. Finally, if you are using a Listener callback to get frames, the callback function is not invoked a second time until the first invocation returns. Thus, your application will miss frames if the callback function takes too long to process a frame. In this case, the missed frame is inserted in a history buffer.

Getting Data from a Frame¶

The Frame class defines several functions that provide access to the data in the frame. For example, the following code illustrates how to get the basic objects tracked by the Leap Motion system:

controller = Leap.Controller()
# wait until Controller.isConnected() evaluates to true
#...

frame = controller.frame()
hands = frame.hands
pointables = frame.pointables
fingers = frame.fingers
tools = frame.tools

The objects returned by the Frame object are all read-only. You can safely store them and use them in the future. They are thread-safe. Internally, the objects use the C++ Boost library shared pointer class.

Getting Frames by Polling¶

Polling the Controller object for frames is the simplest and often best strategy when your application has a natural frame rate. You just call the Controller frame() function when your application is ready to process a frame of data.

When you use polling, there is a chance that you will get the same frame twice in a row (if the application frame rate exceeds the Leap frame rate) or skip a frame (if the Leap frame rate exceeds the application frame rate). In many cases, missed or duplicated frames are not important. For example, if you are moving an object on the screen in response to hand movement, the movement should still be smooth (assuming the overall frame rate of your application is high enough). In those cases where it does matter, you can use the history parameter of the frame() function.

To detect whether you have already processed a frame, save the ID value assigned to the last frame processed and compare it to the current frame:

lastFrameID = 0
def processFrame(self, frame):
    if(frame.id == self.lastFrameID):
        return
    #...
    self.lastFrameID = frame.id

If your application has skipped frames, use the history parameter of the frame() function to access the skipped frames (as long as the Frame object is still in the history buffer):

lastProcessedFrameID = 0
def nextFrame(self, controller):
    currentID = controller.frame().id
    for history in range(0, currentID - self.lastProcessedFrameID):
        self.processFrame(controller.frame(history))
        self.lastProcessedFrameID = currentID

def processNextFrame(self, frame ):
    if(frame.is_valid):
        #...process

Getting Frames with Callbacks¶

Alternatively, you can use a Listener object to get frames at the Leap Motion controller’s frame rate. The Controller object calls the listener’s onFrame() function when a new frame is available. In the onFrame handler, you can call the Controller frame() function to get the Frame object itself.

Using listener callbacks is more complex because the callbacks are multi-threaded; each callback is invoked on an independent thread. You must ensure that any application data accessed by multiple threads is handled in a thread-safe manner. Even though the tracking data objects you get from the API are thread safe, other parts of your application may not be. A common problem is updating objects owned by a GUI layer that can only be modified by a specific thread. In such cases you must arrange to update the non-thread safe portions of your application from the appropriate thread rather than from the callback thread.

The following example defines a minimal Listener subclass that handles new frames of data:

class FrameListener(Leap.Listener):

    def onFrame(self, controller):
        frame = controller.frame() #The latest frame
        previous = controller.frame(1) #The previous frame
        #...

As you can see, getting the tracking data through a Listener object is otherwise the same as polling the controller.

Note that it is possible to skip a frame even when using Listener callbacks. If your onFrame callback function takes too long to complete, then the next frame is added to the history, but the onFrame callback is skipped. Less commonly, if the Leap software itself cannot finish processing a frame in time, that frame can be abandoned and not added to the history. This problem can occur when a computer is bogged down with too many other computing tasks.

Following entities across frames¶

If you have an ID of an entity from a different frame, you can get the object representing that entity in the current frame. Pass the ID to the Frame object function of the appropriate type:

hand = frame.hand(handID)
pointable = frame.pointable(pointableID)
finger = frame.finger(fingerID)
tool = frame.tool(toolID)

If an object with the same ID cannot be found – perhaps a hand moved out of the field of view – then a special, invalid object is returned instead. Invalid objects are instances of the appropriate class, but all their members return 0 values, zero vectors, or other invalid objects. This technique makes it more convenient to chain method calls together. For example, the following code snippet averages finger tip positions over several frames:

#Average a finger position for the last 10 frames
count = 0
average = Leap.Vector()
finger_to_average = frame.fingers[0]
for i in range(0,9):
    finger_from_frame = controller.frame(i).finger(finger_to_average.id)
    if(finger_from_frame.is_valid):
        average = average + finger_from_frame.tip_position
        count += 1
average = average/count

Without invalid objects, this code would have to check each Frame object before checking the returned Finger objects. Invalid objects reduce the amount of null checking you have to do when accessing Leap Motion tracking data.

Serialization¶

The Frame class provides serialize and deserialize() functions that allow you to store the data from a frame and later reconstruct it as a valid Frame object. See Serializing Tracking Data