WebSocket Communication

The Leap Motion software installed on any computer with a Leap Motion controller provides tracking data through a WebSocket server. The WebSocket server listens to port 6347 on the localhost domain (http://127.0.0.1:6437). Any client application, including Web clients, that can make a WebSocket connection can access the Leap Motion tracking data in the form of JSON-formatted messages. The WebSocket server is provided by the leapd process, which runs as a service on Windows and a daemon on OS X and Linux.

In most cases, you should use an existing client library rather than implement this protocol directly in an application. This article is provided to help you if you are developing your own library. Existing libraries include:

Connecting to the WebSocket server

The following example uses JavaScript and the standard browser WebSocket class to manually connect to the WebSocket server. If the connection is successful, the example enables gesture recognition in the ws.onopen() function.

var ws;

// Support both the WebSocket and MozWebSocket objects
if ((typeof(WebSocket) == 'undefined') &&
    (typeof(MozWebSocket) != 'undefined')) {
  WebSocket = MozWebSocket;
}

// Create the socket with event handlers
function connectToWebSocket() {
  // Create and open the socket
  ws = new WebSocket("ws://localhost:6437/v4.json");

  // On successful connection
  ws.onopen = function(event) {
    var enableMessage = JSON.stringify({enableGestures: true});
    ws.send(enableMessage); // Enable gestures
    var backgroundMessage = JSON.stringify({background: true});
    ws.send(backgroundMessage); // Get frames in background
    console.log("open");
  };

  // On message received
  ws.onmessage = function(event) {
      var obj = JSON.parse(event.data);
      var str = JSON.stringify(obj, undefined, 2);
      if(obj.id){
          console.log("Frame data for " + obj.id);
      } else {
          console.log("message " + event.data);
      }
  };

  // On socket close
  ws.onclose = function(event) {
    ws = null;
    console.log("close");
  }

  // On socket error
  ws.onerror = function(event) {
    console.log("error");
  };
}

connectToWebSocket();

Note that the Leap.js library automatically manages the WebSocket connection for you, so you do not need to make calls directly to the WebSocket object if you are using Leap.js (or another client library that manages the connection).

Protocols

The communication between the WebSocket server and a client is mediated by a data exchange subprotocol. This subprotocol defines which messages can be sent between the client and the server and the format of the JSON data. The protocol version defines which feature set the server uses to communicate with a client.

You can request a specific version of the Leap Motion subprotocol by adding a string of the form v6.json to the WebSocket URL, where the digit, in this case 6 indicates which version to use. If you omit this part of the URL, you will get version 1. Note that the Leap Motion leap.js JavaScript library automatically selects the protocol version corresponding to the library version. When you successfully connect to the Leap Motion WebSocket server, it responds with a message reporting the protocol version. If the client’s Leap Motion software is out-of-date, then this version could be less than the one you requested.

The current version of the protocol is v6.json.

Client-Server Messages

The WebSocket server and a client can exchange the following messages:

  • Server to client

    version

    Indicates the version numbers of the Leap Motion service and WebSocket subprotocol:

    {"serviceVersion":"2.3.1+33747", "version":6}
    

    deviceEvent

    Sent when the Leap Motion when the service/daemon is paused or resumed and when the controller hardware is plugged in or unplugged:

    {
      "event": {
        "state": {
          "attached": false,
          "id": "NNNNNNNNNNN",
          "streaming": false,
          "type": "peripheral"
        },
        "type": "deviceEvent"
      }
    }
    
    id A string containing the ID of the Leap Mption device.
    attached Whether the hardware device is plugged in or attached.
    streaming Whether the service/daemon is sending tracking data. Always false if attached is false.
    type The hardware form factor. One of: “peripheral”, “keyboard”, “laptop”
    Tracking data

    Contains a set of tracking data. See JSON tracking data format.

  • Client to server

    background

    Specifies whether the client wants to receive frames when it is not the focused application:

    {"background": true}
    {"background": false}
    
    focused

    Specifies whether the application is active. Setting focused to true, stops other WebSocket clients from getting data. Setting focused to false stops your WebSocket client from getting data (and potentially unwanted input):

    {"focused": true}
    {"focused": false}
    
    enableGestures

    Enables or disables gesture recognition data:

    {"enableGestures": true}
    {"enableGestures": false}
    
    optimizeHMD

    Specifies whether the client application expects the Leap Motion hardware to be attached to a head-mounted display:

    {"optimizeHMD": true}
    {"optimizeHMD": false}
    

To send a message to the WebSocket server, first encode the desired message as a JSON string. In JavaScript, for example, you can use the JSON stringify() function and then send the message using the WebSocket send function:

//ws is a connected WebSocket object
var backgroundMessage = JSON.stringify({background: true});
ws.send(backgroundMessage);

JSON tracking data format

Each frame of data from the WebSocket server contains JSON defining a frame. The attributes of a frame in the JSON message are similar, but not identical to those of a |Frame|_ object obtained through the native library. They include:

"currentFrameRate": float
"id": float
"r": array of floats (Matrix)
"s": float
"t":  array of floats (vector)
"timestamp": integer
"devices": not implemented (always an empty array)

"gestures": array of Gesture objects
    (Attributes vary by gesture type)
    "center": array of floats (vector) -- circle only
    "direction": array of floats (vector) -- swipe, keyTap, screenTap only
    "duration": integer microseconds
    "handIds": array of integers
    "id": integer
    "normal": array of floats -- circle only
    "pointableIds": array
    "position": array of floats (vector) -- swipe, keyTap, screenTap only
    "progress": float -- circle, keyTap, screenTap only
    "radius": float -- circle only
    "speed": float -- swipe only
    "startPosition": array of float (vector) -- swipe only
    "state": string - one of "start", "update", "stop"
    "type": string - one of "circle", "swipe", "keyTap", "screenTap"

"hands": array of Hand objects
   "armBasis: the 3 basis vectors of the arm (array of vectors)
   "armWidth: float
   "confidence: float
   "direction": array of floats (vector)
   "elbow: array of floats (vector)
   "grabStrength: float
   "id": integer
   "palmNormal": array of floats (vector)
   "palmPosition": array of floats (vector)
   "palmVelocity": array of floats (vector)
   "pinchStrength: float
   "r": array of floats (Matrix)
   "s": float
   "sphereCenter": array of floats (vector)
   "sphereRadius": float
   "stabilizedPalmPosition": array of floats (vector)
   "t": array of floats (vector)
   "timeVisible": float
   "type": string - one of "right", "left"
   "wrist: array of floats (vector)

"interactionBox": object
   "center": array of floats (vector)
   "size": array of floats  (vector)

"pointables": array of Pointable objects
   "bases": the 3 basis vectors for each bone, in index order, wrist to tip, (array of vectors).
   "btipPosition": the position of the tip of the distal phalanx as an array of 3 floats.
   "carpPosition": the position of the base of metacarpal bone as an array of 3 floats.
   "dipPosition:" the position of the base of the distal phalanx as an array of 3 floats.
   "direction": array of floats (vector)
   "extended": boolean (true or false)
   "handId": integer
   "id": integer
   "length": float
   "mcpPosition": a position vector as an array of 3 floating point numbers
   "pipPosition": a position vector as an array of 3 floating point numbers
   "stabilizedTipPosition": array of floats (vector)
   "timeVisible": float
   "tipPosition":  array of floats (vector)
   "tipVelocity":  array of floats (vector)
   "tool": boolean (true or false)
   "touchDistance": float
   "touchZone": string - one of "none", "hovering", "touching"
   "type": integer - 0 is thumb; 4 is pinky
   "width": float

Notes:

  • Gesture data is only included in a frame of data if gesture recognition is enabled by sending the enableGestures message to the WebSocket server.
  • Hand.basis is computed from the palmNormal and direction vectors. It is not included in the JSON data.

Motion factors

The motion factors, r, s, t, attached to Hand and |Frame| objects are snapshots of the motion occuring across frames. These factors must be combined with those of a previous frame to derive the relative motion.

  • r – a 3x3 rotation matrix
  • s – a scale factor
  • t – a 3-element translation vector

Rotation factor

The matrix expressing the relative rotation between two frames can be calculated by multiplying the r matrix from the current frame by the inverse of the r matrix of the starting frame.

\[\mathbf{rotation} = \mathbf{r_{current frame}} * \mathbf{r_{since frame}^{-1}}\]

Scale factor

The relative scale factor between two frames can be calculated by subtracting the s value from the starting frame from the current s value and computing the natural exponential function of the result.

\[scalefactor = e^{s_{current frame} - s_{sinceframe}}\]

Translation factor

The relative translation factor between two frames can be calculated by subtracting the t vector from the starting frame from the current t vector.

\[\mathbf{\overrightarrow{translation}} = \mathbf{\vec{t}}_{current frame} - \mathbf{\vec{t}}_{since frame}\]

Protocol Changes

This document describes the most recent version of the protocol.

Each version of the protocol corresponds to particular versions of the Leap Motion software installed on a client computer as well as a versions of the leap.js library which use the protocol:

Protocol Leap Motion Client Software Leap.js
v1.json prerelease 0.1.n
v2.json prerelease 0.2.n
v3.json 1.0+ 0.3.n
v4.json 1.0.9+ 0.4.n
v5.json 1.2.0+ 0.5.n
v6.json 2.0+ 0.6.n

The following changes were made from previous versions of the protocol:

Version 6

  • Added additional data from the Leap Motion v2, skeletal tracking model:
    • Hand.confidence
    • Hand.grabStrength
    • Hand.pinchStrength
    • Hand.type
    • Pointable.dipPosition
    • Pointable.extended
    • Pointable.mcpPosition
    • Pointable.pipPosition
    • Pointable.type
  • Added Pointable.width for fingers
  • Added optimizeHMD client-to-server message.

Version 5

  • Added deviceEvent messages.
  • Removed deviceConnected events in favore of deviceEvent messages.

Version 4

  • Added focused and background messages.
  • Removed heartbeat messages. Servers supporting protocol version 4 or above ignore heartbeat messages, even when connecting to clients requesting protocol versions 2 or 3.

Version 3

  • Added deviceConnected events.

Version 2

  • Added heartbeat messages. If your application supports protocol version 2 or 3, you should send a heartbeat message every 100ms from clients connecting with version 2 or 3. If a heartbeat is not received by the server within this time period, it assumes the client is not active and stops sending data until it receives another heartbeat. Heartbeats were removed in version 4 in favor of focused and background messages.