Handsfree.js

Build handsfree User Experiences and add face, hand, and pose tracking to your projects in a snap ✨👌

// Enable Mediapipe's "Hands" model
const handsfree = new Handsfree({hands: true})
// Enable plugins tagged with "browser"
handsfree.enablePlugins('browser')
// Start tracking
handsfree.start()

Demo: Scroll pages handsfree

Scroll pages handsfree!

👌 Pinch your thumb and index to grab the page
↕ While pinched, move hand up and down to scroll page

Powered by

Special thanks to: The STUDIO for Creative Inquiry, Glitch.com, Google PAIR, and you!

# Installing

CDN

NPM

Note: Some models are over 10Mb+ and may take a few seconds to load.

<head>
  <!-- Include Handsfree.js -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@8.5.1/build/lib/assets/handsfree.css" />
  <script src="https://unpkg.com/handsfree@8.5.1/build/lib/handsfree.js"></script>
</head>
 
<body>
  <!-- Instantiate and start it -->
  <script>
    const handsfree = new Handsfree({hands: true})
    handsfree.enablePlugins('browser')
    handsfree.start()
  </script>
</body>

# From your projects root
npm i handsfree

// Inside your app
import Handsfree from 'handsfree'
 
const handsfree = new Handsfree({hands: true})
handsfree.enablePlugins('browser')
handsfree.start()

Hosting the models yourself

By default, and for simplicity, the above will load models from the Unpkg CDN. This can be very slow. To host the models yourself (and to use offline) you can eject the models from the npm package into your project's public folder:

# Move the models into your project's public directory
# - change PUBLIC below to where you keep your project's assets
 
# ON WINDOWS
xcopy /e node_modules\handsfree\build\lib PUBLIC
# EVERYWHERE ELSE
cp -r node_modules/handsfree/build/lib/* PUBLIC

import Handsfree from 'handsfree'
 
const handsfree = new Handsfree({
  hands: true,
  // Set this to your where you moved the models into
  assetsPath: '/PUBLIC/assets',
})
handsfree.enablePlugins('browser')
handsfree.start()

# Models

Each of the following models can be combined and reconfigured in real time.

Model: MediaPipe Hands (2D)

📚 MediaPipe Hands documentation

21 2D hand landmarks per hand
Track up to 4 hands at once
Pinching states, hand pointers, and gestures

Model: TensorFlow Handpose (3D)

📚 TensorFlow Handpose documentation

21 3D hand landmarks
Can only track 1 hand at a time
📅 Extra helpers and plugins coming soon

Model: MediaPipe FaceMesh

📚 MediaPipe FaceMesh documentation

468 2D face landmarks
Track up to 4 faces at once
📅 Extra helpers and plugins coming soon

Model: MediaPipe Pose

📚 MediaPipe Pose documentation

Full body mode with 33 2D pose landmarks
Upper body mode with 25 2D upper pose landmarks
📅 Extra helpers and plugins coming soon

Model: TensorFlow Handpose

📚 Jeeliz Weboji documentation

6DOF head pose estimations
11 face morphs and 16 helper states
Comes with "Face Pointer" based plugins

# Quickstart Workflow

The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, and feel free to post on the Google Groups (opens new window) or Discord (opens new window) if you get stuck!

// Let's enable face tracking with the default Face Pointer
const handsfree = new Handsfree({weboji: true})
handsfree.enablePlugins('browser')

// Now let's start things up
handsfree.start()

// Let's create a plugin called "logger"
// - Plugins run on every frame and is how you "plug in" to the main loop
// - "this" context is the plugin itself. In this case, handsfree.plugin.logger
handsfree.use('logger', data => {
  console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this)
})

// Let's switch to hand tracking now. To demonstrate that you can do this live,
// let's create a plugin that switches to hand tracking when both eyebrows go up
handsfree.use('handTrackingSwitcher', ({weboji}) => {
  if (weboji.state.browsUp) {
    // Disable this plugin
    // Same as handsfree.plugin.handTrackingSwitcher.disable()
    this.disable()

    // Turn off face tracking and enable hand tracking
    handsfree.update({
      weboji: false,
      hands: true
    })
  }
})

// You can enable and disable any combination of models and plugins
handsfree.update({
  // Disable weboji which is currently running
  weboji: false,
  // Start the pose model
  pose: true,

  // This is also how you configure (or pre-configure) a bunch of plugins at once
  plugin: {
    fingerPointer: {enabled: false},
    faceScroll: {
      vertScroll: {
        scrollSpeed: 0.01
      }
    }
  }
})

// Disable all plugins
handsfree.disablePlugins()
// Enable only the plugins for making music (not actually implemented yet)
handsfree.enablePlugins('music')

// Overwrite our logger to display the original model APIs
handsfree.plugin.logger.onFrame = (data) => {
  console.log(handsfree.model.pose?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api)
}

📋 Guides →