Handsfree.js

# Model: Weboji

A person moving a virtual mouse pointer by moving their head

Overview and basic demo
  • 😀 6DOF head pose estimations
  • 😜 11 face morphs and 16 helper states
  • 🔌 Comes with "Face Pointer" based plugins

This model uses morphs to help estimate various face states simultaneously and includes assistive tech plugins for browsing pages with face gestures.

# Usage

# With defaults

const handsfree = new Handsfree({weboji: true})
handsfree.start()

# With config

// These are all the default values
handsfree = new Handsfree({
  weboji: {
    // Whether the model is enabled or not
    enabled: false,

    // Custom video settings
    videoSettings: {
      // The video, canvas, or image element
      // Omit this to auto create a <VIDEO> with the webcam
      videoElement: null,

      // ID of the device to use
      // Omit this to use the system default
      deviceId: null,

      // Which camera to use on the device
      // Possible values: 'user' (front), 'environment' (back)
      facingMode: 'user',

      // Video dimensions
      idealWidth: 320,
      idealHeight: 240,
      minWidth: 240,
      maxWidth: 1280,
      minHeight: 240,
      maxHeight: 1280
    },

    // Thresholds needed before these are considered "activated"
    // - Ranges from 0 (not active) to 1 (fully active)
    morphs: {
      threshold: {
        smileRight: 0.7,
        smileLeft: 0.7,
        browLeftDown: 0.8,
        browRightDown: 0.8,
        browLeftUp: 0.8,
        browRightUp: 0.8,
        eyeLeftClosed: 0.4,
        eyeRightClosed: 0.4,
        mouthOpen: 0.3,
        mouthRound: 0.8,
        upperLip: 0.5
      }
    }
  }
})

# Data

/**
 * {Boolean} Whether the face is detected or not
 */
handsfree.data.weboji.isDetected

/**
 * {Array} Face morphs, from 0 (not activated) to 1 (fully activated)
 * 
 * 0: smileRight → closed mouth smile right
 * 1: smileLeft → closed mouth smile left
 * 2: eyeBrowLeftDown → left eyebrow frowned
 * 3: eyeBrowRightDown → right eyebrow frowned
 * 4: eyeBrowLeftUp → raise left eyebrow (surprise)
 * 5: eyeBrowRightUp → raise right eyebrow (surprise)
 * 6: mouthOpen → open mouth
 * 7: mouthRound → o shaped mouth
 * 8: eyeRightClose → close right eye
 * 9: eyeLeftClose → close left eye
 * 10: mouthNasty → nasty mouth (show teeth)
 */
handsfree.data.weboji.morphs

/**
 * {Array} Head rotation [pitch, yaw, roll]
 * - in radians where [0, 0, 0] is the head pointed directly at camera
 */
handsfree.data.weboji.rotation

/**
 * {Array} Head rotation [pitch, yaw, roll]
 * - in degrees where [0, 0, 0] is the head pointed directly at camera
 */
handsfree.data.weboji.degree

/**
 * {Array} Head translation [x, y, s]
 * - These are each between 0 and 1
 * - Scale refers to the size of the head in relation to the webcam frame
 */
handsfree.data.weboji.translation

/**
 * {Object} Where on the screen the head is pointed at {x, y}
 * - This is updated by: handsfree.plugin.facePointer
 */
handsfree.data.weboji.pointer

/**
 * {Object} Helper booleans checking if the morph has reached a threshold
 * 
 * .smileRight      Smirking lips to the right
 * .smileLeft       Smirking lips to the left
 * .smile           Smiling equally to both sides
 * .smirk           Smiling either to the right or left, but not both
 * .pursed          Kiss face
 * 
 * .browLeftUp      Left eyebrow raised up
 * .browRightUp     Right eyebrow raised up
 * .browsUp         Both eyebrows raised up
 * .browLeftDown    Left eyebrow frowning down
 * .browRightDown   Right eyebrow frowning down
 * .browsDown       Both eyebrows frowning down
 * .browseUpDown    One eyebrow down and the other up ("The Rock eyebrows")
 * 
 * .eyeLeftClosed   The left eye closed
 * .eyeRightClosed  The right eye closed
 * .eyesClosed      Both eyes closed
 * 
 * .mouthClosed
 * .mouthOpen
 */
handsfree.data.weboji.state

# API

Please see the Weboji Docs (opens new window) to see available methods exposed through handsfree.model.weboji.api:

// Check if the head is detected or not
handsfree.model.weboji.api.is_detected()

# Using a pre-recorded video instead of a webcam

By default, setting {weboji: true} adds a new <video> element to the DOM to grab the webcam:

handsfree = new Handsfree({weboji: true})

To use a pre-recorded video or video stream, a canvas, or an image instead of a webcam set the .videoSettings.videoElement property:

handsfree = new Handsfree({
  weboji: {
    enabled: true,
    videoSettings: {
      videoElement: document.querySelector('#my-video')
    }
  }
})

# Examples

Add your project
If you've made something with this model I'd love to showcase it here! DM me on Twitter, make a pull request, or find us on Discord.

# See also

← ⚡ Events →

Debugger