Plugin: pinchers

• 👌 Pinch your thumb with any finger to set that fingers "click" state
• Unpinched fingers are black, pinched fingers are red

Models: MediaPipe Hands

About: This plugin emits events, adds new properties to the hand data, and sets classes on the body to help you style elements based on which fingers were pinched

Activate: This plugin is automatically activated when the Hands Model is enabled

Tags: ['core']

Hand	Index [0]	Middle [1]	Ring [2]	Pinky [3]
Left
Right

Properties

Pinch States with .pinchState

This plugin adds handsfree.data.hands.pinchState to the Hands Model. It is a 2D array with the following:

handsfree.data.hands.pinchState = [
  // Left hand
  // index, middle, ring, pinky
  ['', '', '', ''],
  // Right hand
  // index, middle, ring, pinky
  ['', '', '', '']
]

Each index can be of one of the following states:

State	Note
start	When the pinch first starts
held	Every frame the pinch is held
released	When the pinch is released

const handsfree = new Handsfree({hands: true})
handsfree.use('logger', ({hands}) => {
  console.log(hands.pinchState)
})

Original Pinch Locations with .origPos

In addition the the .pinchState, you also have access to the original pixel {x, y} that the pinch occurred within the webcam through .origPinch. This is very useful for determining how far a pinch was "dragged". Like with .pinchState, handsfree.data.hands.origPinch contains one value per finger per hand:

// Log the original point of pinch
handsfree.on('finger-pinched-0-1', () => {
  // Display the x and y of the left pointer finger
  console.log(
    handsfree.data.hands.origPinch[0][0].x,
    handsfree.data.hands.origPinch[0][0].y
  )
})

Current Pinch Locations with .curPinch

Like .origPinch, .curPinch lists the current pixel {x, y} that the pinch is happening at. This is useful for calculating the distance since the .origPinch:

// Log the original point of pinch
handsfree.on('finger-pinched-1-3', () => {
  // Display the x and y of the right pinky
  console.log(
    handsfree.data.hands.curPinch[1][3].x,
    handsfree.data.hands.curPinch[1][3].y
  )
})

Events

Currently this plugin emits an event for every individual finger, which you can listen to. There are a total of 8 possible events, where h represents the hand (0 = left, 1 = right) and f represents the index:

/* SPECIFIC HAND */
/* Any event */
handsfree-finger-pinched-h-f
/* start event */
handsfree-finger-pinched-start-h-f
/* held event */
handsfree-finger-pinched-held-h-f
/* released event */
handsfree-finger-pinched-released-h-f

/* ANY HAND */
/* Any event */
handsfree-finger-pinched-f
/* start event */
handsfree-finger-pinched-start-f
/* held event */
handsfree-finger-pinched-held-f
/* released event */
handsfree-finger-pinched-released-f

Here are a few examples for listening to these events:

// ## SPECIFIC HAND
// Listen to any event from left hand (0), index finger (0)
document.addEventListener('handsfree-finger-pinched-0-0')
// Listen to any event right hand (1), pinky finger (3)
handsfree.on('finger-pinched-1-3')

// Listen to a specific event from left hand (0), middle finger (1)
handsfree.on('finger-pinched-start-0-1')
// Listen to a specific event from right hand (1), ring finger (2)
document.addEventListener('handsfree-finger-pinched-1-2')

// ## ANY HAND
// Listen to any event from with any index finger (0)
document.addEventListener('handsfree-finger-pinched-0')
// Listen to any event with any pinky finger (3)
handsfree.on('finger-pinched-3')

// Listen to a specific event with any middle finger (1)
handsfree.on('finger-pinched-start-1')
// Listen to a specific event with any ring finger (2)
document.addEventListener('handsfree-finger-pinched-2')

Classes

This plugin comes with many helper classes to help you style your app based on the pinching fingers, they look like this:

/* ## SPECIFIC HAND */
/* Left hand (0), index finger (0) */
.handsfree-show-when-finger-pinched-0-0
.handsfree-hide-when-finger-pinched-0-0

/* Right hand (1), pinky finger (3) */
.handsfree-show-when-finger-pinched-1-3
.handsfree-hide-when-finger-pinched-1-3

/* ## ANY HAND */
/* Any index finger (0) */
.handsfree-show-when-finger-pinched-0
.handsfree-hide-when-finger-pinched-0

/* Any pinky finger (3) */
.handsfree-show-when-finger-pinched-3
.handsfree-hide-when-finger-pinched-3

Simply apply these classes to the elements you'd like to show/hide. If you'd like some more styling then you can take advantage of the body classes that get added:

/* Left (0) middle finger (1) */
body.handsfree-finger-pinched-0-1

/* Right (1) ring finger (2) */
body.handsfree-finger-pinched-1-2

/* ANY middle finger (1) */
body.handsfree-finger-pinched-1

/* ANY ring finger (2) */
body.handsfree-finger-pinched-2

Full plugin code

export default {
  models: 'hands',
  enabled: true,
  tags: ['core'],

  // Index of fingertips
  fingertipIndex: [8, 12, 16, 20],

  // Number of frames the current element is the same as the last
  // [left, right]
  // [index, middle, ring, pinky]
  numFramesFocused: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // Whether the fingers are touching
  thresholdMet: [[0, 0, 0, 0,], [0, 0, 0, 0]],
  framesSinceLastGrab: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // The original grab point for each finger
  origPinch: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],
  curPinch: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],

  // Just downel
  pinchDowned: [
    [0, 0, 0, 0],
    [0, 0, 0, 0]
  ],
  pinchDown: [
    [false, false, false, false],
    [false, false, false, false]
  ],
  pinchUp: [
    [false, false, false, false],
    [false, false, false, false]
  ],

  // The tweened scrollTop, used to smoothen out scroll
  // [[leftHand], [rightHand]]
  tween: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],

  // Number of frames that has passed since the last grab
  numFramesFocused: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // Number of frames mouse has been downed
  mouseDowned: 0,
  // Is the mouse up?
  mouseUp: false,
  // Whether one of the morph confidences have been met
  mouseThresholdMet: false,

  config: {
    // Number of frames over the same element before activating that element
    framesToFocus: 10,

    // Number of pixels the middle and thumb tips must be near each other to drag
    threshold: 50,

    // Number of frames where a hold is not registered before releasing a drag
    numThresholdErrorFrames: 5,

    maxMouseDownedFrames: 1
  },

  onUse () {
    this.$target = window
  },

  /**
   * Scroll the page when the cursor goes above/below the threshold
   */
  onFrame ({hands}) {
    if (!hands.multiHandLandmarks) return

    const height = this.handsfree.debug.$canvas.hands.height
    const leftVisible = hands.multiHandedness.some(hand => hand.label === 'Right')
    const rightVisible = hands.multiHandedness.some(hand => hand.label === 'Left')
    
    // Detect if the threshold for clicking is met with specific morphs
    for (let n = 0; n < hands.multiHandLandmarks.length; n++) {
      // Set the hand index
      let hand = hands.multiHandedness[n].label === 'Right' ? 0 : 1
      
      for (let finger = 0; finger < 4; finger++) {
        // Check if fingers are touching
        const a = hands.multiHandLandmarks[n][4].x - hands.multiHandLandmarks[n][this.fingertipIndex[finger]].x
        const b = hands.multiHandLandmarks[n][4].y - hands.multiHandLandmarks[n][this.fingertipIndex[finger]].y
        const c = Math.sqrt(a*a + b*b) * height
        const thresholdMet = this.thresholdMet[hand][finger] = c < this.config.threshold

        if (thresholdMet) {
          // Set the current pinch
          this.curPinch[hand][finger] = hands.multiHandLandmarks[n][4]
          
          // Store the original pinch
          if (this.framesSinceLastGrab[hand][finger] > this.config.numThresholdErrorFrames) {
            this.origPinch[hand][finger] = hands.multiHandLandmarks[n][4]
            this.handsfree.TweenMax.killTweensOf(this.tween[hand][finger])
          }
          this.framesSinceLastGrab[hand][finger] = 0
        }
        ++this.framesSinceLastGrab[hand][finger]
      }
    }

    // Update the hands object
    hands.origPinch = this.origPinch
    hands.curPinch = this.curPinch
    this.handsfree.data.hands = this.getPinchStates(hands, leftVisible, rightVisible)
  },

  /**
   * Check if we are "mouse clicking"
   */
  getPinchStates (hands, leftVisible, rightVisible) {
    const visible = [leftVisible, rightVisible]

    // Make sure states are available
    hands.pinchState = [
      ['', '', '', ''],
      ['', '', '', '']
    ]
    
    // Loop through every hand and finger
    for (let hand = 0; hand < 2; hand++) {
      for (let finger = 0; finger < 4; finger++) {
        // Click
        if (visible[hand] && this.thresholdMet[hand][finger]) {
          this.pinchDowned[hand][finger]++
          document.body.classList.add(`handsfree-finger-pinched-${hand}-${finger}`, `handsfree-finger-pinched-${finger}`)
        } else {
          this.pinchUp[hand][finger] = this.pinchDowned[hand][finger]
          this.pinchDowned[hand][finger] = 0
          document.body.classList.remove(`handsfree-finger-pinched-${hand}-${finger}`, `handsfree-finger-pinched-${finger}`)
        }
        
        // Set the state
        if (this.pinchDowned[hand][finger] > 0 && this.pinchDowned[hand][finger] <= this.config.maxMouseDownedFrames) {
          hands.pinchState[hand][finger] = 'start'
        } else if (this.pinchDowned[hand][finger] > this.config.maxMouseDownedFrames) {
          hands.pinchState[hand][finger] = 'held'
        } else if (this.pinchUp[hand][finger]) {
          hands.pinchState[hand][finger] = 'released'
        } else {
          hands.pinchState[hand][finger] = ''
        }

        // Emit an event
        if (hands.pinchState[hand][finger]) {
          // Specific hand
          this.handsfree.emit(`finger-pinched-${hand}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          this.handsfree.emit(`finger-pinched-${hands.pinchState[hand][finger]}-${hand}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          // Any hand
          this.handsfree.emit(`finger-pinched-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          this.handsfree.emit(`finger-pinched-${hands.pinchState[hand][finger]}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
        }
      }
    }

    return hands
  }
}