Hands2MIDI

This converts your hand movements directly to MIDI. Intended for live music performance, but can be used other purposes.

Mediapipe hand tracking is incredibly fast and accurate, but it only outputs a list of detected keypoints. What if you wanted to track the bend of each finger individually? 🤘 Detect when you tap a finger to your thumb? 👌

Demo webpage here (Open in a browser supporting Web-MIDI, e.g. Chrome)

While making my Camera Pitch Bend project I realised that most people have very individual ways of using MIDI in their performances or composing process. So I wanted to make a more general tool that would help people map different hand movements and positions to different MIDI messages, and let them then map those to performance effects in their DAW or software of choice. (It doesn’t even have to be music - you could use this to control a robot or something else if you like!)

For example try mapping an index finger to the cutoff, and hand Y-position to filter position - you can now do a wah effect just by waving your hand.