Performative Detector
A fun Python project that uses MediaPipe and computer vision to detect when you're holding a matcha (or any cup) and plays your favorite songs on Spotify while displaying "PERFORMATIVE" on screen.
Click below to start live hand detection
not performative
↑ Live demo — make a holding/gripping gesture with your hand
// How It Works
Detection Pipeline
Camera Capture
Captures real-time video from your webcam
Hand Detection
MediaPipe identifies hand landmarks in each frame
Grip Analysis
Analyzes finger positions to detect holding gesture
Trigger Action
Plays music and displays 'PERFORMATIVE' overlay
// Capabilities
Features
// Code Sample
Hand Detection Logic
def is_holding_with_one_hand(self, hand_landmarks) -> bool:
"""
Detect if a single hand is in a holding/gripping position
by checking if fingers are curled (bent)
"""
# Get fingertip and PIP (middle joint) landmarks
finger_tips = [
self.mp_hands.HandLandmark.INDEX_FINGER_TIP,
self.mp_hands.HandLandmark.MIDDLE_FINGER_TIP,
self.mp_hands.HandLandmark.RING_FINGER_TIP,
self.mp_hands.HandLandmark.PINKY_TIP,
]
finger_pips = [
self.mp_hands.HandLandmark.INDEX_FINGER_PIP,
self.mp_hands.HandLandmark.MIDDLE_FINGER_PIP,
self.mp_hands.HandLandmark.RING_FINGER_PIP,
self.mp_hands.HandLandmark.PINKY_PIP,
]
curled_fingers = 0
for tip, pip in zip(finger_tips, finger_pips):
tip_y = hand_landmarks.landmark[tip].y
pip_y = hand_landmarks.landmark[pip].y
# If fingertip is below PIP, finger is curled
if tip_y > pip_y:
curled_fingers += 1
# Holding detected if 3+ fingers are curled
return curled_fingers >= 3// Technical Deep Dive
Challenges & Solutions
Hand Detection Accuracy
Problem
Distinguishing between casual hand movements and intentional cup-holding gestures
Solution
Implemented multi-point finger curl detection and two-hand proximity checking for reliable grip recognition
Spotify Integration
Problem
Handling authentication flow and managing playback across different devices
Solution
Built OAuth flow with token caching, added AppleScript fallback for non-Premium accounts on macOS
Real-time Performance
Problem
Processing video frames fast enough while running ML models
Solution
Optimized frame processing with adaptive confidence thresholds and efficient NumPy operations
// Integration
Spotify Features
Playlist Support
Configure your favorite playlist or individual tracks to play when holding is detected.
Shuffle Mode
Enable shuffle to randomize playback from your configured playlist or track list.
Free Account Fallback
AppleScript fallback for non-Premium accounts, controlling Spotify desktop directly on macOS.