Case Study

Performative Detector

A fun Python project that uses MediaPipe and computer vision to detect when you're holding a matcha (or any cup) and plays your favorite songs on Spotify while displaying "PERFORMATIVE" on screen.

2025
~2 weeks
Personal Project
PythonMediaPipeOpenCVSpotify APISpotipyNumPy
performative_detector.py — Live

Click below to start live hand detection

OFF
OFF

not performative

↑ Live demo — make a holding/gripping gesture with your hand

// How It Works

Detection Pipeline

1

Camera Capture

Captures real-time video from your webcam

2

Hand Detection

MediaPipe identifies hand landmarks in each frame

3

Grip Analysis

Analyzes finger positions to detect holding gesture

4

Trigger Action

Plays music and displays 'PERFORMATIVE' overlay

// Capabilities

Features

Real-time camera input processing
Hand gesture detection using MediaPipe
Spotify integration for music playback
Live video display with text overlays
Two-hand and single-hand detection modes
Customizable playlists and tracks

// Code Sample

Hand Detection Logic

performative_detector.py
def is_holding_with_one_hand(self, hand_landmarks) -> bool:
    """
    Detect if a single hand is in a holding/gripping position
    by checking if fingers are curled (bent)
    """
    # Get fingertip and PIP (middle joint) landmarks
    finger_tips = [
        self.mp_hands.HandLandmark.INDEX_FINGER_TIP,
        self.mp_hands.HandLandmark.MIDDLE_FINGER_TIP,
        self.mp_hands.HandLandmark.RING_FINGER_TIP,
        self.mp_hands.HandLandmark.PINKY_TIP,
    ]
    
    finger_pips = [
        self.mp_hands.HandLandmark.INDEX_FINGER_PIP,
        self.mp_hands.HandLandmark.MIDDLE_FINGER_PIP,
        self.mp_hands.HandLandmark.RING_FINGER_PIP,
        self.mp_hands.HandLandmark.PINKY_PIP,
    ]
    
    curled_fingers = 0
    for tip, pip in zip(finger_tips, finger_pips):
        tip_y = hand_landmarks.landmark[tip].y
        pip_y = hand_landmarks.landmark[pip].y
        
        # If fingertip is below PIP, finger is curled
        if tip_y > pip_y:
            curled_fingers += 1
    
    # Holding detected if 3+ fingers are curled
    return curled_fingers >= 3

// Technical Deep Dive

Challenges & Solutions

Hand Detection Accuracy

Problem

Distinguishing between casual hand movements and intentional cup-holding gestures

Solution

Implemented multi-point finger curl detection and two-hand proximity checking for reliable grip recognition

Spotify Integration

Problem

Handling authentication flow and managing playback across different devices

Solution

Built OAuth flow with token caching, added AppleScript fallback for non-Premium accounts on macOS

Real-time Performance

Problem

Processing video frames fast enough while running ML models

Solution

Optimized frame processing with adaptive confidence thresholds and efficient NumPy operations

// Integration

Spotify Features

Playlist Support

Configure your favorite playlist or individual tracks to play when holding is detected.

Shuffle Mode

Enable shuffle to randomize playback from your configured playlist or track list.

Free Account Fallback

AppleScript fallback for non-Premium accounts, controlling Spotify desktop directly on macOS.

Want to Try It?

Clone the repo and detect your own performative moments! ✨🍵

Enjoyed this project? Consider supporting my work ☕