William Zhang2026-09-28

One-Hand Pinch 3×3 Puzzle Game Based on MediaPipe Hand Landmarker

Real-time detection of 21 single-hand landmarks and pinch state using the MediaPipe Tasks Hand Landmarker model mapping "thumb + index finger pinch" directly to "drag / release" semantics — the player draws the ROI and plays the puzzle in front of the camera using only pinch gestures, with no keyboard, mouse, or touch screen required in the entire interaction chain.

CVHand TrackingGithub

A one-hand pinch puzzle game built on top of reComputer-RK / Docker / RKNN / YOLO

https://github.com/doublelf/puzzle_game

Project Overview

This project is a 217-line pure-puzzle application: gesture.py detects pinches → drawing.RectDrawer collects two rectangles → puzzle_game.PuzzleGame scrambles them into a 3×3 grid → the same state machine drives puzzle manipulation via pinch-and-drag.

puzzle-draw

Quick Start

1. Clone and Enter the Project Directory

bash
git clone https://github.com/doublelf/puzzle_game 
cd puzzle_game

2. Install Dependencies

bash
pip install -r requirements.txt

requirements.txt pins four dependencies:

text
opencv-python>=4.8.0
mediapipe>=0.10.14
numpy>=1.26.0
Pillow>=10.4.0

3. Launch the Game

bash
python main.py                  # Default camera 0, horizontal mirror enabled
python main.py --cam 1          # Use the 2nd camera
python main.py --no-flip        # Disable horizontal mirror
python main.py --no-landmarks   # Disable the landmark overlay (enabled by default)
python main.py --hide-trails    # Disable the gesture trail (enabled by default)

After launch you will first see the Pinch-Puzzle menu screen. Press any key to enter the game.

4. Operation Guide

PhasePlayer ActionProgram Feedback
DRAWPinch thumb + index finger, drag, then release to draw the first rectangleRectangle filled with semi-transparent green; top-left shows Draw rectangles: 1/2
DRAWRepeat once to draw the second rectangle2/2 → READY - generating puzzle..., auto-enters PLAY
PLAYPinch to snap to the nearest puzzle tile (gold pulsing border)Toast Tile #N selected + drag toward blank tile hint
PLAYDrag the index fingertip near an adjacent blank cellBoth tiles flash green simultaneously + Toast SWAP!, moves increments
PLAYAll 9 tiles return to place80 rainbow particles burst in the center + large Toast SOLVED!
AlwaysR resets the current puzzleReturns to DRAW phase, rectangles cleared
AlwaysM returns to the menuCamera keeps capturing, only the game loop is closed
AlwaysESC quitsReleases the camera, shuts down MediaPipe, destroys the window

Running Without a Camera

The recomputer-rk3576-devkit ships without a USB camera plugged in by default; this project ships three utilities to guarantee an end-to-end run without a camera:

tools/verify_all.py — 8-step Verification Suite

bash
cd /home/seeed/hand/puzzle_game
python3 -m tools.verify_all

Prints each step:

  • Module imports (gesture / drawing / puzzle_game / overlay / utils)
  • utils geometry + text functions
  • RectDrawer.update(None) does not throw
  • shuffle_solvable produces a legal board
  • Real MediaPipe Hand Landmarker loads the model on aarch64 (XNNPACK delegate, ~110 ms warm-up)
  • HandOverlay.add_event + render
  • Synthetic pipeline completes 60 frames without error

tools/run_synthetic.py — Offline Render to PNG

bash
python3 -m tools.run_synthetic --frames 180 --out /tmp/puzzle_demo
ls /tmp/puzzle_demo | wc -l      # → 180

Drives the full PuzzleGame + HandOverlay, writing every frame to out/puzzle_XXXX.png. In a no-camera scenario, browse frame by frame with feh / eog / an image viewer, or stitch them into an mp4 with ffmpeg:

bash
ffmpeg -framerate 30 -i /tmp/puzzle_demo/puzzle_%04d.png -c:v libx264 -pix_fmt yuv420p puzzle.mp4

tools/synthetic_source.py + tools/fake_tracker.py

A synthetic video source and a synthetic HandTracker exposed with an interface fully equivalent to cv2.VideoCapture / gesture.HandTracker; if you later want to plug in a real camera, just replace these two lines with:

python
cap = cv2.VideoCapture(args.cam)               # Replace SyntheticHandSource
tracker = HandTracker(num_hands=2)              # Replace SyntheticHandTracker

…and tools/run_synthetic.py upgrades into a real-camera-driven script.