GhostHand

GhostHand is an open-source, AI-powered gesture controller that eliminates the need for physical hardware by turning any standard webcam into a full game input device.

The Problem Traditional game controllers create a barrier — they're expensive, inaccessible to people with certain physical limitations, and limit the way humans can naturally interact with games. We asked: what if your hands were the controller?

What It Does GhostHand uses your webcam and computer vision to track both hands in real time. By analyzing the angle and distance between your hands 30 times per second, it maps natural human gestures directly to gamepad inputs — no gloves, no sensors, no hardware whatsoever.

• Tilt your hands like a steering wheel → full analog steering

• Push your hands apart → throttle/gas

• Bring your hands closer → brake

• Everything runs through a virtual Xbox 360 controller, so any game that supports a gamepad works instantly

How It Works Under the hood, GhostHand uses MediaPipe's 21-point hand landmark detection model on both hands simultaneously. Rather than tracking a single noisy point, we average all 21 landmarks per hand to compute a rock-solid hand center that stays stable even during fast movement or partial occlusion. The output is fed into a smoothing and deadzone pipeline before being sent to a virtual Xbox 360 controller via vgamepad — meaning the game sees it as real hardware.

Current State The current prototype fully supports racing games with analog steering, throttle, and braking. It includes a live calibration system, color-coded landmark visualization, and real-time steering feedback on screen.

What's Next GhostHand is designed to grow into a full multi-game platform. Upcoming profiles include FPS and shooter controls using pinch-to-shoot gestures and hand position for aiming, a dedicated desktop GUI app for switching profiles and tuning sensitivity, a custom trained gesture classification model for complex inputs like handbrake and horn, and cross-platform support beyond Windows.

Why It Matters GhostHand makes gaming more accessible, more intuitive, and opens the door to a future where physical controllers are optional — not required. It's a foundation for gesture-based interaction that extends far beyond gaming into accessibility tools, VR interfaces, and beyond.

Tech Stack: Python · OpenCV · MediaPipe · vgamepad · NumPy