Beyond Words
an interactive exhibit creating Non-Verbal Connections
Year
Type
Tools
2025 - 2026
Individual
Machine Learning, Dataset Creation, JavaScript
Beyond Words invites participants to shape their hands into animal and object forms. A custom-trained gesture-recognition model reads the shapes and maps them to a library of 3D animations turning a wordless hand gesture into a shared, animated story.
Recipient of MIT Reality Hack Art Grant, 2026
Context
What if we could connect without a single word?
Beyond Words started from one question: what if two people could collaborate and create something together with no shared language, no typing, no talking, just their hands.
The piece asks participants to shape hand-shadow gestures, animal and object forms prompted on screen, and turns those gestures into a shared, evolving story.
Success Criteria
"Connection doesn't require language, it requires a shared gesture or environment."
- Recognize gestures reliably from first-time users.
- Keep the gesture vocabulary discoverable without instruction.
- Make the resulting experience feel genuinely playful.
- Hold up across a full public exhibition run.
The Principle
Four rules shaped the build.
01.
Discoverable over comprehensive
A smaller gesture set people can find on their own beats a large one they need instructions for.
02.
Custom over generic
A model trained specifically on hand-shadow gestures, not a general-purpose gesture library.
03.
Shared, not parallel
The output had to feel like a direct feedback to the input gesture, not two separate things.
04.
Test with strangers
Design decisions were validated against people who'd never seen the piece before, not the design team.
Building and Training the Dataset
20,000+ images, trained from scratch
A custom dataset of more than 20000 images of gestures taken of 6 people forming a series of gestures as they are able to, forms the basis for the dataset used to train the machine learning model.
These images were then used for training a custom model.
.png)
The Experience
Participants choose from a selection of gestures visible on the interface or make one of their own.
The system recognizes their gesture and maps it to a database of 3D model animations.
.png)
User Testing
What testing with strangers revealed
I conducted multiple rounds of playtesting with participants unfamiliar with the project. Testing focused on gesture discoverability and the clarity of feedback. These sessions revealed moments where gestures were difficult to interpret and where participants relied on unintended cues. Based on these findings, I refined the gesture set, adjusted visual feedback, and simplified interactions to create a smoother experience.
01.
Gestures were hard to interpret
Issue: Some gestures in the original vocabulary were ambiguous even to the model, let alone to a first-time participant guessing at hand shapes.
Resolution: Narrowed the gesture set to shapes people could reliably form and the model could reliably recognize.
02.
Unintended cues confused the system
Issue: Participants sometimes relied on visual cues the design hadn't intended as signals, leading to mismatched recognition.
Resolution: Reworked visual feedback to make the actual intended cues clearer and more prominent.
.jpg)
Outcome
What the installation taught
A custom model can outperform a generic one for a specific interaction.
Training on hand-shadows specifically made an ambiguous input recognizable.
Testing with real strangers changes the design.
The gesture vocabulary and feedback system both shifted meaningfully based on playtesting, not assumption.
Wordless interaction is possible with the right shared vocabulary.
Strangers could co-create their own experiences using only gestures.
