Beyond Words
an interactive exhibit creating Non-Verbal Connections
Year
Type
Tools
2025 - 2026
Individual
Machine Learning, JavaScript
The project invites two participants to use their hands to create different animal and object shapes, prompted by a screen. These gestures are recognized by a gesture-detection system built using a machine learning model that is trained on a custom database of images of gestures.
Recipient of MIT Reality Hack Art Grant, 2026
What if we could connect and collaborate with one another without having to speak to each other?
This project started with this one question.
The Experience
Participants choose from a selection of gestures visible on the interface or make one of their own.
The system recognizes their gesture and maps it to a database of 3D model animations.
.png)
Building and Training the Dataset
A custom dataset of more than 20000 images of gestures taken of 6 people forming a series of gestures as they are able to, forms the basis for the dataset used to train the machine learning model.
These images were then used for training a custom model.
.png)
User Testing
I conducted multiple rounds of playtesting with participants unfamiliar with the project. Testing focused on gesture discoverability and the clarity of feedback. These sessions revealed moments where gestures were difficult to interpret and where participants relied on unintended cues. Based on these findings, I refined the gesture set, adjusted visual feedback, and simplified interactions to create a smoother experience.
.jpg)
