Hello Monday / DEPT®

Fingerspelling with
Machine Learning

Best User Interface / Webby Winner
Hello Monday / DEPT®

Best User Interface

"Refining and debugging the technology as you go along becomes a given but pays off as the technology begins to provide experiences that previously were not possible in a browser context." Hello Monday / DEPT®
image description

Q: Can you describe your project and the concept behind it? A: Every year 2-3 out of every 1,000 children born in the US are deaf or hard-of-hearing. 90% of these children are born to hearing parents, and in many cases, their child is the first deaf person these parents have ever encountered. Fingerspelling.xyz is a browser-based app that uses a webcam and machine learning to analyze your hand shapes so you can learn to sign the ASL alphabet correctly.
image description

Q: Tell us about your initial moodboard, wireframe, or prototype. How did things change throughout the process? A: Refining and debugging the technology as you go along becomes a given but pays off as the technology begins to provide experiences that previously were not possible in a browser context.
image description

Q: What influenced your chosen technical approach, and how did it go beyond past methods? A: The fingerspelling game is a great way to introduce the basics of ASL in a fun and playful way. Instead of having to read or watch videos about fingerspelling, we offer an online teaching tool that guides you step by step in how to master fingerspelling — hands on!

When did you experience a breakthrough or an "a-ha" moment during this project?

Mixing fairly new types of input, like webcam-based hand tracking, into a browser experience comes with a long process of trail and error, since we were working in uncharted territory. Refining and debugging the technology as you go along becomes a given but pays off as the technology begins to provide experiences that previously were not possible in a browser context.

Watch

image description

Q: What web technologies, tools, and resources did you use to develop this? A: The game leverages advanced hand recognition technology, matched with machine learning, to give you real time feedback via the webcam for each sign and word you spell correctly. We use MediaPipe Hands for hand tracking. The game is designed for desktop, primarily to be used by parents for deaf children, but kids will also find it a fun way to practice and improve their fingerspelling technique.
image description

Q: How did the final product meet or exceed your expectations? A: How did the final product meet or exceed your expectations? The fingerspelling game is a great way to introduce the basics of ASL in a fun and playful way. It guides you step by step in how to master fingerspelling — hands on and in the browser.