Skip to content
Piyush Chugeja
All work

Case study

Gesturely

Indian Sign Language read from video, then written out as a sentence

  • Python
  • TensorFlow
  • MediaPipe
  • OpenCV
  • Gemini API
  • Streamlit

How it works

Gesturely turns Indian Sign Language into readable sentences. Each sign is captured as a short clip, reduced to a sequence of body-pose landmarks rather than pixels, and classified against a 40-word vocabulary. The recognised words then go to a language model with one instruction: make a grammatical sentence out of these and add nothing. The sentence can then be translated and spoken aloud.

  1. Sign videoOne sign per clip, from a webcam
  2. Pose landmarksMediaPipe joint coordinates per frame
  3. Sequence modelKeras classifier over 40 word classes
  4. Recognised wordsCollected in the order they were signed
  5. SentenceComposed by an LLM; optional speech output

The gap

Recognising a sign is not the same as reading a sentence

Most sign-language demos stop at the label: identify the gesture, print its name. Four signs in, what you have is four words with no grammar between them. Closing that last gap, from a list of correct words to something a hearing reader would actually write, is the half of the problem this project spent its time on.

The input

Joints, not pixels

Every frame goes through MediaPipe’s pose estimator first, so a sign arrives at the model as a short sequence of joint coordinates rather than a stack of images. The classifier is much smaller for it, and it can’t learn the signer’s shirt or the room behind them. Sequences are padded and truncated to a fixed 25 frames, so a quick sign and a slow one are the same shape, and the final word is the mode of the predictions across the clip rather than any single window’s guess.

Honest limits

Forty words, not an open dictionary

The vocabulary is a fixed list checked into the repository: forty everyday signs: greetings, places around a college, common objects and states. That makes this a working demonstration rather than a general ISL translator. Adding a word means recording it and retraining, and the last round of exactly that is in the commit history.

The sentence

Grammar handled by a model, not by rules

ISL word order is not English word order, so the recognised words need arranging before they read as a sentence. Rather than hand-write reordering rules, the word list goes to a prompted language model told to return one grammatical sentence and invent nothing. Because the handoff between the two halves is only ever a list of words, either side can be swapped without touching the other, and the repository ends up carrying two different sentence backends behind the same interface.

At a glance

Scope
Team project at VESIT, published as a paper
Recognition
Pose-landmark sequences, 40 word classes
Input
Webcam video, one sign per clip
Output
A sentence, optionally translated and spoken

My part: The gesture models and the path from a landmark sequence to a finished sentence. Three of us on the project at VESIT; the approach was published as a paper.