SignBridge AI

A desktop app that sits beside a video call and bridges both directions. A webcam watches the signer, MediaPipe extracts hand and body landmarks, and a Transformer I trained classifies the sign — which is spoken into the call through a neural voice routed as a virtual microphone. Going the other way, the call's audio is captured, transcribed with streaming speech recognition, and painted as live captions over the call window. No audio or video leaves the machine.

• ▸Trained a ~1M-parameter Transformer (3 pre-norm layers, d=192, depthwise-conv stem) in PyTorch on 94,477 landmark sequences across 250 ASL signs — 74% top-1 / 91% top-5 on held-out signers (participant-grouped split; chance is 0.4%).
• ▸Built a fingerspelling recognizer at 98% validation accuracy across 28 classes (A–Z, space, delete) by extracting 33,000 hand-landmark samples from an image dataset with MediaPipe — unlimited vocabulary, letter by letter.
• ▸Feature contract: the Python training features and the TypeScript runtime features are held provably identical — a pytest fixture bundles the real app TypeScript with esbuild and asserts parity to <1e-6; ONNX exports embed SHA-256 hashes of the feature spec and refuse to load on mismatch.
• ▸Live captions at <1.5 s latency via streaming Zipformer STT (~22× real time on CPU) in an isolated Electron utility process; neural TTS (~6× real time) routed into calls as a virtual microphone with echo-safe gating.
• ▸Landmark pipeline at 37–82 FPS; inference in a Web Worker on ONNX Runtime Web with WebGPU and a WASM-threads fallback.

Recognizes isolated signs and joins glosses; it does not translate ASL grammar into English, which is an open research problem.