Multi-Modal Synchronous Intelligence
AI that leverages environment and tone — and redacts PII in-stream.
Human voice is a multi-modal medium. Meaning travels in tone, pitch, rhythm, hesitation and the environment around the words as much as in the words — and a system that transcribes only the words is deciding on a fraction of the evidence it was given.
Almost every conversational stack discards that fraction by construction: speech becomes a string, and everything the string cannot carry is gone before the first decision is made. Recovering it afterwards means a second model, a second pass and a second bill — and it still arrives too late to change the turn it belongs to.
MMSI is the alternative: one owned pass that hears, decides and speaks inside the same turn, with every signal kept as a distribution the next stage can actually reason over — and identifiers structured as they arrive, so PII is redacted before a transcript ever exists. It ships today as Meta-1, our first released model, running inside the Whissle voice agent.
One owned model hears, decides and speaks
The ear & eyes
A 23-language audio-visual model. Emits words, intent, emotion, speaker and entities as distributions with confidence — every interim, not once at the end. Released as Meta-1, our first model.
A primitive, not a wrapper.
The action
A small owned model reads all six as distributions and gates the turn: answer, verify, or escalate. Deterministic, not vibes.
Auditable by construction.
The voice
Speaks back with the emotion the signal calls for, aligned to how the caller actually feels — not a flat readback of the text.
Affect survives to output.
The frontier model that writes a long answer is rented on purpose and swappable, so the system inherits no single lab's roadmap. The three models above are the ones we own — and they are the ones the signal passes through.
What the pass emits
- Emotion
- Affect as a distribution with confidence, updated on every interim — so a turn can be softened while it is still being spoken, not after.
- Intent
- What the speaker is trying to do, scored. A promise, a refusal and a question ask for three different next moves.
- Speaker
- Who is talking, as a probability rather than a yes / no — which is what lets a decision carry its own uncertainty instead of hiding it.
- Entities
- Amounts, dates and identifiers captured as spoken, in-stream. Tagged in-stream means they can be redacted in-stream.
- Sight
- On-device visual signal — presence, attention, gaze — at roughly nine numbers a second. No face recognition, by design.
- Environment
- Acoustic conditions the speech sits in, so the system can tell a hesitant caller from a noisy room.
Why it holds
Distributions, not tags
Every signal carries confidence and is updated each interim. A tag throws away the one thing a decision needs — how sure it should be.
PII redacted in-stream
Identifiers are structured as they arrive, so they can be dropped before a transcript exists. Extract downstream and the card number is already on disk.
Replayable decisions
Each action can be traced back to the evidence it fired on — which turn, which signal, which confidence.
Runs on your hardware
The data plane self-hosts inside your boundary. The audio does not have to leave to be understood.
Publications
Visual-Aware Speech Recognition for Noisy Scenarios
We propose a model that improves transcription by correlating noise sources to visual cues from the environment, enabling the system to naturally filter speech from noise — much like humans do. Our method re-purposes pretrained speech and visual encoders linked with multi-headed attention, showing significant improvements over audio-only models in noisy scenarios.
Authors: Karan Singla, Yash Khasbage, Mark Turner
Read paperNews Reporter: A Multi-lingual LLM Framework for Broadcast T.V News
We present a fine-tuned LLM framework for verifiable T.V. news QA pairs, surpassing similar base models and enhancing answer contextualization.
Authors: Tarun Jain, Yufei Gao, Sridhar Vanga, Karan Singla
Read paper1SPU: 1-step speech processing unit
1SPU extends ASR with tagged placeholders for semantic events, achieving improved transcription quality on SLUE and SLURP benchmarks.
Authors: Karan Singla, Shahab Jalavand, Andrej Ljolje, Antonio Moreno Daniel, Srinivas Bangalore, Yeon-Jun Kim, Ben Stern
Read paperE2E spoken entity extraction for virtual agents
This study refines entity extraction directly from speech, optimizing ASR encoders to transcribe only relevant content in virtual agent dialogs.
Authors: Karan Singla, Yeon-Jun Kim, Srinivas Bangalore
Read paperCombining pretrained speech and text encoders for continuous spoken language Processing
We introduce a multi-modal model for token-level classification using cross-modal attention, efficient for single GPU training.
Authors: Karan Singla, Mahnoosh Mehrabani, Daniel Pressel, Ryan Price, Bhargav S. Chinnari, Yeon-Jun Kim, Srinivas Bangalore
Read paperThe Red Hen anonymizer and the red hen protocol for de-identifying audiovisual recordings
The Red Hen Anonymizer enables de-identification of audiovisual data, ensuring privacy while supporting machine learning and research.
Authors: Yash Khasbage, Daniel Alcaraz Carrión, Jennifer Hinnell, Frankie Robertson, Karan Singla, Peter Uhrig, Mark Turner
Read paperNotes, experiments and write-ups from the same work.
Read the blog