Host it yourself

The API surface is identical either way — the same agents, calls, sessions, KB and model endpoints. Code written against the cloud runs against your own gateway by changing one base URL.

Whissle is a lightweight gateway plus an optional GPU models service. The gateway is one container (~500 MB) carrying no model weights — it uses external STT/TTS/LLM APIs by default and reverse-proxies everything behind a single port:

• :9000 — nginx reverse proxy (HTTPS + WebRTC), the one door your clients talk to
• :8000 — the real-time voice pipeline (served under /bot/*)
• :8765 — the agent runtime (served under /agent/*)
• :8080 — the backend API (agents, calls, KB, billing)

For fully on-prem speech you add the separate GPU models service (~5 GB, on your GPU) and point the backend at it with WHISSLE_ASR_URL + WHISSLE_GRPC_TARGET (see below). Until you wire those, the backend uses its default speech provider and no camera track is accepted server-side.

The fastest way to a working stack — frontend, gateway and backend, with SQLite instead of a managed database and the local filesystem instead of cloud object storage. Requires ~16 GB RAM.

git clone https://github.com/WhissleAI/live_assist_full_docker   # requires self-host access
cd live_assist_full_docker
cp .env.example .env          # add your GEMINI_API_KEY
docker compose up -d

This brings up three services:

• :3000 — the frontend
• :9000 — the gateway (ASR + agent + nginx)
• :8080 — the backend (SQLite)

If you only want the backend gateway (bring your own frontend, or drive it purely over the API), run the prebuilt unified image — ASR, agent runtime, backend and nginx in one container:

docker run -d --name whissle-unified \
  -p 9000:9000 -p 8001:8001 -p 8765:8765 -p 8082:8082 \
  -e VARIANT=en-full \
  whissleasr/whissle-unified:latest

VARIANT picks which ASR models download on first boot: en-tiny (~310 MB, lightest), en-full (~600 MB, production English), multi-full (~2.8 GB, 23 languages), or all.

To keep speech entirely on your own hardware — private ASR/TTS and the video perception engine — run the GPU GPU models service on a GPU host and point the gateway at it. A single T4-class GPU is the cheapest fit.

# on the GPU host — build the image, then run it (see whissle_gateway_models)
docker build -f Dockerfile.gpu -t whissle/gateway-models:gpu .
docker run -d --name whissle-models --gpus all \
  -p 8001:8001 -p 8003:8003 -p 8002:8002 -p 50051:50051 \
  -v whissle-models:/models \           # weights persist across restarts
  -e VARIANT=en-full -e ASR_DEVICE=cuda \
  whissle/gateway-models:gpu
# ports: 8001 ASR (WS /listen + HTTP /transcribe) · 8003 TTS · 8002 video · 50051 ASR gRPC

Then, on the gateway/backend, wire it in with the real env vars:

WHISSLE_ASR_URL=ws://<models-host>:8001/listen   # on-prem ASR (words)
WHISSLE_STT_TRANSPORT=grpc                        # use the gRPC head for words+metadata
WHISSLE_GRPC_TARGET=<models-host>:50051           # emotion / intent / entities

Some teams don't want the agent platform at all — they already have their own app and only want Whissle's on-prem speech: ASR, TTS, and the metadata head (emotion / intent / entities). Run whissle_gateway_models on its own and call it directly — no gateway, no backend, no metering.

The service exposes the engines over REST, WebSocket and gRPC:

# streaming ASR (words)     ws://<host>:8001/listen
# batch ASR (a whole file)  POST http://<host>:8001/transcribe
# streaming TTS (kokoro)    ws://<host>:8003/stream
# video / scene perception  http://<host>:8002
# ASR over gRPC             <host>:50051   (unary + bidi streaming)

gRPC is the enterprise-integration path — the same engines in the same process, sharing the already-loaded models, and it's the transport that carries the metadata distributions. There's a stdlib-only test client in the repo:

python scripts/grpc_client.py --mode status
python scripts/grpc_client.py --mode unary path.wav --language en --no-lm
python scripts/grpc_client.py --mode stream path.wav --realtime

Everything that talks to the cloud takes a base-URL override. Point it at your own gateway (the :9000 door), and the SDK, CLI and browser widget behave identically.

SDK (@whissle/sdk)

import { Whissle } from "@whissle/sdk";

const whissle = new Whissle({
  apiKey: process.env.WHISSLE_API_KEY,   // your workspace secret (wsk_...)
  baseUrl: "https://your-gateway.example.com/bot",
});

CLI (@whissle/cli)

export WHISSLE_BASE_URL="https://your-gateway.example.com/bot"
# or per-command:
whissle agents list --base-url https://your-gateway.example.com/bot

To let agents send email, book calendars, or reach a CRM, connect the provider once — each is an OAuth app in that provider's own console (Google Cloud, Salesforce, Spotify…), whose client id / secret go in your .env like every other key. The one part that trips people up is the redirect URI: it points at your gateway, and it must carry the /bot prefix. Register these exact paths (host = your gateway's public URL):

https://<your-gateway>/bot/api/auth/google/callback            # Google sign-in (the base)
https://<your-gateway>/bot/api/calendar/google/callback        # Google Calendar
https://<your-gateway>/bot/api/datasources/email/callback      # Gmail send (scope gmail.send)
https://<your-gateway>/bot/api/datasources/google-sheets/callback
https://<your-gateway>/bot/api/datasources/salesforce/callback # Salesforce
https://<your-gateway>/bot/api/music/spotify/callback          # Spotify
https://<your-gateway>/bot/api/mcp/oauth/callback              # external MCP servers