Build a voice agent app

One folder — a JSON file, a 200-line server and a single HTML page — that stands up four working voice agents and the screens around them. It ships inside the public SDK repository as examples/interview-platform, and it is deliberately small enough to read in a sitting.

Small is not the same as partial. It does everything a real integration does: creates and configures agents, ingests the knowledge that makes them know your domain, authenticates a user before minting them a session, carries a live conversation with a rendered avatar, and reads back transcripts and grades afterwards. What it leaves out is a database, a build step and a login screen — the three things you already have.

Three packages, and which is which

Where the code runs decides the package, not preference. Getting this wrong is the one mistake with a security consequence.

How a session starts

browser :4000          your server :4000              Whissle
   |                       |                            |
   |--- POST /api/sessions ----->|                      |
   |                    who is this user?               |
   |               may they talk to this agent?         |
   |                            |--- wsk_ key --------->|
   |<-- token + transport + ICE -|                      |
   |                            |                       |
   +---------- joins the session directly -------------->|
                voice - avatar - transcripts

The server's job ends when the token is issued; media never passes through it. Keep that shape whatever else you change — your application decides who may talk to which agent, because it is the only thing that knows who the user is.

Node 20 or newer and a Whissle key are the whole prerequisite list. No database, no Docker, no private repository access — the SDK repo is public and the packages install from npm.

Get your own key rather than sharing one. Sessions are billed to the workspace the key belongs to, and call records are scoped to it.

1. Sign in at whissle.ai.
2. Settings → API Keys → Create.
3. Grant agents:read, agents:write, kb:read, kb:write, calls:read.
4. Copy the secret. It is shown once and cannot be retrieved later.

Two minutes here saves an afternoon later: if the key is wrong you want to learn it from whoami, not from an app that starts and then fails at the first call.

npm i -g @whissle/cli

whissle login          # paste the wsk_ key when prompted
                       # saved to ~/.whissle/config.json (0600)
whissle whoami         # confirms the workspace and your role

A healthy reply names your workspace:

{
  "base_url": "https://aws-gateway-backend.whissle.ai/bot",
  "organization": { "id": "65c2980e-…", "name": "Your Workspace" },
  "role": "owner"
}

Two more, to confirm the workspace can actually run a session:

whissle agents types   # the blueprint catalogue: skills_exam, ai_tutor, …
whissle usage          # wallet balance; a 402 later means this hit zero

git clone https://github.com/WhissleAI/agents_js_sdk.git
cd agents_js_sdk/examples/interview-platform

npm install
export WHISSLE_API_KEY=wsk_live_…
npm start

It provisions on boot and tells you what it did:

Whissle example app → http://localhost:4000

provisioned "Interview — Electrician"   (a41f…)
provisioned "Interview — Line Cook"     (b7c2…)
provisioned "Interview coach"           (c93d…)
provisioned "Scheduling assistant"      (d05e…)

4/4 agent(s) ready.

You are running when

• The boot log says 4/4 agent(s) ready
• Agents lists four, two of them with an avatar code
• A call connects and the agent speaks first, unprompted
• The face's mouth moves on that greeting — not only on later replies
• After hanging up, the call appears under Sessions

Everything specific to this app is in agents.json. Editing that file is how you change what the application is; the server and the page do not mention electricians or line cooks anywhere.

{
  "id": "line-cook",
  "name": "Interview — Line Cook",
  "type": "skills_exam",          // whissle agents types
  "avatar": "F2-TL",              // omit → voice only
  "summary": "Trade interview for a busy service line.",
  "knowledge": "Poultry to 74 °C, ground meat 71 °C, …",
  "interview": {                  // present → examiner + rubric
    "level": "entry",
    "skills":    ["Food safety and HACCP", "Cooking to temperature", …],
    "questions": ["What are the safe internal temperatures…", …]
  }
}

On boot, each entry is turned into a real agent — or adopted, if one with that name already exists. Restarting the server never produces a second copy. This is @whissle/sdk, server-side:

const existing = (await whissle.agents.list()).find((a) => a.name === def.name);
const agent = existing ?? await whissle.agents.create({
  name:         def.name,
  agentType:    def.type,
  systemPrompt: def.interview ? interviewPrompt(def) : def.prompt,
  greeting:     def.greeting,
  ...(def.interview ? { scoring_prompt: interviewRubric(def) } : {}),
});

await whissle.embed.enable(agent.id, { origins: ["http://localhost:4000"] });
await whissle.kb.addSnippet(agent.id, def.knowledge, `${def.name} — reference`);

Three things that surprise people

• Creating an agent does not apply its type's default prompt. Whatever you send is its brain. To keep a blueprint's behaviour, read whissle.agentTypes() and compose your section onto its default_prompt.
• Embedding must be enabled or the mint refuses, and enabling it needs an origin list even though a secret-key mint ignores origins.
• Adoption matches on name. An agent your workspace already has under the same name is reused — and its past calls will show up in this app's Sessions view.

The browser makes exactly one request before a conversation, and it is to your server:

// server — behind YOUR auth
const session = await whissle.embed.sessionToken(agentId, {
  metadata: { user, agent: def.id },   // yours; lands on the call record
});
return json(res, 200, session);        // the WHOLE descriptor

// browser — no key, ever
const agent = new WhissleAgent({
  getToken: () => fetch("/api/sessions", {
    method: "POST", body: JSON.stringify({ agent: id }),
  }).then((r) => r.json()),
  ...(withAvatar ? { avatar: { id: "F2-TL", container: faceEl } } : {}),
});
await agent.start();

No backend at all? A publishable wpk_ key can go in the page and the SDK mints for itself — origin-bound and single-use. Never a wsk_: the SDK throws if you try, because the mint would otherwise accept it and the mistake would be silent.

New in 0.5.0. An embedded agent used to be strictly worse than the one on whissle.ai, and three of the reasons were silent by construction — no error, no log, nothing in the console. The events below are what closed that gap.

agent.on("tool-started",  (t) => showSpinner(t.name));
agent.on("tool-finished", (t) => showResult(t.name, t.ok, t.citations));
agent.on("thinking",      (on) => setPaused(on));
agent.on("agent-partial", (t) => renderLive(t));       // as it speaks
agent.on("error",         (e) => explain(e.code));     // 402 vs 403 vs 404

await agent.sendText("can you repeat the last question?");  // type instead of talk

Changed in 0.3.0. calls means “rows from the calls table” and structurally cannot see a text thread. If your product has any text in it, read sessions instead — it is the union of voice calls and text threads, on the same calls:read scope.

const list   = await whissle.sessions.list({ limit: 50 });   // voice + text
const detail = await whissle.sessions.get(id);               // transcript, disposition
const trace  = await whissle.sessions.trace(id);             // which model answered
const result = await whissle.calls.result(id);               // the grade

for await (const s of whissle.sessions.iterate()) { … }       // auto-paging

The same records from a terminal, which is the fastest way to debug a session:

whissle sessions list --limit 10     # voice AND text
whissle sessions trace <id>          # which provider answered, and failovers
whissle calls transcript <call-id>
whissle calls result <call-id> --wait

Nothing about trades or interviewing is in the code — it is all in the JSON. These are the four places real change happens, roughly in the order you will want them.

1. Replace agents.json. An app for accountants, triage nurses or a support line is this code with different entries.
2. Rewrite the prompt builders. Two small functions in server.mjs decide how strict an examiner is and what its rubric rewards — the highest-leverage twenty lines in the example.
3. Swap userFrom() for real authentication. It only has to answer one question before a token is minted: who is this, and may they talk to this agent?
4. Put the id map in your database, so a restart re-provisions nothing.

It hangs on “connecting”, with no error anywhere

Almost always ICE, and the silence is the diagnosis: a connection with no viable candidates never completes and never fails. Use the ice_servers the mint returns. If you find yourself adding an ICE constant, that is this bug arriving.

The agent goes silent mid-call

It is probably running a tool. On @whissle/agents 0.5.0 or later, listen for tool-started and thinking and say so in your UI. Before 0.5.0 there was no signal to hang that on.

The avatar does not move, or barely moves

Check the page is on @whissle/agents 0.3.1 or later, and confirm the container element exists before start().

The microphone is refused

localhost counts as a secure context. Serve it on a LAN IP to test from a phone and the browser will refuse the microphone until it is HTTPS. Since 0.5.0 this is checked before connecting, with the fix in the message — previously the session came up and ignored the visitor for the whole call.

402, 403 and 404

The workspace is out of credit (whissle usage); the key lacks a scope, which are fixed at creation; or the id is wrong. Since 0.5.0 the browser error event carries a code, and since sdk 0.3.0 the server throws a distinct error class for each.

The Sessions list shows calls you did not make

Adoption matched an agent that already existed under the same name, and it brought its history. Rename the entry in agents.json to get a separate agent.

CLI, by what you are trying to do

Every command takes --json for scripting.