Compare commits

..
Author SHA1 Message Date
Peter Steinberger 382681178b fix: enforce reasoning tags across providers (#801) (thanks @mcinteerj) 2026-01-12 22:58:44 +00:00
Jake cd169aceb5 fix(agent): restore reasoning tag enforcement for non-ollama providers
This restores the fix from PR #754 which was accidentally reverted in bf11a42c3.
2026-01-12 22:49:18 +00:00
Peter Steinberger 6ed2d69ff9 docs: clarify telegram allowFrom 2026-01-12 22:45:39 +00:00
Peter Steinberger e7e544174f docs: add region guidance for hosted minimax 2026-01-12 22:45:00 +00:00
Peter Steinberger 9b0d9db3a3 docs: detail memory tools and local models 2026-01-12 22:35:19 +00:00
Peter Steinberger fd768334a9 refactor: fast-lane directives helpers 2026-01-12 22:34:13 +00:00
Peter Steinberger 9f90d0721a docs: note relative plugin install paths 2026-01-12 22:34:13 +00:00
Peter Steinberger 1920138122 docs: add voice-call plugin changelog 2026-01-12 22:34:13 +00:00
Peter Steinberger 27d940f5b6 refactor: reuse streaming text normalizer across callbacks 2026-01-12 22:27:56 +00:00
Peter Steinberger 7ba72aeb6c fix: make pw download tests platform-safe 2026-01-12 22:27:19 +00:00
Peter Steinberger f19d37c7bb build: harden installer smoke apt 2026-01-12 22:21:50 +00:00
Peter Steinberger 9d5bf38416 style(telegram): format bot.ts 2026-01-12 22:19:02 +00:00
Peter Steinberger e0c1f2fdc0 test(agents): avoid Copilot token fetch in image-tool tests 2026-01-12 22:19:02 +00:00
Peter Steinberger 08fdac0561 fix(telegram): guard setMyCommands in native commands 2026-01-12 22:19:02 +00:00
Peter Steinberger d3eeddfc2f chore: fix lint after rebase 2026-01-12 22:19:02 +00:00
Peter Steinberger 0a24bc0427 docs(changelog): note scrollintoview 2026-01-12 22:19:02 +00:00
Peter Steinberger 9da97d1a41 test(browser): expand scrollintoview coverage 2026-01-12 22:19:02 +00:00
Peter Steinberger 29b7b2068a refactor: centralize streaming text normalization 2026-01-12 22:17:14 +00:00
Peter Steinberger f4ab057807 fix: start typing on partial deltas 2026-01-12 22:16:29 +00:00
Peter Steinberger cb35db0c7e docs: note to echo docs links after edits 2026-01-12 22:13:12 +00:00
Peter Steinberger f13db1c836 test: guard telegram native commands when mock lacks .command 2026-01-12 22:12:16 +00:00
Peter Steinberger 59063a7c15 test: skip setMyCommands when API mock lacks it 2026-01-12 22:08:53 +00:00
Peter Steinberger 5bc4971432 chore: fix lint warnings 2026-01-12 22:07:39 +00:00
Peter Steinberger 59c8d2d17f docs: clarify sandbox bind mounts (#790) 2026-01-12 22:06:35 +00:00
Peter Steinberger 21405b0dfc fix(discord): rebalance reasoning italics 2026-01-12 22:01:48 +00:00
Peter Steinberger 256304037e fix: keep Claude file_path aliases validated 2026-01-12 22:00:08 +00:00
Peter Steinberger d8ae905d54 Merge pull request #790 from akonyer/feature/custom-sandbox-binds
Custom sandbox binds for docker sandbox
2026-01-12 21:58:31 +00:00
Peter Steinberger bbc34215a2 fix: land sandbox binds (#790) (thanks @akonyer) 2026-01-12 21:58:16 +00:00
Aaron Konyer 583fc4fb11 test(sandbox): add coverage for binds -v flag emission 2026-01-12 21:57:51 +00:00
Aaron Konyer 0b2b8c7c52 Add docker bind mounds for sandboxing 2026-01-12 21:57:51 +00:00
Peter Steinberger 5d83be76c9 test: cover mixed directive fast-lane 2026-01-12 21:57:10 +00:00
Peter Steinberger 877bc61b53 docs(browser): document scrollintoview 2026-01-12 21:56:27 +00:00
Peter Steinberger fcaeee7073 test(browser): cover scrollintoview 2026-01-12 21:56:27 +00:00
Peter Steinberger 6857f16609 feat(browser): add scrollintoview action 2026-01-12 21:56:27 +00:00
Peter Steinberger 2faf7cea93 feat(sandbox): add tool-policy groups 2026-01-12 21:51:49 +00:00
Peter Steinberger 26d5cca97c feat: auto native commands defaults 2026-01-12 21:49:44 +00:00
Peter Steinberger 99fea64823 fix: fast-lane directives bypass queue dedupe 2026-01-12 21:44:19 +00:00
Peter Steinberger 42c17adb5e feat: restore voice-call plugin parity 2026-01-12 21:44:19 +00:00
Zach KnickerbockerandShadow 3467b0ba07 Discord: add allowBots config option (#802)
Co-authored-by: Shadow <shadow@clawd.bot>
2026-01-12 15:30:05 -06:00
Peter Steinberger 490cb834e5 style: italicize reasoning output 2026-01-12 21:24:36 +00:00
Peter Steinberger cd12ad8aab fix(image): accept @-prefixed file paths 2026-01-12 20:53:16 +00:00
Peter Steinberger 6f496b7739 fix(macos): treat tests as preview 2026-01-12 20:38:34 +00:00
Peter Steinberger e961e02f71 fix(gateway): quiet loopback ws closes 2026-01-12 20:38:34 +00:00
Peter Steinberger 36a02b3e67 fix(image): route MiniMax vision to VLM 2026-01-12 20:38:34 +00:00
David GuttmanandShadow b73042500e Discord: per-channel autoThread (#800)
Co-authored-by: Shadow <shadow@clawd.bot>
2026-01-12 14:33:07 -06:00
Peter Steinberger 6406ed869a docs(browser): document downloads + responsebody 2026-01-12 19:41:12 +00:00
Peter Steinberger f839d949b2 test(browser): cover downloads + responsebody 2026-01-12 19:41:12 +00:00
Peter Steinberger d4f7dc067e feat(browser): add downloads + response bodies 2026-01-12 19:41:12 +00:00
Peter Steinberger 3dbfe65eea Merge pull request #791 from roshanasingh4/fix/788-implicit-delivery-leak
Prevent onboarding TUI from auto-delivering to lastProvider/lastTo
2026-01-12 19:39:05 +00:00
Peter Steinberger ddd4b55cf6 fix: prevent onboarding TUI auto-delivery (#791) (thanks @roshanasingh4) 2026-01-12 19:38:52 +00:00
Roshan Singh 298c6eea1f Fix: prevent onboarding TUI auto-delivery 2026-01-12 19:31:04 +00:00
Peter Steinberger 74806aa5e3 docs: add dashboard faq 2026-01-12 19:14:08 +00:00
Peter Steinberger 55aeb8a0d3 fix(image): drop temperature for OpenAI 2026-01-12 19:09:15 +00:00
Peter Steinberger 86ea00dc21 fix(tools): accept legacy bash tool calls 2026-01-12 19:09:15 +00:00
Peter Steinberger a0a7e74a62 fix(models): preserve implicit vision models 2026-01-12 19:09:15 +00:00
Peter Steinberger bb7397c636 feat: add dashboard command 2026-01-12 19:08:29 +00:00
Peter Steinberger 7dc44b04c1 fix: close memory index and refresh protocol outputs 2026-01-12 18:49:24 +00:00
Peter Steinberger 45232137a2 fix(logging): honor silent console level 2026-01-12 18:46:40 +00:00
Peter Steinberger b1c3e38df0 refactor(models): share implicit providers 2026-01-12 18:46:40 +00:00
Peter Steinberger 0be62c3542 fix(image): fail over on empty output 2026-01-12 18:46:16 +00:00
Peter Steinberger 523f91758d test(browser): extend automation coverage 2026-01-12 18:42:46 +00:00
Peter Steinberger 5baba5f84e docs(browser): expand automation docs 2026-01-12 18:42:46 +00:00
Peter Steinberger ffbcd83d1e chore: log elevated and reasoning toggles 2026-01-12 18:37:44 +00:00
Peter Steinberger 1baf9f6a83 fix(image): normalize mime type handling 2026-01-12 18:24:11 +00:00
Peter Steinberger 77b20377cc fix: stabilize session tools and tests 2026-01-12 18:21:20 +00:00
Peter Steinberger 3ffb9a3b5e fix: keep session sanitizer stable 2026-01-12 18:19:30 +00:00
Peter Steinberger 29807119d5 chore: format tool guard files 2026-01-12 18:19:30 +00:00
Peter Steinberger b88ea39b83 fix: add subagent default model typing 2026-01-12 18:18:15 +00:00
Peter Steinberger 0a2dcd844b fix(image): support data URLs 2026-01-12 18:17:43 +00:00
Peter Steinberger 2ed95634fe fix: relax image tool agentDir guard 2026-01-12 18:12:51 +00:00
Peter Steinberger 9526f9861a chore: mark 2026.1.12 unreleased 2026-01-12 18:09:35 +00:00
Peter Steinberger 7b93356fb7 feat: subagent model defaults 2026-01-12 18:08:30 +00:00
Peter Steinberger 17ff25bd20 fix(sandbox): always allow image tool 2026-01-12 18:07:34 +00:00
Peter Steinberger d24de1ec3b feat(sandbox): allow image tool 2026-01-12 17:56:51 +00:00
Peter Steinberger 44e1f271c8 fix: keep image sanitizer scoped 2026-01-12 17:55:45 +00:00
Peter Steinberger 8ff09f8337 feat(image): auto-pair image model 2026-01-12 17:50:47 +00:00
Peter Steinberger e91aa0657e fix: add copilot tests and lint fixes 2026-01-12 17:48:08 +00:00
Ayaan Zaidi 14801b46fc Merge pull request #792 from clawdbot/fix/gateway-auth-off
fix: clear gateway auth when set to off
2026-01-12 23:10:15 +05:30
Peter Steinberger 99a7548c07 docs(browser): update browser tool docs 2026-01-12 17:32:44 +00:00
Peter Steinberger 35bbc2ba87 feat(cli): expand browser commands 2026-01-12 17:32:44 +00:00
Peter Steinberger cf78d28d74 test(browser): add regression coverage 2026-01-12 17:32:44 +00:00
Peter Steinberger eeca541dde feat(browser): expand browser control surface 2026-01-12 17:32:44 +00:00
Ayaan Zaidi 4bba49770d fix: clear gateway auth on off selection 2026-01-12 22:59:37 +05:30
Peter Steinberger f5d5661adf fix: guard session tool results 2026-01-12 17:28:46 +00:00
Peter Steinberger f83fb70360 fix: keep main agent in list output 2026-01-12 17:24:03 +00:00
Peter Steinberger 355c13564c fix: restore heartbeat defaults and model listing 2026-01-12 17:17:54 +00:00
Peter Steinberger f1dd59bf82 test: update heartbeat and agent list thresholds 2026-01-12 17:14:04 +00:00
Peter Steinberger fd1e959c2d fix: clean up models-config provider normalization 2026-01-12 17:14:04 +00:00
Peter Steinberger 1b2c1545a0 docs: harden local model guidance 2026-01-12 17:11:04 +00:00
Peter Steinberger 05ac67c520 refactor: split models-config provider helpers 2026-01-12 17:05:53 +00:00
Peter Steinberger f5ee2b3a4f Merge pull request #705 from TAGOOZ/feat/github-copilot-onboard
feat: add GitHub Copilot provider
2026-01-12 16:56:51 +00:00
Peter Steinberger 8afdf75e2b fix: honor copilot config and profiles (#705) (thanks @TAGOOZ) 2026-01-12 16:55:47 +00:00
Peter Steinberger 6aed3c0fd3 docs: add changelog entry for minimax fix 2026-01-12 16:53:53 +00:00
Peter Steinberger f0fbd8b012 docs: note minimax unknown-model fix 2026-01-12 16:52:49 +00:00
Peter Steinberger 5a3eb5ad62 test: cover models.json apiKey fill 2026-01-12 16:52:43 +00:00
Peter Steinberger 79beb20ba2 fix: make models.json generation fill apiKey 2026-01-12 16:52:43 +00:00
Mustafa Tag Eldeen 3da1afed68 feat: add GitHub Copilot provider
Copilot device login + onboarding option; model list auth detection.
2026-01-12 16:52:15 +00:00
Peter Steinberger 717a259056 docs: add local models guide 2026-01-12 16:50:37 +00:00
Ayaan Zaidi adaa30c73a test(telegram): cover General topic typing fallback 2026-01-12 22:03:42 +05:30
Azade ff292e67ce fix(telegram): show typing indicator in General forum topic
In forum supergroups, messages from the General topic arrive without
message_thread_id in updates, but sendChatAction requires one to display
the typing indicator in the correct topic.

Use message_thread_id=1 (Telegram's internal ID for General topic) as
fallback when messageThreadId is undefined in a forum chat.
2026-01-12 22:01:21 +05:30
Peter Steinberger 31752aa944 chore: remove qmd skill 2026-01-12 11:31:34 +00:00
Peter Steinberger bf11a42c37 feat: add memory vector search 2026-01-12 11:23:44 +00:00
Peter Steinberger 8049f33435 chore: sanitize onboarding api keys 2026-01-12 11:18:31 +00:00
Peter Steinberger 115591c5b6 feat: add cron agent binding 2026-01-12 11:07:38 +00:00
Peter Steinberger a3938d62f6 chore: raise heartbeat ack window 2026-01-12 11:06:46 +00:00
Peter Steinberger 3c7a8579ad test: cover minimax env provider injection 2026-01-12 11:05:08 +00:00
Peter Steinberger f5a9421b10 fix: auto-add minimax provider from auth 2026-01-12 11:05:08 +00:00
Peter Steinberger 562d0e3b5f fix: avoid duplicate status replies 2026-01-12 11:04:03 +00:00
Peter Steinberger bf7e813573 chore: release 2026.1.11-4 2026-01-12 10:52:34 +00:00
Peter Steinberger c69abe08eb chore: update appcast for 2026.1.11-3 2026-01-12 10:40:31 +00:00
Peter Steinberger 5a29ec78ca chore: release 2026.1.11-3 2026-01-12 10:35:50 +00:00
Peter Steinberger 42b43f8c58 chore: update appcast for 2026.1.11-2 2026-01-12 10:26:01 +00:00
Peter Steinberger c1f8f1d9d0 chore: release 2026.1.11-2 2026-01-12 10:14:24 +00:00
Peter Steinberger 35f8be33d2 docs: remove git source install snippet 2026-01-12 10:00:26 +00:00
194 changed files with 19159 additions and 1926 deletions

No files matched your search

+1
View File
@@ -11,6 +11,7 @@
- Internal doc links in `docs/**/*.md`: root-relative, no `.md`/`.mdx` (example: `[Config](/configuration)`).
- Section cross-references: use anchors on root-relative paths (example: `[Hooks](/configuration#hooks)`).
- When Peter asks for links, reply with full `https://docs.clawd.bot/...` URLs (not root-relative).
- When you touch docs, end the reply with the `https://docs.clawd.bot/...` URLs you referenced.
- README (GitHub): keep absolute docs URLs (`https://docs.clawd.bot/...`) so links work on GitHub.
- Docs content must be generic: no personal device names/hostnames/paths; use placeholders like `user@gateway-host` and “gateway host”.
+47 -25
View File
@@ -1,9 +1,53 @@
# Changelog
## 2026.1.11-1
## 2026.1.12-2
### Changes
- Subagents: add config to set default sub-agent model (`agents.defaults.subagents.model` + per-agent override); still overridden by `sessions_spawn.model`.
- Plugins: restore full voice-call plugin parity (Telnyx/Twilio, streaming, inbound policies, tools/CLI).
- Sandbox: support tool-policy groups in `tools.sandbox.tools` (e.g. `group:memory`, `group:fs`) to reduce config churn.
### Fixes
- Installer: include `patches/` in the npm package so postinstall patching works for npm/bun installs.
- Tools/Models: MiniMax vision now uses the Coding Plan VLM endpoint (`/v1/coding_plan/vlm`) so the `image` tool works with MiniMax keys (also accepts `@/path/to/file.png`-style inputs).
- Gateway/macOS: reduce noisy loopback WS “closed before connect” logs during tests.
## 2026.1.12-1
### Changes
- Heartbeat: raise default `ackMaxChars` to 300 so any `HEARTBEAT_OK` replies with short padding stay internal (fewer noisy heartbeat posts on providers).
- Onboarding: normalize API key inputs (strip `export KEY=...` wrappers) so shell-style entries paste cleanly.
## 2026.1.12 (Unreleased)
### Highlights
- Memory: add vector search for agent memories (Markdown-only scope) with SQLite index, chunking, lazy sync + file watch, and per-agent enablement/fallback.
### Changes
- Browser: add `scrollintoview` action to scroll refs into view before click/type.
- Memory: embedding providers support OpenAI or local `node-llama-cpp`; config adds defaults + per-agent overrides, provider/fallback metadata surfaced in tools/CLI.
- CLI/Tools: new `clawdbot memory` commands plus `memory_search`/`memory_get` tools returning snippets + line ranges and provider info.
- Runtime: memory index stored under `~/.clawdbot/memory/{agentId}.sqlite` with watch-on-by-default; inline status replies now stay auth-gated while inline prompts continue to the agent.
- Discord: add `discord.allowBots` to permit bot-authored messages (still ignores its own messages) with docs warning about bot loops. (#802) — thanks @zknicker.
- CLI/Onboarding: `clawdbot dashboard` prints/copies the tokenized Control UI link and opens it; onboarding now auto-opens the dashboard with your token and keeps the link in the summary.
- Commands: native slash commands now default to `"auto"` (on for Discord/Telegram, off for Slack) with per-provider overrides (`discord/telegram/slack.commands.native`) and docs updated.
- Sandbox: allow Docker bind mounts via `docker.binds`; merges global + per-agent binds (per-agent ignored under shared scope) for custom host paths. (#790 — thanks @akonyer)
### Fixes
- Auto-reply: inline `/status` now honors allowlists (authorized stripped + replied inline; unauthorized leaves text for the agent) to match command gating tests.
- Auto-reply: enforce `<final>` tag for all reasoning-tag providers (Gemini Antigravity, MiniMax, etc.), not just Ollama. (#801 — thanks @mcinteerj)
- Models: normalize `${ENV_VAR}` apiKey config values and auto-fill missing provider `apiKey` from env/auth when custom provider models are configured (fixes MiniMax “Unknown model” on fresh installs).
- Models/Tools: include `MiniMax-VL-01` in implicit MiniMax provider so image pairing uses a real vision model.
- Telegram: show typing indicator in General forum topics. (#779) — thanks @azade-c.
- Discord: keep reasoning italics intact when messages are chunked, so reasoning stays italic across multi-part sends.
- Models: keep explicit GitHub Copilot provider config and honor agent-dir auth profiles for auto-injection. (#705) — thanks @TAGOOZ.
- Auto-reply: restore 300-char heartbeat ack limit and keep >300 char replies instead of dropping them; adjust long heartbeat test content accordingly.
- Gateway: `agents.list` now honors explicit `agents.list` config without pulling stray agents from disk; GitHub Copilot CLI auth path uses the updated provider build.
- Google: apply patched pi-ai `google-gemini-cli` function call handling (strips ids) after upgrading to pi-ai 0.43.0.
- Tools: allow Gemini/Claude-style aliases (`file_path`, `old_string`, `new_string`) in read/write/edit schemas while still enforcing required params at runtime to avoid validation loops.
- Auto-reply: elevated/reasoning toggles now enqueue system events so the model sees the mode change immediately.
- Tools: keep `image` available in sandbox and fail over when image models return empty output (fixes “(no text returned)”).
- Discord: add per-channel `autoThread` to auto-create threads for top-level messages. (#800) — thanks @davidguttman.
- Onboarding: TUI defaults to `deliver: false` to avoid cross-provider auto-delivery leaks; onboarding spawns the TUI with explicit `deliver: false`. (#791 — thanks @roshanasingh4)
## 2026.1.11
@@ -14,9 +58,6 @@
- Agents: automatic pre-compaction memory flush turn to store durable memories before compaction.
### Changes
- Deps: update pi-agent-core/pi-ai/pi-coding-agent/pi-tui and refresh the pi-ai patch.
- Dev: bump @types/node.
- macOS: add wizard debug CLI and share wizard parsing helpers.
- CLI/Onboarding: simplify MiniMax auth choice to a single M2.1 option.
- CLI: configure section selection now loops until Continue.
- Docs: explain MiniMax vs MiniMax Lightning (speed vs cost) and restore LM Studio example.
@@ -24,20 +65,16 @@
- Onboarding/CLI: group model/auth choice by provider and label Z.AI as GLM 4.7.
- Onboarding/Docs: add Moonshot AI (Kimi K2) auth choice + config example.
- CLI/Onboarding: prompt to reuse detected API keys for Moonshot/MiniMax/Z.AI/Gemini/Anthropic/OpenCode.
- CLI/Onboarding: move MiniMax to the top of the provider list.
- CLI/Onboarding: add MiniMax M2.1 Lightning auth choice.
- CLI/Onboarding: show key previews when reusing detected API keys.
- Auto-reply: add compact `/model` picker (models + available providers) and show provider endpoints in `/model status`.
- Control UI: add Config tab model presets (MiniMax M2.1, GLM 4.7, Kimi) for one-click setup.
- Plugins: add extension loader (tools/RPC/CLI/services), discovery paths, and config schema + Control UI labels (uiHints).
- Plugins: add `clawdbot plugins install` (path/tgz/npm), plus `list|info|enable|disable|doctor` UX.
- Plugins: voice-call plugin now real (Twilio/log), adds start/status RPC/CLI/tool + tests.
- Docs: add plugins doc + cross-links from tools/skills/gateway config.
- Docs: clarify memory flush behavior + writable workspace requirement in Memory/Session/FAQ.
- Docs: add beginner-friendly plugin quick start + expand Voice Call plugin docs.
- Tests: add Docker plugin loader + tgz-install smoke test.
- Tests: extend Docker plugin E2E to cover installing from local folders (`plugins.load.paths`) and `file:` npm specs.
- Tests: add coverage for pre-compaction memory flush settings (including read-only/CLI skips).
- Tests: add coverage for pre-compaction memory flush settings.
- Tests: modernize live model smoke selection for current releases and enforce tools/images/thinking-high coverage. (#769) — thanks @steipete.
- Agents/Tools: add `apply_patch` tool for multi-file edits (experimental; gated by tools.exec.applyPatch; OpenAI-only).
- Agents/Tools: rename the bash tool to exec (config alias maintained). (#748) — thanks @myfunc.
@@ -64,17 +101,9 @@
- Installer UX: add `--install-method git|npm` and auto-detect source checkouts (prompt to update git checkout vs migrate to npm).
### Fixes
- Control UI: flatten nav into a single horizontal scroll row on tablet/mobile (and always show collapsed group items). (#771) — thanks @carlulsoe.
- macOS: start + await local gateway before onboarding wizard begins.
- macOS: cancel onboarding wizard on close, recover if the gateway drops the session, and time out stalled gateway connects.
- macOS: wizard debug CLI now surfaces error status instead of exiting as complete.
- Models/Onboarding: configure MiniMax (minimax.io) via Anthropic-compatible `/anthropic` endpoint by default (keep `minimax-api` as a legacy alias).
- Agents/Browser: cap Playwright AI snapshots for tool calls (maxChars); CLI snapshots remain full. (#763) — thanks @thesash.
- Models: normalize Gemini 3 Pro/Flash IDs to preview names for live model lookups. (#769) — thanks @steipete.
- CLI: fix guardCancel typing for configure prompts. (#769) — thanks @steipete.
- Providers: default groupPolicy to allowlist across providers and warn in doctor when groups are open.
- MS Teams: add groupPolicy/groupAllowFrom gating for group chats and warn when groups are open.
- Providers: strip tool call/result ids from Gemini CLI payloads to avoid API 400s. (#756)
- Gateway/WebChat: include handshake validation details in the WebSocket close reason for easier debugging; preserve close codes.
- Gateway/Auth: send invalid connect responses before closing the handshake; stabilize invalid-connect auth test.
- Gateway: tighten gateway listener detection.
@@ -86,23 +115,16 @@
- Config: expand `~` in `CLAWDBOT_CONFIG_PATH` and common path-like config fields (including `plugins.load.paths`); guard invalid `$include` paths. (#731) — thanks @pasogott.
- Agents: stop pre-creating session transcripts so first user messages persist in JSONL history.
- Agents: skip pre-compaction memory flush when the session workspace is read-only.
- Auto-reply: allow inline `/status` for allowlisted senders (stripped before the model); unauthorized senders see it as plain text.
- Auto-reply: include config-only allowlisted models in `/model` even when the catalog is partial.
- Auto-reply: allow fuzzy `/model` matches (e.g. `/model kimi` or `/model moonshot/kimi`) when unambiguous.
- Auto-reply: ignore inline `/status` directives unless the message is directive-only.
- CLI/Configure: enter the selected section immediately, then return to the section picker.
- CLI/Configure: apply the chosen auth model as default (skip the extra picker) and refresh the model catalog for new providers.
- Auto-reply: align `/think` default display with model reasoning defaults. (#751) — thanks @gabriel-trigo.
- Auto-reply: flush block reply buffers on tool boundaries. (#750) — thanks @sebslight.
- Auto-reply: allow sender fallback for command authorization when `SenderId` is empty (WhatsApp self-chat). (#755) — thanks @juanpablodlc.
- Heartbeat: refresh prompt text for updated defaults.
- Agents/Tools: use PowerShell on Windows to capture system utility output. (#748) — thanks @myfunc.
- Agents/Tools: normalize Claude Code-style read/write/edit params (file_path/old_string/new_string) and keep sandbox guards in place. (#768) — thanks @hsrvc.
- Docker: tolerate unset optional env vars in docker-setup.sh under strict mode. (#725) — thanks @petradonka.
- CLI/Update: preserve base environment when passing overrides to update subprocesses. (#713) — thanks @danielz1z.
- Agents: treat message tool errors as failures so fallback replies still send; require `to` + `message` for `action=send`. (#717) — thanks @theglove44.
- Agents: preserve reasoning items on tool-only turns.
- Agents: enforce `<final>` gating for reasoning-tag providers to prevent tag/reasoning leaks. (#754) — thanks @mcinteerj.
- Agents/Subagents: wait for completion before announcing, align wait timeout with run timeout, and make announce prompts more emphatic.
- Agents: route subagent transcripts to the target agent sessions directory and add regression coverage. (#708) — thanks @xMikeMickelson.
- Agents/Tools: preserve action enums when flattening tool schemas. (#708) — thanks @xMikeMickelson.
+34 -223
View File
@@ -2,6 +2,40 @@
<rss xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle" version="2.0">
<channel>
<title>Clawdbot</title>
<item>
<title>2026.1.11-3</title>
<pubDate>Mon, 12 Jan 2026 10:40:23 +0000</pubDate>
<link>https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml</link>
<sparkle:version>5212</sparkle:version>
<sparkle:shortVersionString>2026.1.11-3</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>15.0</sparkle:minimumSystemVersion>
<description><![CDATA[<h2>Clawdbot 2026.1.11-3</h2>
<h3>Fixes</h3>
<ul>
<li>CLI: avoid top-level await warnings in the entrypoint on fresh installs.</li>
<li>CLI: show a commit hash in the banner for npm installs (package.json gitHead fallback).</li>
</ul>
<p><a href="https://github.com/clawdbot/clawdbot/blob/main/CHANGELOG.md">View full changelog</a></p>
]]></description>
<enclosure url="https://github.com/clawdbot/clawdbot/releases/download/v2026.1.11-3/Clawdbot-2026.1.11-3.zip" length="19860758" type="application/octet-stream" sparkle:edSignature="LbvGUSjc3jGO7aVo2UVA0nEkaJbb3O4iwRBo1TBqoapdTtxnDlS3s6N+Z4vOSLRAoAm22EoZOwbpK9085c7HAQ=="/>
</item>
<item>
<title>2026.1.11-2</title>
<pubDate>Mon, 12 Jan 2026 10:25:53 +0000</pubDate>
<link>https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml</link>
<sparkle:version>5210</sparkle:version>
<sparkle:shortVersionString>2026.1.11-2</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>15.0</sparkle:minimumSystemVersion>
<description><![CDATA[<h2>Clawdbot 2026.1.11-2</h2>
<h3>Fixes</h3>
<ul>
<li>Installer: ensure the CLI entrypoint is executable after npm installs.</li>
<li>Packaging: include <code>dist/plugins/</code> in the npm package to avoid missing module errors.</li>
</ul>
<p><a href="https://github.com/clawdbot/clawdbot/blob/main/CHANGELOG.md">View full changelog</a></p>
]]></description>
<enclosure url="https://github.com/clawdbot/clawdbot/releases/download/v2026.1.11-2/Clawdbot-2026.1.11-2.zip" length="19860732" type="application/octet-stream" sparkle:edSignature="0UG+d9v3Qf5F9vs/KozUB404WpHjFBQRVoRuhwtzF8kpU7jJmmGlQzh1c61E+LMN4fHcljpxIwHHrvvIfRyrCw=="/>
</item>
<item>
<title>2026.1.11-1</title>
<pubDate>Mon, 12 Jan 2026 09:53:46 +0000</pubDate>
@@ -18,228 +52,5 @@
]]></description>
<enclosure url="https://github.com/clawdbot/clawdbot/releases/download/v2026.1.11-1/Clawdbot-2026.1.11-1.zip" length="19860761" type="application/octet-stream" sparkle:edSignature="CXKzzha/s6cGBeF0TMz+cV8/pfqoAL9ZyNVacYRLnnHEwA1cMbOWRftpGRhYe4HknVQYYBgNQqZK2lBxpOZgBg=="/>
</item>
<item>
<title>2026.1.11</title>
<pubDate>Mon, 12 Jan 2026 09:37:49 +0000</pubDate>
<link>https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml</link>
<sparkle:version>5205</sparkle:version>
<sparkle:shortVersionString>2026.1.11</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>15.0</sparkle:minimumSystemVersion>
<description><![CDATA[<h2>Clawdbot 2026.1.11</h2>
<h3>Highlights</h3>
<ul>
<li>Plugins are now first-class: loader + CLI management, plus the new Voice Call plugin.</li>
<li>Config: modular <code>$include</code> support for split config files. (#731) — thanks @pasogott.</li>
<li>Agents/Pi: reserve compaction headroom so pre-compaction memory writes can run before auto-compaction.</li>
<li>Agents: automatic pre-compaction memory flush turn to store durable memories before compaction.</li>
</ul>
<h3>Changes</h3>
<ul>
<li>Deps: update pi-agent-core/pi-ai/pi-coding-agent/pi-tui and refresh the pi-ai patch.</li>
<li>Dev: bump @types/node.</li>
<li>macOS: add wizard debug CLI and share wizard parsing helpers.</li>
<li>CLI/Onboarding: simplify MiniMax auth choice to a single M2.1 option.</li>
<li>CLI: configure section selection now loops until Continue.</li>
<li>Docs: explain MiniMax vs MiniMax Lightning (speed vs cost) and restore LM Studio example.</li>
<li>Docs: add Cerebras GLM 4.6/4.7 config example (OpenAI-compatible endpoint).</li>
<li>Onboarding/CLI: group model/auth choice by provider and label Z.AI as GLM 4.7.</li>
<li>Onboarding/Docs: add Moonshot AI (Kimi K2) auth choice + config example.</li>
<li>CLI/Onboarding: prompt to reuse detected API keys for Moonshot/MiniMax/Z.AI/Gemini/Anthropic/OpenCode.</li>
<li>CLI/Onboarding: move MiniMax to the top of the provider list.</li>
<li>CLI/Onboarding: add MiniMax M2.1 Lightning auth choice.</li>
<li>CLI/Onboarding: show key previews when reusing detected API keys.</li>
<li>Auto-reply: add compact <code>/model</code> picker (models + available providers) and show provider endpoints in <code>/model status</code>.</li>
<li>Control UI: add Config tab model presets (MiniMax M2.1, GLM 4.7, Kimi) for one-click setup.</li>
<li>Plugins: add extension loader (tools/RPC/CLI/services), discovery paths, and config schema + Control UI labels (uiHints).</li>
<li>Plugins: add <code>clawdbot plugins install</code> (path/tgz/npm), plus <code>list|info|enable|disable|doctor</code> UX.</li>
<li>Plugins: voice-call plugin now real (Twilio/log), adds start/status RPC/CLI/tool + tests.</li>
<li>Docs: add plugins doc + cross-links from tools/skills/gateway config.</li>
<li>Docs: clarify memory flush behavior + writable workspace requirement in Memory/Session/FAQ.</li>
<li>Docs: add beginner-friendly plugin quick start + expand Voice Call plugin docs.</li>
<li>Tests: add Docker plugin loader + tgz-install smoke test.</li>
<li>Tests: extend Docker plugin E2E to cover installing from local folders (<code>plugins.load.paths</code>) and <code>file:</code> npm specs.</li>
<li>Tests: add coverage for pre-compaction memory flush settings (including read-only/CLI skips).</li>
<li>Tests: modernize live model smoke selection for current releases and enforce tools/images/thinking-high coverage. (#769) — thanks @steipete.</li>
<li>Agents/Tools: add <code>apply_patch</code> tool for multi-file edits (experimental; gated by tools.exec.applyPatch; OpenAI-only).</li>
<li>Agents/Tools: rename the bash tool to exec (config alias maintained). (#748) — thanks @myfunc.</li>
<li>Agents: add pre-compaction memory flush config (<code>agents.defaults.compaction.*</code>) with a soft threshold + system prompt.</li>
<li>Config: add <code>$include</code> directive for modular config files. (#731) — thanks @pasogott.</li>
<li>Build: set pnpm minimum release age to 2880 minutes (2 days). (#718) — thanks @dan-dr.</li>
<li>macOS: prompt to install the global <code>clawdbot</code> CLI when missing in local mode; install via <code>clawd.bot/install-cli.sh</code> (no onboarding) and use external launchd/CLI instead of the embedded gateway runtime.</li>
<li>Docs: add gog calendar event color IDs from <code>gog calendar colors</code>. (#715) — thanks @mjrussell.</li>
<li>Cron/CLI: add <code>--model</code> flag to cron add/edit commands. (#711) — thanks @mjrussell.</li>
<li>Cron/CLI: trim model overrides on cron edits and document main-session guidance. (#711) — thanks @mjrussell.</li>
<li>Skills: bundle <code>skill-creator</code> to guide creating and packaging skills.</li>
<li>Providers: add per-DM history limit overrides (<code>dmHistoryLimit</code>) with provider-level config. (#728) — thanks @pkrmf.</li>
<li>Discord: expose channel/category management actions in the message tool. (#730) — thanks @NicholasSpisak.</li>
<li>Docs: rename README “macOS app” section to “Apps”. (#733) — thanks @AbhisekBasu1.</li>
<li>Gateway: require <code>client.id</code> in WebSocket connect params; use <code>client.instanceId</code> for presence de-dupe; update docs/tests.</li>
<li>macOS: remove the attach-only gateway setting; local mode now always manages launchd while still attaching to an existing gateway if present.</li>
</ul>
<h3>Installer</h3>
<ul>
<li>Postinstall: replace <code>git apply</code> with builtin JS patcher (works npm/pnpm/bun; no git dependency) plus regression tests.</li>
<li>Postinstall: skip pnpm patch fallback when the new patcher is active.</li>
<li>Installer tests: add root+non-root docker smokes, CI workflow to fetch clawd.bot scripts and run install sh/cli with onboarding skipped.</li>
<li>Installer UX: support <code>CLAWDBOT_NO_ONBOARD=1</code> for non-interactive installs; fix npm prefix on Linux and auto-install git.</li>
<li>Installer UX: add <code>install.sh --help</code> with flags/env and git install hint.</li>
<li>Installer UX: add <code>--install-method git|npm</code> and auto-detect source checkouts (prompt to update git checkout vs migrate to npm).</li>
</ul>
<h3>Fixes</h3>
<ul>
<li>Control UI: flatten nav into a single horizontal scroll row on tablet/mobile (and always show collapsed group items). (#771) — thanks @carlulsoe.</li>
<li>macOS: start + await local gateway before onboarding wizard begins.</li>
<li>macOS: cancel onboarding wizard on close, recover if the gateway drops the session, and time out stalled gateway connects.</li>
<li>macOS: wizard debug CLI now surfaces error status instead of exiting as complete.</li>
<li>Models/Onboarding: configure MiniMax (minimax.io) via Anthropic-compatible <code>/anthropic</code> endpoint by default (keep <code>minimax-api</code> as a legacy alias).</li>
<li>Agents/Browser: cap Playwright AI snapshots for tool calls (maxChars); CLI snapshots remain full. (#763) — thanks @thesash.</li>
<li>Models: normalize Gemini 3 Pro/Flash IDs to preview names for live model lookups. (#769) — thanks @steipete.</li>
<li>CLI: fix guardCancel typing for configure prompts. (#769) — thanks @steipete.</li>
<li>Providers: default groupPolicy to allowlist across providers and warn in doctor when groups are open.</li>
<li>MS Teams: add groupPolicy/groupAllowFrom gating for group chats and warn when groups are open.</li>
<li>Providers: strip tool call/result ids from Gemini CLI payloads to avoid API 400s. (#756)</li>
<li>Gateway/WebChat: include handshake validation details in the WebSocket close reason for easier debugging; preserve close codes.</li>
<li>Gateway/Auth: send invalid connect responses before closing the handshake; stabilize invalid-connect auth test.</li>
<li>Gateway: tighten gateway listener detection.</li>
<li>Control UI: hide onboarding chat when configured and guard the mobile chat sidebar overlay.</li>
<li>Auth: read Codex keychain credentials and make the lookup platform-aware.</li>
<li>macOS/Release: avoid bundling dist artifacts in relay builds and generate appcasts from zip-only sources.</li>
<li>Doctor: surface plugin diagnostics in the report.</li>
<li>Plugins: treat <code>plugins.load.paths</code> directory entries as package roots when they contain <code>package.json</code> + <code>clawdbot.extensions</code>; load plugin packages from config dirs; extract archives without system tar.</li>
<li>Config: expand <code>~</code> in <code>CLAWDBOT_CONFIG_PATH</code> and common path-like config fields (including <code>plugins.load.paths</code>); guard invalid <code>$include</code> paths. (#731) — thanks @pasogott.</li>
<li>Agents: stop pre-creating session transcripts so first user messages persist in JSONL history.</li>
<li>Agents: skip pre-compaction memory flush when the session workspace is read-only.</li>
<li>Auto-reply: allow inline <code>/status</code> for allowlisted senders (stripped before the model); unauthorized senders see it as plain text.</li>
<li>Auto-reply: include config-only allowlisted models in <code>/model</code> even when the catalog is partial.</li>
<li>Auto-reply: allow fuzzy <code>/model</code> matches (e.g. <code>/model kimi</code> or <code>/model moonshot/kimi</code>) when unambiguous.</li>
<li>Auto-reply: ignore inline <code>/status</code> directives unless the message is directive-only.</li>
<li>CLI/Configure: enter the selected section immediately, then return to the section picker.</li>
<li>CLI/Configure: apply the chosen auth model as default (skip the extra picker) and refresh the model catalog for new providers.</li>
<li>Auto-reply: align <code>/think</code> default display with model reasoning defaults. (#751) — thanks @gabriel-trigo.</li>
<li>Auto-reply: flush block reply buffers on tool boundaries. (#750) — thanks @sebslight.</li>
<li>Auto-reply: allow sender fallback for command authorization when <code>SenderId</code> is empty (WhatsApp self-chat). (#755) — thanks @juanpablodlc.</li>
<li>Heartbeat: refresh prompt text for updated defaults.</li>
<li>Agents/Tools: use PowerShell on Windows to capture system utility output. (#748) — thanks @myfunc.</li>
<li>Agents/Tools: normalize Claude Code-style read/write/edit params (file_path/old_string/new_string) and keep sandbox guards in place. (#768) — thanks @hsrvc.</li>
<li>Docker: tolerate unset optional env vars in docker-setup.sh under strict mode. (#725) — thanks @petradonka.</li>
<li>CLI/Update: preserve base environment when passing overrides to update subprocesses. (#713) — thanks @danielz1z.</li>
<li>Agents: treat message tool errors as failures so fallback replies still send; require <code>to</code> + <code>message</code> for <code>action=send</code>. (#717) — thanks @theglove44.</li>
<li>Agents: preserve reasoning items on tool-only turns.</li>
<li>Agents: enforce <code><final></code> gating for reasoning-tag providers to prevent tag/reasoning leaks. (#754) — thanks @mcinteerj.</li>
<li>Agents/Subagents: wait for completion before announcing, align wait timeout with run timeout, and make announce prompts more emphatic.</li>
<li>Agents: route subagent transcripts to the target agent sessions directory and add regression coverage. (#708) — thanks @xMikeMickelson.</li>
<li>Agents/Tools: preserve action enums when flattening tool schemas. (#708) — thanks @xMikeMickelson.</li>
<li>Gateway/Agents: canonicalize main session aliases for store writes and add regression coverage. (#709) — thanks @xMikeMickelson.</li>
<li>Agents: reset sessions and retry when auto-compaction overflows instead of crashing the gateway.</li>
<li>Providers/Telegram: normalize command mentions for consistent parsing. (#729) — thanks @obviyus.</li>
<li>Providers: skip DM history limit handling for non-DM sessions. (#728) — thanks @pkrmf.</li>
<li>Sandbox: fix non-main mode incorrectly sandboxing the main DM session and align <code>/status</code> runtime reporting with effective sandbox state.</li>
<li>Sandbox/Gateway: treat <code>agent:<id>:main</code> as a main-session alias when <code>session.mainKey</code> is customized (backwards compatible).</li>
<li>Auto-reply: fast-path allowlisted slash commands (inline <code>/help</code>/<code>/commands</code>/<code>/status</code>/<code>/whoami</code> stripped before model).</li>
</ul>
<p><a href="https://github.com/clawdbot/clawdbot/blob/main/CHANGELOG.md">View full changelog</a></p>
]]></description>
<enclosure url="https://github.com/clawdbot/clawdbot/releases/download/v2026.1.11/Clawdbot-2026.1.11.zip" length="19860746" type="application/octet-stream" sparkle:edSignature="zmN6RovfpvPNm7PPYKDQ2c8nUtlq4k1+S6c3fvZwEHFRwKOxjhQtaW0phriDJwcqfguU3ibAQd/cpRtP4LYdCQ=="/>
</item>
<item>
<title>2026.1.10</title>
<pubDate>Sun, 11 Jan 2026 10:26:34 +0000</pubDate>
<link>https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml</link>
<sparkle:version>4946</sparkle:version>
<sparkle:shortVersionString>2026.1.10</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>15.0</sparkle:minimumSystemVersion>
<description><![CDATA[<h2>Clawdbot 2026.1.10</h2>
<h3>Highlights</h3>
<ul>
<li>CLI: <code>clawdbot status</code> now table-based + shows OS/update/gateway/daemon/agents/sessions; <code>status --all</code> adds a full read-only debug report (tables, log tails, Tailscale summary, and scan progress via OSC-9 + spinner).</li>
<li>CLI Backends: add Codex CLI fallback with resume support (text output) and JSONL parsing for new runs, plus a live CLI resume probe.</li>
<li>CLI: add <code>clawdbot update</code> (safe-ish git checkout update) + <code>--update</code> shorthand. (#673) — thanks @fm1randa.</li>
<li>Gateway: add OpenAI-compatible <code>/v1/chat/completions</code> HTTP endpoint (auth, SSE streaming, per-agent routing). (#680) — thanks @steipete.</li>
</ul>
<h3>Changes</h3>
<ul>
<li>Onboarding/Models: add first-class Z.AI (GLM) auth choice (<code>zai-api-key</code>) + <code>--zai-api-key</code> flag.</li>
<li>CLI/Onboarding: add OpenRouter API key auth option in configure/onboard. (#703) — thanks @mteam88.</li>
<li>Agents: add human-delay pacing between block replies (modes: off/natural/custom, per-agent configurable). (#446) — thanks @tony-freedomology.</li>
<li>Agents/Browser: add <code>browser.target</code> (sandbox/host/custom) with sandbox host-control gating via <code>agents.defaults.sandbox.browser.allowHostControl</code>, allowlists for custom control URLs/hosts/ports, and expand browser tool docs (remote control, profiles, internals).</li>
<li>Onboarding/Models: add catalog-backed default model picker to onboarding + configure. (#611) — thanks @jonasjancarik.</li>
<li>Agents/OpenCode Zen: update fallback models + defaults, keep legacy alias mappings. (#669) — thanks @magimetal.</li>
<li>Providers: unify group history context wrappers across providers with per-provider/per-account <code>historyLimit</code> overrides (fallback to <code>messages.groupChat.historyLimit</code>). Set <code>0</code> to disable. (#672) — thanks @steipete.</li>
<li>Gateway/Heartbeat: optionally deliver heartbeat <code>Reasoning:</code> output (<code>agents.defaults.heartbeat.includeReasoning</code>). (#690)</li>
<li>Docker: allow optional home volume + extra bind mounts in <code>docker-setup.sh</code>. (#679) — thanks @gabriel-trigo.</li>
</ul>
<h3>Fixes</h3>
<ul>
<li>CLI/Status: expand tables to full terminal width; clarify provider setup vs runtime warnings; richer per-provider detail; token previews in <code>status</code> while keeping <code>status --all</code> redacted; add troubleshooting link footer; keep log tails pasteable; show gateway auth used when reachable; surface provider runtime errors (Signal/iMessage/Slack); harden <code>tailscale status --json</code> parsing; make <code>status --all</code> scan progress determinate; and replace the footer with a 3-line “Next steps” recommendation (share/debug/probe).</li>
<li>CLI/Gateway: clarify that <code>clawdbot gateway status</code> reports RPC health (connect + RPC) and shows RPC failures separately from connect failures.</li>
<li>CLI/Update: gate progress spinner on stdout TTY and align clean-check step label. (#701) — thanks @bjesuiter.</li>
<li>Telegram: add <code>/whoami</code> + <code>/id</code> commands to reveal sender id for allowlists; allow <code>@username</code> and prefixed ids in <code>allowFrom</code> prompts (with stability warning).</li>
<li>Control UI: stop auto-writing <code>telegram.groups["*"]</code> and warn/confirm before enabling wildcard groups.</li>
<li>WhatsApp: send ack reactions only for handled messages and ignore legacy <code>messages.ackReaction</code> (doctor copies to <code>whatsapp.ackReaction</code>). (#629) — thanks @pasogott.</li>
<li>Sandbox/Skills: mirror skills into sandbox workspaces for read-only mounts so SKILL.md stays accessible.</li>
<li>Terminal/Table: ANSI-safe wrapping to prevent table clipping/color loss; add regression coverage.</li>
<li>Docker: allow optional apt packages during image build and document the build arg. (#697) — thanks @gabriel-trigo.</li>
<li>Gateway/Heartbeat: deliver reasoning even when the main heartbeat reply is <code>HEARTBEAT_OK</code>. (#694) — thanks @antons.</li>
<li>macOS: clear unsigned launchd overrides on signed restarts and warn via doctor when attach-only/disable markers are set. (#695) — thanks @jeffersonwarrior.</li>
<li>Agents: enforce single-writer session locks and drop orphan tool results to prevent tool-call ID failures (MiniMax/Anthropic-compatible APIs).</li>
<li>Docs: make <code>clawdbot status</code> the first diagnostic step, clarify <code>status --deep</code> behavior, and document <code>/whoami</code> + <code>/id</code>.</li>
<li>Docs/Testing: clarify live tool+image probes and how to list your testable <code>provider/model</code> ids.</li>
<li>Tests/Live: make gateway bash+read probes resilient to provider formatting while still validating real tool calls.</li>
<li>WhatsApp: detect @lid mentions in groups using authDir reverse mapping + resolve self JID E.164 for mention gating. (#692) — thanks @peschee.</li>
<li>Gateway/Auth: default to token auth on loopback during onboarding, add doctor token generation flow, and tighten audio transcription config to Whisper-only.</li>
<li>Providers: dedupe inbound messages across providers to avoid duplicate LLM runs on redeliveries/reconnects. (#689) — thanks @adam91holt.</li>
<li>Agents: strip <code><thought></code>/<code><antthinking></code> tags from hidden reasoning output and cover tag variants in tests. (#688) — thanks @theglove44.</li>
<li>macOS: save model picker selections as normalized provider/model IDs and keep manual entries aligned. (#683) — thanks @benithors.</li>
<li>Agents: recognize "usage limit" errors as rate limits for failover. (#687) — thanks @evalexpr.</li>
<li>CLI: avoid success message when daemon restart is skipped. (#685) — thanks @carlulsoe.</li>
<li>Commands: disable <code>/config</code> + <code>/debug</code> by default; gate via <code>commands.config</code>/<code>commands.debug</code> and hide from native registration/help output.</li>
<li>Agents/System: clarify that sub-agents remain sandboxed and cannot use elevated host access.</li>
<li>Gateway: disable the OpenAI-compatible <code>/v1/chat/completions</code> endpoint by default; enable via <code>gateway.http.endpoints.chatCompletions.enabled=true</code>.</li>
<li>macOS: stabilize bridge tunnels, guard invoke senders on disconnect, and drain stdout/stderr to avoid deadlocks. (#676) — thanks @ngutman.</li>
<li>Agents/System: clarify sandboxed runtime in system prompt and surface elevated availability when sandboxed.</li>
<li>Auto-reply: prefer <code>RawBody</code> for command/directive parsing (WhatsApp + Discord) and prevent fallback runs from clobbering concurrent session updates. (#643) — thanks @mcinteerj.</li>
<li>WhatsApp: fix group reactions by preserving message IDs and sender JIDs in history; normalize participant phone numbers to JIDs in outbound reactions. (#640) — thanks @mcinteerj.</li>
<li>WhatsApp: expose group participant IDs to the model so reactions can target the right sender.</li>
<li>Cron: <code>wakeMode: "now"</code> waits for heartbeat completion (and retries when the main lane is busy). (#666) — thanks @roshanasingh4.</li>
<li>Agents/OpenAI: fix Responses tool-only → follow-up turn handling (avoid standalone <code>reasoning</code> items that trigger 400 “required following item”) and replay reasoning items in Responses/Codex Responses history for tool-call-only turns.</li>
<li>Sandbox: add <code>clawdbot sandbox explain</code> (effective policy inspector + fix-it keys); improve “sandbox jail” tool-policy/elevated errors with actionable config key paths; link to docs.</li>
<li>Hooks/Gmail: keep Tailscale serve path at <code>/</code> while preserving the public path. (#668) — thanks @antons.</li>
<li>Hooks/Gmail: allow Tailscale target URLs to preserve internal serve paths.</li>
<li>Auth: update Claude Code keychain credentials in-place during refresh sync; share JSON file helpers; add CLI fallback coverage.</li>
<li>Auth: throttle external CLI credential syncs (Claude/Codex), reduce Keychain reads, and skip sync when cached credentials are still fresh.</li>
<li>CLI: respect <code>CLAWDBOT_STATE_DIR</code> for node pairing + voice wake settings storage. (#664) — thanks @azade-c.</li>
<li>Onboarding/Gateway: persist non-interactive gateway token auth in config; add WS wizard + gateway tool-calling regression coverage.</li>
<li>Gateway/Control UI: make <code>chat.send</code> non-blocking, wire Stop to <code>chat.abort</code>, and treat <code>/stop</code> as an out-of-band abort. (#653)</li>
<li>Gateway/Control UI: allow <code>chat.abort</code> without <code>runId</code> (abort active runs), suppress post-abort chat streaming, and prune stuck chat runs. (#653)</li>
<li>Gateway/Control UI: sniff image attachments for chat.send, drop non-images, and log mismatches. (#670) — thanks @cristip73.</li>
<li>macOS: force <code>restart-mac.sh --sign</code> to require identities and keep bundled Node signed for relay verification. (#580) — thanks @jeffersonwarrior.</li>
<li>Gateway/Agent: accept image attachments on <code>agent</code> (multimodal message) and add live gateway image probe (<code>CLAWDBOT_LIVE_GATEWAY_IMAGE_PROBE=1</code>).</li>
<li>CLI: <code>clawdbot sessions</code> now includes <code>elev:*</code> + <code>usage:*</code> flags in the table output.</li>
<li>CLI/Pairing: accept positional provider for <code>pairing list|approve</code> (npm-run compatible); update docs/bot hints.</li>
<li>Branding: normalize user-facing “ClawdBot”/“CLAWDBOT” → “Clawdbot” (CLI, status, docs).</li>
<li>Auto-reply: fix native <code>/model</code> not updating the actual chat session (Telegram/Slack/Discord). (#646)</li>
<li>Doctor: offer to run <code>clawdbot update</code> first on git installs (keeps doctor output aligned with latest).</li>
<li>Doctor: avoid false legacy workspace warning when install dir is <code>~/clawdbot</code>. (#660)</li>
<li>iMessage: fix reasoning persistence across DMs; avoid partial/duplicate replies when reasoning is enabled. (#655) — thanks @antons.</li>
<li>Models/Auth: allow MiniMax API configs without <code>models.providers.minimax.apiKey</code> (auth profiles / <code>MINIMAX_API_KEY</code>). (#656) — thanks @mneves75.</li>
<li>Agents: avoid duplicate replies when the message tool sends. (#659) — thanks @mickahouan.</li>
<li>Agents: harden Cloud Code Assist tool ID sanitization (toolUse/toolCall/toolResult) and scrub extra JSON Schema constraints. (#665) — thanks @sebslight.</li>
<li>Agents: sanitize tool results + Cloud Code Assist tool IDs at context-build time (prevents mid-run strict-provider request rejects).</li>
<li>Agents/Tools: resolve workspace-relative Read/Write/Edit paths; align bash default cwd. (#642) — thanks @mukhtharcm.</li>
<li>Discord: include forwarded message snapshots in agent session context. (#667) — thanks @rubyrunsstuff.</li>
<li>Telegram: add <code>telegram.draftChunk</code> to tune draft streaming chunking for <code>streamMode: "block"</code>. (#667) — thanks @rubyrunsstuff.</li>
<li>Tests/Agents: add regression coverage for workspace tool path resolution and bash cwd defaults.</li>
<li>iOS/Android: enable stricter concurrency/lint checks; fix Swift 6 strict concurrency issues + Android lint errors (ExifInterface, obsolete SDK check). (#662) — thanks @KristijanJovanovski.</li>
<li>iOS/macOS: share <code>AsyncTimeout</code>, require explicit <code>bridgeStableID</code> on connect, and harden tool display defaults (avoids missing-resource label fallbacks).</li>
<li>Telegram: serialize media-group processing to avoid missed albums under load.</li>
<li>Signal: handle <code>dataMessage.reaction</code> events (signal-cli SSE) to avoid broken attachment errors. (#637) — thanks @neist.</li>
<li>Docs: showcase entries for ParentPay, R2 Upload, iOS TestFlight, and Oura Health. (#650) — thanks @henrino3.</li>
<li>Agents: repair session transcripts by dropping duplicate tool results across the whole history (unblocks Anthropic-compatible APIs after retries).</li>
<li>Tests/Live: reset the gateway session between model runs to avoid cross-provider transcript incompatibilities (notably OpenAI Responses reasoning replay rules).</li>
</ul>
<p><a href="https://github.com/clawdbot/clawdbot/blob/main/CHANGELOG.md">View full changelog</a></p>
]]></description>
<enclosure url="https://github.com/clawdbot/clawdbot/releases/download/v2026.1.10/Clawdbot-2026.1.10.zip" length="199586798" type="application/octet-stream" sparkle:edSignature="Ntb22WL4b0Hu3Ntqm1iqnD+5bUqqa36iSQ8vbT/50C5FnL5IjF4GoHWrfanJ58JhBCQ4eKdK+K0U4OBZiSzZBw=="/>
</item>
</channel>
</rss>
+2 -2
View File
@@ -21,8 +21,8 @@ android {
applicationId = "com.clawdbot.android"
minSdk = 31
targetSdk = 36
versionCode = 202601111
versionName = "2026.1.11-1"
versionCode = 202601114
versionName = "2026.1.11-4"
}
buildTypes {
+2 -2
View File
@@ -19,9 +19,9 @@
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>2026.1.11-1</string>
<string>2026.1.11-4</string>
<key>CFBundleVersion</key>
<string>202601111</string>
<string>202601113</string>
<key>NSAppTransportSecurity</key>
<dict>
<key>NSAllowsArbitraryLoadsInWebContent</key>
+2 -2
View File
@@ -17,8 +17,8 @@
<key>CFBundlePackageType</key>
<string>BNDL</string>
<key>CFBundleShortVersionString</key>
<string>2026.1.11-1</string>
<string>2026.1.11-4</string>
<key>CFBundleVersion</key>
<string>202601111</string>
<string>202601113</string>
</dict>
</plist>
+1 -1
View File
@@ -204,7 +204,7 @@ final class AppState {
private var earBoostTask: Task<Void, Never>?
init(preview: Bool = false) {
self.isPreview = preview
self.isPreview = preview || ProcessInfo.processInfo.isRunningTests
let onboardingSeen = UserDefaults.standard.bool(forKey: "clawdbot.onboardingSeen")
self.isPaused = UserDefaults.standard.bool(forKey: pauseDefaultsKey)
self.launchAtLogin = false
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>2026.1.11-1</string>
<string>2026.1.11-4</string>
<key>CFBundleVersion</key>
<string>202601111</string>
<string>202601113</string>
<key>CFBundleIconFile</key>
<string>Clawdbot</string>
<key>CFBundleURLTypes</key>
@@ -1352,6 +1352,7 @@ public struct SkillsUpdateParams: Codable, Sendable {
public struct CronJob: Codable, Sendable {
public let id: String
public let agentid: String?
public let name: String
public let description: String?
public let enabled: Bool
@@ -1366,6 +1367,7 @@ public struct CronJob: Codable, Sendable {
public init(
id: String,
agentid: String?,
name: String,
description: String?,
enabled: Bool,
@@ -1379,6 +1381,7 @@ public struct CronJob: Codable, Sendable {
state: [String: AnyCodable]
) {
self.id = id
self.agentid = agentid
self.name = name
self.description = description
self.enabled = enabled
@@ -1393,6 +1396,7 @@ public struct CronJob: Codable, Sendable {
}
private enum CodingKeys: String, CodingKey {
case id
case agentid = "agentId"
case name
case description
case enabled
@@ -1425,6 +1429,7 @@ public struct CronStatusParams: Codable, Sendable {
public struct CronAddParams: Codable, Sendable {
public let name: String
public let agentid: AnyCodable?
public let description: String?
public let enabled: Bool?
public let schedule: AnyCodable
@@ -1435,6 +1440,7 @@ public struct CronAddParams: Codable, Sendable {
public init(
name: String,
agentid: AnyCodable?,
description: String?,
enabled: Bool?,
schedule: AnyCodable,
@@ -1444,6 +1450,7 @@ public struct CronAddParams: Codable, Sendable {
isolation: [String: AnyCodable]?
) {
self.name = name
self.agentid = agentid
self.description = description
self.enabled = enabled
self.schedule = schedule
@@ -1454,6 +1461,7 @@ public struct CronAddParams: Codable, Sendable {
}
private enum CodingKeys: String, CodingKey {
case name
case agentid = "agentId"
case description
case enabled
case schedule
+12
View File
@@ -27,6 +27,8 @@ A cron job is a stored record with:
- a **schedule** (when it should run),
- a **payload** (what it should do),
- optional **delivery** (where output should be sent).
- optional **agent binding** (`agentId`): run the job under a specific agent; if
missing or unknown, the gateway falls back to the default agent.
Jobs are identified by a stable `jobId` (used by CLI/Gateway APIs).
In agent tool calls, `jobId` is canonical; legacy `id` is accepted for compatibility.
@@ -190,6 +192,16 @@ clawdbot cron add \
--deliver \
--provider whatsapp \
--to "+15551234567"
Agent selection (multi-agent setups):
```bash
# Pin a job to agent "ops" (falls back to default if that agent is missing)
clawdbot cron add --name "Ops sweep" --cron "0 6 * * *" --session isolated --message "Check ops queue" --agent ops
# Switch or clear the agent on an existing job
clawdbot cron edit <jobId> --agent ops
clawdbot cron edit <jobId> --clear-agent
```
```
Manual run (debug):
+12
View File
@@ -70,6 +70,10 @@ clawdbot [--dev] [--profile <name>] <command>
enable
disable
doctor
memory
status
index
search
message
agent
agents
@@ -188,6 +192,14 @@ Manage extensions and their config:
Most plugin changes require a gateway restart. See [/plugin](/plugin).
## Memory
Vector search over `MEMORY.md` + `memory/*.md`:
- `clawdbot memory status` — show index stats.
- `clawdbot memory index` — reindex memory files.
- `clawdbot memory search "<query>"` — semantic search over memory.
## Chat slash commands
Chat messages support `/...` commands (text and native). See [/tools/slash-commands](/tools/slash-commands).
+54
View File
@@ -67,3 +67,57 @@ Details:
For the full compaction lifecycle, see
[Session management + compaction](/reference/session-management-compaction).
## Vector memory search
Clawdbot can build a small vector index over `MEMORY.md` and `memory/*.md` so
semantic queries can find related notes even when wording differs.
Defaults:
- Enabled by default.
- Watches memory files for changes (debounced).
- Uses remote embeddings (OpenAI) unless configured for local.
- Local mode uses node-llama-cpp and may require `pnpm approve-builds`.
Config example:
```json5
agents: {
defaults: {
memorySearch: {
provider: "openai",
model: "text-embedding-3-small",
fallback: "openai",
sync: { watch: true }
}
}
}
```
Tools:
- `memory_search` — returns snippets with file + line ranges.
- `memory_get` — read memory file content by path.
Local mode:
- Set `agents.defaults.memorySearch.provider = "local"`.
- Provide `agents.defaults.memorySearch.local.modelPath` (GGUF or `hf:` URI).
- Optional: set `agents.defaults.memorySearch.fallback = "none"` to avoid remote fallback.
### How the memory tools work
- `memory_search` semantically searches Markdown chunks (~400 token target, 80-token overlap) from `MEMORY.md` + `memory/**/*.md`. It returns snippet text (capped ~700 chars), file path, line range, score, provider/model, and whether we fell back from local → remote embeddings. No full file payload is returned.
- `memory_get` reads a specific memory Markdown file (workspace-relative), optionally from a starting line and for N lines. Paths outside `MEMORY.md` / `memory/` are rejected.
- Both tools are enabled only when `memorySearch.enabled` resolves true for the agent.
### What gets indexed (and when)
- File type: Markdown only (`MEMORY.md`, `memory/**/*.md`).
- Index storage: per-agent SQLite at `~/.clawdbot/state/memory/<agentId>.sqlite` (configurable via `agents.defaults.memorySearch.store.path`, supports `{agentId}` token).
- Freshness: watcher on `MEMORY.md` + `memory/` marks the index dirty (debounce 1.5s). Sync runs on session start, on first search when dirty, and optionally on an interval. Reindex triggers when embedding model/provider or chunk sizes change.
### Local embedding auto-download
- Default local embedding model: `hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf` (~0.6 GB).
- When `memorySearch.provider = "local"`, `node-llama-cpp` resolves `modelPath`; if the GGUF is missing it **auto-downloads** to the cache (or `local.modelCacheDir` if set), then loads it. Downloads resume on retry.
- Native build requirement: run `pnpm approve-builds`, pick `node-llama-cpp`, then `pnpm rebuild node-llama-cpp`.
- Fallback: if local setup fails and `memorySearch.fallback = "openai"`, we automatically switch to remote embeddings (`openai/text-embedding-3-small` unless overridden) and record the reason.
+1 -1
View File
@@ -239,7 +239,7 @@ Save to `~/.clawdbot/clawdbot.json` and you can DM the bot from that number.
target: "last",
to: "+15555550123",
prompt: "HEARTBEAT",
ackMaxChars: 30
ackMaxChars: 300
},
sandbox: {
mode: "non-main",
+13 -46
View File
@@ -756,8 +756,8 @@ Controls how chat commands are enabled across connectors.
Notes:
- Text commands must be sent as a **standalone** message and use the leading `/` (no plain-text aliases).
- `commands.text: false` disables parsing chat messages for commands.
- `commands.native: true` registers native commands on supported connectors (Discord/Slack/Telegram). Platforms without native commands still rely on text commands.
- `commands.native: false` skips native registration; Discord/Telegram clear previously registered commands on startup. Slack commands are managed in the Slack app.
- `commands.native: "auto"` (default) turns on native commands for Discord/Telegram and leaves Slack off; unsupported providers stay text-only.
- Set `commands.native: true|false` to force all, or override per provider with `discord.commands.native`, `telegram.commands.native`, `slack.commands.native` (bool or `"auto"`). `false` clears previously registered commands on Discord/Telegram at startup; Slack commands are managed in the Slack app.
- `commands.config: true` enables `/config` (reads/writes `clawdbot.json`).
- `commands.debug: true` enables `/debug` (runtime-only overrides).
- `commands.restart: true` enables `/restart` and the gateway tool restart action.
@@ -852,6 +852,7 @@ Multi-account support lives under `discord.accounts` (see the multi-account sect
enabled: true,
token: "your-bot-token",
mediaMaxMb: 8, // clamp inbound media size
allowBots: false, // allow bot-authored messages
actions: { // tool action gates (false disables)
reactions: true,
stickers: true,
@@ -910,6 +911,7 @@ Multi-account support lives under `discord.accounts` (see the multi-account sect
Clawdbot starts Discord only when a `discord` config section exists. The token is resolved from `DISCORD_BOT_TOKEN` or `discord.token` (unless `discord.enabled` is `false`). Use `user:<id>` (DM) or `channel:<id>` (guild channel) when specifying delivery targets for cron/CLI commands; bare numeric IDs are ambiguous and rejected.
Guild slugs are lowercase with spaces replaced by `-`; channel keys use the slugged channel name (no leading `#`). Prefer guild ids as keys to avoid rename ambiguity.
Bot-authored messages are ignored by default. Enable with `discord.allowBots` (own messages are still filtered to prevent self-reply loops).
Reaction notification modes:
- `off`: no reaction events.
- `own`: reactions on the bot's own messages (default).
@@ -1276,6 +1278,7 @@ Example:
},
maxConcurrent: 3,
subagents: {
model: "minimax/MiniMax-M2.1",
maxConcurrent: 1,
archiveAfterMinutes: 60
},
@@ -1463,7 +1466,7 @@ Z.AI models are available as `zai/<model>` (e.g. `zai/glm-4.7`) and require
- `target`: optional delivery provider (`last`, `whatsapp`, `telegram`, `discord`, `slack`, `signal`, `imessage`, `none`). Default: `last`.
- `to`: optional recipient override (provider-specific id, e.g. E.164 for WhatsApp, chat id for Telegram).
- `prompt`: optional override for the heartbeat body (default: `Read HEARTBEAT.md if exists. Consider outstanding tasks. Checkup sometimes on your human during (user local) day time.`). Overrides are sent verbatim; include a `Read HEARTBEAT.md if exists` line if you still want the file read.
- `ackMaxChars`: max chars allowed after `HEARTBEAT_OK` before delivery (default: 30).
- `ackMaxChars`: max chars allowed after `HEARTBEAT_OK` before delivery (default: 300).
Heartbeats run full agent turns. Shorter intervals burn more tokens; be mindful
of `every`, keep `HEARTBEAT.md` tiny, and/or choose a cheaper `model`.
@@ -1478,6 +1481,7 @@ Note: `applyPatch` is only under `tools.exec` (no `tools.bash` alias).
Legacy: `tools.bash` is still accepted as an alias.
`agents.defaults.subagents` configures sub-agent defaults:
- `model`: default model for spawned sub-agents (string or `{ primary, fallbacks }`). If omitted, sub-agents inherit the caller’s model unless overridden per agent or per call.
- `maxConcurrent`: max concurrent sub-agent runs (default 1)
- `archiveAfterMinutes`: auto-archive sub-agent sessions after N minutes (default 60; set `0` to disable)
- Per-subagent tool policy: `tools.subagents.tools.allow` / `tools.subagents.tools.deny` (deny wins)
@@ -1560,6 +1564,7 @@ Defaults (if enabled):
- auto-prune: idle > 24h OR age > 7d
- tool policy: allow only `exec`, `process`, `read`, `write`, `edit`, `apply_patch`, `sessions_list`, `sessions_history`, `sessions_send`, `sessions_spawn`, `session_status` (deny wins)
- configure via `tools.sandbox.tools`, override per-agent via `agents.list[].tools.sandbox.tools`
- tool group shorthands supported in sandbox policy: `group:runtime`, `group:fs`, `group:sessions`, `group:memory` (see [Sandbox vs Tool Policy vs Elevated](/gateway/sandbox-vs-tool-policy-vs-elevated#tool-groups-shorthands))
- optional sandboxed browser (Chromium + CDP, noVNC observer)
- hardening knobs: `network`, `user`, `pidsLimit`, `memory`, `cpus`, `ulimits`, `seccompProfile`, `apparmorProfile`
@@ -1601,7 +1606,8 @@ Legacy: `perSession` is still supported (`true` → `scope: "session"`,
seccompProfile: "/path/to/seccomp.json",
apparmorProfile: "clawdbot-sandbox",
dns: ["1.1.1.1", "8.8.8.8"],
extraHosts: ["internal.service:10.0.0.5"]
extraHosts: ["internal.service:10.0.0.5"],
binds: ["/var/run/docker.sock:/var/run/docker.sock", "/home/user/source:/source:rw"]
},
browser: {
enabled: false,
@@ -1647,6 +1653,8 @@ to `"bridge"` (or your custom network) if the agent needs outbound access.
Note: inbound attachments are staged into the active workspace at `media/inbound/*`. With `workspaceAccess: "rw"`, that means files are written into the agent workspace.
Note: `docker.binds` mounts additional host directories; global and per-agent binds are merged.
Build the optional browser image with:
```bash
scripts/sandbox-browser-setup.sh
@@ -1813,48 +1821,7 @@ Notes:
### Local models (LM Studio) — recommended setup
Best current local setup (what we’re running): **MiniMax M2.1** on a powerful local machine
via **LM Studio** using the **Responses API**.
```json5
{
agents: {
defaults: {
model: { primary: "lmstudio/minimax-m2.1-gs32" },
models: {
"anthropic/claude-opus-4-5": { alias: "Opus" },
"lmstudio/minimax-m2.1-gs32": { alias: "Minimax" }
}
}
},
models: {
mode: "merge",
providers: {
lmstudio: {
baseUrl: "http://127.0.0.1:1234/v1",
apiKey: "lmstudio",
api: "openai-responses",
models: [
{
id: "minimax-m2.1-gs32",
name: "MiniMax M2.1 GS32",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 196608,
maxTokens: 8192
}
]
}
}
}
}
```
Notes:
- LM Studio must have the model loaded and the local server enabled (default URL above).
- Responses API enables clean reasoning/output separation; WhatsApp sees only final text.
- Adjust `contextWindow`/`maxTokens` if your LM Studio context length differs.
See [/gateway/local-models](/gateway/local-models) for the current local guidance. TL;DR: run MiniMax M2.1 via LM Studio Responses API on serious hardware; keep hosted models merged for fallback.
### MiniMax M2.1
+2 -2
View File
@@ -57,7 +57,7 @@ custom body (sent verbatim).
- If nothing needs attention, reply with **`HEARTBEAT_OK`**.
- During heartbeat runs, Clawdbot treats `HEARTBEAT_OK` as an ack when it appears
at the **start or end** of the reply. The token is stripped and the reply is
dropped if the remaining content is **≤ `ackMaxChars`** (default: 30).
dropped if the remaining content is **≤ `ackMaxChars`** (default: 300).
- If `HEARTBEAT_OK` appears in the **middle** of a reply, it is not treated
specially.
- For alerts, **do not** include `HEARTBEAT_OK`; return only the alert text.
@@ -78,7 +78,7 @@ and logged; a message that is only `HEARTBEAT_OK` is dropped.
target: "last", // last | whatsapp | telegram | discord | slack | signal | imessage | none
to: "+15551234567", // optional provider-specific override
prompt: "Read HEARTBEAT.md if exists. Consider outstanding tasks. Checkup sometimes on your human during (user local) day time.",
ackMaxChars: 30 // max chars allowed after HEARTBEAT_OK
ackMaxChars: 300 // max chars allowed after HEARTBEAT_OK
}
}
}
+157
View File
@@ -0,0 +1,157 @@
---
summary: "Run Clawdbot on local LLMs (LM Studio, vLLM, LiteLLM, custom OpenAI endpoints)"
read_when:
- You want to serve models from your own GPU box
- You are wiring LM Studio or an OpenAI-compatible proxy
- You need the safest local model guidance
---
# Local models
Local is doable, but Clawdbot expects large context + strong defenses against prompt injection. Small cards truncate context and leak safety. Aim high: **≥2 maxed-out Mac Studios or equivalent GPU rig (~$30k+)**. A single **24 GB** GPU works only for lighter prompts with higher latency. Use the **largest / full-size model variant you can run**; aggressively quantized or “small” checkpoints raise prompt-injection risk (see [Security](/gateway/security)).
## Recommended: LM Studio + MiniMax M2.1 (Responses API, full-size)
Best current local stack. Load MiniMax M2.1 in LM Studio, enable the local server (default `http://127.0.0.1:1234`), and use Responses API to keep reasoning separate from final text.
```json5
{
agents: {
defaults: {
model: { primary: "lmstudio/minimax-m2.1-gs32" },
models: {
"anthropic/claude-opus-4-5": { alias: "Opus" },
"lmstudio/minimax-m2.1-gs32": { alias: "Minimax" }
}
}
},
models: {
mode: "merge",
providers: {
lmstudio: {
baseUrl: "http://127.0.0.1:1234/v1",
apiKey: "lmstudio",
api: "openai-responses",
models: [
{
id: "minimax-m2.1-gs32",
name: "MiniMax M2.1 GS32",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 196608,
maxTokens: 8192
}
]
}
}
}
}
```
**Setup checklist**
- Install LM Studio: https://lmstudio.ai
- In LM Studio, download the **largest MiniMax M2.1 build available** (avoid “small”/heavily quantized variants), start the server, confirm `http://127.0.0.1:1234/v1/models` lists it.
- Keep the model loaded; cold-load adds startup latency.
- Adjust `contextWindow`/`maxTokens` if your LM Studio build differs.
- For WhatsApp, stick to Responses API so only final text is sent.
## Model catalog (remote + local)
| Tier | Model ID | Context | Who downloads | Notes |
| --- | --- | --- | --- | --- |
| Hosted | `anthropic/claude-opus-4-5` | 200k | Provider | Latest Claude; keep as high-quality fallback. |
| Hosted | `anthropic/claude-sonnet-4-5` | 200k | Provider | Cheaper Claude; good default. |
| Hosted | `openai/gpt-4.1` | 128k | Provider | Latest GPT-4; strong tools + reasoning. |
| Hosted | `openai/gpt-4.1-mini` | 128k | Provider | Fast/cheap GPT-4 family; good fallback. |
| Local | `lmstudio/minimax-m2.1-gs32` | ~196k (build-dependent) | You (LM Studio UI) | Recommended local heavy model; keep loaded. |
| Local | Custom `vllm` / `litellm` model | server-defined | You (server) | Any OpenAI-compatible endpoint; align context settings. |
Keep hosted models configured even when running local; use `models.mode: "merge"` so fallbacks stay available.
### Hybrid config: hosted primary, local fallback
```json5
{
agents: {
defaults: {
model: {
primary: "anthropic/claude-sonnet-4-5",
fallbacks: ["lmstudio/minimax-m2.1-gs32", "openai/gpt-4.1-mini"]
},
models: {
"anthropic/claude-sonnet-4-5": { alias: "Sonnet" },
"lmstudio/minimax-m2.1-gs32": { alias: "MiniMax Local" },
"openai/gpt-4.1-mini": { alias: "GPT-4.1 mini" }
}
}
},
models: {
mode: "merge",
providers: {
lmstudio: {
baseUrl: "http://127.0.0.1:1234/v1",
apiKey: "lmstudio",
api: "openai-responses",
models: [
{
id: "minimax-m2.1-gs32",
name: "MiniMax M2.1 GS32",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 196608,
maxTokens: 8192
}
]
}
}
}
}
```
### Local-first with hosted safety net
Swap the primary and fallback order; keep the same providers block and `models.mode: "merge"` so you can fall back to Sonnet or GPT-4.1 when the local box is down.
### Regional hosting / data routing
- Hosted MiniMax/Kimi/GLM variants also exist on OpenRouter with region-pinned endpoints (e.g., US-hosted). Pick the regional variant there to keep traffic in your chosen jurisdiction while still using `models.mode: "merge"` for Anthropic/OpenAI fallbacks.
- Local-only remains the strongest privacy path; hosted regional routing is the middle ground when you need provider features but want control over data flow.
## Other OpenAI-compatible local proxies
vLLM, LiteLLM, OAI-proxy, or custom gateways work if they expose an OpenAI-style `/v1` endpoint. Replace the provider block above with your endpoint and model ID:
```json5
{
models: {
mode: "merge",
providers: {
local: {
baseUrl: "http://127.0.0.1:8000/v1",
apiKey: "sk-local",
api: "openai-responses",
models: [
{
id: "my-local-model",
name: "Local Model",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 120000,
maxTokens: 8192
}
]
}
}
}
}
```
Keep `models.mode: "merge"` so hosted models stay available as fallbacks.
## Troubleshooting
- Gateway can reach the proxy? `curl http://127.0.0.1:1234/v1/models`.
- LM Studio model unloaded? Reload; cold start is a common “hanging” cause.
- Context errors? Lower `contextWindow` or raise your server limit.
- Safety: local models skip provider-side filters; keep agents narrow and compaction on to limit prompt injection blast radius.
@@ -39,6 +39,14 @@ Sandboxing is controlled by `agents.defaults.sandbox.mode`:
See [Sandboxing](/gateway/sandboxing) for the full matrix (scope, workspace mounts, images).
### Bind mounts (security quick check)
- `docker.binds` *pierces* the sandbox filesystem: whatever you mount is visible inside the container with the mode you set (`:ro` or `:rw`).
- Default is read-write if you omit the mode; prefer `:ro` for source/secrets.
- `scope: "shared"` ignores per-agent binds (only global binds apply).
- Binding `/var/run/docker.sock` effectively hands host control to the sandbox; only do this intentionally.
- Workspace access (`workspaceAccess: "ro"`/`"rw"`) is independent of bind modes.
## Tool policy: which tools exist/are callable
Two layers matter:
@@ -49,6 +57,30 @@ Rules of thumb:
- `deny` always wins.
- If `allow` is non-empty, everything else is treated as blocked.
### Tool groups (shorthands)
For sandbox tool policy, you can use `group:*` entries that expand to multiple tools:
```json5
{
tools: {
sandbox: {
tools: {
allow: ["group:runtime", "group:fs", "group:sessions", "group:memory"]
}
}
}
}
```
Available groups:
- `group:runtime`: `exec`, `bash`, `process`
- `group:fs`: `read`, `write`, `edit`, `apply_patch`
- `group:sessions`: `sessions_list`, `sessions_history`, `sessions_send`, `sessions_spawn`, `session_status`
- `group:memory`: `memory_search`, `memory_get`
Legacy shorthand: `memory` expands to `group:memory`.
## Elevated: exec-only “run on host”
Elevated does **not** grant extra tools; it only affects `exec`.
+41
View File
@@ -56,6 +56,47 @@ Clawdbot mirrors eligible skills into the sandbox workspace (`.../skills`) so
they can be read. With `"rw"`, workspace skills are readable from
`/workspace/skills`.
## Custom bind mounts
`agents.defaults.sandbox.docker.binds` mounts additional host directories into the container.
Format: `host:container:mode` (e.g., `"/home/user/source:/source:rw"`).
Global and per-agent binds are **merged** (not replaced). Under `scope: "shared"`, per-agent binds are ignored.
Example (read-only source + docker socket):
```json5
{
agents: {
defaults: {
sandbox: {
docker: {
binds: [
"/home/user/source:/source:ro",
"/var/run/docker.sock:/var/run/docker.sock"
]
}
}
},
list: [
{
id: "build",
sandbox: {
docker: {
binds: ["/mnt/cache:/cache:rw"]
}
}
}
]
}
}
```
Security notes:
- Binds bypass the sandbox filesystem: they expose host paths with whatever mode you set (`:ro` or `:rw`).
- Sensitive mounts (e.g., `docker.sock`, secrets, SSH keys) should be `:ro` unless absolutely required.
- Combine with `workspaceAccess: "ro"` if you only need read access to the workspace; bind modes stay independent.
- See [Sandbox vs Tool Policy vs Elevated](/gateway/sandbox-vs-tool-policy-vs-elevated) for how binds interact with tool policy and elevated exec.
## Images + setup
Default image: `clawdbot-sandbox:bookworm-slim`
-20
View File
@@ -69,26 +69,6 @@ Equivalent env vars (useful for automation):
- `CLAWDBOT_NO_ONBOARD=1`
- `SHARP_IGNORE_GLOBAL_LIBVIPS=0|1` (default: `1`; avoids `sharp` building against system libvips)
## Install from GitHub (from source)
Use this if you want a source checkout you can edit, or you prefer `clawdbot update`.
```bash
curl -fsSL https://clawd.bot/install.sh | bash -s -- --install-method git --no-onboard
```
What it does:
- Clones `https://github.com/clawdbot/clawdbot.git` into `~/clawdbot` (or `--git-dir`)
- Runs `pnpm install`, `pnpm ui:build` (best-effort), and `pnpm build`
- Adds a wrapper at `~/.local/bin/clawdbot` pointing at `<checkout>/dist/entry.js`
Update later:
```bash
clawdbot update --restart
```
## Global install (manual)
If you already have Node:
+11
View File
@@ -173,6 +173,17 @@ The filtering order is:
Each level can further restrict tools, but cannot grant back denied tools from earlier levels.
If `agents.list[].tools.sandbox.tools` is set, it replaces `tools.sandbox.tools` for that agent.
### Tool groups (shorthands)
Sandbox tool policy supports `group:*` entries that expand to multiple concrete tools:
- `group:runtime`: `exec`, `bash`, `process`
- `group:fs`: `read`, `write`, `edit`, `apply_patch`
- `group:sessions`: `sessions_list`, `sessions_history`, `sessions_send`, `sessions_spawn`, `session_status`
- `group:memory`: `memory_search`, `memory_get`
Legacy shorthand: `memory` expands to `group:memory`.
### Elevated Mode
`tools.elevated` is the global baseline (sender-based allowlist). `agents.list[].tools.elevated` can further restrict elevated for specific agents (both must allow).
+7 -7
View File
@@ -29,17 +29,17 @@ Notes:
# From repo root; set release IDs so Sparkle feed is enabled.
# APP_BUILD must be numeric + monotonic for Sparkle compare.
BUNDLE_ID=com.clawdbot.mac \
APP_VERSION=2026.1.11-1 \
APP_VERSION=2026.1.11-4 \
APP_BUILD="$(git rev-list --count HEAD)" \
BUILD_CONFIG=release \
SIGN_IDENTITY="Developer ID Application: Peter Steinberger (Y5PE65HELJ)" \
scripts/package-mac-app.sh
# Zip for distribution (includes resource forks for Sparkle delta support)
ditto -c -k --sequesterRsrc --keepParent dist/Clawdbot.app dist/Clawdbot-2026.1.11-1.zip
ditto -c -k --sequesterRsrc --keepParent dist/Clawdbot.app dist/Clawdbot-2026.1.11-4.zip
# Optional: also build a styled DMG for humans (drag to /Applications)
scripts/create-dmg.sh dist/Clawdbot.app dist/Clawdbot-2026.1.11-1.dmg
scripts/create-dmg.sh dist/Clawdbot.app dist/Clawdbot-2026.1.11-4.dmg
# Recommended: build + notarize/staple zip + DMG
# First, create a keychain profile once:
@@ -47,26 +47,26 @@ scripts/create-dmg.sh dist/Clawdbot.app dist/Clawdbot-2026.1.11-1.dmg
# --apple-id "<apple-id>" --team-id "<team-id>" --password "<app-specific-password>"
NOTARIZE=1 NOTARYTOOL_PROFILE=clawdbot-notary \
BUNDLE_ID=com.clawdbot.mac \
APP_VERSION=2026.1.11-1 \
APP_VERSION=2026.1.11-4 \
APP_BUILD="$(git rev-list --count HEAD)" \
BUILD_CONFIG=release \
SIGN_IDENTITY="Developer ID Application: Peter Steinberger (Y5PE65HELJ)" \
scripts/package-mac-dist.sh
# Optional: ship dSYM alongside the release
ditto -c -k --keepParent apps/macos/.build/release/Clawdbot.app.dSYM dist/Clawdbot-2026.1.11-1.dSYM.zip
ditto -c -k --keepParent apps/macos/.build/release/Clawdbot.app.dSYM dist/Clawdbot-2026.1.11-4.dSYM.zip
```
## Appcast entry
Use the release note generator so Sparkle renders formatted HTML notes:
```bash
SPARKLE_PRIVATE_KEY_FILE=/Users/steipete/Library/CloudStorage/Dropbox/Backup/Sparkle/ed25519-private-key scripts/make_appcast.sh dist/Clawdbot-2026.1.11-1.zip https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml
SPARKLE_PRIVATE_KEY_FILE=/Users/steipete/Library/CloudStorage/Dropbox/Backup/Sparkle/ed25519-private-key scripts/make_appcast.sh dist/Clawdbot-2026.1.11-4.zip https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml
```
Generates HTML release notes from `CHANGELOG.md` (via [`scripts/changelog-to-html.sh`](https://github.com/clawdbot/clawdbot/blob/main/scripts/changelog-to-html.sh)) and embeds them in the appcast entry.
Commit the updated `appcast.xml` alongside the release assets (zip + dSYM) when publishing.
## Publish & verify
- Upload `Clawdbot-2026.1.11-1.zip` (and `Clawdbot-2026.1.11-1.dSYM.zip`) to the GitHub release for tag `v2026.1.11-1`.
- Upload `Clawdbot-2026.1.11-4.zip` (and `Clawdbot-2026.1.11-4.dSYM.zip`) to the GitHub release for tag `v2026.1.11-4`.
- Ensure the raw appcast URL matches the baked feed: `https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml`.
- Sanity checks:
- `curl -I https://raw.githubusercontent.com/clawdbot/clawdbot/main/appcast.xml` returns 200.
+1
View File
@@ -148,6 +148,7 @@ export default {
clawdbot plugins list
clawdbot plugins info <id>
clawdbot plugins install <path> # add a local file/dir to plugins.load.paths
clawdbot plugins install ./extensions/voice-call # relative path ok
clawdbot plugins install ./plugin.tgz # install from a local tarball
clawdbot plugins install @clawdbot/voice-call # install from npm
clawdbot plugins enable <id>
+69 -65
View File
@@ -1,5 +1,5 @@
---
summary: "Voice Call plugin: Twilio-backed outbound calls (plugin install + config + CLI)"
summary: "Voice Call plugin: outbound + inbound calls via Twilio/Telnyx (plugin install + config + CLI)"
read_when:
- You want to place an outbound voice call from Clawdbot
- You are configuring or developing the voice-call plugin
@@ -7,20 +7,19 @@ read_when:
# Voice Call (plugin)
Outbound voice calls for Clawdbot via a plugin.
If you’re new to plugins, start with [Plugins](/plugin): what they are, where
they live on disk, and how install/config works.
Voice calls for Clawdbot via a plugin. Supports outbound notifications and
multi-turn conversations with inbound policies.
Current providers:
- `twilio` (real calls)
- `log` (dev fallback; no network)
- `twilio` (Programmable Voice + Media Streams)
- `telnyx` (Call Control v2)
- `mock` (dev/no network)
Quick mental model:
- Install plugin
- Restart Gateway
- Configure under `plugins.entries.voice-call.config`
- Use `clawdbot voicecall …` or the `voice_call` tool
- Use `clawdbot voicecall ...` or the `voice_call` tool
## Where it runs (local vs remote)
@@ -28,23 +27,6 @@ The Voice Call plugin runs **inside the Gateway process**.
If you use a remote Gateway, install/configure the plugin on the **machine running the Gateway**, then restart the Gateway to load it.
## Twilio setup (getting the credentials)
You’ll need:
- **Account SID** (`AC...`)
- **Auth Token** (treat like a password)
- A Twilio **Voice-capable phone number** to use as `from` (E.164, like `+15551234567`)
Steps:
1) Create/sign in to Twilio: `https://www.twilio.com/`
2) Open the Twilio Console: `https://console.twilio.com/`
3) Find your **Account SID** + **Auth Token** (typically on the Console dashboard / Account settings).
4) Buy (or select) a Twilio phone number with **Voice** capability:
- Console → Phone Numbers → Manage → Buy a number
5) (Trial accounts) You may need to verify the destination `to` numbers before Twilio allows outbound calling.
Tip: keep `twilio.authToken` out of git; store it in `~/.clawdbot/clawdbot.json` or your Control UI config, not in repo files.
## Install
### Option A: install from npm (recommended)
@@ -53,34 +35,13 @@ Tip: keep `twilio.authToken` out of git; store it in `~/.clawdbot/clawdbot.json`
clawdbot plugins install @clawdbot/voice-call
```
This downloads the package, extracts it into `~/.clawdbot/extensions/`, and enables it in `clawdbot.json`.
Restart the Gateway afterwards.
### Option B: install from a local folder (dev, no copying)
This keeps the plugin in-place (great for iterating locally) and adds the folder
to `plugins.load.paths`.
```bash
clawdbot plugins install /absolute/path/to/voice-call
```
If your plugin has dependencies, install them in that folder (so it has a
`node_modules`):
```bash
cd /absolute/path/to/voice-call && pnpm install
```
Restart the Gateway afterwards.
### Option C: copy into the global extensions folder (dev)
```bash
mkdir -p ~/.clawdbot/extensions
cp -R extensions/voice-call ~/.clawdbot/extensions/voice-call
cd ~/.clawdbot/extensions/voice-call && pnpm install
clawdbot plugins install ./extensions/voice-call
cd ./extensions/voice-call && pnpm install
```
Restart the Gateway afterwards.
@@ -96,13 +57,33 @@ Set config under `plugins.entries.voice-call.config`:
"voice-call": {
enabled: true,
config: {
provider: "twilio",
provider: "twilio", // or "telnyx" | "mock"
fromNumber: "+15550001234",
toNumber: "+15550005678",
twilio: {
accountSid: "ACxxxxxxxx",
authToken: "…",
from: "+15551234567",
statusCallbackUrl: "https://example.com/twilio-status", // optional
twimlUrl: "https://example.com/twiml" // optional
authToken: "..."
},
// Webhook server
serve: {
port: 3334,
path: "/voice/webhook"
},
// Public exposure (pick one)
// publicUrl: "https://example.ngrok.app/voice/webhook",
// tunnel: { provider: "ngrok" },
// tailscale: { mode: "funnel", path: "/voice/webhook" }
outbound: {
defaultMode: "notify" // notify | conversation
},
streaming: {
enabled: true,
streamPath: "/voice/stream"
}
}
}
@@ -111,34 +92,57 @@ Set config under `plugins.entries.voice-call.config`:
}
```
Dev fallback:
Notes:
- Twilio/Telnyx require a **publicly reachable** webhook URL.
- `mock` is a local dev provider (no network calls).
- `skipSignatureVerification` is for local testing only.
## Inbound calls
Inbound policy defaults to `disabled`. To enable inbound calls, set:
```json5
{ provider: "log" }
{
inboundPolicy: "allowlist",
allowFrom: ["+15550001234"],
inboundGreeting: "Hello! How can I help?"
}
```
Notes:
- `twilio.authToken` is treated as sensitive in the Control UI schema hints.
Auto-responses use the agent system. Tune with:
- `responseModel`
- `responseSystemPrompt`
- `responseTimeoutMs`
## CLI
```bash
clawdbot voicecall start --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall status --sid CAxxxxxxxx
clawdbot voicecall call --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall continue --call-id <id> --message "Any questions?"
clawdbot voicecall speak --call-id <id> --message "One moment"
clawdbot voicecall end --call-id <id>
clawdbot voicecall status --call-id <id>
clawdbot voicecall tail
clawdbot voicecall expose --mode funnel
```
## Agent tool
Tool name: `voice_call`
- `mode`: `"call" | "status"` (default: `call`)
- `to`: required for `call`
- `sid`: required for `status`
- `message`: optional
Actions:
- `initiate_call` (message, to?, mode?)
- `continue_call` (callId, message)
- `speak_to_user` (callId, message)
- `end_call` (callId)
- `get_status` (callId)
This repo ships a matching skill doc at `skills/voice-call/SKILL.md`.
## Gateway RPC
- `voicecall.start` (`to`, optional `message`)
- `voicecall.status` (`sid`)
- `voicecall.initiate` (`to?`, `message`, `mode?`)
- `voicecall.continue` (`callId`, `message`)
- `voicecall.speak` (`callId`, `message`)
- `voicecall.end` (`callId`)
- `voicecall.status` (`callId`)
+3 -1
View File
@@ -47,7 +47,7 @@ Minimal config:
- To ignore all DMs: set `discord.dm.enabled=false` or `discord.dm.policy="disabled"`.
8. Group DMs are ignored by default; enable via `discord.dm.groupEnabled` and optionally restrict by `discord.dm.groupChannels`.
9. Optional guild rules: set `discord.guilds` keyed by guild id (preferred) or slug, with per-channel rules.
10. Optional native commands: set `commands.native: true` to register native commands in Discord; set `commands.native: false` to clear previously registered native commands. Text commands are controlled by `commands.text` and must be sent as standalone `/...` messages. Use `commands.useAccessGroups: false` to bypass access-group checks for commands.
10. Optional native commands: `commands.native` defaults to `"auto"` (on for Discord/Telegram, off for Slack). Override with `discord.commands.native: true|false|"auto"`; `false` clears previously registered commands. Text commands are controlled by `commands.text` and must be sent as standalone `/...` messages. Use `commands.useAccessGroups: false` to bypass access-group checks for commands.
- Full command list + config: [Slash commands](/tools/slash-commands)
11. Optional guild context history: set `discord.historyLimit` (default 20, falls back to `messages.groupChat.historyLimit`) to include the last N guild messages as context when replying to a mention. Set `0` to disable.
12. Reactions: the agent can trigger reactions via the `discord` tool (gated by `discord.actions.*`).
@@ -158,6 +158,8 @@ Notes:
- `agents.list[].groupChat.mentionPatterns` (or `messages.groupChat.mentionPatterns`) also count as mentions for guild messages.
- Multi-agent override: set per-agent patterns on `agents.list[].groupChat.mentionPatterns`.
- If `channels` is present, any channel not listed is denied by default.
- Bot-authored messages are ignored by default; set `discord.allowBots=true` to allow them (own messages remain filtered).
- Warning: If you allow replies to other bots (`discord.allowBots=true`), prevent bot-to-bot reply loops with `requireMention`, `discord.guilds.*.channels.<id>.users` allowlists, and/or clear guardrails in `AGENTS.md` and `SOUL.md`.
### 6) Verify it works
1. Start the gateway.
+4 -3
View File
@@ -33,7 +33,7 @@ Minimal config:
- `channel_rename`
- `pin_added`, `pin_removed`
5) Invite the bot to channels you want it to read.
6) Slash Commands → create `/clawd` if you use `slack.slashCommand`. If you enable `commands.native`, add slash commands for the built-in chat commands (same names as `/help`).
6) Slash Commands → create `/clawd` if you use `slack.slashCommand`. If you enable native commands, add one slash command per built-in command (same names as `/help`). Native defaults to off for Slack unless you set `slack.commands.native: true` (global `commands.native` is `"auto"` which leaves Slack off).
7) App Home → enable the **Messages Tab** so users can DM the bot.
Use the manifest below so scopes and events stay in sync.
@@ -138,7 +138,7 @@ Use this Slack app manifest to create the app quickly (adjust the name/command i
}
```
If you enable `commands.native`, add one `slash_commands` entry per command you want to expose (matching the `/help` list).
If you enable native commands, add one `slash_commands` entry per command you want to expose (matching the `/help` list). Override with `slack.commands.native`.
## Scopes (current vs optional)
Slack's Conversations API is type-scoped: you only need the scopes for the
@@ -257,7 +257,7 @@ For fine-grained control, use these tags in agent responses:
- DMs share the `main` session (like WhatsApp/Telegram).
- Channels map to `agent:<agentId>:slack:channel:<channelId>` sessions.
- Slash commands use `agent:<agentId>:slack:slash:<userId>` sessions (prefix configurable via `slack.slashCommand.sessionPrefix`).
- Native command registration is controlled by `commands.native`; text commands require standalone `/...` messages and can be disabled with `commands.text: false`. Slack slash commands are managed in the Slack app and are not removed automatically. Use `commands.useAccessGroups: false` to bypass access-group checks for commands.
- Native command registration uses `commands.native` (global default `"auto"` → Slack off) and can be overridden per-workspace with `slack.commands.native`. Text commands require standalone `/...` messages and can be disabled with `commands.text: false`. Slack slash commands are managed in the Slack app and are not removed automatically. Use `commands.useAccessGroups: false` to bypass access-group checks for commands.
- Full command list + config: [Slash commands](/tools/slash-commands)
## DM security (pairing)
@@ -299,5 +299,6 @@ Slack tool actions can be gated with `slack.actions.*`:
- Multi-agent override: set per-agent patterns on `agents.list[].groupChat.mentionPatterns`.
- Reaction notifications follow `slack.reactionNotifications` (use `reactionAllowlist` with mode `allowlist`).
- Bot-authored messages are ignored by default; enable via `slack.allowBots` or `slack.channels.<id>.allowBots`.
- Warning: If you allow replies to other bots (`slack.allowBots=true` or `slack.channels.<id>.allowBots=true`), prevent bot-to-bot reply loops with `requireMention`, `slack.channels.<id>.users` allowlists, and/or clear guardrails in `AGENTS.md` and `SOUL.md`.
- For the Slack tool, reaction removal semantics are in [/tools/reactions](/tools/reactions).
- Attachments are downloaded to the media store when permitted and under the size limit.
+2 -2
View File
@@ -174,7 +174,7 @@ Private chats can include `message_thread_id` in some edge cases. Clawdbot keeps
- `clawdbot pairing list telegram`
- `clawdbot pairing approve telegram <CODE>`
- Pairing is the default token exchange used for Telegram DMs. Details: [Pairing](/start/pairing)
- `telegram.allowFrom` accepts numeric user IDs (recommended) or `@username` entries.
- `telegram.allowFrom` accepts numeric user IDs (recommended) or `@username` entries. It is **not** the bot username; use the human sender’s ID (get it from `@userinfobot` or the `from.id` field in the gateway log).
### Group access
@@ -315,5 +315,5 @@ Provider options:
Related global options:
- `agents.list[].groupChat.mentionPatterns` (mention gating patterns).
- `messages.groupChat.mentionPatterns` (global fallback).
- `commands.native`, `commands.text`, `commands.useAccessGroups` (command behavior).
- `commands.native` (defaults to `"auto"` → on for Telegram/Discord, off for Slack), `commands.text`, `commands.useAccessGroups` (command behavior). Override with `telegram.commands.native`.
- `messages.responsePrefix`, `messages.ackReaction`, `messages.ackReactionScope`, `messages.removeAckAfterReply`.
+127
View File
@@ -0,0 +1,127 @@
---
summary: "Vector memory search design plan (per-agent, watch/lazy sync, storage)"
read_when:
- Designing or implementing vector memory search
- Adding embedding providers or sync behavior
---
# Vector Memory Search — Design Plan
Goal: semantic search over **agent memory files** only, with minimal deps and
good UX defaults. Default enabled. Per-agent overrides.
## Scope
- Sources: `MEMORY.md` + `memory/YYYY-MM-DD.md` inside the agent workspace.
- No indexing outside the workspace. No hidden paths.
- No QMD-style query expansion or rerank in v1.
## Config Shape
Location: `agents.defaults.memorySearch` + `agents.list[].memorySearch`.
```json5
agents: {
defaults: {
memorySearch: {
enabled: true,
provider: "openai", // "openai" | "local"
fallback: "openai", // "openai" | "none"
model: "text-embedding-3-small",
store: {
driver: "sqlite",
path: "~/.clawdbot/memory/{agentId}.sqlite"
},
chunking: {
tokens: 400,
overlap: 80
},
sync: {
onSessionStart: true,
onSearch: true, // LazySync
watch: true, // default on
watchDebounceMs: 1500,
intervalMinutes: 0
},
query: {
maxResults: 6,
minScore: 0.35
}
}
},
list: [
{ id: "peter", memorySearch: { provider: "local", sync: { watch: false } } }
]
}
```
## Storage
Per-agent DB (default): `~/.clawdbot/memory/{agentId}.sqlite`.
Tables (v1):
- `files(path PRIMARY KEY, hash, mtime, size)`
- `chunks(id PRIMARY KEY, path, start_line, end_line, hash, text, embedding, updated_at)`
Notes:
- `hash` = content hash of chunk text.
- `embedding` stored as float[] (sqlite vec extension optional); if not using vec,
store as JSON and do linear scan in memory for small corpora.
## Embedding Providers
Interface (core):
- `embedQuery(text): number[]`
- `embedBatch(texts[]): number[][]`
Providers:
- `openai` (default): OpenAI embeddings via existing keys.
- `local` (optional): node-llama-cpp (GGUF).
- Fallback: when `provider: "local"` fails, fallback to OpenAI unless `fallback: "none"`.
## Index Pipeline
1) Resolve memory file list (workspace only).
2) Read file, compute file hash/mtime.
3) Chunk by headings + token cap (overlap).
4) Embed only changed chunks (hash compare).
5) Upsert `chunks` rows, prune deleted files.
Chunking:
- Prefer heading-aware splits.
- Max tokens + overlap; keep line ranges for snippets.
## Sync Strategy
Default: **watch + lazy + session-start**
- `watch`: chokidar on `MEMORY.md` + `memory/**/*.md` (debounced).
- `onSearch`: if dirty, sync before search (LazySync).
- `onSessionStart`: warm index once per session.
- `intervalMinutes`: optional for long-lived sessions.
If workspace access is read-only or missing: disable writes; return “not indexed”.
## Query Flow
1) Embed query.
2) Cosine similarity over all chunk embeddings.
3) Return top K with `{path, startLine, endLine, snippet, score}`.
4) Model may call `memory_get` when full context needed.
Optional v2: add FTS5 + RRF merge (FTS + vector) for quality.
## Tool + CLI
Tools:
- `memory_search { query, maxResults?, minScore? }`
- `memory_get { path, from?, lines? }`
CLI (optional):
- `clawdbot memory index|search|status`
## Security + Permissions
- Indexer reads only memory files in workspace.
- No scanning outside workspace; no “sneak” reads.
- Respect sandbox `workspaceAccess` (ro = read-only; none = disabled).
## Tests
- Chunking boundaries + line ranges.
- Hash-based incremental updates.
- Search ranking (cosine).
- Watcher debounce (fake fs).
## Rollout
- Default enabled; if no memory files, index is empty (silent).
- No migration needed.
-1
View File
@@ -88,7 +88,6 @@ git commit -m "Add Clawd workspace"
- **Peekaboo** — Fast macOS screenshots with optional AI vision analysis.
- **camsnap** — Capture frames, clips, or motion alerts from RTSP/ONVIF security cams.
- **oracle** — OpenAI-ready agent CLI with session replay and browser control.
- **qmd** — Hybrid markdown search (BM25 + vectors + rerank) with a local server for agents.
- **eightctl** — Control your sleep, from the terminal.
- **imsg** — Send, read, stream iMessage & SMS.
- **wacli** — WhatsApp CLI: sync, search, send.
-10
View File
@@ -46,16 +46,6 @@ Capture what matters. Decisions, context, things to remember. Skip the secrets u
- When you make a mistake → document it so future-you doesn't repeat it
- **Text > Brain** 📝
### 🧠 Memory Recall - Use qmd!
When you need to remember something from the past, use `qmd` instead of grepping files:
```bash
qmd query "what happened at Christmas" # Semantic search with reranking
qmd search "specific phrase" # BM25 keyword search
qmd vsearch "conceptual question" # Pure vector similarity
```
Index your memory folder: `qmd index memory/`
Vectors + BM25 + reranking finds things even with different wording.
## Safety
- Don't exfiltrate private data. Ever.
+2
View File
@@ -89,6 +89,8 @@ clawdbot gateway --port 18789
Now message the assistant number from your allowlisted phone.
When onboarding finishes, we auto-open the dashboard with your gateway token and print the tokenized link. To reopen later: `clawdbot dashboard`.
## Give the agent a workspace (AGENTS)
Clawd reads operating instructions and “memory” from its workspace directory.
+22
View File
@@ -83,6 +83,10 @@ pnpm clawdbot onboard
The wizard can also build UI assets automatically. After onboarding, you typically run the Gateway on port **18789**.
### How do I open the dashboard after onboarding?
The wizard now opens your browser with a tokenized dashboard URL right after onboarding and also prints the full link (with token) in the summary. Keep that tab open; if it didn’t launch, copy/paste the printed URL on the same machine. Tokens stay local to your host—nothing is fetched from the browser.
### What runtime do I need?
Node **>= 22** is required. `pnpm` is recommended; `bun` is optional.
@@ -116,10 +120,22 @@ Not currently. Clawdbot doesn’t ship a Bedrock provider today. If you must use
Clawdbot supports **OpenAI Code (Codex)** via OAuth or by reusing your Codex CLI login (`~/.codex/auth.json`). The wizard can import the CLI login or run the OAuth flow and will set the default model to `openai-codex/gpt-5.2` when appropriate. See [Model providers](/concepts/model-providers) and [Wizard](/start/wizard).
### Is a local model OK for casual chats?
Usually no. Clawdbot needs large context + strong safety; small cards truncate and leak. If you must, run the **largest** MiniMax M2.1 build you can locally (LM Studio) and see [/gateway/local-models](/gateway/local-models). Smaller/quantized models increase prompt-injection risk — see [Security](/gateway/security).
### How do I keep hosted model traffic in a specific region?
Pick region-pinned endpoints. OpenRouter exposes US-hosted options for MiniMax, Kimi, and GLM; choose the US-hosted variant to keep data in-region. You can still list Anthropic/OpenAI alongside these by using `models.mode: "merge"` so fallbacks stay available while respecting the regioned provider you select.
### Can I use Bun?
Bun is supported for faster TypeScript execution, but **WhatsApp requires Node** in this ecosystem. The wizard lets you pick the runtime; choose **Node** if you use WhatsApp.
### Telegram: what goes in `allowFrom`?
`telegram.allowFrom` is **the human sender’s Telegram user ID** (numeric, recommended) or `@username`. It is not the bot username. To find your ID, DM `@userinfobot` or read the `from.id` in the gateway log for a DM. See [/providers/telegram](/providers/telegram#access-control-dms--groups).
### Can multiple people use one WhatsApp number with different Clawdbots?
Yes, via **multi‑agent routing**. Bind each sender’s WhatsApp **DM** (peer `kind: "dm"`, sender E.164 like `+15551234567`) to a different `agentId`, so each person gets their own workspace and session store. Replies still come from the **same WhatsApp account**, and DM access control (`whatsapp.dmPolicy` / `whatsapp.allowFrom`) is global per WhatsApp account. See [Multi-Agent Routing](/concepts/multi-agent) and [WhatsApp](/providers/whatsapp).
@@ -201,6 +217,10 @@ ClawdHub installs into `./skills` under your current directory; Clawdbot treats
Yes. See [Sandboxing](/gateway/sandboxing). For Docker-specific setup (full gateway in Docker or sandbox images), see [Docker](/install/docker).
### How do I bind a host folder into the sandbox?
Set `agents.defaults.sandbox.docker.binds` to `["host:path:mode"]` (e.g., `"/home/user/src:/src:ro"`). Global + per-agent binds merge; per-agent binds are ignored when `scope: "shared"`. Use `:ro` for anything sensitive and remember binds bypass the sandbox filesystem walls. See [Sandboxing](/gateway/sandboxing#custom-bind-mounts) and [Sandbox vs Tool Policy vs Elevated](/gateway/sandbox-vs-tool-policy-vs-elevated#bind-mounts-security-quick-check) for examples and safety notes.
### How does memory work?
Clawdbot memory is just Markdown files in the agent workspace:
@@ -756,6 +776,8 @@ Facts (from code):
- The UI can import `?token=...` (and/or `?password=...`) once, then strips it from the URL.
Fix:
- Fastest: `clawdbot dashboard` (prints + copies tokenized link, tries to open; shows SSH hint if headless).
- If remote, tunnel first: `ssh -N -L 18789:127.0.0.1:18789 user@host` then open `http://127.0.0.1:18789/?token=...`.
- Set `gateway.auth.token` (or `CLAWDBOT_GATEWAY_TOKEN`) on the gateway host.
- In the Control UI settings, paste the same token (or refresh with a one-time `?token=...` link).
+128
View File
@@ -150,7 +150,13 @@ HTTP API:
- Snapshot/screenshot: `GET /snapshot`, `POST /screenshot`
- Actions: `POST /navigate`, `POST /act`
- Hooks: `POST /hooks/file-chooser`, `POST /hooks/dialog`
- Downloads: `POST /download`, `POST /wait/download`
- Debugging: `GET /console`, `POST /pdf`
- Debugging: `GET /errors`, `GET /requests`, `POST /trace/start`, `POST /trace/stop`, `POST /highlight`
- Network: `POST /response/body`
- State: `GET /cookies`, `POST /cookies/set`, `POST /cookies/clear`
- State: `GET /storage/:kind`, `POST /storage/:kind/set`, `POST /storage/:kind/clear`
- Settings: `POST /set/offline`, `POST /set/headers`, `POST /set/credentials`, `POST /set/geolocation`, `POST /set/media`, `POST /set/timezone`, `POST /set/locale`, `POST /set/device`
All endpoints accept `?profile=<name>`.
@@ -175,12 +181,17 @@ you swap local/remote browsers and profiles.
## CLI quick reference
All commands accept `--browser-profile <name>` to target a specific profile.
All commands also accept `--json` for machine-readable output (stable payloads).
Basics:
- `clawdbot browser status`
- `clawdbot browser start`
- `clawdbot browser stop`
- `clawdbot browser tabs`
- `clawdbot browser tab`
- `clawdbot browser tab new`
- `clawdbot browser tab select 2`
- `clawdbot browser tab close 2`
- `clawdbot browser open https://example.com`
- `clawdbot browser focus abcd1234`
- `clawdbot browser close abcd1234`
@@ -194,8 +205,12 @@ Inspection:
- `clawdbot browser snapshot --format aria --limit 200`
- `clawdbot browser snapshot --interactive --compact --depth 6`
- `clawdbot browser snapshot --selector "#main" --interactive`
- `clawdbot browser snapshot --frame "iframe#main" --interactive`
- `clawdbot browser console --level error`
- `clawdbot browser errors --clear`
- `clawdbot browser requests --filter api --clear`
- `clawdbot browser pdf`
- `clawdbot browser responsebody "**/api" --max-chars 5000`
Actions:
- `clawdbot browser navigate https://example.com`
@@ -205,13 +220,38 @@ Actions:
- `clawdbot browser type 23 "hello" --submit`
- `clawdbot browser press Enter`
- `clawdbot browser hover 44`
- `clawdbot browser scrollintoview e12`
- `clawdbot browser drag 10 11`
- `clawdbot browser select 9 OptionA OptionB`
- `clawdbot browser download e12 /tmp/report.pdf`
- `clawdbot browser waitfordownload /tmp/report.pdf`
- `clawdbot browser upload /tmp/file.pdf`
- `clawdbot browser fill --fields '[{"ref":"1","type":"text","value":"Ada"}]'`
- `clawdbot browser dialog --accept`
- `clawdbot browser wait --text "Done"`
- `clawdbot browser wait "#main" --url "**/dash" --load networkidle --fn "window.ready===true"`
- `clawdbot browser evaluate --fn '(el) => el.textContent' --ref 7`
- `clawdbot browser highlight e12`
- `clawdbot browser trace start`
- `clawdbot browser trace stop`
State:
- `clawdbot browser cookies`
- `clawdbot browser cookies set session abc123 --url "https://example.com"`
- `clawdbot browser cookies clear`
- `clawdbot browser storage local get`
- `clawdbot browser storage local set theme dark`
- `clawdbot browser storage session clear`
- `clawdbot browser set offline on`
- `clawdbot browser set headers --json '{"X-Debug":"1"}'`
- `clawdbot browser set credentials user pass`
- `clawdbot browser set credentials --clear`
- `clawdbot browser set geo 37.7749 -122.4194 --origin "https://example.com"`
- `clawdbot browser set geo --clear`
- `clawdbot browser set media dark`
- `clawdbot browser set timezone America/New_York`
- `clawdbot browser set locale en-US`
- `clawdbot browser set device "iPhone 14"`
Notes:
- `upload` and `dialog` are **arming** calls; run them before the click/press
@@ -221,10 +261,98 @@ Notes:
- `--format ai` (default when Playwright is installed): returns an AI snapshot with numeric refs (`aria-ref="<n>"`).
- `--format aria`: returns the accessibility tree (no refs; inspection only).
- Role snapshot options (`--interactive`, `--compact`, `--depth`, `--selector`) force a role-based snapshot with refs like `ref=e12`.
- `--frame "<iframe selector>"` scopes role snapshots to an iframe (pairs with role refs like `e12`).
- `--interactive` outputs a flat, easy-to-pick list of interactive elements (best for driving actions).
- `click`/`type`/etc require a `ref` from `snapshot` (either numeric `12` or role ref `e12`).
CSS selectors are intentionally not supported for actions.
## Snapshots and refs
Clawdbot supports two “snapshot” styles:
- **AI snapshot (numeric refs)**: `clawdbot browser snapshot` (default; `--format ai`)
- Output: a text snapshot that includes numeric refs.
- Actions: `clawdbot browser click 12`, `clawdbot browser type 23 "hello"`.
- Internally, the ref is resolved via Playwright’s `aria-ref`.
- **Role snapshot (role refs like `e12`)**: `clawdbot browser snapshot --interactive` (or `--compact`, `--depth`, `--selector`, `--frame`)
- Output: a role-based list/tree with `[ref=e12]` (and optional `[nth=1]`).
- Actions: `clawdbot browser click e12`, `clawdbot browser highlight e12`.
- Internally, the ref is resolved via `getByRole(...)` (plus `nth()` for duplicates).
Ref behavior:
- Refs are **not stable across navigations**; if something fails, re-run `snapshot` and use a fresh ref.
- If the role snapshot was taken with `--frame`, role refs are scoped to that iframe until the next role snapshot.
## Wait power-ups
You can wait on more than just time/text:
- Wait for URL (globs supported by Playwright):
- `clawdbot browser wait --url "**/dash"`
- Wait for load state:
- `clawdbot browser wait --load networkidle`
- Wait for a JS predicate:
- `clawdbot browser wait --fn "window.ready===true"`
- Wait for a selector to become visible:
- `clawdbot browser wait "#main"`
These can be combined:
```bash
clawdbot browser wait "#main" \
--url "**/dash" \
--load networkidle \
--fn "window.ready===true" \
--timeout-ms 15000
```
## Debug workflows
When an action fails (e.g. “not visible”, “strict mode violation”, “covered”):
1. `clawdbot browser snapshot --interactive`
2. Use `click <ref>` / `type <ref>` (prefer role refs in interactive mode)
3. If it still fails: `clawdbot browser highlight <ref>` to see what Playwright is targeting
4. If the page behaves oddly:
- `clawdbot browser errors --clear`
- `clawdbot browser requests --filter api --clear`
5. For deep debugging: record a trace:
- `clawdbot browser trace start`
- reproduce the issue
- `clawdbot browser trace stop` (prints `TRACE:<path>`)
## JSON output
`--json` is for scripting and structured tooling.
Examples:
```bash
clawdbot browser status --json
clawdbot browser snapshot --interactive --json
clawdbot browser requests --filter api --json
clawdbot browser cookies --json
```
Role snapshots in JSON include `refs` plus a small `stats` block (lines/chars/refs/interactive) so tools can reason about payload size and density.
## State and environment knobs
These are useful for “make the site behave like X” workflows:
- Cookies: `cookies`, `cookies set`, `cookies clear`
- Storage: `storage local|session get|set|clear`
- Offline: `set offline on|off`
- Headers: `set headers --json '{"X-Debug":"1"}'` (or `--clear`)
- HTTP basic auth: `set credentials user pass` (or `--clear`)
- Geolocation: `set geo <lat> <lon> --origin "https://example.com"` (or `--clear`)
- Media: `set media dark|light|no-preference|none`
- Timezone / locale: `set timezone ...`, `set locale ...`
- Device / viewport:
- `set device "iPhone 14"` (Playwright device presets)
- `set viewport 1280 720`
## Security & privacy
- The clawd browser profile may contain logged-in sessions; treat it as sensitive.
+1 -1
View File
@@ -153,7 +153,7 @@ Core parameters:
- `maxBytesMb` (optional size cap)
Notes:
- Only available when `agents.defaults.imageModel` is configured (primary or fallbacks).
- Only available when `agents.defaults.imageModel` is configured (primary or fallbacks), or when an implicit image model can be inferred from your default model + configured auth (best-effort pairing).
- Uses the image model directly (independent of the main chat model).
### `message`
+5 -4
View File
@@ -24,7 +24,7 @@ They run immediately, are stripped before the model sees the message, and the re
```json5
{
commands: {
native: false,
native: "auto",
text: true,
config: false,
debug: false,
@@ -36,9 +36,10 @@ They run immediately, are stripped before the model sees the message, and the re
- `commands.text` (default `true`) enables parsing `/...` in chat messages.
- On surfaces without native commands (WhatsApp/WebChat/Signal/iMessage/MS Teams), text commands still work even if you set this to `false`.
- `commands.native` (default `false`) registers native commands on Discord/Slack/Telegram.
- `false` clears previously registered commands on Discord/Telegram at startup.
- Slack commands are managed in the Slack app and are not removed automatically.
- `commands.native` (default `"auto"`) registers native commands.
- Auto: on for Discord/Telegram; off for Slack (until you add slash commands); ignored for providers without native support.
- Set `discord.commands.native`, `telegram.commands.native`, or `slack.commands.native` to override per provider (bool or `"auto"`).
- `false` clears previously registered commands on Discord/Telegram at startup. Slack commands are managed in the Slack app and are not removed automatically.
- `commands.config` (default `false`) enables `/config` (reads/writes `clawdbot.json`).
- `commands.debug` (default `false`) enables `/debug` (runtime-only overrides).
- `commands.useAccessGroups` (default `true`) enforces allowlists/policies for commands.
+1
View File
@@ -20,6 +20,7 @@ Primary goals:
Use `sessions_spawn`:
- Starts a sub-agent run (`deliver: false`, global lane: `subagent`)
- Then runs an announce step and posts the announce reply to the requester chat provider
- Default model: inherits the caller unless you set `agents.defaults.subagents.model` (or per-agent `agents.list[].subagents.model`); an explicit `sessions_spawn.model` still wins.
Tool params:
- `task` (required)
+12
View File
@@ -18,3 +18,15 @@ Key references:
Authentication is enforced at the WebSocket handshake via `connect.params.auth`
(token or password). See `gateway.auth` in [Gateway configuration](/gateway/configuration).
## Fast path (recommended)
- After onboarding, the CLI now auto-opens the dashboard with your token and prints the same tokenized link.
- Re-open anytime: `clawdbot dashboard` (copies link, opens browser if possible, shows SSH hint if headless).
- The token stays local (query param only); the UI strips it after first load and saves it in localStorage.
## If you see “unauthorized” / 1008
- Run `clawdbot dashboard` to get a fresh tokenized link.
- Ensure the gateway is reachable (local: `clawdbot status`; remote: SSH tunnel `ssh -N -L 18789:127.0.0.1:18789 user@host` then open `http://127.0.0.1:18789/?token=...`).
- In the dashboard settings, paste the same token you configured in `gateway.auth.token` (or `CLAWDBOT_GATEWAY_TOKEN`).
+19
View File
@@ -0,0 +1,19 @@
# Changelog
## 0.1.0
### Highlights
- Initial release of the @clawdbot/voice-call plugin.
### Changes
- Providers: Twilio (Programmable Voice + Media Streams), Telnyx (Call Control v2), and mock dev provider.
- Outbound calls: notify mode with auto-hangup + conversation mode for multi-turn calls.
- Inbound calls: policies (disabled/allowlist/open), allowlist matching, and inbound greeting.
- Webhooks: local server with configurable bind/port/path plus publicUrl override.
- Exposure: ngrok and Tailscale (serve/funnel) tunnel helpers; dev-only signature bypass for ngrok free tier.
- Streaming: OpenAI Realtime STT with media stream WebSocket + partial/final transcripts.
- Speech: OpenAI TTS (model/voice/instructions) with Twilio <Say> fallback.
- Tooling: `voice_call` tool with action-based API for initiate/continue/speak/end/status.
- CLI: `clawdbot voicecall` commands (call/start/continue/speak/end/status/tail/expose).
- RPC: gateway methods (`voicecall.initiate|continue|speak|end|status` + legacy `voicecall.start`).
- Persistence: JSONL call logs with `voicecall tail` for live inspection.
+53 -54
View File
@@ -2,8 +2,10 @@
Official Voice Call plugin for **Clawdbot**.
- Provider: **Twilio** (real outbound calls)
- Dev fallback: `log` (no network)
Providers:
- **Twilio** (Programmable Voice + Media Streams)
- **Telnyx** (Call Control v2)
- **Mock** (dev/no network)
Docs: `https://docs.clawd.bot/plugins/voice-call`
Plugin system: `https://docs.clawd.bot/plugin`
@@ -26,83 +28,80 @@ cp -R extensions/voice-call ~/.clawdbot/extensions/voice-call
cd ~/.clawdbot/extensions/voice-call && pnpm install
```
### Option C: add via config (custom path)
```json5
{
plugins: {
load: { paths: ["/absolute/path/to/voice-call/index.ts"] },
entries: { "voice-call": { enabled: true, config: { provider: "log" } } }
}
}
```
Restart the Gateway after changes.
## Config
Put under `plugins.entries.voice-call.config`:
```json5
{
provider: "twilio",
provider: "twilio", // or "telnyx" | "mock"
fromNumber: "+15550001234",
toNumber: "+15550005678",
twilio: {
accountSid: "ACxxxxxxxx",
authToken: "your_token",
from: "+15551234567",
statusCallbackUrl: "https://example.com/twilio-status", // optional
twimlUrl: "https://example.com/twiml" // optional, else auto-generates <Say>
authToken: "your_token"
},
// Webhook server
serve: {
port: 3334,
path: "/voice/webhook"
},
// Public exposure (pick one):
// publicUrl: "https://example.ngrok.app/voice/webhook",
// tunnel: { provider: "ngrok" },
// tailscale: { mode: "funnel", path: "/voice/webhook" }
outbound: {
defaultMode: "notify" // or "conversation"
},
streaming: {
enabled: true,
streamPath: "/voice/stream"
}
}
```
Dev fallback (no network):
```json5
{ provider: "log" }
```
## Twilio credentials (quick notes)
You’ll need a Twilio account, your **Account SID**, your **Auth Token**, and a Twilio **Voice-capable** phone number to use as `from`.
- Signup: `https://www.twilio.com/`
- Console: `https://console.twilio.com/`
Full setup guide: `https://docs.clawd.bot/plugins/voice-call`
Notes:
- Twilio/Telnyx require a **publicly reachable** webhook URL.
- `mock` is a local dev provider (no network calls).
## CLI
```bash
clawdbot voicecall start --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall status --sid CAxxxxxxxx
clawdbot voicecall call --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall continue --call-id <id> --message "Any questions?"
clawdbot voicecall speak --call-id <id> --message "One moment"
clawdbot voicecall end --call-id <id>
clawdbot voicecall status --call-id <id>
clawdbot voicecall tail
clawdbot voicecall expose --mode funnel
```
## Tool
Tool name: `voice_call`
Parameters:
- `mode`: `"call" | "status"` (default: `call`)
- `to`: target string (required for call)
- `sid`: call SID (required for status)
- `message`: optional intro text
Actions:
- `initiate_call` (message, to?, mode?)
- `continue_call` (callId, message)
- `speak_to_user` (callId, message)
- `end_call` (callId)
- `get_status` (callId)
## Gateway RPC
- `voicecall.start` (to, message?)
- `voicecall.status` (sid)
## Skill
The repo includes `skills/voice-call/SKILL.md` for agent guidance. Enable it by
setting:
```json5
{ plugins: { entries: { "voice-call": { enabled: true } } } }
```
- `voicecall.initiate` (to?, message, mode?)
- `voicecall.continue` (callId, message)
- `voicecall.speak` (callId, message)
- `voicecall.end` (callId)
- `voicecall.status` (callId)
## Notes
- Uses Twilio REST API via fetch (no SDK). Provide valid SID/token/from.
- Use `voicecall.*` for RPC names and `voice_call` for tool naming consistency.
- Uses webhook signature verification for Twilio/Telnyx.
- `responseModel` / `responseSystemPrompt` control AI auto-responses.
- Media streaming requires `ws` and OpenAI Realtime API key.
+422 -273
View File
@@ -1,325 +1,474 @@
import { Type } from "@sinclair/typebox";
type VoiceCallConfig =
| {
provider: "twilio";
twilio: {
accountSid: string;
authToken: string;
from: string;
statusCallbackUrl?: string;
twimlUrl?: string;
};
}
| {
provider?: "log";
};
type VoiceCallStartParams = {
to: string;
message?: string;
};
type VoiceCallStatus = {
sid: string;
status: string;
provider: string;
to?: string;
from?: string;
};
import type { CoreConfig } from "./src/core-bridge.js";
import {
VoiceCallConfigSchema,
validateProviderConfig,
type VoiceCallConfig,
} from "./src/config.js";
import { registerVoiceCallCli } from "./src/cli.js";
import { createVoiceCallRuntime, type VoiceCallRuntime } from "./src/runtime.js";
const voiceCallConfigSchema = {
parse(value: unknown): VoiceCallConfig {
if (!value || typeof value !== "object" || Array.isArray(value)) {
return { provider: "log" };
}
const cfg = value as Record<string, unknown>;
const provider =
cfg.provider === "twilio" || cfg.provider === "log"
? cfg.provider
: "log";
if (provider === "twilio") {
const twilio = cfg.twilio as Record<string, unknown> | undefined;
if (
!twilio ||
typeof twilio.accountSid !== "string" ||
typeof twilio.authToken !== "string" ||
typeof twilio.from !== "string"
) {
throw new Error(
"twilio provider requires twilio.accountSid, twilio.authToken, twilio.from",
);
}
return {
provider: "twilio",
twilio: {
accountSid: twilio.accountSid,
authToken: twilio.authToken,
from: twilio.from,
statusCallbackUrl:
typeof twilio.statusCallbackUrl === "string"
? twilio.statusCallbackUrl
: undefined,
twimlUrl:
typeof twilio.twimlUrl === "string" ? twilio.twimlUrl : undefined,
},
};
}
return { provider: "log" };
const raw =
value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
const twilio = raw.twilio as Record<string, unknown> | undefined;
const legacyFrom = typeof twilio?.from === "string" ? twilio.from : undefined;
const enabled = typeof raw.enabled === "boolean" ? raw.enabled : true;
const providerRaw = raw.provider === "log" ? "mock" : raw.provider;
const provider = providerRaw ?? (enabled ? "mock" : undefined);
return VoiceCallConfigSchema.parse({
...raw,
enabled,
provider,
fromNumber: raw.fromNumber ?? legacyFrom,
});
},
uiHints: {
provider: {
label: "Provider",
help: 'Use "twilio" for real calls or "log" for dev/no-network.',
help: "Use twilio, telnyx, or mock for dev/no-network.",
},
"twilio.accountSid": {
label: "Twilio Account SID",
placeholder: "ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
fromNumber: { label: "From Number", placeholder: "+15550001234" },
toNumber: { label: "Default To Number", placeholder: "+15550001234" },
inboundPolicy: { label: "Inbound Policy" },
allowFrom: { label: "Inbound Allowlist" },
inboundGreeting: { label: "Inbound Greeting", advanced: true },
"telnyx.apiKey": { label: "Telnyx API Key", sensitive: true },
"telnyx.connectionId": { label: "Telnyx Connection ID" },
"telnyx.publicKey": { label: "Telnyx Public Key", sensitive: true },
"twilio.accountSid": { label: "Twilio Account SID" },
"twilio.authToken": { label: "Twilio Auth Token", sensitive: true },
"outbound.defaultMode": { label: "Default Call Mode" },
"outbound.notifyHangupDelaySec": {
label: "Notify Hangup Delay (sec)",
advanced: true,
},
"twilio.authToken": {
label: "Twilio Auth Token",
"serve.port": { label: "Webhook Port" },
"serve.bind": { label: "Webhook Bind" },
"serve.path": { label: "Webhook Path" },
"tailscale.mode": { label: "Tailscale Mode", advanced: true },
"tailscale.path": { label: "Tailscale Path", advanced: true },
"tunnel.provider": { label: "Tunnel Provider", advanced: true },
"tunnel.ngrokAuthToken": {
label: "ngrok Auth Token",
sensitive: true,
placeholder: "••••••••••••••••",
},
"twilio.from": {
label: "Twilio From (E.164)",
placeholder: "+15551234567",
},
"twilio.statusCallbackUrl": {
label: "Status Callback URL",
placeholder: "https://example.com/twilio-status",
advanced: true,
},
"twilio.twimlUrl": {
label: "TwiML URL",
placeholder: "https://example.com/twiml",
"tunnel.ngrokDomain": { label: "ngrok Domain", advanced: true },
"tunnel.allowNgrokFreeTier": {
label: "Allow ngrok Free Tier",
advanced: true,
},
"streaming.enabled": { label: "Enable Streaming", advanced: true },
"streaming.openaiApiKey": {
label: "OpenAI Realtime API Key",
sensitive: true,
advanced: true,
},
"streaming.sttModel": { label: "Realtime STT Model", advanced: true },
"streaming.streamPath": { label: "Media Stream Path", advanced: true },
"tts.model": { label: "TTS Model", advanced: true },
"tts.voice": { label: "TTS Voice", advanced: true },
"tts.instructions": { label: "TTS Instructions", advanced: true },
publicUrl: { label: "Public Webhook URL", advanced: true },
skipSignatureVerification: {
label: "Skip Signature Verification",
advanced: true,
},
store: { label: "Call Log Store Path", advanced: true },
responseModel: { label: "Response Model", advanced: true },
responseSystemPrompt: { label: "Response System Prompt", advanced: true },
responseTimeoutMs: { label: "Response Timeout (ms)", advanced: true },
},
};
const escapeXml = (input: string): string =>
input
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;")
.replace(/'/g, "&apos;");
async function startTwilioCall(
cfg: Exclude<VoiceCallConfig, { provider?: "log" }>["twilio"],
params: VoiceCallStartParams,
): Promise<VoiceCallStatus> {
const body = new URLSearchParams();
body.set("To", params.to);
body.set("From", cfg.from);
if (cfg.twimlUrl) {
body.set("Url", cfg.twimlUrl);
} else {
const say = escapeXml(params.message ?? "Hello from Clawdbot.");
body.set("Twiml", `<Response><Say>${say}</Say></Response>`);
}
if (cfg.statusCallbackUrl) {
body.set("StatusCallback", cfg.statusCallbackUrl);
}
const res = await fetch(
`https://api.twilio.com/2010-04-01/Accounts/${cfg.accountSid}/Calls.json`,
{
method: "POST",
headers: {
Authorization: `Basic ${Buffer.from(`${cfg.accountSid}:${cfg.authToken}`).toString("base64")}`,
"Content-Type": "application/x-www-form-urlencoded",
},
body,
},
);
if (!res.ok) {
const text = await res.text();
throw new Error(`twilio call failed: ${res.status} ${text}`);
}
const payload = (await res.json()) as Record<string, unknown>;
return {
sid: String(payload.sid ?? ""),
status: String(payload.status ?? "unknown"),
provider: "twilio",
to: String(payload.to ?? params.to),
from: String(payload.from ?? cfg.from),
};
}
async function getTwilioStatus(
cfg: Exclude<VoiceCallConfig, { provider?: "log" }>["twilio"],
sid: string,
): Promise<VoiceCallStatus> {
const res = await fetch(
`https://api.twilio.com/2010-04-01/Accounts/${cfg.accountSid}/Calls/${sid}.json`,
{
method: "GET",
headers: {
Authorization: `Basic ${Buffer.from(`${cfg.accountSid}:${cfg.authToken}`).toString("base64")}`,
},
},
);
if (!res.ok) {
const text = await res.text();
throw new Error(`twilio status failed: ${res.status} ${text}`);
}
const payload = (await res.json()) as Record<string, unknown>;
return {
sid,
status: String(payload.status ?? "unknown"),
provider: "twilio",
to: String(payload.to ?? ""),
from: String(payload.from ?? cfg.from),
};
}
async function startCall(
cfg: VoiceCallConfig,
params: VoiceCallStartParams,
): Promise<VoiceCallStatus> {
if (cfg.provider === "twilio") {
return startTwilioCall(cfg.twilio, params);
}
return {
sid: `log-${Date.now()}`,
status: "queued",
provider: "log",
to: params.to,
};
}
async function getStatus(
cfg: VoiceCallConfig,
sid: string,
): Promise<VoiceCallStatus> {
if (cfg.provider === "twilio") {
return getTwilioStatus(cfg.twilio, sid);
}
return { sid, status: "mock", provider: "log" };
}
const VoiceCallToolSchema = Type.Union([
Type.Object({
action: Type.Literal("initiate_call"),
to: Type.Optional(Type.String({ description: "Call target" })),
message: Type.String({ description: "Intro message" }),
mode: Type.Optional(Type.Union([Type.Literal("notify"), Type.Literal("conversation")])),
}),
Type.Object({
action: Type.Literal("continue_call"),
callId: Type.String({ description: "Call ID" }),
message: Type.String({ description: "Follow-up message" }),
}),
Type.Object({
action: Type.Literal("speak_to_user"),
callId: Type.String({ description: "Call ID" }),
message: Type.String({ description: "Message to speak" }),
}),
Type.Object({
action: Type.Literal("end_call"),
callId: Type.String({ description: "Call ID" }),
}),
Type.Object({
action: Type.Literal("get_status"),
callId: Type.String({ description: "Call ID" }),
}),
Type.Object({
mode: Type.Optional(Type.Union([Type.Literal("call"), Type.Literal("status")])),
to: Type.Optional(Type.String({ description: "Call target" })),
sid: Type.Optional(Type.String({ description: "Call SID" })),
message: Type.Optional(Type.String({ description: "Optional intro message" })),
}),
]);
const voiceCallPlugin = {
id: "voice-call",
name: "Voice Call",
description: "Voice-call plugin with Twilio/log providers",
description: "Voice-call plugin with Telnyx/Twilio providers",
configSchema: voiceCallConfigSchema,
register(api) {
const cfg = voiceCallConfigSchema.parse(api.pluginConfig);
const validation = validateProviderConfig(cfg);
api.registerGatewayMethod("voicecall.start", async ({ params, respond }) => {
const to = typeof params?.to === "string" ? params.to.trim() : "";
const message =
typeof params?.message === "string" ? params.message.trim() : undefined;
if (!to) {
respond(false, { error: "to required" });
return;
if (api.pluginConfig && typeof api.pluginConfig === "object") {
const raw = api.pluginConfig as Record<string, unknown>;
const twilio = raw.twilio as Record<string, unknown> | undefined;
if (raw.provider === "log") {
api.logger.warn(
"[voice-call] provider \"log\" is deprecated; use \"mock\" instead",
);
}
if (typeof twilio?.from === "string") {
api.logger.warn(
"[voice-call] twilio.from is deprecated; use fromNumber instead",
);
}
}
let runtimePromise: Promise<VoiceCallRuntime> | null = null;
let runtime: VoiceCallRuntime | null = null;
const ensureRuntime = async () => {
if (!cfg.enabled) {
throw new Error("Voice call disabled in plugin config");
}
if (!validation.valid) {
throw new Error(validation.errors.join("; "));
}
if (runtime) return runtime;
if (!runtimePromise) {
runtimePromise = createVoiceCallRuntime({
config: cfg,
coreConfig: api.config as CoreConfig,
logger: api.logger,
});
}
runtime = await runtimePromise;
return runtime;
};
const sendError = (respond: (ok: boolean, payload?: unknown) => void, err: unknown) => {
respond(false, { error: err instanceof Error ? err.message : String(err) });
};
api.registerGatewayMethod("voicecall.initiate", async ({ params, respond }) => {
try {
const result = await startCall(cfg, { to, message });
respond(true, result);
const message =
typeof params?.message === "string" ? params.message.trim() : "";
if (!message) {
respond(false, { error: "message required" });
return;
}
const rt = await ensureRuntime();
const to =
typeof params?.to === "string" && params.to.trim()
? params.to.trim()
: rt.config.toNumber;
if (!to) {
respond(false, { error: "to required" });
return;
}
const mode =
params?.mode === "notify" || params?.mode === "conversation"
? params.mode
: undefined;
const result = await rt.manager.initiateCall(to, undefined, {
message,
mode,
});
if (!result.success) {
respond(false, { error: result.error || "initiate failed" });
return;
}
respond(true, { callId: result.callId, initiated: true });
} catch (err) {
respond(false, { error: String(err) });
sendError(respond, err);
}
});
api.registerGatewayMethod(
"voicecall.status",
async ({ params, respond }) => {
const sid = typeof params?.sid === "string" ? params.sid.trim() : "";
if (!sid) {
respond(false, { error: "sid required" });
api.registerGatewayMethod("voicecall.continue", async ({ params, respond }) => {
try {
const callId =
typeof params?.callId === "string" ? params.callId.trim() : "";
const message =
typeof params?.message === "string" ? params.message.trim() : "";
if (!callId || !message) {
respond(false, { error: "callId and message required" });
return;
}
try {
const result = await getStatus(cfg, sid);
respond(true, result);
} catch (err) {
respond(false, { error: String(err) });
const rt = await ensureRuntime();
const result = await rt.manager.continueCall(callId, message);
if (!result.success) {
respond(false, { error: result.error || "continue failed" });
return;
}
},
);
respond(true, { success: true, transcript: result.transcript });
} catch (err) {
sendError(respond, err);
}
});
api.registerTool(
{
name: "voice_call",
label: "Voice Call",
description: "Start or inspect a voice call via the voice-call plugin",
parameters: Type.Object({
mode: Type.Optional(
Type.Union([Type.Literal("call"), Type.Literal("status")]),
),
to: Type.Optional(Type.String({ description: "Call target" })),
sid: Type.Optional(Type.String({ description: "Call SID" })),
message: Type.Optional(
Type.String({ description: "Optional intro message" }),
),
}),
async execute(_toolCallId, params) {
const mode = params.mode ?? "call";
if (mode === "status") {
if (typeof params.sid !== "string") {
throw new Error("sid required for status");
api.registerGatewayMethod("voicecall.speak", async ({ params, respond }) => {
try {
const callId =
typeof params?.callId === "string" ? params.callId.trim() : "";
const message =
typeof params?.message === "string" ? params.message.trim() : "";
if (!callId || !message) {
respond(false, { error: "callId and message required" });
return;
}
const rt = await ensureRuntime();
const result = await rt.manager.speak(callId, message);
if (!result.success) {
respond(false, { error: result.error || "speak failed" });
return;
}
respond(true, { success: true });
} catch (err) {
sendError(respond, err);
}
});
api.registerGatewayMethod("voicecall.end", async ({ params, respond }) => {
try {
const callId =
typeof params?.callId === "string" ? params.callId.trim() : "";
if (!callId) {
respond(false, { error: "callId required" });
return;
}
const rt = await ensureRuntime();
const result = await rt.manager.endCall(callId);
if (!result.success) {
respond(false, { error: result.error || "end failed" });
return;
}
respond(true, { success: true });
} catch (err) {
sendError(respond, err);
}
});
api.registerGatewayMethod("voicecall.status", async ({ params, respond }) => {
try {
const raw =
typeof params?.callId === "string"
? params.callId.trim()
: typeof params?.sid === "string"
? params.sid.trim()
: "";
if (!raw) {
respond(false, { error: "callId required" });
return;
}
const rt = await ensureRuntime();
const call =
rt.manager.getCall(raw) || rt.manager.getCallByProviderCallId(raw);
if (!call) {
respond(true, { found: false });
return;
}
respond(true, { found: true, call });
} catch (err) {
sendError(respond, err);
}
});
api.registerGatewayMethod("voicecall.start", async ({ params, respond }) => {
try {
const to = typeof params?.to === "string" ? params.to.trim() : "";
const message =
typeof params?.message === "string" ? params.message.trim() : "";
if (!to) {
respond(false, { error: "to required" });
return;
}
const rt = await ensureRuntime();
const result = await rt.manager.initiateCall(to, undefined, {
message: message || undefined,
});
if (!result.success) {
respond(false, { error: result.error || "initiate failed" });
return;
}
respond(true, { callId: result.callId, initiated: true });
} catch (err) {
sendError(respond, err);
}
});
api.registerTool({
name: "voice_call",
label: "Voice Call",
description:
"Make phone calls and have voice conversations via the voice-call plugin.",
parameters: VoiceCallToolSchema,
async execute(_toolCallId, params) {
const json = (payload: unknown) => ({
content: [
{ type: "text", text: JSON.stringify(payload, null, 2) },
],
details: payload,
});
try {
const rt = await ensureRuntime();
if (typeof params?.action === "string") {
switch (params.action) {
case "initiate_call": {
const message = String(params.message || "").trim();
if (!message) throw new Error("message required");
const to =
typeof params.to === "string" && params.to.trim()
? params.to.trim()
: rt.config.toNumber;
if (!to) throw new Error("to required");
const result = await rt.manager.initiateCall(to, undefined, {
message,
mode:
params.mode === "notify" || params.mode === "conversation"
? params.mode
: undefined,
});
if (!result.success) {
throw new Error(result.error || "initiate failed");
}
return json({ callId: result.callId, initiated: true });
}
case "continue_call": {
const callId = String(params.callId || "").trim();
const message = String(params.message || "").trim();
if (!callId || !message) {
throw new Error("callId and message required");
}
const result = await rt.manager.continueCall(callId, message);
if (!result.success) {
throw new Error(result.error || "continue failed");
}
return json({ success: true, transcript: result.transcript });
}
case "speak_to_user": {
const callId = String(params.callId || "").trim();
const message = String(params.message || "").trim();
if (!callId || !message) {
throw new Error("callId and message required");
}
const result = await rt.manager.speak(callId, message);
if (!result.success) {
throw new Error(result.error || "speak failed");
}
return json({ success: true });
}
case "end_call": {
const callId = String(params.callId || "").trim();
if (!callId) throw new Error("callId required");
const result = await rt.manager.endCall(callId);
if (!result.success) {
throw new Error(result.error || "end failed");
}
return json({ success: true });
}
case "get_status": {
const callId = String(params.callId || "").trim();
if (!callId) throw new Error("callId required");
const call =
rt.manager.getCall(callId) ||
rt.manager.getCallByProviderCallId(callId);
return json(call ? { found: true, call } : { found: false });
}
}
const status = await getStatus(cfg, params.sid.trim());
return {
content: [{ type: "text", text: JSON.stringify(status, null, 2) }],
details: status,
};
}
const mode = params?.mode ?? "call";
if (mode === "status") {
const sid =
typeof params.sid === "string" ? params.sid.trim() : "";
if (!sid) throw new Error("sid required for status");
const call =
rt.manager.getCall(sid) || rt.manager.getCallByProviderCallId(sid);
return json(call ? { found: true, call } : { found: false });
}
const to =
typeof params.to === "string" && params.to.trim()
? params.to.trim()
: null;
: rt.config.toNumber;
if (!to) throw new Error("to required for call");
const result = await startCall(cfg, { to, message: params.message });
return {
content: [{ type: "text", text: JSON.stringify(result, null, 2) }],
details: result,
};
},
},
{ name: "voice_call" },
);
api.registerCli(({ program }) => {
const voicecall = program
.command("voicecall")
.description("Voice call plugin commands");
voicecall
.command("start")
.description("Start a voice call")
.requiredOption("--to <target>", "Target to call")
.option("--message <text>", "Optional intro message")
.action(async (opts) => {
const result = await startCall(cfg, {
to: opts.to,
message: opts.message,
const result = await rt.manager.initiateCall(to, undefined, {
message:
typeof params.message === "string" && params.message.trim()
? params.message.trim()
: undefined,
});
console.log(JSON.stringify(result, null, 2));
});
if (!result.success) {
throw new Error(result.error || "initiate failed");
}
return json({ callId: result.callId, initiated: true });
} catch (err) {
return json({
error: err instanceof Error ? err.message : String(err),
});
}
},
});
voicecall
.command("status")
.description("Show voice-call status")
.requiredOption("--sid <sid>", "Call SID")
.action(async (opts) => {
const result = await getStatus(cfg, opts.sid);
console.log(JSON.stringify(result, null, 2));
});
}, { commands: ["voicecall"] });
api.registerCli(
({ program }) =>
registerVoiceCallCli({
program,
config: cfg,
ensureRuntime,
logger: api.logger,
}),
{ commands: ["voicecall"] },
);
api.registerService({
id: "voicecall",
start: () => {
api.logger.info(`voice-call provider: ${cfg.provider}`);
start: async () => {
if (!cfg.enabled) return;
try {
await ensureRuntime();
} catch (err) {
api.logger.error(
`[voice-call] Failed to start runtime: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
},
stop: () => {
api.logger.info("voice-call service stopped");
stop: async () => {
if (!runtimePromise) return;
try {
const rt = await runtimePromise;
await rt.stop();
} finally {
runtimePromise = null;
runtime = null;
}
},
});
},
+4 -2
View File
@@ -2,9 +2,11 @@
"name": "@clawdbot/voice-call",
"version": "0.0.1",
"type": "module",
"description": "Clawdbot voice-call plugin (example)",
"description": "Clawdbot voice-call plugin",
"dependencies": {
"@sinclair/typebox": "0.34.47"
"@sinclair/typebox": "0.34.47",
"ws": "^8.19.0",
"zod": "^4.3.5"
},
"clawdbot": {
"extensions": ["./index.ts"]
+297
View File
@@ -0,0 +1,297 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import type { Command } from "commander";
import type { VoiceCallConfig } from "./config.js";
import type { VoiceCallRuntime } from "./runtime.js";
import { resolveUserPath } from "./utils.js";
import {
cleanupTailscaleExposureRoute,
getTailscaleSelfInfo,
setupTailscaleExposureRoute,
} from "./webhook.js";
type Logger = {
info: (message: string) => void;
warn: (message: string) => void;
error: (message: string) => void;
};
function resolveMode(input: string): "off" | "serve" | "funnel" {
const raw = input.trim().toLowerCase();
if (raw === "serve" || raw === "off") return raw;
return "funnel";
}
function resolveDefaultStorePath(config: VoiceCallConfig): string {
const base =
config.store?.trim() || path.join(os.homedir(), "clawd", "voice-calls");
return path.join(resolveUserPath(base), "calls.jsonl");
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export function registerVoiceCallCli(params: {
program: Command;
config: VoiceCallConfig;
ensureRuntime: () => Promise<VoiceCallRuntime>;
logger: Logger;
}) {
const { program, config, ensureRuntime, logger } = params;
const root = program.command("voicecall").description("Voice call utilities");
root
.command("call")
.description("Initiate an outbound voice call")
.requiredOption(
"-m, --message <text>",
"Message to speak when call connects",
)
.option(
"-t, --to <phone>",
"Phone number to call (E.164 format, uses config toNumber if not set)",
)
.option(
"--mode <mode>",
"Call mode: notify (hangup after message) or conversation (stay open)",
"conversation",
)
.action(
async (options: { message: string; to?: string; mode?: string }) => {
const rt = await ensureRuntime();
const to = options.to ?? rt.config.toNumber;
if (!to) {
throw new Error("Missing --to and no toNumber configured");
}
const result = await rt.manager.initiateCall(to, undefined, {
message: options.message,
mode:
options.mode === "notify" || options.mode === "conversation"
? options.mode
: undefined,
});
if (!result.success) {
throw new Error(result.error || "initiate failed");
}
// eslint-disable-next-line no-console
console.log(JSON.stringify({ callId: result.callId }, null, 2));
},
);
root
.command("start")
.description("Alias for voicecall call")
.requiredOption("--to <phone>", "Phone number to call")
.option("--message <text>", "Message to speak when call connects")
.option(
"--mode <mode>",
"Call mode: notify (hangup after message) or conversation (stay open)",
"conversation",
)
.action(
async (options: { to: string; message?: string; mode?: string }) => {
const rt = await ensureRuntime();
const result = await rt.manager.initiateCall(options.to, undefined, {
message: options.message,
mode:
options.mode === "notify" || options.mode === "conversation"
? options.mode
: undefined,
});
if (!result.success) {
throw new Error(result.error || "initiate failed");
}
// eslint-disable-next-line no-console
console.log(JSON.stringify({ callId: result.callId }, null, 2));
},
);
root
.command("continue")
.description("Speak a message and wait for a response")
.requiredOption("--call-id <id>", "Call ID")
.requiredOption("--message <text>", "Message to speak")
.action(async (options: { callId: string; message: string }) => {
const rt = await ensureRuntime();
const result = await rt.manager.continueCall(
options.callId,
options.message,
);
if (!result.success) {
throw new Error(result.error || "continue failed");
}
// eslint-disable-next-line no-console
console.log(JSON.stringify(result, null, 2));
});
root
.command("speak")
.description("Speak a message without waiting for response")
.requiredOption("--call-id <id>", "Call ID")
.requiredOption("--message <text>", "Message to speak")
.action(async (options: { callId: string; message: string }) => {
const rt = await ensureRuntime();
const result = await rt.manager.speak(options.callId, options.message);
if (!result.success) {
throw new Error(result.error || "speak failed");
}
// eslint-disable-next-line no-console
console.log(JSON.stringify(result, null, 2));
});
root
.command("end")
.description("Hang up an active call")
.requiredOption("--call-id <id>", "Call ID")
.action(async (options: { callId: string }) => {
const rt = await ensureRuntime();
const result = await rt.manager.endCall(options.callId);
if (!result.success) {
throw new Error(result.error || "end failed");
}
// eslint-disable-next-line no-console
console.log(JSON.stringify(result, null, 2));
});
root
.command("status")
.description("Show call status")
.requiredOption("--call-id <id>", "Call ID")
.action(async (options: { callId: string }) => {
const rt = await ensureRuntime();
const call = rt.manager.getCall(options.callId);
// eslint-disable-next-line no-console
console.log(JSON.stringify(call ?? { found: false }, null, 2));
});
root
.command("tail")
.description(
"Tail voice-call JSONL logs (prints new lines; useful during provider tests)",
)
.option("--file <path>", "Path to calls.jsonl", resolveDefaultStorePath(config))
.option("--since <n>", "Print last N lines first", "25")
.option("--poll <ms>", "Poll interval in ms", "250")
.action(
async (options: { file: string; since?: string; poll?: string }) => {
const file = options.file;
const since = Math.max(0, Number(options.since ?? 0));
const pollMs = Math.max(50, Number(options.poll ?? 250));
if (!fs.existsSync(file)) {
logger.error(`No log file at ${file}`);
process.exit(1);
}
const initial = fs.readFileSync(file, "utf8");
const lines = initial.split("\n").filter(Boolean);
for (const line of lines.slice(Math.max(0, lines.length - since))) {
// eslint-disable-next-line no-console
console.log(line);
}
let offset = Buffer.byteLength(initial, "utf8");
for (;;) {
try {
const stat = fs.statSync(file);
if (stat.size < offset) {
offset = 0;
}
if (stat.size > offset) {
const fd = fs.openSync(file, "r");
try {
const buf = Buffer.alloc(stat.size - offset);
fs.readSync(fd, buf, 0, buf.length, offset);
offset = stat.size;
const text = buf.toString("utf8");
for (const line of text.split("\n").filter(Boolean)) {
// eslint-disable-next-line no-console
console.log(line);
}
} finally {
fs.closeSync(fd);
}
}
} catch {
// ignore and retry
}
await sleep(pollMs);
}
},
);
root
.command("expose")
.description("Enable/disable Tailscale serve/funnel for the webhook")
.option("--mode <mode>", "off | serve (tailnet) | funnel (public)", "funnel")
.option(
"--path <path>",
"Tailscale path to expose (recommend matching serve.path)",
)
.option("--port <port>", "Local webhook port")
.option("--serve-path <path>", "Local webhook path")
.action(
async (options: {
mode?: string;
port?: string;
path?: string;
servePath?: string;
}) => {
const mode = resolveMode(options.mode ?? "funnel");
const servePort = Number(options.port ?? config.serve.port ?? 3334);
const servePath = String(
options.servePath ?? config.serve.path ?? "/voice/webhook",
);
const tsPath = String(
options.path ?? config.tailscale?.path ?? servePath,
);
const localUrl = `http://127.0.0.1:${servePort}`;
if (mode === "off") {
await cleanupTailscaleExposureRoute({ mode: "serve", path: tsPath });
await cleanupTailscaleExposureRoute({ mode: "funnel", path: tsPath });
// eslint-disable-next-line no-console
console.log(JSON.stringify({ ok: true, mode: "off", path: tsPath }, null, 2));
return;
}
const publicUrl = await setupTailscaleExposureRoute({
mode,
path: tsPath,
localUrl,
});
const tsInfo = publicUrl ? null : await getTailscaleSelfInfo();
const enableUrl = tsInfo?.nodeId
? `https://login.tailscale.com/f/${mode}?node=${tsInfo.nodeId}`
: null;
// eslint-disable-next-line no-console
console.log(
JSON.stringify(
{
ok: Boolean(publicUrl),
mode,
path: tsPath,
localUrl,
publicUrl,
hint: publicUrl
? undefined
: {
note: "Tailscale serve/funnel may be disabled on this tailnet (or require admin enable).",
enableUrl,
},
},
null,
2,
),
);
},
);
}
+355
View File
@@ -0,0 +1,355 @@
import { z } from "zod";
// -----------------------------------------------------------------------------
// Phone Number Validation
// -----------------------------------------------------------------------------
/**
* E.164 phone number format: +[country code][number]
* Examples use 555 prefix (reserved for fictional numbers)
*/
export const E164Schema = z
.string()
.regex(/^\+[1-9]\d{1,14}$/, "Expected E.164 format, e.g. +15550001234");
// -----------------------------------------------------------------------------
// Inbound Policy
// -----------------------------------------------------------------------------
/**
* Controls how inbound calls are handled:
* - "disabled": Block all inbound calls (outbound only)
* - "allowlist": Only accept calls from numbers in allowFrom
* - "pairing": Unknown callers can request pairing (future)
* - "open": Accept all inbound calls (dangerous!)
*/
export const InboundPolicySchema = z.enum([
"disabled",
"allowlist",
"pairing",
"open",
]);
export type InboundPolicy = z.infer<typeof InboundPolicySchema>;
// -----------------------------------------------------------------------------
// Provider-Specific Configuration
// -----------------------------------------------------------------------------
export const TelnyxConfigSchema = z.object({
/** Telnyx API v2 key */
apiKey: z.string().min(1).optional(),
/** Telnyx connection ID (from Call Control app) */
connectionId: z.string().min(1).optional(),
/** Public key for webhook signature verification */
publicKey: z.string().min(1).optional(),
});
export type TelnyxConfig = z.infer<typeof TelnyxConfigSchema>;
export const TwilioConfigSchema = z.object({
/** Twilio Account SID */
accountSid: z.string().min(1).optional(),
/** Twilio Auth Token */
authToken: z.string().min(1).optional(),
});
export type TwilioConfig = z.infer<typeof TwilioConfigSchema>;
// -----------------------------------------------------------------------------
// STT/TTS Configuration
// -----------------------------------------------------------------------------
export const SttConfigSchema = z
.object({
/** STT provider (currently only OpenAI supported) */
provider: z.literal("openai").default("openai"),
/** Whisper model to use */
model: z.string().min(1).default("whisper-1"),
})
.default({ provider: "openai", model: "whisper-1" });
export type SttConfig = z.infer<typeof SttConfigSchema>;
export const TtsConfigSchema = z
.object({
/** TTS provider (currently only OpenAI supported) */
provider: z.literal("openai").default("openai"),
/**
* TTS model to use:
* - gpt-4o-mini-tts: newest, supports instructions for tone/style control (recommended)
* - tts-1: lower latency
* - tts-1-hd: higher quality
*/
model: z.string().min(1).default("gpt-4o-mini-tts"),
/**
* Voice ID. For best quality, use marin or cedar.
* All voices: alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar
*/
voice: z.string().min(1).default("coral"),
/**
* Instructions for speech style (only works with gpt-4o-mini-tts).
* Examples: "Speak in a cheerful tone", "Talk like a sympathetic customer service agent"
*/
instructions: z.string().optional(),
})
.default({ provider: "openai", model: "gpt-4o-mini-tts", voice: "coral" });
export type TtsConfig = z.infer<typeof TtsConfigSchema>;
// -----------------------------------------------------------------------------
// Webhook Server Configuration
// -----------------------------------------------------------------------------
export const VoiceCallServeConfigSchema = z
.object({
/** Port to listen on */
port: z.number().int().positive().default(3334),
/** Bind address */
bind: z.string().default("127.0.0.1"),
/** Webhook path */
path: z.string().min(1).default("/voice/webhook"),
})
.default({ port: 3334, bind: "127.0.0.1", path: "/voice/webhook" });
export type VoiceCallServeConfig = z.infer<typeof VoiceCallServeConfigSchema>;
export const VoiceCallTailscaleConfigSchema = z
.object({
/**
* Tailscale exposure mode:
* - "off": No Tailscale exposure
* - "serve": Tailscale serve (private to tailnet)
* - "funnel": Tailscale funnel (public HTTPS)
*/
mode: z.enum(["off", "serve", "funnel"]).default("off"),
/** Path for Tailscale serve/funnel (should usually match serve.path) */
path: z.string().min(1).default("/voice/webhook"),
})
.default({ mode: "off", path: "/voice/webhook" });
export type VoiceCallTailscaleConfig = z.infer<
typeof VoiceCallTailscaleConfigSchema
>;
// -----------------------------------------------------------------------------
// Tunnel Configuration (unified ngrok/tailscale)
// -----------------------------------------------------------------------------
export const VoiceCallTunnelConfigSchema = z
.object({
/**
* Tunnel provider:
* - "none": No tunnel (use publicUrl if set, or manual setup)
* - "ngrok": Use ngrok for public HTTPS tunnel
* - "tailscale-serve": Tailscale serve (private to tailnet)
* - "tailscale-funnel": Tailscale funnel (public HTTPS)
*/
provider: z
.enum(["none", "ngrok", "tailscale-serve", "tailscale-funnel"])
.default("none"),
/** ngrok auth token (optional, enables longer sessions and more features) */
ngrokAuthToken: z.string().min(1).optional(),
/** ngrok custom domain (paid feature, e.g., "myapp.ngrok.io") */
ngrokDomain: z.string().min(1).optional(),
/**
* Allow ngrok free tier compatibility mode.
* When true, signature verification failures on ngrok-free.app URLs
* will be logged but allowed through. Less secure, but necessary
* for ngrok free tier which may modify URLs.
*/
allowNgrokFreeTier: z.boolean().default(true),
})
.default({ provider: "none", allowNgrokFreeTier: true });
export type VoiceCallTunnelConfig = z.infer<typeof VoiceCallTunnelConfigSchema>;
// -----------------------------------------------------------------------------
// Outbound Call Configuration
// -----------------------------------------------------------------------------
/**
* Call mode determines how outbound calls behave:
* - "notify": Deliver message and auto-hangup after delay (one-way notification)
* - "conversation": Stay open for back-and-forth until explicit end or timeout
*/
export const CallModeSchema = z.enum(["notify", "conversation"]);
export type CallMode = z.infer<typeof CallModeSchema>;
export const OutboundConfigSchema = z
.object({
/** Default call mode for outbound calls */
defaultMode: CallModeSchema.default("notify"),
/** Seconds to wait after TTS before auto-hangup in notify mode */
notifyHangupDelaySec: z.number().int().nonnegative().default(3),
})
.default({ defaultMode: "notify", notifyHangupDelaySec: 3 });
export type OutboundConfig = z.infer<typeof OutboundConfigSchema>;
// -----------------------------------------------------------------------------
// Streaming Configuration (OpenAI Realtime STT)
// -----------------------------------------------------------------------------
export const VoiceCallStreamingConfigSchema = z
.object({
/** Enable real-time audio streaming (requires WebSocket support) */
enabled: z.boolean().default(false),
/** STT provider for real-time transcription */
sttProvider: z.enum(["openai-realtime"]).default("openai-realtime"),
/** OpenAI API key for Realtime API (uses OPENAI_API_KEY env if not set) */
openaiApiKey: z.string().min(1).optional(),
/** OpenAI transcription model (default: gpt-4o-transcribe) */
sttModel: z.string().min(1).default("gpt-4o-transcribe"),
/** VAD silence duration in ms before considering speech ended */
silenceDurationMs: z.number().int().positive().default(800),
/** VAD threshold 0-1 (higher = less sensitive) */
vadThreshold: z.number().min(0).max(1).default(0.5),
/** WebSocket path for media stream connections */
streamPath: z.string().min(1).default("/voice/stream"),
})
.default({
enabled: false,
sttProvider: "openai-realtime",
sttModel: "gpt-4o-transcribe",
silenceDurationMs: 800,
vadThreshold: 0.5,
streamPath: "/voice/stream",
});
export type VoiceCallStreamingConfig = z.infer<
typeof VoiceCallStreamingConfigSchema
>;
// -----------------------------------------------------------------------------
// Main Voice Call Configuration
// -----------------------------------------------------------------------------
export const VoiceCallConfigSchema = z.object({
/** Enable voice call functionality */
enabled: z.boolean().default(false),
/** Active provider (telnyx, twilio, or mock) */
provider: z.enum(["telnyx", "twilio", "mock"]).optional(),
/** Telnyx-specific configuration */
telnyx: TelnyxConfigSchema.optional(),
/** Twilio-specific configuration */
twilio: TwilioConfigSchema.optional(),
/** Phone number to call from (E.164) */
fromNumber: E164Schema.optional(),
/** Default phone number to call (E.164) */
toNumber: E164Schema.optional(),
/** Inbound call policy */
inboundPolicy: InboundPolicySchema.default("disabled"),
/** Allowlist of phone numbers for inbound calls (E.164) */
allowFrom: z.array(E164Schema).default([]),
/** Greeting message for inbound calls */
inboundGreeting: z.string().optional(),
/** Outbound call configuration */
outbound: OutboundConfigSchema,
/** Maximum call duration in seconds */
maxDurationSeconds: z.number().int().positive().default(300),
/** Silence timeout for end-of-speech detection (ms) */
silenceTimeoutMs: z.number().int().positive().default(800),
/** Timeout for user transcript (ms) */
transcriptTimeoutMs: z.number().int().positive().default(180000),
/** Ring timeout for outbound calls (ms) */
ringTimeoutMs: z.number().int().positive().default(30000),
/** Maximum concurrent calls */
maxConcurrentCalls: z.number().int().positive().default(1),
/** Webhook server configuration */
serve: VoiceCallServeConfigSchema,
/** Tailscale exposure configuration (legacy, prefer tunnel config) */
tailscale: VoiceCallTailscaleConfigSchema,
/** Tunnel configuration (unified ngrok/tailscale) */
tunnel: VoiceCallTunnelConfigSchema,
/** Real-time audio streaming configuration */
streaming: VoiceCallStreamingConfigSchema,
/** Public webhook URL override (if set, bypasses tunnel auto-detection) */
publicUrl: z.string().url().optional(),
/** Skip webhook signature verification (development only, NOT for production) */
skipSignatureVerification: z.boolean().default(false),
/** STT configuration */
stt: SttConfigSchema,
/** TTS configuration */
tts: TtsConfigSchema,
/** Store path for call logs */
store: z.string().optional(),
/** Model for generating voice responses (e.g., "anthropic/claude-sonnet-4", "openai/gpt-4o") */
responseModel: z.string().default("openai/gpt-4o-mini"),
/** System prompt for voice responses */
responseSystemPrompt: z.string().optional(),
/** Timeout for response generation in ms (default 30s) */
responseTimeoutMs: z.number().int().positive().default(30000),
});
export type VoiceCallConfig = z.infer<typeof VoiceCallConfigSchema>;
// -----------------------------------------------------------------------------
// Configuration Helpers
// -----------------------------------------------------------------------------
/**
* Validate that the configuration has all required fields for the selected provider.
*/
export function validateProviderConfig(config: VoiceCallConfig): {
valid: boolean;
errors: string[];
} {
const errors: string[] = [];
if (!config.enabled) {
return { valid: true, errors: [] };
}
if (!config.provider) {
errors.push("plugins.entries.voice-call.config.provider is required");
}
if (!config.fromNumber && config.provider !== "mock") {
errors.push("plugins.entries.voice-call.config.fromNumber is required");
}
if (config.provider === "telnyx") {
if (!config.telnyx?.apiKey) {
errors.push(
"plugins.entries.voice-call.config.telnyx.apiKey is required (or set TELNYX_API_KEY env)",
);
}
if (!config.telnyx?.connectionId) {
errors.push(
"plugins.entries.voice-call.config.telnyx.connectionId is required (or set TELNYX_CONNECTION_ID env)",
);
}
}
if (config.provider === "twilio") {
if (!config.twilio?.accountSid) {
errors.push(
"plugins.entries.voice-call.config.twilio.accountSid is required (or set TWILIO_ACCOUNT_SID env)",
);
}
if (!config.twilio?.authToken) {
errors.push(
"plugins.entries.voice-call.config.twilio.authToken is required (or set TWILIO_AUTH_TOKEN env)",
);
}
}
return { valid: errors.length === 0, errors };
}
+190
View File
@@ -0,0 +1,190 @@
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
export type CoreConfig = {
session?: {
store?: string;
};
};
type CoreAgentDeps = {
resolveAgentDir: (cfg: CoreConfig, agentId: string) => string;
resolveAgentWorkspaceDir: (cfg: CoreConfig, agentId: string) => string;
resolveAgentIdentity: (
cfg: CoreConfig,
agentId: string,
) => { name?: string | null } | null | undefined;
resolveThinkingDefault: (params: {
cfg: CoreConfig;
provider?: string;
model?: string;
}) => string;
runEmbeddedPiAgent: (params: {
sessionId: string;
sessionKey?: string;
messageProvider?: string;
sessionFile: string;
workspaceDir: string;
config?: CoreConfig;
prompt: string;
provider?: string;
model?: string;
thinkLevel?: string;
verboseLevel?: string;
timeoutMs: number;
runId: string;
lane?: string;
extraSystemPrompt?: string;
agentDir?: string;
}) => Promise<{
payloads?: Array<{ text?: string; isError?: boolean }>;
meta?: { aborted?: boolean };
}>;
resolveAgentTimeoutMs: (opts: { cfg: CoreConfig }) => number;
ensureAgentWorkspace: (params?: { dir: string }) => Promise<void>;
resolveStorePath: (store?: string, opts?: { agentId?: string }) => string;
loadSessionStore: (storePath: string) => Record<string, unknown>;
saveSessionStore: (
storePath: string,
store: Record<string, unknown>,
) => Promise<void>;
resolveSessionFilePath: (
sessionId: string,
entry: unknown,
opts?: { agentId?: string },
) => string;
DEFAULT_MODEL: string;
DEFAULT_PROVIDER: string;
};
let coreRootCache: string | null = null;
let coreDepsPromise: Promise<CoreAgentDeps> | null = null;
function findPackageRoot(startDir: string, name: string): string | null {
let dir = startDir;
for (;;) {
const pkgPath = path.join(dir, "package.json");
try {
if (fs.existsSync(pkgPath)) {
const raw = fs.readFileSync(pkgPath, "utf8");
const pkg = JSON.parse(raw) as { name?: string };
if (pkg.name === name) return dir;
}
} catch {
// ignore parse errors and keep walking
}
const parent = path.dirname(dir);
if (parent === dir) return null;
dir = parent;
}
}
function resolveClawdbotRoot(): string {
if (coreRootCache) return coreRootCache;
const override = process.env.CLAWDBOT_ROOT?.trim();
if (override) {
coreRootCache = override;
return override;
}
const candidates = new Set<string>();
if (process.argv[1]) {
candidates.add(path.dirname(process.argv[1]));
}
candidates.add(process.cwd());
try {
const urlPath = fileURLToPath(import.meta.url);
candidates.add(path.dirname(urlPath));
} catch {
// ignore
}
for (const start of candidates) {
const found = findPackageRoot(start, "clawdbot");
if (found) {
coreRootCache = found;
return found;
}
}
throw new Error(
"Unable to resolve Clawdbot root. Set CLAWDBOT_ROOT to the package root.",
);
}
async function importCoreModule<T>(relativePath: string): Promise<T> {
const root = resolveClawdbotRoot();
const distPath = path.join(root, "dist", relativePath);
if (!fs.existsSync(distPath)) {
throw new Error(
`Missing core module at ${distPath}. Run \`pnpm build\` or install the official package.`,
);
}
return (await import(pathToFileURL(distPath).href)) as T;
}
export async function loadCoreAgentDeps(): Promise<CoreAgentDeps> {
if (coreDepsPromise) return coreDepsPromise;
coreDepsPromise = (async () => {
const [
agentScope,
defaults,
identity,
modelSelection,
piEmbedded,
timeout,
workspace,
sessions,
] = await Promise.all([
importCoreModule<{
resolveAgentDir: CoreAgentDeps["resolveAgentDir"];
resolveAgentWorkspaceDir: CoreAgentDeps["resolveAgentWorkspaceDir"];
}>("agents/agent-scope.js"),
importCoreModule<{
DEFAULT_MODEL: string;
DEFAULT_PROVIDER: string;
}>("agents/defaults.js"),
importCoreModule<{
resolveAgentIdentity: CoreAgentDeps["resolveAgentIdentity"];
}>("agents/identity.js"),
importCoreModule<{
resolveThinkingDefault: CoreAgentDeps["resolveThinkingDefault"];
}>("agents/model-selection.js"),
importCoreModule<{
runEmbeddedPiAgent: CoreAgentDeps["runEmbeddedPiAgent"];
}>("agents/pi-embedded.js"),
importCoreModule<{
resolveAgentTimeoutMs: CoreAgentDeps["resolveAgentTimeoutMs"];
}>("agents/timeout.js"),
importCoreModule<{
ensureAgentWorkspace: CoreAgentDeps["ensureAgentWorkspace"];
}>("agents/workspace.js"),
importCoreModule<{
resolveStorePath: CoreAgentDeps["resolveStorePath"];
loadSessionStore: CoreAgentDeps["loadSessionStore"];
saveSessionStore: CoreAgentDeps["saveSessionStore"];
resolveSessionFilePath: CoreAgentDeps["resolveSessionFilePath"];
}>("config/sessions.js"),
]);
return {
resolveAgentDir: agentScope.resolveAgentDir,
resolveAgentWorkspaceDir: agentScope.resolveAgentWorkspaceDir,
resolveAgentIdentity: identity.resolveAgentIdentity,
resolveThinkingDefault: modelSelection.resolveThinkingDefault,
runEmbeddedPiAgent: piEmbedded.runEmbeddedPiAgent,
resolveAgentTimeoutMs: timeout.resolveAgentTimeoutMs,
ensureAgentWorkspace: workspace.ensureAgentWorkspace,
resolveStorePath: sessions.resolveStorePath,
loadSessionStore: sessions.loadSessionStore,
saveSessionStore: sessions.saveSessionStore,
resolveSessionFilePath: sessions.resolveSessionFilePath,
DEFAULT_MODEL: defaults.DEFAULT_MODEL,
DEFAULT_PROVIDER: defaults.DEFAULT_PROVIDER,
};
})();
return coreDepsPromise;
}
+846
View File
@@ -0,0 +1,846 @@
import crypto from "node:crypto";
import fs from "node:fs";
import fsp from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { resolveUserPath } from "./utils.js";
import type { CallMode, VoiceCallConfig } from "./config.js";
import type { VoiceCallProvider } from "./providers/base.js";
import {
type CallId,
type CallRecord,
CallRecordSchema,
type CallState,
type NormalizedEvent,
type OutboundCallOptions,
TerminalStates,
type TranscriptEntry,
} from "./types.js";
import { escapeXml, mapVoiceToPolly } from "./voice-mapping.js";
/**
* Manages voice calls: state machine, persistence, and provider coordination.
*/
export class CallManager {
private activeCalls = new Map<CallId, CallRecord>();
private providerCallIdMap = new Map<string, CallId>(); // providerCallId -> internal callId
private processedEventIds = new Set<string>();
private provider: VoiceCallProvider | null = null;
private config: VoiceCallConfig;
private storePath: string;
private webhookUrl: string | null = null;
private transcriptWaiters = new Map<
CallId,
{
resolve: (text: string) => void;
reject: (err: Error) => void;
timeout: NodeJS.Timeout;
}
>();
/** Max duration timers to auto-hangup calls after configured timeout */
private maxDurationTimers = new Map<CallId, NodeJS.Timeout>();
constructor(config: VoiceCallConfig, storePath?: string) {
this.config = config;
// Resolve store path with tilde expansion (like other config values)
const rawPath =
storePath ||
config.store ||
path.join(os.homedir(), "clawd", "voice-calls");
this.storePath = resolveUserPath(rawPath);
}
/**
* Initialize the call manager with a provider.
*/
initialize(provider: VoiceCallProvider, webhookUrl: string): void {
this.provider = provider;
this.webhookUrl = webhookUrl;
// Ensure store directory exists
fs.mkdirSync(this.storePath, { recursive: true });
// Load any persisted active calls
this.loadActiveCalls();
}
/**
* Get the current provider.
*/
getProvider(): VoiceCallProvider | null {
return this.provider;
}
/**
* Initiate an outbound call.
* @param to - The phone number to call
* @param sessionKey - Optional session key for context
* @param options - Optional call options (message, mode)
*/
async initiateCall(
to: string,
sessionKey?: string,
options?: OutboundCallOptions | string,
): Promise<{ callId: CallId; success: boolean; error?: string }> {
// Support legacy string argument for initialMessage
const opts: OutboundCallOptions =
typeof options === "string" ? { message: options } : (options ?? {});
const initialMessage = opts.message;
const mode = opts.mode ?? this.config.outbound.defaultMode;
if (!this.provider) {
return { callId: "", success: false, error: "Provider not initialized" };
}
if (!this.webhookUrl) {
return {
callId: "",
success: false,
error: "Webhook URL not configured",
};
}
// Check concurrent call limit
const activeCalls = this.getActiveCalls();
if (activeCalls.length >= this.config.maxConcurrentCalls) {
return {
callId: "",
success: false,
error: `Maximum concurrent calls (${this.config.maxConcurrentCalls}) reached`,
};
}
const callId = crypto.randomUUID();
const from =
this.config.fromNumber ||
(this.provider?.name === "mock" ? "+15550000000" : undefined);
if (!from) {
return { callId: "", success: false, error: "fromNumber not configured" };
}
// Create call record with mode in metadata
const callRecord: CallRecord = {
callId,
provider: this.provider.name,
direction: "outbound",
state: "initiated",
from,
to,
sessionKey,
startedAt: Date.now(),
transcript: [],
processedEventIds: [],
metadata: {
...(initialMessage && { initialMessage }),
mode,
},
};
this.activeCalls.set(callId, callRecord);
this.persistCallRecord(callRecord);
try {
// For notify mode with a message, use inline TwiML with <Say>
let inlineTwiml: string | undefined;
if (mode === "notify" && initialMessage) {
const pollyVoice = mapVoiceToPolly(this.config.tts.voice);
inlineTwiml = this.generateNotifyTwiml(initialMessage, pollyVoice);
console.log(
`[voice-call] Using inline TwiML for notify mode (voice: ${pollyVoice})`,
);
}
const result = await this.provider.initiateCall({
callId,
from,
to,
webhookUrl: this.webhookUrl,
inlineTwiml,
});
callRecord.providerCallId = result.providerCallId;
this.providerCallIdMap.set(result.providerCallId, callId); // Map providerCallId to internal callId
this.persistCallRecord(callRecord);
return { callId, success: true };
} catch (err) {
callRecord.state = "failed";
callRecord.endedAt = Date.now();
callRecord.endReason = "failed";
this.persistCallRecord(callRecord);
this.activeCalls.delete(callId);
if (callRecord.providerCallId) {
this.providerCallIdMap.delete(callRecord.providerCallId);
}
return {
callId,
success: false,
error: err instanceof Error ? err.message : String(err),
};
}
}
/**
* Speak to user in an active call.
*/
async speak(
callId: CallId,
text: string,
): Promise<{ success: boolean; error?: string }> {
const call = this.activeCalls.get(callId);
if (!call) {
return { success: false, error: "Call not found" };
}
if (!this.provider || !call.providerCallId) {
return { success: false, error: "Call not connected" };
}
if (TerminalStates.has(call.state)) {
return { success: false, error: "Call has ended" };
}
try {
// Update state
call.state = "speaking";
this.persistCallRecord(call);
// Add to transcript
this.addTranscriptEntry(call, "bot", text);
// Play TTS
await this.provider.playTts({
callId,
providerCallId: call.providerCallId,
text,
voice: this.config.tts.voice,
});
return { success: true };
} catch (err) {
return {
success: false,
error: err instanceof Error ? err.message : String(err),
};
}
}
/**
* Speak the initial message for a call (called when media stream connects).
* This is used to auto-play the message passed to initiateCall.
* In notify mode, auto-hangup after the message is delivered.
*/
async speakInitialMessage(providerCallId: string): Promise<void> {
const call = this.getCallByProviderCallId(providerCallId);
if (!call) {
console.warn(
`[voice-call] speakInitialMessage: no call found for ${providerCallId}`,
);
return;
}
const initialMessage = call.metadata?.initialMessage as string | undefined;
const mode = (call.metadata?.mode as CallMode) ?? "conversation";
if (!initialMessage) {
console.log(
`[voice-call] speakInitialMessage: no initial message for ${call.callId}`,
);
return;
}
// Clear the initial message so we don't speak it again
if (call.metadata) {
delete call.metadata.initialMessage;
this.persistCallRecord(call);
}
console.log(
`[voice-call] Speaking initial message for call ${call.callId} (mode: ${mode})`,
);
const result = await this.speak(call.callId, initialMessage);
if (!result.success) {
console.warn(
`[voice-call] Failed to speak initial message: ${result.error}`,
);
return;
}
// In notify mode, auto-hangup after delay
if (mode === "notify") {
const delaySec = this.config.outbound.notifyHangupDelaySec;
console.log(
`[voice-call] Notify mode: auto-hangup in ${delaySec}s for call ${call.callId}`,
);
setTimeout(async () => {
const currentCall = this.getCall(call.callId);
if (currentCall && !TerminalStates.has(currentCall.state)) {
console.log(
`[voice-call] Notify mode: hanging up call ${call.callId}`,
);
await this.endCall(call.callId);
}
}, delaySec * 1000);
}
}
/**
* Start max duration timer for a call.
* Auto-hangup when maxDurationSeconds is reached.
*/
private startMaxDurationTimer(callId: CallId): void {
// Clear any existing timer
this.clearMaxDurationTimer(callId);
const maxDurationMs = this.config.maxDurationSeconds * 1000;
console.log(
`[voice-call] Starting max duration timer (${this.config.maxDurationSeconds}s) for call ${callId}`,
);
const timer = setTimeout(async () => {
this.maxDurationTimers.delete(callId);
const call = this.getCall(callId);
if (call && !TerminalStates.has(call.state)) {
console.log(
`[voice-call] Max duration reached (${this.config.maxDurationSeconds}s), ending call ${callId}`,
);
call.endReason = "timeout";
this.persistCallRecord(call);
await this.endCall(callId);
}
}, maxDurationMs);
this.maxDurationTimers.set(callId, timer);
}
/**
* Clear max duration timer for a call.
*/
private clearMaxDurationTimer(callId: CallId): void {
const timer = this.maxDurationTimers.get(callId);
if (timer) {
clearTimeout(timer);
this.maxDurationTimers.delete(callId);
}
}
private clearTranscriptWaiter(callId: CallId): void {
const waiter = this.transcriptWaiters.get(callId);
if (!waiter) return;
clearTimeout(waiter.timeout);
this.transcriptWaiters.delete(callId);
}
private rejectTranscriptWaiter(callId: CallId, reason: string): void {
const waiter = this.transcriptWaiters.get(callId);
if (!waiter) return;
this.clearTranscriptWaiter(callId);
waiter.reject(new Error(reason));
}
private resolveTranscriptWaiter(callId: CallId, transcript: string): void {
const waiter = this.transcriptWaiters.get(callId);
if (!waiter) return;
this.clearTranscriptWaiter(callId);
waiter.resolve(transcript);
}
private waitForFinalTranscript(callId: CallId): Promise<string> {
// Only allow one in-flight waiter per call.
this.rejectTranscriptWaiter(callId, "Transcript waiter replaced");
const timeoutMs = this.config.transcriptTimeoutMs;
return new Promise((resolve, reject) => {
const timeout = setTimeout(() => {
this.transcriptWaiters.delete(callId);
reject(
new Error(`Timed out waiting for transcript after ${timeoutMs}ms`),
);
}, timeoutMs);
this.transcriptWaiters.set(callId, { resolve, reject, timeout });
});
}
/**
* Continue call: speak prompt, then wait for user's final transcript.
*/
async continueCall(
callId: CallId,
prompt: string,
): Promise<{ success: boolean; transcript?: string; error?: string }> {
const call = this.activeCalls.get(callId);
if (!call) {
return { success: false, error: "Call not found" };
}
if (!this.provider || !call.providerCallId) {
return { success: false, error: "Call not connected" };
}
if (TerminalStates.has(call.state)) {
return { success: false, error: "Call has ended" };
}
try {
await this.speak(callId, prompt);
call.state = "listening";
this.persistCallRecord(call);
await this.provider.startListening({
callId,
providerCallId: call.providerCallId,
});
const transcript = await this.waitForFinalTranscript(callId);
// Best-effort: stop listening after final transcript.
await this.provider.stopListening({
callId,
providerCallId: call.providerCallId,
});
return { success: true, transcript };
} catch (err) {
return {
success: false,
error: err instanceof Error ? err.message : String(err),
};
} finally {
this.clearTranscriptWaiter(callId);
}
}
/**
* End an active call.
*/
async endCall(callId: CallId): Promise<{ success: boolean; error?: string }> {
const call = this.activeCalls.get(callId);
if (!call) {
return { success: false, error: "Call not found" };
}
if (!this.provider || !call.providerCallId) {
return { success: false, error: "Call not connected" };
}
if (TerminalStates.has(call.state)) {
return { success: true }; // Already ended
}
try {
await this.provider.hangupCall({
callId,
providerCallId: call.providerCallId,
reason: "hangup-bot",
});
call.state = "hangup-bot";
call.endedAt = Date.now();
call.endReason = "hangup-bot";
this.persistCallRecord(call);
this.clearMaxDurationTimer(callId);
this.rejectTranscriptWaiter(callId, "Call ended: hangup-bot");
this.activeCalls.delete(callId);
if (call.providerCallId) {
this.providerCallIdMap.delete(call.providerCallId);
}
return { success: true };
} catch (err) {
return {
success: false,
error: err instanceof Error ? err.message : String(err),
};
}
}
/**
* Check if an inbound call should be accepted based on policy.
*/
private shouldAcceptInbound(from: string | undefined): boolean {
const { inboundPolicy: policy, allowFrom } = this.config;
switch (policy) {
case "disabled":
console.log("[voice-call] Inbound call rejected: policy is disabled");
return false;
case "open":
console.log("[voice-call] Inbound call accepted: policy is open");
return true;
case "allowlist":
case "pairing": {
const normalized = from?.replace(/\D/g, "") || "";
const allowed = (allowFrom || []).some((num) => {
const normalizedAllow = num.replace(/\D/g, "");
return (
normalized.endsWith(normalizedAllow) ||
normalizedAllow.endsWith(normalized)
);
});
const status = allowed ? "accepted" : "rejected";
console.log(
`[voice-call] Inbound call ${status}: ${from} ${allowed ? "is in" : "not in"} allowlist`,
);
return allowed;
}
default:
return false;
}
}
/**
* Create a call record for an inbound call.
*/
private createInboundCall(
providerCallId: string,
from: string,
to: string,
): CallRecord {
const callId = crypto.randomUUID();
const callRecord: CallRecord = {
callId,
providerCallId,
provider: this.provider?.name || "twilio",
direction: "inbound",
state: "ringing",
from,
to,
startedAt: Date.now(),
transcript: [],
processedEventIds: [],
metadata: {
initialMessage:
this.config.inboundGreeting || "Hello! How can I help you today?",
},
};
this.activeCalls.set(callId, callRecord);
this.providerCallIdMap.set(providerCallId, callId); // Map providerCallId to internal callId
this.persistCallRecord(callRecord);
console.log(
`[voice-call] Created inbound call record: ${callId} from ${from}`,
);
return callRecord;
}
/**
* Look up a call by either internal callId or providerCallId.
*/
private findCall(callIdOrProviderCallId: string): CallRecord | undefined {
// Try direct lookup by internal callId
const directCall = this.activeCalls.get(callIdOrProviderCallId);
if (directCall) return directCall;
// Try lookup by providerCallId
return this.getCallByProviderCallId(callIdOrProviderCallId);
}
/**
* Process a webhook event.
*/
processEvent(event: NormalizedEvent): void {
// Idempotency check
if (this.processedEventIds.has(event.id)) {
return;
}
this.processedEventIds.add(event.id);
let call = this.findCall(event.callId);
// Handle inbound calls - create record if it doesn't exist
if (!call && event.direction === "inbound" && event.providerCallId) {
// Check if we should accept this inbound call
if (!this.shouldAcceptInbound(event.from)) {
// TODO: Could hang up the call here
return;
}
// Create a new call record for this inbound call
call = this.createInboundCall(
event.providerCallId,
event.from || "unknown",
event.to || this.config.fromNumber || "unknown",
);
// Update the event's callId to use our internal ID
event.callId = call.callId;
}
if (!call) {
// Still no call record - ignore event
return;
}
// Update provider call ID if we got it
if (event.providerCallId && !call.providerCallId) {
call.providerCallId = event.providerCallId;
}
// Track processed event
call.processedEventIds.push(event.id);
// Process event based on type
switch (event.type) {
case "call.initiated":
this.transitionState(call, "initiated");
break;
case "call.ringing":
this.transitionState(call, "ringing");
break;
case "call.answered":
call.answeredAt = event.timestamp;
this.transitionState(call, "answered");
// Start max duration timer when call is answered
this.startMaxDurationTimer(call.callId);
break;
case "call.active":
this.transitionState(call, "active");
break;
case "call.speaking":
this.transitionState(call, "speaking");
break;
case "call.speech":
if (event.isFinal) {
this.addTranscriptEntry(call, "user", event.transcript);
this.resolveTranscriptWaiter(call.callId, event.transcript);
}
this.transitionState(call, "listening");
break;
case "call.ended":
call.endedAt = event.timestamp;
call.endReason = event.reason;
this.transitionState(call, event.reason as CallState);
this.clearMaxDurationTimer(call.callId);
this.rejectTranscriptWaiter(call.callId, `Call ended: ${event.reason}`);
this.activeCalls.delete(call.callId);
if (call.providerCallId) {
this.providerCallIdMap.delete(call.providerCallId);
}
break;
case "call.error":
if (!event.retryable) {
call.endedAt = event.timestamp;
call.endReason = "error";
this.transitionState(call, "error");
this.clearMaxDurationTimer(call.callId);
this.rejectTranscriptWaiter(
call.callId,
`Call error: ${event.error}`,
);
this.activeCalls.delete(call.callId);
if (call.providerCallId) {
this.providerCallIdMap.delete(call.providerCallId);
}
}
break;
}
this.persistCallRecord(call);
}
/**
* Get an active call by ID.
*/
getCall(callId: CallId): CallRecord | undefined {
return this.activeCalls.get(callId);
}
/**
* Get an active call by provider call ID (e.g., Twilio CallSid).
*/
getCallByProviderCallId(providerCallId: string): CallRecord | undefined {
// Fast path: use the providerCallIdMap for O(1) lookup
const callId = this.providerCallIdMap.get(providerCallId);
if (callId) {
return this.activeCalls.get(callId);
}
// Fallback: linear search for cases where map wasn't populated
// (e.g., providerCallId set directly on call record)
for (const call of this.activeCalls.values()) {
if (call.providerCallId === providerCallId) {
return call;
}
}
return undefined;
}
/**
* Get all active calls.
*/
getActiveCalls(): CallRecord[] {
return Array.from(this.activeCalls.values());
}
/**
* Get call history (from persisted logs).
*/
async getCallHistory(limit = 50): Promise<CallRecord[]> {
const logPath = path.join(this.storePath, "calls.jsonl");
try {
await fsp.access(logPath);
} catch {
return [];
}
const content = await fsp.readFile(logPath, "utf-8");
const lines = content.trim().split("\n").filter(Boolean);
const calls: CallRecord[] = [];
// Parse last N lines
for (const line of lines.slice(-limit)) {
try {
const parsed = CallRecordSchema.parse(JSON.parse(line));
calls.push(parsed);
} catch {
// Skip invalid lines
}
}
return calls;
}
// States that can cycle during multi-turn conversations
private static readonly ConversationStates = new Set<CallState>([
"speaking",
"listening",
]);
// Non-terminal state order for monotonic transitions
private static readonly StateOrder: readonly CallState[] = [
"initiated",
"ringing",
"answered",
"active",
"speaking",
"listening",
];
/**
* Transition call state with monotonic enforcement.
*/
private transitionState(call: CallRecord, newState: CallState): void {
// No-op for same state or already terminal
if (call.state === newState || TerminalStates.has(call.state)) return;
// Terminal states can always be reached from non-terminal
if (TerminalStates.has(newState)) {
call.state = newState;
return;
}
// Allow cycling between speaking and listening (multi-turn conversations)
if (
CallManager.ConversationStates.has(call.state) &&
CallManager.ConversationStates.has(newState)
) {
call.state = newState;
return;
}
// Only allow forward transitions in state order
const currentIndex = CallManager.StateOrder.indexOf(call.state);
const newIndex = CallManager.StateOrder.indexOf(newState);
if (newIndex > currentIndex) {
call.state = newState;
}
}
/**
* Add an entry to the call transcript.
*/
private addTranscriptEntry(
call: CallRecord,
speaker: "bot" | "user",
text: string,
): void {
const entry: TranscriptEntry = {
timestamp: Date.now(),
speaker,
text,
isFinal: true,
};
call.transcript.push(entry);
}
/**
* Persist a call record to disk (fire-and-forget async).
*/
private persistCallRecord(call: CallRecord): void {
const logPath = path.join(this.storePath, "calls.jsonl");
const line = `${JSON.stringify(call)}\n`;
// Fire-and-forget async write to avoid blocking event loop
fsp.appendFile(logPath, line).catch((err) => {
console.error("[voice-call] Failed to persist call record:", err);
});
}
/**
* Load active calls from persistence (for crash recovery).
* Uses streaming to handle large log files efficiently.
*/
private loadActiveCalls(): void {
const logPath = path.join(this.storePath, "calls.jsonl");
if (!fs.existsSync(logPath)) return;
// Read file synchronously and parse lines
const content = fs.readFileSync(logPath, "utf-8");
const lines = content.split("\n");
// Build map of latest state per call
const callMap = new Map<CallId, CallRecord>();
for (const line of lines) {
if (!line.trim()) continue;
try {
const call = CallRecordSchema.parse(JSON.parse(line));
callMap.set(call.callId, call);
} catch {
// Skip invalid lines
}
}
// Only keep non-terminal calls
for (const [callId, call] of callMap) {
if (!TerminalStates.has(call.state)) {
this.activeCalls.set(callId, call);
// Populate providerCallId mapping for lookups
if (call.providerCallId) {
this.providerCallIdMap.set(call.providerCallId, callId);
}
// Populate processed event IDs
for (const eventId of call.processedEventIds) {
this.processedEventIds.add(eventId);
}
}
}
}
/**
* Generate TwiML for notify mode (speak message and hang up).
*/
private generateNotifyTwiml(message: string, voice: string): string {
return `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say voice="${voice}">${escapeXml(message)}</Say>
<Hangup/>
</Response>`;
}
}
+279
View File
@@ -0,0 +1,279 @@
/**
* Media Stream Handler
*
* Handles bidirectional audio streaming between Twilio and the AI services.
* - Receives mu-law audio from Twilio via WebSocket
* - Forwards to OpenAI Realtime STT for transcription
* - Sends TTS audio back to Twilio
*/
import type { IncomingMessage } from "node:http";
import type { Duplex } from "node:stream";
import { WebSocket, WebSocketServer } from "ws";
import type {
OpenAIRealtimeSTTProvider,
RealtimeSTTSession,
} from "./providers/stt-openai-realtime.js";
/**
* Configuration for the media stream handler.
*/
export interface MediaStreamConfig {
/** STT provider for transcription */
sttProvider: OpenAIRealtimeSTTProvider;
/** Callback when transcript is received */
onTranscript?: (callId: string, transcript: string) => void;
/** Callback for partial transcripts (streaming UI) */
onPartialTranscript?: (callId: string, partial: string) => void;
/** Callback when stream connects */
onConnect?: (callId: string, streamSid: string) => void;
/** Callback when stream disconnects */
onDisconnect?: (callId: string) => void;
}
/**
* Active media stream session.
*/
interface StreamSession {
callId: string;
streamSid: string;
ws: WebSocket;
sttSession: RealtimeSTTSession;
}
/**
* Manages WebSocket connections for Twilio media streams.
*/
export class MediaStreamHandler {
private wss: WebSocketServer | null = null;
private sessions = new Map<string, StreamSession>();
private config: MediaStreamConfig;
constructor(config: MediaStreamConfig) {
this.config = config;
}
/**
* Handle WebSocket upgrade for media stream connections.
*/
handleUpgrade(request: IncomingMessage, socket: Duplex, head: Buffer): void {
if (!this.wss) {
this.wss = new WebSocketServer({ noServer: true });
this.wss.on("connection", (ws, req) => this.handleConnection(ws, req));
}
this.wss.handleUpgrade(request, socket, head, (ws) => {
this.wss?.emit("connection", ws, request);
});
}
/**
* Handle new WebSocket connection from Twilio.
*/
private async handleConnection(
ws: WebSocket,
_request: IncomingMessage,
): Promise<void> {
let session: StreamSession | null = null;
ws.on("message", async (data: Buffer) => {
try {
const message = JSON.parse(data.toString()) as TwilioMediaMessage;
switch (message.event) {
case "connected":
console.log("[MediaStream] Twilio connected");
break;
case "start":
session = await this.handleStart(ws, message);
break;
case "media":
if (session && message.media?.payload) {
// Forward audio to STT
const audioBuffer = Buffer.from(message.media.payload, "base64");
session.sttSession.sendAudio(audioBuffer);
}
break;
case "stop":
if (session) {
this.handleStop(session);
session = null;
}
break;
}
} catch (error) {
console.error("[MediaStream] Error processing message:", error);
}
});
ws.on("close", () => {
if (session) {
this.handleStop(session);
}
});
ws.on("error", (error) => {
console.error("[MediaStream] WebSocket error:", error);
});
}
/**
* Handle stream start event.
*/
private async handleStart(
ws: WebSocket,
message: TwilioMediaMessage,
): Promise<StreamSession> {
const streamSid = message.streamSid || "";
const callSid = message.start?.callSid || "";
console.log(
`[MediaStream] Stream started: ${streamSid} (call: ${callSid})`,
);
// Create STT session
const sttSession = this.config.sttProvider.createSession();
// Set up transcript callbacks
sttSession.onPartial((partial) => {
this.config.onPartialTranscript?.(callSid, partial);
});
sttSession.onTranscript((transcript) => {
this.config.onTranscript?.(callSid, transcript);
});
const session: StreamSession = {
callId: callSid,
streamSid,
ws,
sttSession,
};
this.sessions.set(streamSid, session);
// Notify connection BEFORE STT connect so TTS can work even if STT fails
this.config.onConnect?.(callSid, streamSid);
// Connect to OpenAI STT (non-blocking, log errors but don't fail the call)
sttSession.connect().catch((err) => {
console.warn(
`[MediaStream] STT connection failed (TTS still works):`,
err.message,
);
});
return session;
}
/**
* Handle stream stop event.
*/
private handleStop(session: StreamSession): void {
console.log(`[MediaStream] Stream stopped: ${session.streamSid}`);
session.sttSession.close();
this.sessions.delete(session.streamSid);
this.config.onDisconnect?.(session.callId);
}
/**
* Get an active session with an open WebSocket, or undefined if unavailable.
*/
private getOpenSession(streamSid: string): StreamSession | undefined {
const session = this.sessions.get(streamSid);
return session?.ws.readyState === WebSocket.OPEN ? session : undefined;
}
/**
* Send a message to a stream's WebSocket if available.
*/
private sendToStream(streamSid: string, message: unknown): void {
const session = this.getOpenSession(streamSid);
session?.ws.send(JSON.stringify(message));
}
/**
* Send audio to a specific stream (for TTS playback).
* Audio should be mu-law encoded at 8kHz mono.
*/
sendAudio(streamSid: string, muLawAudio: Buffer): void {
this.sendToStream(streamSid, {
event: "media",
streamSid,
media: { payload: muLawAudio.toString("base64") },
});
}
/**
* Send a mark event to track audio playback position.
*/
sendMark(streamSid: string, name: string): void {
this.sendToStream(streamSid, {
event: "mark",
streamSid,
mark: { name },
});
}
/**
* Clear audio buffer (interrupt playback).
*/
clearAudio(streamSid: string): void {
this.sendToStream(streamSid, { event: "clear", streamSid });
}
/**
* Get active session by call ID.
*/
getSessionByCallId(callId: string): StreamSession | undefined {
return [...this.sessions.values()].find(
(session) => session.callId === callId,
);
}
/**
* Close all sessions.
*/
closeAll(): void {
for (const session of this.sessions.values()) {
session.sttSession.close();
session.ws.close();
}
this.sessions.clear();
}
}
/**
* Twilio Media Stream message format.
*/
interface TwilioMediaMessage {
event: "connected" | "start" | "media" | "stop" | "mark" | "clear";
sequenceNumber?: string;
streamSid?: string;
start?: {
streamSid: string;
accountSid: string;
callSid: string;
tracks: string[];
mediaFormat: {
encoding: string;
sampleRate: number;
channels: number;
};
};
media?: {
track?: string;
chunk?: string;
timestamp?: string;
payload?: string;
};
mark?: {
name: string;
};
}
@@ -0,0 +1,67 @@
import type {
HangupCallInput,
InitiateCallInput,
InitiateCallResult,
PlayTtsInput,
ProviderName,
ProviderWebhookParseResult,
StartListeningInput,
StopListeningInput,
WebhookContext,
WebhookVerificationResult,
} from "../types.js";
/**
* Abstract base interface for voice call providers.
*
* Each provider (Telnyx, Twilio, etc.) implements this interface to provide
* a consistent API for the call manager.
*
* Responsibilities:
* - Webhook verification and event parsing
* - Outbound call initiation and hangup
* - Media control (TTS playback, STT listening)
*/
export interface VoiceCallProvider {
/** Provider identifier */
readonly name: ProviderName;
/**
* Verify webhook signature/HMAC before processing.
* Must be called before parseWebhookEvent.
*/
verifyWebhook(ctx: WebhookContext): WebhookVerificationResult;
/**
* Parse provider-specific webhook payload into normalized events.
* Returns events and optional response to send back to provider.
*/
parseWebhookEvent(ctx: WebhookContext): ProviderWebhookParseResult;
/**
* Initiate an outbound call.
* @returns Provider call ID and status
*/
initiateCall(input: InitiateCallInput): Promise<InitiateCallResult>;
/**
* Hang up an active call.
*/
hangupCall(input: HangupCallInput): Promise<void>;
/**
* Play TTS audio to the caller.
* The provider should handle streaming if supported.
*/
playTts(input: PlayTtsInput): Promise<void>;
/**
* Start listening for user speech (activate STT).
*/
startListening(input: StartListeningInput): Promise<void>;
/**
* Stop listening for user speech (deactivate STT).
*/
stopListening(input: StopListeningInput): Promise<void>;
}
@@ -0,0 +1,9 @@
export type { VoiceCallProvider } from "./base.js";
export { MockProvider } from "./mock.js";
export {
OpenAIRealtimeSTTProvider,
type RealtimeSTTConfig,
type RealtimeSTTSession,
} from "./stt-openai-realtime.js";
export { TelnyxProvider } from "./telnyx.js";
export { TwilioProvider } from "./twilio.js";
+168
View File
@@ -0,0 +1,168 @@
import crypto from "node:crypto";
import type {
EndReason,
HangupCallInput,
InitiateCallInput,
InitiateCallResult,
NormalizedEvent,
PlayTtsInput,
ProviderWebhookParseResult,
StartListeningInput,
StopListeningInput,
WebhookContext,
WebhookVerificationResult,
} from "../types.js";
import type { VoiceCallProvider } from "./base.js";
/**
* Mock voice call provider for local testing.
*
* Events are driven via webhook POST with JSON body:
* - { events: NormalizedEvent[] } for bulk events
* - { event: NormalizedEvent } for single event
*/
export class MockProvider implements VoiceCallProvider {
readonly name = "mock" as const;
verifyWebhook(_ctx: WebhookContext): WebhookVerificationResult {
return { ok: true };
}
parseWebhookEvent(ctx: WebhookContext): ProviderWebhookParseResult {
try {
const payload = JSON.parse(ctx.rawBody);
const events: NormalizedEvent[] = [];
if (Array.isArray(payload.events)) {
for (const evt of payload.events) {
const normalized = this.normalizeEvent(evt);
if (normalized) events.push(normalized);
}
} else if (payload.event) {
const normalized = this.normalizeEvent(payload.event);
if (normalized) events.push(normalized);
}
return { events, statusCode: 200 };
} catch {
return { events: [], statusCode: 400 };
}
}
private normalizeEvent(
evt: Partial<NormalizedEvent>,
): NormalizedEvent | null {
if (!evt.type || !evt.callId) return null;
const base = {
id: evt.id || crypto.randomUUID(),
callId: evt.callId,
providerCallId: evt.providerCallId,
timestamp: evt.timestamp || Date.now(),
};
switch (evt.type) {
case "call.initiated":
case "call.ringing":
case "call.answered":
case "call.active":
return { ...base, type: evt.type };
case "call.speaking": {
const payload = evt as Partial<NormalizedEvent & { text?: string }>;
return {
...base,
type: evt.type,
text: payload.text || "",
};
}
case "call.speech": {
const payload = evt as Partial<
NormalizedEvent & {
transcript?: string;
isFinal?: boolean;
confidence?: number;
}
>;
return {
...base,
type: evt.type,
transcript: payload.transcript || "",
isFinal: payload.isFinal ?? true,
confidence: payload.confidence,
};
}
case "call.silence": {
const payload = evt as Partial<
NormalizedEvent & { durationMs?: number }
>;
return {
...base,
type: evt.type,
durationMs: payload.durationMs || 0,
};
}
case "call.dtmf": {
const payload = evt as Partial<NormalizedEvent & { digits?: string }>;
return {
...base,
type: evt.type,
digits: payload.digits || "",
};
}
case "call.ended": {
const payload = evt as Partial<
NormalizedEvent & { reason?: EndReason }
>;
return {
...base,
type: evt.type,
reason: payload.reason || "completed",
};
}
case "call.error": {
const payload = evt as Partial<
NormalizedEvent & { error?: string; retryable?: boolean }
>;
return {
...base,
type: evt.type,
error: payload.error || "unknown error",
retryable: payload.retryable,
};
}
default:
return null;
}
}
async initiateCall(input: InitiateCallInput): Promise<InitiateCallResult> {
return {
providerCallId: `mock-${input.callId}`,
status: "initiated",
};
}
async hangupCall(_input: HangupCallInput): Promise<void> {
// No-op for mock
}
async playTts(_input: PlayTtsInput): Promise<void> {
// No-op for mock
}
async startListening(_input: StartListeningInput): Promise<void> {
// No-op for mock
}
async stopListening(_input: StopListeningInput): Promise<void> {
// No-op for mock
}
}
@@ -0,0 +1,303 @@
/**
* OpenAI Realtime STT Provider
*
* Uses the OpenAI Realtime API for streaming transcription with:
* - Direct mu-law audio support (no conversion needed)
* - Built-in server-side VAD for turn detection
* - Low-latency streaming transcription
* - Partial transcript callbacks for real-time UI updates
*/
import WebSocket from "ws";
/**
* Configuration for OpenAI Realtime STT.
*/
export interface RealtimeSTTConfig {
/** OpenAI API key */
apiKey: string;
/** Model to use (default: gpt-4o-transcribe) */
model?: string;
/** Silence duration in ms before considering speech ended (default: 800) */
silenceDurationMs?: number;
/** VAD threshold 0-1 (default: 0.5) */
vadThreshold?: number;
}
/**
* Session for streaming audio and receiving transcripts.
*/
export interface RealtimeSTTSession {
/** Connect to the transcription service */
connect(): Promise<void>;
/** Send mu-law audio data (8kHz mono) */
sendAudio(audio: Buffer): void;
/** Wait for next complete transcript (after VAD detects end of speech) */
waitForTranscript(timeoutMs?: number): Promise<string>;
/** Set callback for partial transcripts (streaming) */
onPartial(callback: (partial: string) => void): void;
/** Set callback for final transcripts */
onTranscript(callback: (transcript: string) => void): void;
/** Close the session */
close(): void;
/** Check if session is connected */
isConnected(): boolean;
}
/**
* Provider factory for OpenAI Realtime STT sessions.
*/
export class OpenAIRealtimeSTTProvider {
readonly name = "openai-realtime";
private apiKey: string;
private model: string;
private silenceDurationMs: number;
private vadThreshold: number;
constructor(config: RealtimeSTTConfig) {
if (!config.apiKey) {
throw new Error("OpenAI API key required for Realtime STT");
}
this.apiKey = config.apiKey;
this.model = config.model || "gpt-4o-transcribe";
this.silenceDurationMs = config.silenceDurationMs || 800;
this.vadThreshold = config.vadThreshold || 0.5;
}
/**
* Create a new realtime transcription session.
*/
createSession(): RealtimeSTTSession {
return new OpenAIRealtimeSTTSession(
this.apiKey,
this.model,
this.silenceDurationMs,
this.vadThreshold,
);
}
}
/**
* WebSocket-based session for real-time speech-to-text.
*/
class OpenAIRealtimeSTTSession implements RealtimeSTTSession {
private static readonly MAX_RECONNECT_ATTEMPTS = 5;
private static readonly RECONNECT_DELAY_MS = 1000;
private ws: WebSocket | null = null;
private connected = false;
private closed = false;
private reconnectAttempts = 0;
private pendingTranscript = "";
private onTranscriptCallback: ((transcript: string) => void) | null = null;
private onPartialCallback: ((partial: string) => void) | null = null;
constructor(
private readonly apiKey: string,
private readonly model: string,
private readonly silenceDurationMs: number,
private readonly vadThreshold: number,
) {}
async connect(): Promise<void> {
this.closed = false;
this.reconnectAttempts = 0;
return this.doConnect();
}
private async doConnect(): Promise<void> {
return new Promise((resolve, reject) => {
const url = "wss://api.openai.com/v1/realtime?intent=transcription";
this.ws = new WebSocket(url, {
headers: {
Authorization: `Bearer ${this.apiKey}`,
"OpenAI-Beta": "realtime=v1",
},
});
this.ws.on("open", () => {
console.log("[RealtimeSTT] WebSocket connected");
this.connected = true;
this.reconnectAttempts = 0;
// Configure the transcription session
this.sendEvent({
type: "transcription_session.update",
session: {
input_audio_format: "g711_ulaw",
input_audio_transcription: {
model: this.model,
},
turn_detection: {
type: "server_vad",
threshold: this.vadThreshold,
prefix_padding_ms: 300,
silence_duration_ms: this.silenceDurationMs,
},
},
});
resolve();
});
this.ws.on("message", (data: Buffer) => {
try {
const event = JSON.parse(data.toString());
this.handleEvent(event);
} catch (e) {
console.error("[RealtimeSTT] Failed to parse event:", e);
}
});
this.ws.on("error", (error) => {
console.error("[RealtimeSTT] WebSocket error:", error);
if (!this.connected) reject(error);
});
this.ws.on("close", (code, reason) => {
console.log(
`[RealtimeSTT] WebSocket closed (code: ${code}, reason: ${reason?.toString() || "none"})`,
);
this.connected = false;
// Attempt reconnection if not intentionally closed
if (!this.closed) {
void this.attemptReconnect();
}
});
setTimeout(() => {
if (!this.connected) {
reject(new Error("Realtime STT connection timeout"));
}
}, 10000);
});
}
private async attemptReconnect(): Promise<void> {
if (this.closed) {
return;
}
if (
this.reconnectAttempts >= OpenAIRealtimeSTTSession.MAX_RECONNECT_ATTEMPTS
) {
console.error(
`[RealtimeSTT] Max reconnect attempts (${OpenAIRealtimeSTTSession.MAX_RECONNECT_ATTEMPTS}) reached`,
);
return;
}
this.reconnectAttempts++;
const delay =
OpenAIRealtimeSTTSession.RECONNECT_DELAY_MS *
2 ** (this.reconnectAttempts - 1);
console.log(
`[RealtimeSTT] Reconnecting ${this.reconnectAttempts}/${OpenAIRealtimeSTTSession.MAX_RECONNECT_ATTEMPTS} in ${delay}ms...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
if (this.closed) {
return;
}
try {
await this.doConnect();
console.log("[RealtimeSTT] Reconnected successfully");
} catch (error) {
console.error("[RealtimeSTT] Reconnect failed:", error);
}
}
private handleEvent(event: {
type: string;
delta?: string;
transcript?: string;
error?: unknown;
}): void {
switch (event.type) {
case "transcription_session.created":
case "transcription_session.updated":
case "input_audio_buffer.speech_stopped":
case "input_audio_buffer.committed":
console.log(`[RealtimeSTT] ${event.type}`);
break;
case "conversation.item.input_audio_transcription.delta":
if (event.delta) {
this.pendingTranscript += event.delta;
this.onPartialCallback?.(this.pendingTranscript);
}
break;
case "conversation.item.input_audio_transcription.completed":
if (event.transcript) {
console.log(`[RealtimeSTT] Transcript: ${event.transcript}`);
this.onTranscriptCallback?.(event.transcript);
}
this.pendingTranscript = "";
break;
case "input_audio_buffer.speech_started":
console.log("[RealtimeSTT] Speech started");
this.pendingTranscript = "";
break;
case "error":
console.error("[RealtimeSTT] Error:", event.error);
break;
}
}
private sendEvent(event: unknown): void {
if (this.ws?.readyState === WebSocket.OPEN) {
this.ws.send(JSON.stringify(event));
}
}
sendAudio(muLawData: Buffer): void {
if (!this.connected) return;
this.sendEvent({
type: "input_audio_buffer.append",
audio: muLawData.toString("base64"),
});
}
onPartial(callback: (partial: string) => void): void {
this.onPartialCallback = callback;
}
onTranscript(callback: (transcript: string) => void): void {
this.onTranscriptCallback = callback;
}
async waitForTranscript(timeoutMs = 30000): Promise<string> {
return new Promise((resolve, reject) => {
const timeout = setTimeout(() => {
this.onTranscriptCallback = null;
reject(new Error("Transcript timeout"));
}, timeoutMs);
this.onTranscriptCallback = (transcript) => {
clearTimeout(timeout);
this.onTranscriptCallback = null;
resolve(transcript);
};
});
}
close(): void {
this.closed = true;
if (this.ws) {
this.ws.close();
this.ws = null;
}
this.connected = false;
}
isConnected(): boolean {
return this.connected;
}
}
@@ -0,0 +1,364 @@
import crypto from "node:crypto";
import type { TelnyxConfig } from "../config.js";
import type {
EndReason,
HangupCallInput,
InitiateCallInput,
InitiateCallResult,
NormalizedEvent,
PlayTtsInput,
ProviderWebhookParseResult,
StartListeningInput,
StopListeningInput,
WebhookContext,
WebhookVerificationResult,
} from "../types.js";
import type { VoiceCallProvider } from "./base.js";
/**
* Telnyx Voice API provider implementation.
*
* Uses Telnyx Call Control API v2 for managing calls.
* @see https://developers.telnyx.com/docs/api/v2/call-control
*/
export class TelnyxProvider implements VoiceCallProvider {
readonly name = "telnyx" as const;
private readonly apiKey: string;
private readonly connectionId: string;
private readonly publicKey: string | undefined;
private readonly baseUrl = "https://api.telnyx.com/v2";
constructor(config: TelnyxConfig) {
if (!config.apiKey) {
throw new Error("Telnyx API key is required");
}
if (!config.connectionId) {
throw new Error("Telnyx connection ID is required");
}
this.apiKey = config.apiKey;
this.connectionId = config.connectionId;
this.publicKey = config.publicKey;
}
/**
* Make an authenticated request to the Telnyx API.
*/
private async apiRequest<T = unknown>(
endpoint: string,
body: Record<string, unknown>,
options?: { allowNotFound?: boolean },
): Promise<T> {
const response = await fetch(`${this.baseUrl}${endpoint}`, {
method: "POST",
headers: {
Authorization: `Bearer ${this.apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!response.ok) {
if (options?.allowNotFound && response.status === 404) {
return undefined as T;
}
const errorText = await response.text();
throw new Error(`Telnyx API error: ${response.status} ${errorText}`);
}
const text = await response.text();
return text ? (JSON.parse(text) as T) : (undefined as T);
}
/**
* Verify Telnyx webhook signature using Ed25519.
*/
verifyWebhook(ctx: WebhookContext): WebhookVerificationResult {
if (!this.publicKey) {
// No public key configured, skip verification (not recommended for production)
return { ok: true };
}
const signature = ctx.headers["telnyx-signature-ed25519"];
const timestamp = ctx.headers["telnyx-timestamp"];
if (!signature || !timestamp) {
return { ok: false, reason: "Missing signature or timestamp header" };
}
const signatureStr = Array.isArray(signature) ? signature[0] : signature;
const timestampStr = Array.isArray(timestamp) ? timestamp[0] : timestamp;
if (!signatureStr || !timestampStr) {
return { ok: false, reason: "Empty signature or timestamp" };
}
try {
const signedPayload = `${timestampStr}|${ctx.rawBody}`;
const signatureBuffer = Buffer.from(signatureStr, "base64");
const publicKeyBuffer = Buffer.from(this.publicKey, "base64");
const isValid = crypto.verify(
null, // Ed25519 doesn't use a digest
Buffer.from(signedPayload),
{
key: publicKeyBuffer,
format: "der",
type: "spki",
},
signatureBuffer,
);
if (!isValid) {
return { ok: false, reason: "Invalid signature" };
}
// Check timestamp is within 5 minutes
const eventTime = parseInt(timestampStr, 10) * 1000;
const now = Date.now();
if (Math.abs(now - eventTime) > 5 * 60 * 1000) {
return { ok: false, reason: "Timestamp too old" };
}
return { ok: true };
} catch (err) {
return {
ok: false,
reason: `Verification error: ${err instanceof Error ? err.message : String(err)}`,
};
}
}
/**
* Parse Telnyx webhook event into normalized format.
*/
parseWebhookEvent(ctx: WebhookContext): ProviderWebhookParseResult {
try {
const payload = JSON.parse(ctx.rawBody);
const data = payload.data;
if (!data || !data.event_type) {
return { events: [], statusCode: 200 };
}
const event = this.normalizeEvent(data);
return {
events: event ? [event] : [],
statusCode: 200,
};
} catch {
return { events: [], statusCode: 400 };
}
}
/**
* Convert Telnyx event to normalized event format.
*/
private normalizeEvent(data: TelnyxEvent): NormalizedEvent | null {
// Decode client_state from Base64 (we encode it in initiateCall)
let callId = "";
if (data.payload?.client_state) {
try {
callId = Buffer.from(data.payload.client_state, "base64").toString(
"utf8",
);
} catch {
// Fallback if not valid Base64
callId = data.payload.client_state;
}
}
if (!callId) {
callId = data.payload?.call_control_id || "";
}
const baseEvent = {
id: data.id || crypto.randomUUID(),
callId,
providerCallId: data.payload?.call_control_id,
timestamp: Date.now(),
};
switch (data.event_type) {
case "call.initiated":
return { ...baseEvent, type: "call.initiated" };
case "call.ringing":
return { ...baseEvent, type: "call.ringing" };
case "call.answered":
return { ...baseEvent, type: "call.answered" };
case "call.bridged":
return { ...baseEvent, type: "call.active" };
case "call.speak.started":
return {
...baseEvent,
type: "call.speaking",
text: data.payload?.text || "",
};
case "call.transcription":
return {
...baseEvent,
type: "call.speech",
transcript: data.payload?.transcription || "",
isFinal: data.payload?.is_final ?? true,
confidence: data.payload?.confidence,
};
case "call.hangup":
return {
...baseEvent,
type: "call.ended",
reason: this.mapHangupCause(data.payload?.hangup_cause),
};
case "call.dtmf.received":
return {
...baseEvent,
type: "call.dtmf",
digits: data.payload?.digit || "",
};
default:
return null;
}
}
/**
* Map Telnyx hangup cause to normalized end reason.
* @see https://developers.telnyx.com/docs/api/v2/call-control/Call-Commands#hangup-causes
*/
private mapHangupCause(cause?: string): EndReason {
switch (cause) {
case "normal_clearing":
case "normal_unspecified":
return "completed";
case "originator_cancel":
return "hangup-bot";
case "call_rejected":
case "user_busy":
return "busy";
case "no_answer":
case "no_user_response":
return "no-answer";
case "destination_out_of_order":
case "network_out_of_order":
case "service_unavailable":
case "recovery_on_timer_expire":
return "failed";
case "machine_detected":
case "fax_detected":
return "voicemail";
case "user_hangup":
case "subscriber_absent":
return "hangup-user";
default:
// Unknown cause - log it for debugging and return completed
if (cause) {
console.warn(`[telnyx] Unknown hangup cause: ${cause}`);
}
return "completed";
}
}
/**
* Initiate an outbound call via Telnyx API.
*/
async initiateCall(input: InitiateCallInput): Promise<InitiateCallResult> {
const result = await this.apiRequest<TelnyxCallResponse>("/calls", {
connection_id: this.connectionId,
to: input.to,
from: input.from,
webhook_url: input.webhookUrl,
webhook_url_method: "POST",
client_state: Buffer.from(input.callId).toString("base64"),
timeout_secs: 30,
});
return {
providerCallId: result.data.call_control_id,
status: "initiated",
};
}
/**
* Hang up a call via Telnyx API.
*/
async hangupCall(input: HangupCallInput): Promise<void> {
await this.apiRequest(
`/calls/${input.providerCallId}/actions/hangup`,
{ command_id: crypto.randomUUID() },
{ allowNotFound: true },
);
}
/**
* Play TTS audio via Telnyx speak action.
*/
async playTts(input: PlayTtsInput): Promise<void> {
await this.apiRequest(`/calls/${input.providerCallId}/actions/speak`, {
command_id: crypto.randomUUID(),
payload: input.text,
voice: input.voice || "female",
language: input.locale || "en-US",
});
}
/**
* Start transcription (STT) via Telnyx.
*/
async startListening(input: StartListeningInput): Promise<void> {
await this.apiRequest(
`/calls/${input.providerCallId}/actions/transcription_start`,
{
command_id: crypto.randomUUID(),
language: input.language || "en",
},
);
}
/**
* Stop transcription via Telnyx.
*/
async stopListening(input: StopListeningInput): Promise<void> {
await this.apiRequest(
`/calls/${input.providerCallId}/actions/transcription_stop`,
{ command_id: crypto.randomUUID() },
{ allowNotFound: true },
);
}
}
// -----------------------------------------------------------------------------
// Telnyx-specific types
// -----------------------------------------------------------------------------
interface TelnyxEvent {
id?: string;
event_type: string;
payload?: {
call_control_id?: string;
client_state?: string;
text?: string;
transcription?: string;
is_final?: boolean;
confidence?: number;
hangup_cause?: string;
digit?: string;
[key: string]: unknown;
};
}
interface TelnyxCallResponse {
data: {
call_control_id: string;
call_leg_id: string;
call_session_id: string;
is_alive: boolean;
record_type: string;
};
}
@@ -0,0 +1,264 @@
/**
* OpenAI TTS Provider
*
* Generates speech audio using OpenAI's text-to-speech API.
* Handles audio format conversion for telephony (mu-law 8kHz).
*
* Best practices from OpenAI docs:
* - Use gpt-4o-mini-tts for intelligent realtime applications (supports instructions)
* - Use tts-1 for lower latency, tts-1-hd for higher quality
* - Use marin or cedar voices for best quality
* - Use pcm or wav format for fastest response times
*
* @see https://platform.openai.com/docs/guides/text-to-speech
*/
/**
* OpenAI TTS configuration.
*/
export interface OpenAITTSConfig {
/** OpenAI API key (uses OPENAI_API_KEY env if not set) */
apiKey?: string;
/**
* TTS model:
* - gpt-4o-mini-tts: newest, supports instructions for tone/style control (recommended)
* - tts-1: lower latency
* - tts-1-hd: higher quality
*/
model?: string;
/**
* Voice to use. For best quality, use marin or cedar.
* All 13 voices: alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar
* Note: tts-1/tts-1-hd only support: alloy, ash, coral, echo, fable, onyx, nova, sage, shimmer
*/
voice?: string;
/** Speed multiplier (0.25 to 4.0) */
speed?: number;
/**
* Instructions for speech style (only works with gpt-4o-mini-tts model).
* Examples: "Speak in a cheerful tone", "Talk like a sympathetic customer service agent"
*/
instructions?: string;
}
/**
* Supported OpenAI TTS voices (all 13 built-in voices).
* For best quality, use marin or cedar.
* Note: tts-1 and tts-1-hd support a smaller set.
*/
export const OPENAI_TTS_VOICES = [
"alloy",
"ash",
"ballad",
"coral",
"echo",
"fable",
"nova",
"onyx",
"sage",
"shimmer",
"verse",
"marin",
"cedar",
] as const;
export type OpenAITTSVoice = (typeof OPENAI_TTS_VOICES)[number];
/**
* OpenAI TTS Provider for generating speech audio.
*/
export class OpenAITTSProvider {
private apiKey: string;
private model: string;
private voice: OpenAITTSVoice;
private speed: number;
private instructions?: string;
constructor(config: OpenAITTSConfig = {}) {
this.apiKey = config.apiKey || process.env.OPENAI_API_KEY || "";
// Default to gpt-4o-mini-tts for intelligent realtime applications
this.model = config.model || "gpt-4o-mini-tts";
// Default to coral - good balance of quality and natural tone
this.voice = (config.voice as OpenAITTSVoice) || "coral";
this.speed = config.speed || 1.0;
this.instructions = config.instructions;
if (!this.apiKey) {
throw new Error(
"OpenAI API key required (set OPENAI_API_KEY or pass apiKey)",
);
}
}
/**
* Generate speech audio from text.
* Returns raw PCM audio data (24kHz, mono, 16-bit).
*/
async synthesize(text: string, instructions?: string): Promise<Buffer> {
// Build request body
const body: Record<string, unknown> = {
model: this.model,
input: text,
voice: this.voice,
response_format: "pcm", // Raw PCM audio (24kHz, mono, 16-bit signed LE)
speed: this.speed,
};
// Add instructions if using gpt-4o-mini-tts model
const effectiveInstructions = instructions || this.instructions;
if (effectiveInstructions && this.model.includes("gpt-4o-mini-tts")) {
body.instructions = effectiveInstructions;
}
const response = await fetch("https://api.openai.com/v1/audio/speech", {
method: "POST",
headers: {
Authorization: `Bearer ${this.apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!response.ok) {
const error = await response.text();
throw new Error(`OpenAI TTS failed: ${response.status} - ${error}`);
}
const arrayBuffer = await response.arrayBuffer();
return Buffer.from(arrayBuffer);
}
/**
* Generate speech and convert to mu-law format for Twilio.
* Twilio Media Streams expect 8kHz mono mu-law audio.
*/
async synthesizeForTwilio(text: string): Promise<Buffer> {
// Get raw PCM from OpenAI (24kHz, 16-bit signed LE, mono)
const pcm24k = await this.synthesize(text);
// Resample from 24kHz to 8kHz
const pcm8k = resample24kTo8k(pcm24k);
// Encode to mu-law
return pcmToMulaw(pcm8k);
}
}
/**
* Resample 24kHz PCM to 8kHz using linear interpolation.
* Input/output: 16-bit signed little-endian mono.
*/
function resample24kTo8k(input: Buffer): Buffer {
const inputSamples = input.length / 2;
const outputSamples = Math.floor(inputSamples / 3);
const output = Buffer.alloc(outputSamples * 2);
for (let i = 0; i < outputSamples; i++) {
// Calculate position in input (3:1 ratio)
const srcPos = i * 3;
const srcIdx = srcPos * 2;
if (srcIdx + 3 < input.length) {
// Linear interpolation between samples
const s0 = input.readInt16LE(srcIdx);
const s1 = input.readInt16LE(srcIdx + 2);
const frac = srcPos % 1 || 0;
const sample = Math.round(s0 + frac * (s1 - s0));
output.writeInt16LE(clamp16(sample), i * 2);
} else {
// Last sample
output.writeInt16LE(input.readInt16LE(srcIdx), i * 2);
}
}
return output;
}
/**
* Clamp value to 16-bit signed integer range.
*/
function clamp16(value: number): number {
return Math.max(-32768, Math.min(32767, value));
}
/**
* Convert 16-bit PCM to 8-bit mu-law.
* Standard G.711 mu-law encoding for telephony.
*/
function pcmToMulaw(pcm: Buffer): Buffer {
const samples = pcm.length / 2;
const mulaw = Buffer.alloc(samples);
for (let i = 0; i < samples; i++) {
const sample = pcm.readInt16LE(i * 2);
mulaw[i] = linearToMulaw(sample);
}
return mulaw;
}
/**
* Convert a single 16-bit linear sample to 8-bit mu-law.
* Implements ITU-T G.711 mu-law encoding.
*/
function linearToMulaw(sample: number): number {
const BIAS = 132;
const CLIP = 32635;
// Get sign bit
const sign = sample < 0 ? 0x80 : 0;
if (sample < 0) sample = -sample;
// Clip to prevent overflow
if (sample > CLIP) sample = CLIP;
// Add bias and find segment
sample += BIAS;
let exponent = 7;
for (
let expMask = 0x4000;
(sample & expMask) === 0 && exponent > 0;
exponent--, expMask >>= 1
) {
// Find the segment (exponent)
}
// Extract mantissa bits
const mantissa = (sample >> (exponent + 3)) & 0x0f;
// Combine into mu-law byte (inverted for transmission)
return ~(sign | (exponent << 4) | mantissa) & 0xff;
}
/**
* Convert 8-bit mu-law to 16-bit linear PCM.
* Useful for decoding incoming audio.
*/
export function mulawToLinear(mulaw: number): number {
// mu-law is transmitted inverted
mulaw = ~mulaw & 0xff;
const sign = mulaw & 0x80;
const exponent = (mulaw >> 4) & 0x07;
const mantissa = mulaw & 0x0f;
let sample = ((mantissa << 3) + 132) << exponent;
sample -= 132;
return sign ? -sample : sample;
}
/**
* Chunk audio buffer into 20ms frames for streaming.
* At 8kHz mono, 20ms = 160 samples = 160 bytes (mu-law).
*/
export function chunkAudio(
audio: Buffer,
chunkSize = 160,
): Generator<Buffer, void, unknown> {
return (function* () {
for (let i = 0; i < audio.length; i += chunkSize) {
yield audio.subarray(i, Math.min(i + chunkSize, audio.length));
}
})();
}
@@ -0,0 +1,537 @@
import crypto from "node:crypto";
import type { TwilioConfig } from "../config.js";
import type { MediaStreamHandler } from "../media-stream.js";
import type {
HangupCallInput,
InitiateCallInput,
InitiateCallResult,
NormalizedEvent,
PlayTtsInput,
ProviderWebhookParseResult,
StartListeningInput,
StopListeningInput,
WebhookContext,
WebhookVerificationResult,
} from "../types.js";
import { escapeXml, mapVoiceToPolly } from "../voice-mapping.js";
import { verifyTwilioWebhook } from "../webhook-security.js";
import type { VoiceCallProvider } from "./base.js";
import type { OpenAITTSProvider } from "./tts-openai.js";
import { chunkAudio } from "./tts-openai.js";
/**
* Twilio Voice API provider implementation.
*
* Uses Twilio Programmable Voice API with Media Streams for real-time
* bidirectional audio streaming.
*
* @see https://www.twilio.com/docs/voice
* @see https://www.twilio.com/docs/voice/media-streams
*/
export interface TwilioProviderOptions {
/** Allow ngrok free tier compatibility mode (less secure) */
allowNgrokFreeTier?: boolean;
/** Override public URL for signature verification */
publicUrl?: string;
/** Path for media stream WebSocket (e.g., /voice/stream) */
streamPath?: string;
/** Skip webhook signature verification (development only) */
skipVerification?: boolean;
}
export class TwilioProvider implements VoiceCallProvider {
readonly name = "twilio" as const;
private readonly accountSid: string;
private readonly authToken: string;
private readonly baseUrl: string;
private readonly callWebhookUrls = new Map<string, string>();
private readonly options: TwilioProviderOptions;
/** Current public webhook URL (set when tunnel starts or from config) */
private currentPublicUrl: string | null = null;
/** Optional OpenAI TTS provider for streaming TTS */
private ttsProvider: OpenAITTSProvider | null = null;
/** Optional media stream handler for sending audio */
private mediaStreamHandler: MediaStreamHandler | null = null;
/** Map of call SID to stream SID for media streams */
private callStreamMap = new Map<string, string>();
constructor(config: TwilioConfig, options: TwilioProviderOptions = {}) {
if (!config.accountSid) {
throw new Error("Twilio Account SID is required");
}
if (!config.authToken) {
throw new Error("Twilio Auth Token is required");
}
this.accountSid = config.accountSid;
this.authToken = config.authToken;
this.baseUrl = `https://api.twilio.com/2010-04-01/Accounts/${this.accountSid}`;
this.options = options;
if (options.publicUrl) {
this.currentPublicUrl = options.publicUrl;
}
}
/**
* Set the current public webhook URL (called when tunnel starts).
*/
setPublicUrl(url: string): void {
this.currentPublicUrl = url;
}
/**
* Get the current public webhook URL.
*/
getPublicUrl(): string | null {
return this.currentPublicUrl;
}
/**
* Set the OpenAI TTS provider for streaming TTS.
* When set, playTts will use OpenAI audio via media streams.
*/
setTTSProvider(provider: OpenAITTSProvider): void {
this.ttsProvider = provider;
}
/**
* Set the media stream handler for sending audio.
*/
setMediaStreamHandler(handler: MediaStreamHandler): void {
this.mediaStreamHandler = handler;
}
/**
* Register a call's stream SID for audio routing.
*/
registerCallStream(callSid: string, streamSid: string): void {
this.callStreamMap.set(callSid, streamSid);
}
/**
* Unregister a call's stream SID.
*/
unregisterCallStream(callSid: string): void {
this.callStreamMap.delete(callSid);
}
/**
* Make an authenticated request to the Twilio API.
*/
private async apiRequest<T = unknown>(
endpoint: string,
params: Record<string, string>,
options?: { allowNotFound?: boolean },
): Promise<T> {
const response = await fetch(`${this.baseUrl}${endpoint}`, {
method: "POST",
headers: {
Authorization: `Basic ${Buffer.from(`${this.accountSid}:${this.authToken}`).toString("base64")}`,
"Content-Type": "application/x-www-form-urlencoded",
},
body: new URLSearchParams(params),
});
if (!response.ok) {
if (options?.allowNotFound && response.status === 404) {
return undefined as T;
}
const errorText = await response.text();
throw new Error(`Twilio API error: ${response.status} ${errorText}`);
}
const text = await response.text();
return text ? (JSON.parse(text) as T) : (undefined as T);
}
/**
* Verify Twilio webhook signature using HMAC-SHA1.
*
* Handles reverse proxy scenarios (Tailscale, nginx, ngrok) by reconstructing
* the public URL from forwarding headers.
*
* @see https://www.twilio.com/docs/usage/webhooks/webhooks-security
*/
verifyWebhook(ctx: WebhookContext): WebhookVerificationResult {
const result = verifyTwilioWebhook(ctx, this.authToken, {
publicUrl: this.currentPublicUrl || undefined,
allowNgrokFreeTier: this.options.allowNgrokFreeTier ?? true,
skipVerification: this.options.skipVerification,
});
if (!result.ok) {
console.warn(`[twilio] Webhook verification failed: ${result.reason}`);
if (result.verificationUrl) {
console.warn(`[twilio] Verification URL: ${result.verificationUrl}`);
}
}
return {
ok: result.ok,
reason: result.reason,
};
}
/**
* Parse Twilio webhook event into normalized format.
*/
parseWebhookEvent(ctx: WebhookContext): ProviderWebhookParseResult {
try {
const params = new URLSearchParams(ctx.rawBody);
const callIdFromQuery =
typeof ctx.query?.callId === "string" && ctx.query.callId.trim()
? ctx.query.callId.trim()
: undefined;
const event = this.normalizeEvent(params, callIdFromQuery);
// For Twilio, we must return TwiML. Most actions are driven by Calls API updates,
// so the webhook response is typically a pause to keep the call alive.
const twiml = this.generateTwimlResponse(ctx);
return {
events: event ? [event] : [],
providerResponseBody: twiml,
providerResponseHeaders: { "Content-Type": "application/xml" },
statusCode: 200,
};
} catch {
return { events: [], statusCode: 400 };
}
}
/**
* Parse Twilio direction to normalized format.
*/
private static parseDirection(
direction: string | null,
): "inbound" | "outbound" | undefined {
if (direction === "inbound") return "inbound";
if (direction === "outbound-api" || direction === "outbound-dial")
return "outbound";
return undefined;
}
/**
* Convert Twilio webhook params to normalized event format.
*/
private normalizeEvent(
params: URLSearchParams,
callIdOverride?: string,
): NormalizedEvent | null {
const callSid = params.get("CallSid") || "";
const baseEvent = {
id: crypto.randomUUID(),
callId: callIdOverride || callSid,
providerCallId: callSid,
timestamp: Date.now(),
direction: TwilioProvider.parseDirection(params.get("Direction")),
from: params.get("From") || undefined,
to: params.get("To") || undefined,
};
// Handle speech result (from <Gather>)
const speechResult = params.get("SpeechResult");
if (speechResult) {
return {
...baseEvent,
type: "call.speech",
transcript: speechResult,
isFinal: true,
confidence: parseFloat(params.get("Confidence") || "0.9"),
};
}
// Handle DTMF
const digits = params.get("Digits");
if (digits) {
return { ...baseEvent, type: "call.dtmf", digits };
}
// Handle call status changes
const callStatus = params.get("CallStatus");
switch (callStatus) {
case "initiated":
return { ...baseEvent, type: "call.initiated" };
case "ringing":
return { ...baseEvent, type: "call.ringing" };
case "in-progress":
return { ...baseEvent, type: "call.answered" };
case "completed":
case "busy":
case "no-answer":
case "failed":
return { ...baseEvent, type: "call.ended", reason: callStatus };
case "canceled":
return { ...baseEvent, type: "call.ended", reason: "hangup-bot" };
default:
return null;
}
}
private static readonly EMPTY_TWIML =
'<?xml version="1.0" encoding="UTF-8"?><Response></Response>';
private static readonly PAUSE_TWIML = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Pause length="30"/>
</Response>`;
/**
* Generate TwiML response for webhook.
* When a call is answered, connects to media stream for bidirectional audio.
*/
private generateTwimlResponse(ctx?: WebhookContext): string {
if (!ctx) return TwilioProvider.EMPTY_TWIML;
const params = new URLSearchParams(ctx.rawBody);
const callStatus = params.get("CallStatus");
const direction = params.get("Direction");
console.log(
`[voice-call] generateTwimlResponse: status=${callStatus} direction=${direction}`,
);
// For inbound calls, answer immediately with stream
if (direction === "inbound") {
const streamUrl = this.getStreamUrl();
return streamUrl
? this.getStreamConnectXml(streamUrl)
: TwilioProvider.PAUSE_TWIML;
}
// For outbound calls, only connect to stream when call is in-progress
if (callStatus !== "in-progress") {
return TwilioProvider.EMPTY_TWIML;
}
const streamUrl = this.getStreamUrl();
return streamUrl
? this.getStreamConnectXml(streamUrl)
: TwilioProvider.PAUSE_TWIML;
}
/**
* Get the WebSocket URL for media streaming.
* Derives from the public URL origin + stream path.
*/
private getStreamUrl(): string | null {
if (!this.currentPublicUrl || !this.options.streamPath) {
return null;
}
// Extract just the origin (host) from the public URL, ignoring any path
const url = new URL(this.currentPublicUrl);
const origin = url.origin;
// Convert https:// to wss:// for WebSocket
const wsOrigin = origin
.replace(/^https:\/\//, "wss://")
.replace(/^http:\/\//, "ws://");
// Append the stream path
const path = this.options.streamPath.startsWith("/")
? this.options.streamPath
: `/${this.options.streamPath}`;
return `${wsOrigin}${path}`;
}
/**
* Generate TwiML to connect a call to a WebSocket media stream.
* This enables bidirectional audio streaming for real-time STT/TTS.
*
* @param streamUrl - WebSocket URL (wss://...) for the media stream
*/
getStreamConnectXml(streamUrl: string): string {
return `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect>
<Stream url="${escapeXml(streamUrl)}" />
</Connect>
</Response>`;
}
/**
* Initiate an outbound call via Twilio API.
* If inlineTwiml is provided, uses that directly (for notify mode).
* Otherwise, uses webhook URL for dynamic TwiML.
*/
async initiateCall(input: InitiateCallInput): Promise<InitiateCallResult> {
const url = new URL(input.webhookUrl);
url.searchParams.set("callId", input.callId);
// Build request params
const params: Record<string, string> = {
To: input.to,
From: input.from,
StatusCallback: url.toString(),
StatusCallbackEvent: "initiated ringing answered completed",
Timeout: "30",
};
// Use inline TwiML for notify mode (simpler, no webhook needed)
if (input.inlineTwiml) {
params.Twiml = input.inlineTwiml;
} else {
params.Url = url.toString();
}
const result = await this.apiRequest<TwilioCallResponse>(
"/Calls.json",
params,
);
this.callWebhookUrls.set(result.sid, url.toString());
return {
providerCallId: result.sid,
status: result.status === "queued" ? "queued" : "initiated",
};
}
/**
* Hang up a call via Twilio API.
*/
async hangupCall(input: HangupCallInput): Promise<void> {
this.callWebhookUrls.delete(input.providerCallId);
await this.apiRequest(
`/Calls/${input.providerCallId}.json`,
{ Status: "completed" },
{ allowNotFound: true },
);
}
/**
* Play TTS audio via Twilio.
*
* Two modes:
* 1. OpenAI TTS + Media Streams: If TTS provider and media stream are available,
* generates audio via OpenAI and streams it through WebSocket (preferred).
* 2. TwiML <Say>: Falls back to Twilio's native TTS with Polly voices.
* Note: This may not work on all Twilio accounts.
*/
async playTts(input: PlayTtsInput): Promise<void> {
// Try OpenAI TTS via media stream first (if configured)
const streamSid = this.callStreamMap.get(input.providerCallId);
if (this.ttsProvider && this.mediaStreamHandler && streamSid) {
try {
await this.playTtsViaStream(input.text, streamSid);
return;
} catch (err) {
console.warn(
`[voice-call] OpenAI TTS failed, falling back to Twilio <Say>:`,
err instanceof Error ? err.message : err,
);
// Fall through to TwiML <Say> fallback
}
}
// Fall back to TwiML <Say> (may not work on all accounts)
const webhookUrl = this.callWebhookUrls.get(input.providerCallId);
if (!webhookUrl) {
throw new Error(
"Missing webhook URL for this call (provider state not initialized)",
);
}
console.warn(
"[voice-call] Using TwiML <Say> fallback - OpenAI TTS not configured or media stream not active",
);
const pollyVoice = mapVoiceToPolly(input.voice);
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say voice="${pollyVoice}" language="${input.locale || "en-US"}">${escapeXml(input.text)}</Say>
<Gather input="speech" speechTimeout="auto" action="${escapeXml(webhookUrl)}" method="POST">
<Say>.</Say>
</Gather>
</Response>`;
await this.apiRequest(`/Calls/${input.providerCallId}.json`, {
Twiml: twiml,
});
}
/**
* Play TTS via OpenAI and Twilio Media Streams.
* Generates audio with OpenAI TTS, converts to mu-law, and streams via WebSocket.
* Uses a jitter buffer to smooth out timing variations.
*/
private async playTtsViaStream(
text: string,
streamSid: string,
): Promise<void> {
if (!this.ttsProvider || !this.mediaStreamHandler) {
throw new Error("TTS provider and media stream handler required");
}
// Generate audio with OpenAI TTS (returns mu-law at 8kHz)
const muLawAudio = await this.ttsProvider.synthesizeForTwilio(text);
// Stream audio in 20ms chunks (160 bytes at 8kHz mu-law)
const CHUNK_SIZE = 160;
const CHUNK_DELAY_MS = 20;
for (const chunk of chunkAudio(muLawAudio, CHUNK_SIZE)) {
this.mediaStreamHandler.sendAudio(streamSid, chunk);
// Pace the audio to match real-time playback
await new Promise((resolve) => setTimeout(resolve, CHUNK_DELAY_MS));
}
// Send a mark to track when audio finishes
this.mediaStreamHandler.sendMark(streamSid, `tts-${Date.now()}`);
}
/**
* Start listening for speech via Twilio <Gather>.
*/
async startListening(input: StartListeningInput): Promise<void> {
const webhookUrl = this.callWebhookUrls.get(input.providerCallId);
if (!webhookUrl) {
throw new Error(
"Missing webhook URL for this call (provider state not initialized)",
);
}
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Gather input="speech" speechTimeout="auto" language="${input.language || "en-US"}" action="${escapeXml(webhookUrl)}" method="POST">
</Gather>
</Response>`;
await this.apiRequest(`/Calls/${input.providerCallId}.json`, {
Twiml: twiml,
});
}
/**
* Stop listening - for Twilio this is a no-op as <Gather> auto-ends.
*/
async stopListening(_input: StopListeningInput): Promise<void> {
// Twilio's <Gather> automatically stops on speech end
// No explicit action needed
}
}
// -----------------------------------------------------------------------------
// Twilio-specific types
// -----------------------------------------------------------------------------
interface TwilioCallResponse {
sid: string;
status: string;
direction: string;
from: string;
to: string;
uri: string;
}
@@ -0,0 +1,171 @@
/**
* Voice call response generator - uses the embedded Pi agent for tool support.
* Routes voice responses through the same agent infrastructure as messaging.
*/
import crypto from "node:crypto";
import { loadCoreAgentDeps, type CoreConfig } from "./core-bridge.js";
import type { VoiceCallConfig } from "./config.js";
export type VoiceResponseParams = {
/** Voice call config */
voiceConfig: VoiceCallConfig;
/** Core Clawdbot config */
coreConfig: CoreConfig;
/** Call ID for session tracking */
callId: string;
/** Caller's phone number */
from: string;
/** Conversation transcript */
transcript: Array<{ speaker: "user" | "bot"; text: string }>;
/** Latest user message */
userMessage: string;
};
export type VoiceResponseResult = {
text: string | null;
error?: string;
};
type SessionEntry = {
sessionId: string;
updatedAt: number;
};
/**
* Generate a voice response using the embedded Pi agent with full tool support.
* Uses the same agent infrastructure as messaging for consistent behavior.
*/
export async function generateVoiceResponse(
params: VoiceResponseParams,
): Promise<VoiceResponseResult> {
const { voiceConfig, callId, from, transcript, userMessage, coreConfig } =
params;
if (!coreConfig) {
return { text: null, error: "Core config unavailable for voice response" };
}
let deps: Awaited<ReturnType<typeof loadCoreAgentDeps>>;
try {
deps = await loadCoreAgentDeps();
} catch (err) {
return {
text: null,
error:
err instanceof Error
? err.message
: "Unable to load core agent dependencies",
};
}
const cfg = coreConfig;
// Build voice-specific session key based on phone number
const normalizedPhone = from.replace(/\D/g, "");
const sessionKey = `voice:${normalizedPhone}`;
const agentId = "main";
// Resolve paths
const storePath = deps.resolveStorePath(cfg.session?.store, { agentId });
const agentDir = deps.resolveAgentDir(cfg, agentId);
const workspaceDir = deps.resolveAgentWorkspaceDir(cfg, agentId);
// Ensure workspace exists
await deps.ensureAgentWorkspace({ dir: workspaceDir });
// Load or create session entry
const sessionStore = deps.loadSessionStore(storePath);
const now = Date.now();
let sessionEntry = sessionStore[sessionKey] as SessionEntry | undefined;
if (!sessionEntry) {
sessionEntry = {
sessionId: crypto.randomUUID(),
updatedAt: now,
};
sessionStore[sessionKey] = sessionEntry;
await deps.saveSessionStore(storePath, sessionStore);
}
const sessionId = sessionEntry.sessionId;
const sessionFile = deps.resolveSessionFilePath(sessionId, sessionEntry, {
agentId,
});
// Resolve model from config
const modelRef =
voiceConfig.responseModel ||
`${deps.DEFAULT_PROVIDER}/${deps.DEFAULT_MODEL}`;
const slashIndex = modelRef.indexOf("/");
const provider =
slashIndex === -1 ? deps.DEFAULT_PROVIDER : modelRef.slice(0, slashIndex);
const model = slashIndex === -1 ? modelRef : modelRef.slice(slashIndex + 1);
// Resolve thinking level
const thinkLevel = deps.resolveThinkingDefault({ cfg, provider, model });
// Resolve agent identity for personalized prompt
const identity = deps.resolveAgentIdentity(cfg, agentId);
const agentName = identity?.name?.trim() || "assistant";
// Build system prompt with conversation history
const basePrompt =
voiceConfig.responseSystemPrompt ??
`You are ${agentName}, a helpful voice assistant on a phone call. Keep responses brief and conversational (1-2 sentences max). Be natural and friendly. The caller's phone number is ${from}. You have access to tools - use them when helpful.`;
let extraSystemPrompt = basePrompt;
if (transcript.length > 0) {
const history = transcript
.map(
(entry) =>
`${entry.speaker === "bot" ? "You" : "Caller"}: ${entry.text}`,
)
.join("\n");
extraSystemPrompt = `${basePrompt}\n\nConversation so far:\n${history}`;
}
// Resolve timeout
const timeoutMs =
voiceConfig.responseTimeoutMs ?? deps.resolveAgentTimeoutMs({ cfg });
const runId = `voice:${callId}:${Date.now()}`;
try {
const result = await deps.runEmbeddedPiAgent({
sessionId,
sessionKey,
messageProvider: "voice",
sessionFile,
workspaceDir,
config: cfg,
prompt: userMessage,
provider,
model,
thinkLevel,
verboseLevel: "off",
timeoutMs,
runId,
lane: "voice",
extraSystemPrompt,
agentDir,
});
// Extract text from payloads
const texts = (result.payloads ?? [])
.filter((p) => p.text && !p.isError)
.map((p) => p.text?.trim())
.filter(Boolean);
const text = texts.join(" ") || null;
if (!text && result.meta.aborted) {
return { text: null, error: "Response generation was aborted" };
}
return { text };
} catch (err) {
console.error(`[voice-call] Response generation failed:`, err);
return { text: null, error: String(err) };
}
}
+194
View File
@@ -0,0 +1,194 @@
import type { CoreConfig } from "./core-bridge.js";
import type { VoiceCallConfig } from "./config.js";
import { validateProviderConfig } from "./config.js";
import { CallManager } from "./manager.js";
import type { VoiceCallProvider } from "./providers/base.js";
import { MockProvider } from "./providers/mock.js";
import { TelnyxProvider } from "./providers/telnyx.js";
import { OpenAITTSProvider } from "./providers/tts-openai.js";
import { TwilioProvider } from "./providers/twilio.js";
import { startTunnel, type TunnelResult } from "./tunnel.js";
import {
cleanupTailscaleExposure,
setupTailscaleExposure,
VoiceCallWebhookServer,
} from "./webhook.js";
export type VoiceCallRuntime = {
config: VoiceCallConfig;
provider: VoiceCallProvider;
manager: CallManager;
webhookServer: VoiceCallWebhookServer;
webhookUrl: string;
publicUrl: string | null;
stop: () => Promise<void>;
};
type Logger = {
info: (message: string) => void;
warn: (message: string) => void;
error: (message: string) => void;
debug: (message: string) => void;
};
function resolveProvider(config: VoiceCallConfig): VoiceCallProvider {
switch (config.provider) {
case "telnyx":
return new TelnyxProvider({
apiKey: config.telnyx?.apiKey ?? process.env.TELNYX_API_KEY,
connectionId:
config.telnyx?.connectionId ?? process.env.TELNYX_CONNECTION_ID,
publicKey: config.telnyx?.publicKey ?? process.env.TELNYX_PUBLIC_KEY,
});
case "twilio":
return new TwilioProvider(
{
accountSid:
config.twilio?.accountSid ?? process.env.TWILIO_ACCOUNT_SID,
authToken: config.twilio?.authToken ?? process.env.TWILIO_AUTH_TOKEN,
},
{
allowNgrokFreeTier: config.tunnel?.allowNgrokFreeTier ?? true,
publicUrl: config.publicUrl,
skipVerification: config.skipSignatureVerification,
streamPath: config.streaming?.enabled
? config.streaming.streamPath
: undefined,
},
);
case "mock":
return new MockProvider();
default:
throw new Error(
`Unsupported voice-call provider: ${String(config.provider)}`,
);
}
}
export async function createVoiceCallRuntime(params: {
config: VoiceCallConfig;
coreConfig: CoreConfig;
logger?: Logger;
}): Promise<VoiceCallRuntime> {
const { config, coreConfig, logger } = params;
const log = logger ?? {
info: console.log,
warn: console.warn,
error: console.error,
debug: console.debug,
};
if (!config.enabled) {
throw new Error(
"Voice call disabled. Enable the plugin entry in config.",
);
}
const validation = validateProviderConfig(config);
if (!validation.valid) {
throw new Error(`Invalid voice-call config: ${validation.errors.join("; ")}`);
}
const provider = resolveProvider(config);
const manager = new CallManager(config);
const webhookServer = new VoiceCallWebhookServer(
config,
manager,
provider,
coreConfig,
);
const localUrl = await webhookServer.start();
// Determine public URL - priority: config.publicUrl > tunnel > legacy tailscale
let publicUrl: string | null = config.publicUrl ?? null;
let tunnelResult: TunnelResult | null = null;
if (!publicUrl && config.tunnel?.provider && config.tunnel.provider !== "none") {
try {
tunnelResult = await startTunnel({
provider: config.tunnel.provider,
port: config.serve.port,
path: config.serve.path,
ngrokAuthToken:
config.tunnel.ngrokAuthToken ?? process.env.NGROK_AUTHTOKEN,
ngrokDomain: config.tunnel.ngrokDomain ?? process.env.NGROK_DOMAIN,
});
publicUrl = tunnelResult?.publicUrl ?? null;
} catch (err) {
log.error(
`[voice-call] Tunnel setup failed: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
}
if (!publicUrl && config.tailscale?.mode !== "off") {
publicUrl = await setupTailscaleExposure(config);
}
const webhookUrl = publicUrl ?? localUrl;
if (publicUrl && provider.name === "twilio") {
(provider as TwilioProvider).setPublicUrl(publicUrl);
}
if (provider.name === "twilio" && config.streaming?.enabled) {
const twilioProvider = provider as TwilioProvider;
const openaiApiKey =
config.streaming.openaiApiKey || process.env.OPENAI_API_KEY;
if (openaiApiKey) {
try {
const ttsProvider = new OpenAITTSProvider({
apiKey: openaiApiKey,
voice: config.tts.voice,
model: config.tts.model,
instructions: config.tts.instructions,
});
twilioProvider.setTTSProvider(ttsProvider);
log.info("[voice-call] OpenAI TTS provider configured");
} catch (err) {
log.warn(
`[voice-call] Failed to initialize OpenAI TTS: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
} else {
log.warn("[voice-call] OpenAI TTS key missing; streaming TTS disabled");
}
const mediaHandler = webhookServer.getMediaStreamHandler();
if (mediaHandler) {
twilioProvider.setMediaStreamHandler(mediaHandler);
log.info("[voice-call] Media stream handler wired to provider");
}
}
manager.initialize(provider, webhookUrl);
const stop = async () => {
if (tunnelResult) {
await tunnelResult.stop();
}
await cleanupTailscaleExposure(config);
await webhookServer.stop();
};
log.info("[voice-call] Runtime initialized");
log.info(`[voice-call] Webhook URL: ${webhookUrl}`);
if (publicUrl) {
log.info(`[voice-call] Public URL: ${publicUrl}`);
}
return {
config,
provider,
manager,
webhookServer,
webhookUrl,
publicUrl,
stop,
};
}
+330
View File
@@ -0,0 +1,330 @@
import { spawn } from "node:child_process";
import { getTailscaleDnsName } from "./webhook.js";
/**
* Tunnel configuration for exposing the webhook server.
*/
export interface TunnelConfig {
/** Tunnel provider: ngrok, tailscale-serve, or tailscale-funnel */
provider: "ngrok" | "tailscale-serve" | "tailscale-funnel" | "none";
/** Local port to tunnel */
port: number;
/** Path prefix for the tunnel (e.g., /voice/webhook) */
path: string;
/** ngrok auth token (optional, enables longer sessions) */
ngrokAuthToken?: string;
/** ngrok custom domain (paid feature) */
ngrokDomain?: string;
}
/**
* Result of starting a tunnel.
*/
export interface TunnelResult {
/** The public URL */
publicUrl: string;
/** Function to stop the tunnel */
stop: () => Promise<void>;
/** Tunnel provider name */
provider: string;
}
/**
* Start an ngrok tunnel to expose the local webhook server.
*
* Uses the ngrok CLI which must be installed: https://ngrok.com/download
*
* @example
* const tunnel = await startNgrokTunnel({ port: 3334, path: '/voice/webhook' });
* console.log('Public URL:', tunnel.publicUrl);
* // Later: await tunnel.stop();
*/
export async function startNgrokTunnel(config: {
port: number;
path: string;
authToken?: string;
domain?: string;
}): Promise<TunnelResult> {
// Set auth token if provided
if (config.authToken) {
await runNgrokCommand(["config", "add-authtoken", config.authToken]);
}
// Build ngrok command args
const args = [
"http",
String(config.port),
"--log",
"stdout",
"--log-format",
"json",
];
// Add custom domain if provided (paid ngrok feature)
if (config.domain) {
args.push("--domain", config.domain);
}
return new Promise((resolve, reject) => {
const proc = spawn("ngrok", args, {
stdio: ["ignore", "pipe", "pipe"],
});
let resolved = false;
let publicUrl: string | null = null;
let outputBuffer = "";
const timeout = setTimeout(() => {
if (!resolved) {
resolved = true;
proc.kill("SIGTERM");
reject(new Error("ngrok startup timed out (30s)"));
}
}, 30000);
const processLine = (line: string) => {
try {
const log = JSON.parse(line);
// ngrok logs the public URL in a 'started tunnel' message
if (log.msg === "started tunnel" && log.url) {
publicUrl = log.url;
}
// Also check for the URL field directly
if (log.addr && log.url && !publicUrl) {
publicUrl = log.url;
}
// Check for ready state
if (publicUrl && !resolved) {
resolved = true;
clearTimeout(timeout);
// Add path to the public URL
const fullUrl = publicUrl + config.path;
console.log(`[voice-call] ngrok tunnel active: ${fullUrl}`);
resolve({
publicUrl: fullUrl,
provider: "ngrok",
stop: async () => {
proc.kill("SIGTERM");
await new Promise<void>((res) => {
proc.on("close", () => res());
setTimeout(res, 2000); // Fallback timeout
});
},
});
}
} catch {
// Not JSON, might be startup message
}
};
proc.stdout.on("data", (data: Buffer) => {
outputBuffer += data.toString();
const lines = outputBuffer.split("\n");
outputBuffer = lines.pop() || "";
for (const line of lines) {
if (line.trim()) {
processLine(line);
}
}
});
proc.stderr.on("data", (data: Buffer) => {
const msg = data.toString();
// Check for common errors
if (msg.includes("ERR_NGROK")) {
if (!resolved) {
resolved = true;
clearTimeout(timeout);
reject(new Error(`ngrok error: ${msg}`));
}
}
});
proc.on("error", (err) => {
if (!resolved) {
resolved = true;
clearTimeout(timeout);
reject(new Error(`Failed to start ngrok: ${err.message}`));
}
});
proc.on("close", (code) => {
if (!resolved) {
resolved = true;
clearTimeout(timeout);
reject(new Error(`ngrok exited unexpectedly with code ${code}`));
}
});
});
}
/**
* Run an ngrok command and wait for completion.
*/
async function runNgrokCommand(args: string[]): Promise<string> {
return new Promise((resolve, reject) => {
const proc = spawn("ngrok", args, {
stdio: ["ignore", "pipe", "pipe"],
});
let stdout = "";
let stderr = "";
proc.stdout.on("data", (data) => {
stdout += data.toString();
});
proc.stderr.on("data", (data) => {
stderr += data.toString();
});
proc.on("close", (code) => {
if (code === 0) {
resolve(stdout);
} else {
reject(new Error(`ngrok command failed: ${stderr || stdout}`));
}
});
proc.on("error", reject);
});
}
/**
* Check if ngrok is installed and available.
*/
export async function isNgrokAvailable(): Promise<boolean> {
return new Promise((resolve) => {
const proc = spawn("ngrok", ["version"], {
stdio: ["ignore", "pipe", "pipe"],
});
proc.on("close", (code) => {
resolve(code === 0);
});
proc.on("error", () => {
resolve(false);
});
});
}
/**
* Start a Tailscale serve/funnel tunnel.
*/
export async function startTailscaleTunnel(config: {
mode: "serve" | "funnel";
port: number;
path: string;
}): Promise<TunnelResult> {
// Get Tailscale DNS name
const dnsName = await getTailscaleDnsName();
if (!dnsName) {
throw new Error("Could not get Tailscale DNS name. Is Tailscale running?");
}
const localUrl = `http://127.0.0.1:${config.port}`;
return new Promise((resolve, reject) => {
const proc = spawn(
"tailscale",
[config.mode, "--bg", "--yes", "--set-path", config.path, localUrl],
{ stdio: ["ignore", "pipe", "pipe"] },
);
const timeout = setTimeout(() => {
proc.kill("SIGKILL");
reject(new Error(`Tailscale ${config.mode} timed out`));
}, 10000);
proc.on("close", (code) => {
clearTimeout(timeout);
if (code === 0) {
const publicUrl = `https://${dnsName}${config.path}`;
console.log(
`[voice-call] Tailscale ${config.mode} active: ${publicUrl}`,
);
resolve({
publicUrl,
provider: `tailscale-${config.mode}`,
stop: async () => {
await stopTailscaleTunnel(config.mode, config.path);
},
});
} else {
reject(new Error(`Tailscale ${config.mode} failed with code ${code}`));
}
});
proc.on("error", (err) => {
clearTimeout(timeout);
reject(err);
});
});
}
/**
* Stop a Tailscale serve/funnel tunnel.
*/
async function stopTailscaleTunnel(
mode: "serve" | "funnel",
path: string,
): Promise<void> {
return new Promise((resolve) => {
const proc = spawn("tailscale", [mode, "off", path], {
stdio: "ignore",
});
const timeout = setTimeout(() => {
proc.kill("SIGKILL");
resolve();
}, 5000);
proc.on("close", () => {
clearTimeout(timeout);
resolve();
});
});
}
/**
* Start a tunnel based on configuration.
*/
export async function startTunnel(
config: TunnelConfig,
): Promise<TunnelResult | null> {
switch (config.provider) {
case "ngrok":
return startNgrokTunnel({
port: config.port,
path: config.path,
authToken: config.ngrokAuthToken,
domain: config.ngrokDomain,
});
case "tailscale-serve":
return startTailscaleTunnel({
mode: "serve",
port: config.port,
path: config.path,
});
case "tailscale-funnel":
return startTailscaleTunnel({
mode: "funnel",
port: config.port,
path: config.path,
});
default:
return null;
}
}
+272
View File
@@ -0,0 +1,272 @@
import { z } from "zod";
import type { CallMode } from "./config.js";
// -----------------------------------------------------------------------------
// Provider Identifiers
// -----------------------------------------------------------------------------
export const ProviderNameSchema = z.enum(["telnyx", "twilio", "mock"]);
export type ProviderName = z.infer<typeof ProviderNameSchema>;
// -----------------------------------------------------------------------------
// Core Call Identifiers
// -----------------------------------------------------------------------------
/** Internal call identifier (UUID) */
export type CallId = string;
/** Provider-specific call identifier */
export type ProviderCallId = string;
// -----------------------------------------------------------------------------
// Call Lifecycle States
// -----------------------------------------------------------------------------
export const CallStateSchema = z.enum([
// Non-terminal states
"initiated",
"ringing",
"answered",
"active",
"speaking",
"listening",
// Terminal states
"completed",
"hangup-user",
"hangup-bot",
"timeout",
"error",
"failed",
"no-answer",
"busy",
"voicemail",
]);
export type CallState = z.infer<typeof CallStateSchema>;
export const TerminalStates = new Set<CallState>([
"completed",
"hangup-user",
"hangup-bot",
"timeout",
"error",
"failed",
"no-answer",
"busy",
"voicemail",
]);
export const EndReasonSchema = z.enum([
"completed",
"hangup-user",
"hangup-bot",
"timeout",
"error",
"failed",
"no-answer",
"busy",
"voicemail",
]);
export type EndReason = z.infer<typeof EndReasonSchema>;
// -----------------------------------------------------------------------------
// Normalized Call Events
// -----------------------------------------------------------------------------
const BaseEventSchema = z.object({
id: z.string(),
callId: z.string(),
providerCallId: z.string().optional(),
timestamp: z.number(),
// Optional fields for inbound call detection
direction: z.enum(["inbound", "outbound"]).optional(),
from: z.string().optional(),
to: z.string().optional(),
});
export const NormalizedEventSchema = z.discriminatedUnion("type", [
BaseEventSchema.extend({
type: z.literal("call.initiated"),
}),
BaseEventSchema.extend({
type: z.literal("call.ringing"),
}),
BaseEventSchema.extend({
type: z.literal("call.answered"),
}),
BaseEventSchema.extend({
type: z.literal("call.active"),
}),
BaseEventSchema.extend({
type: z.literal("call.speaking"),
text: z.string(),
}),
BaseEventSchema.extend({
type: z.literal("call.speech"),
transcript: z.string(),
isFinal: z.boolean(),
confidence: z.number().min(0).max(1).optional(),
}),
BaseEventSchema.extend({
type: z.literal("call.silence"),
durationMs: z.number(),
}),
BaseEventSchema.extend({
type: z.literal("call.dtmf"),
digits: z.string(),
}),
BaseEventSchema.extend({
type: z.literal("call.ended"),
reason: EndReasonSchema,
}),
BaseEventSchema.extend({
type: z.literal("call.error"),
error: z.string(),
retryable: z.boolean().optional(),
}),
]);
export type NormalizedEvent = z.infer<typeof NormalizedEventSchema>;
// -----------------------------------------------------------------------------
// Call Direction
// -----------------------------------------------------------------------------
export const CallDirectionSchema = z.enum(["outbound", "inbound"]);
export type CallDirection = z.infer<typeof CallDirectionSchema>;
// -----------------------------------------------------------------------------
// Call Record
// -----------------------------------------------------------------------------
export const TranscriptEntrySchema = z.object({
timestamp: z.number(),
speaker: z.enum(["bot", "user"]),
text: z.string(),
isFinal: z.boolean().default(true),
});
export type TranscriptEntry = z.infer<typeof TranscriptEntrySchema>;
export const CallRecordSchema = z.object({
callId: z.string(),
providerCallId: z.string().optional(),
provider: ProviderNameSchema,
direction: CallDirectionSchema,
state: CallStateSchema,
from: z.string(),
to: z.string(),
sessionKey: z.string().optional(),
startedAt: z.number(),
answeredAt: z.number().optional(),
endedAt: z.number().optional(),
endReason: EndReasonSchema.optional(),
transcript: z.array(TranscriptEntrySchema).default([]),
processedEventIds: z.array(z.string()).default([]),
metadata: z.record(z.string(), z.unknown()).optional(),
});
export type CallRecord = z.infer<typeof CallRecordSchema>;
// -----------------------------------------------------------------------------
// Webhook Types
// -----------------------------------------------------------------------------
export type WebhookVerificationResult = {
ok: boolean;
reason?: string;
};
export type WebhookContext = {
headers: Record<string, string | string[] | undefined>;
rawBody: string;
url: string;
method: "GET" | "POST" | "PUT" | "DELETE" | "PATCH";
query?: Record<string, string | string[] | undefined>;
};
export type ProviderWebhookParseResult = {
events: NormalizedEvent[];
providerResponseBody?: string;
providerResponseHeaders?: Record<string, string>;
statusCode?: number;
};
// -----------------------------------------------------------------------------
// Provider Method Types
// -----------------------------------------------------------------------------
export type InitiateCallInput = {
callId: CallId;
from: string;
to: string;
webhookUrl: string;
clientState?: Record<string, string>;
/** Inline TwiML to execute (skips webhook, used for notify mode) */
inlineTwiml?: string;
};
export type InitiateCallResult = {
providerCallId: ProviderCallId;
status: "initiated" | "queued";
};
export type HangupCallInput = {
callId: CallId;
providerCallId: ProviderCallId;
reason: EndReason;
};
export type PlayTtsInput = {
callId: CallId;
providerCallId: ProviderCallId;
text: string;
voice?: string;
locale?: string;
};
export type StartListeningInput = {
callId: CallId;
providerCallId: ProviderCallId;
language?: string;
};
export type StopListeningInput = {
callId: CallId;
providerCallId: ProviderCallId;
};
// -----------------------------------------------------------------------------
// Outbound Call Options
// -----------------------------------------------------------------------------
export type OutboundCallOptions = {
/** Message to speak when call connects */
message?: string;
/** Call mode (overrides config default) */
mode?: CallMode;
};
// -----------------------------------------------------------------------------
// Tool Result Types
// -----------------------------------------------------------------------------
export type InitiateCallToolResult = {
success: boolean;
callId?: string;
status?: "initiated" | "queued" | "no-answer" | "busy" | "failed";
error?: string;
};
export type ContinueCallToolResult = {
success: boolean;
transcript?: string;
error?: string;
};
export type SpeakToUserToolResult = {
success: boolean;
error?: string;
};
export type EndCallToolResult = {
success: boolean;
error?: string;
};
+12
View File
@@ -0,0 +1,12 @@
import os from "node:os";
import path from "node:path";
export function resolveUserPath(input: string): string {
const trimmed = input.trim();
if (!trimmed) return trimmed;
if (trimmed.startsWith("~")) {
const expanded = trimmed.replace(/^~(?=$|[\\/])/, os.homedir());
return path.resolve(expanded);
}
return path.resolve(trimmed);
}
@@ -0,0 +1,65 @@
/**
* Voice mapping and XML utilities for voice call providers.
*/
/**
* Escape XML special characters for TwiML and other XML responses.
*/
export function escapeXml(text: string): string {
return text
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;")
.replace(/'/g, "&apos;");
}
/**
* Map of OpenAI voice names to similar Twilio Polly voices.
*/
const OPENAI_TO_POLLY_MAP: Record<string, string> = {
alloy: "Polly.Joanna", // neutral, warm
echo: "Polly.Matthew", // male, warm
fable: "Polly.Amy", // British, expressive
onyx: "Polly.Brian", // deep male
nova: "Polly.Salli", // female, friendly
shimmer: "Polly.Kimberly", // female, clear
};
/**
* Default Polly voice when no mapping is found.
*/
export const DEFAULT_POLLY_VOICE = "Polly.Joanna";
/**
* Map OpenAI voice names to Twilio Polly equivalents.
* Falls through if already a valid Polly/Google voice.
*
* @param voice - OpenAI voice name (alloy, echo, etc.) or Polly voice name
* @returns Polly voice name suitable for Twilio TwiML
*/
export function mapVoiceToPolly(voice: string | undefined): string {
if (!voice) return DEFAULT_POLLY_VOICE;
// Already a Polly/Google voice - pass through
if (voice.startsWith("Polly.") || voice.startsWith("Google.")) {
return voice;
}
// Map OpenAI voices to Polly equivalents
return OPENAI_TO_POLLY_MAP[voice.toLowerCase()] || DEFAULT_POLLY_VOICE;
}
/**
* Check if a voice name is a known OpenAI voice.
*/
export function isOpenAiVoice(voice: string): boolean {
return voice.toLowerCase() in OPENAI_TO_POLLY_MAP;
}
/**
* Get all supported OpenAI voice names.
*/
export function getOpenAiVoiceNames(): string[] {
return Object.keys(OPENAI_TO_POLLY_MAP);
}
@@ -0,0 +1,197 @@
import crypto from "node:crypto";
import type { WebhookContext } from "./types.js";
/**
* Validate Twilio webhook signature using HMAC-SHA1.
*
* Twilio signs requests by concatenating the URL with sorted POST params,
* then computing HMAC-SHA1 with the auth token.
*
* @see https://www.twilio.com/docs/usage/webhooks/webhooks-security
*/
export function validateTwilioSignature(
authToken: string,
signature: string | undefined,
url: string,
params: URLSearchParams,
): boolean {
if (!signature) {
return false;
}
// Build the string to sign: URL + sorted params (key+value pairs)
let dataToSign = url;
// Sort params alphabetically and append key+value
const sortedParams = Array.from(params.entries()).sort((a, b) =>
a[0].localeCompare(b[0]),
);
for (const [key, value] of sortedParams) {
dataToSign += key + value;
}
// HMAC-SHA1 with auth token, then base64 encode
const expectedSignature = crypto
.createHmac("sha1", authToken)
.update(dataToSign)
.digest("base64");
// Use timing-safe comparison to prevent timing attacks
return timingSafeEqual(signature, expectedSignature);
}
/**
* Timing-safe string comparison to prevent timing attacks.
*/
function timingSafeEqual(a: string, b: string): boolean {
if (a.length !== b.length) {
// Still do comparison to maintain constant time
const dummy = Buffer.from(a);
crypto.timingSafeEqual(dummy, dummy);
return false;
}
const bufA = Buffer.from(a);
const bufB = Buffer.from(b);
return crypto.timingSafeEqual(bufA, bufB);
}
/**
* Reconstruct the public webhook URL from request headers.
*
* When behind a reverse proxy (Tailscale, nginx, ngrok), the original URL
* used by Twilio differs from the local request URL. We use standard
* forwarding headers to reconstruct it.
*
* Priority order:
* 1. X-Forwarded-Proto + X-Forwarded-Host (standard proxy headers)
* 2. X-Original-Host (nginx)
* 3. Ngrok-Forwarded-Host (ngrok specific)
* 4. Host header (direct connection)
*/
export function reconstructWebhookUrl(ctx: WebhookContext): string {
const { headers } = ctx;
const proto = getHeader(headers, "x-forwarded-proto") || "https";
const forwardedHost =
getHeader(headers, "x-forwarded-host") ||
getHeader(headers, "x-original-host") ||
getHeader(headers, "ngrok-forwarded-host") ||
getHeader(headers, "host") ||
"";
// Extract path from the context URL (fallback to "/" on parse failure)
let path = "/";
try {
const parsed = new URL(ctx.url);
path = parsed.pathname + parsed.search;
} catch {
// URL parsing failed
}
// Remove port from host (ngrok URLs don't have ports)
const host = forwardedHost.split(":")[0] || forwardedHost;
return `${proto}://${host}${path}`;
}
/**
* Get a header value, handling both string and string[] types.
*/
function getHeader(
headers: Record<string, string | string[] | undefined>,
name: string,
): string | undefined {
const value = headers[name.toLowerCase()];
if (Array.isArray(value)) {
return value[0];
}
return value;
}
/**
* Result of Twilio webhook verification with detailed info.
*/
export interface TwilioVerificationResult {
ok: boolean;
reason?: string;
/** The URL that was used for verification (for debugging) */
verificationUrl?: string;
/** Whether we're running behind ngrok free tier */
isNgrokFreeTier?: boolean;
}
/**
* Verify Twilio webhook with full context and detailed result.
*
* Handles the special case of ngrok free tier where signature validation
* may fail due to URL discrepancies (ngrok adds interstitial page handling).
*/
export function verifyTwilioWebhook(
ctx: WebhookContext,
authToken: string,
options?: {
/** Override the public URL (e.g., from config) */
publicUrl?: string;
/** Allow ngrok free tier compatibility mode (less secure) */
allowNgrokFreeTier?: boolean;
/** Skip verification entirely (only for development) */
skipVerification?: boolean;
},
): TwilioVerificationResult {
// Allow skipping verification for development/testing
if (options?.skipVerification) {
return { ok: true, reason: "verification skipped (dev mode)" };
}
const signature = getHeader(ctx.headers, "x-twilio-signature");
if (!signature) {
return { ok: false, reason: "Missing X-Twilio-Signature header" };
}
// Reconstruct the URL Twilio used
const verificationUrl = options?.publicUrl || reconstructWebhookUrl(ctx);
// Parse the body as URL-encoded params
const params = new URLSearchParams(ctx.rawBody);
// Validate signature
const isValid = validateTwilioSignature(
authToken,
signature,
verificationUrl,
params,
);
if (isValid) {
return { ok: true, verificationUrl };
}
// Check if this is ngrok free tier - the URL might have different format
const isNgrokFreeTier =
verificationUrl.includes(".ngrok-free.app") ||
verificationUrl.includes(".ngrok.io");
if (isNgrokFreeTier && options?.allowNgrokFreeTier) {
console.warn(
"[voice-call] Twilio signature validation failed (proceeding for ngrok free tier compatibility)",
);
return {
ok: true,
reason: "ngrok free tier compatibility mode",
verificationUrl,
isNgrokFreeTier: true,
};
}
return {
ok: false,
reason: `Invalid signature for URL: ${verificationUrl}`,
verificationUrl,
isNgrokFreeTier,
};
}
+480
View File
@@ -0,0 +1,480 @@
import { spawn } from "node:child_process";
import http from "node:http";
import { URL } from "node:url";
import type { VoiceCallConfig } from "./config.js";
import type { CoreConfig } from "./core-bridge.js";
import type { CallManager } from "./manager.js";
import type { MediaStreamConfig } from "./media-stream.js";
import { MediaStreamHandler } from "./media-stream.js";
import type { VoiceCallProvider } from "./providers/base.js";
import { OpenAIRealtimeSTTProvider } from "./providers/stt-openai-realtime.js";
import type { TwilioProvider } from "./providers/twilio.js";
import type { NormalizedEvent, WebhookContext } from "./types.js";
/**
* HTTP server for receiving voice call webhooks from providers.
* Supports WebSocket upgrades for media streams when streaming is enabled.
*/
export class VoiceCallWebhookServer {
private server: http.Server | null = null;
private config: VoiceCallConfig;
private manager: CallManager;
private provider: VoiceCallProvider;
private coreConfig: CoreConfig | null;
/** Media stream handler for bidirectional audio (when streaming enabled) */
private mediaStreamHandler: MediaStreamHandler | null = null;
constructor(
config: VoiceCallConfig,
manager: CallManager,
provider: VoiceCallProvider,
coreConfig?: CoreConfig,
) {
this.config = config;
this.manager = manager;
this.provider = provider;
this.coreConfig = coreConfig ?? null;
// Initialize media stream handler if streaming is enabled
if (config.streaming?.enabled) {
this.initializeMediaStreaming();
}
}
/**
* Get the media stream handler (for wiring to provider).
*/
getMediaStreamHandler(): MediaStreamHandler | null {
return this.mediaStreamHandler;
}
/**
* Initialize media streaming with OpenAI Realtime STT.
*/
private initializeMediaStreaming(): void {
const apiKey =
this.config.streaming?.openaiApiKey || process.env.OPENAI_API_KEY;
if (!apiKey) {
console.warn(
"[voice-call] Streaming enabled but no OpenAI API key found",
);
return;
}
const sttProvider = new OpenAIRealtimeSTTProvider({
apiKey,
model: this.config.streaming?.sttModel,
silenceDurationMs: this.config.streaming?.silenceDurationMs,
vadThreshold: this.config.streaming?.vadThreshold,
});
const streamConfig: MediaStreamConfig = {
sttProvider,
onTranscript: (providerCallId, transcript) => {
console.log(
`[voice-call] Transcript for ${providerCallId}: ${transcript}`,
);
// Look up our internal call ID from the provider call ID
const call = this.manager.getCallByProviderCallId(providerCallId);
if (!call) {
console.warn(
`[voice-call] No active call found for provider ID: ${providerCallId}`,
);
return;
}
// Create a speech event and process it through the manager
const event: NormalizedEvent = {
id: `stream-transcript-${Date.now()}`,
type: "call.speech",
callId: call.callId,
providerCallId,
timestamp: Date.now(),
transcript,
isFinal: true,
};
this.manager.processEvent(event);
// Auto-respond in conversation mode (inbound always, outbound if mode is conversation)
const callMode = call.metadata?.mode as string | undefined;
const shouldRespond =
call.direction === "inbound" || callMode === "conversation";
if (shouldRespond) {
this.handleInboundResponse(call.callId, transcript).catch((err) => {
console.warn(`[voice-call] Failed to auto-respond:`, err);
});
}
},
onPartialTranscript: (callId, partial) => {
console.log(`[voice-call] Partial for ${callId}: ${partial}`);
},
onConnect: (callId, streamSid) => {
console.log(
`[voice-call] Media stream connected: ${callId} -> ${streamSid}`,
);
// Register stream with provider for TTS routing
if (this.provider.name === "twilio") {
(this.provider as TwilioProvider).registerCallStream(
callId,
streamSid,
);
}
// Speak initial message if one was provided when call was initiated
// Use setTimeout to allow stream setup to complete
setTimeout(() => {
this.manager.speakInitialMessage(callId).catch((err) => {
console.warn(`[voice-call] Failed to speak initial message:`, err);
});
}, 500);
},
onDisconnect: (callId) => {
console.log(`[voice-call] Media stream disconnected: ${callId}`);
if (this.provider.name === "twilio") {
(this.provider as TwilioProvider).unregisterCallStream(callId);
}
},
};
this.mediaStreamHandler = new MediaStreamHandler(streamConfig);
console.log("[voice-call] Media streaming initialized");
}
/**
* Start the webhook server.
*/
async start(): Promise<string> {
const { port, bind, path: webhookPath } = this.config.serve;
const streamPath = this.config.streaming?.streamPath || "/voice/stream";
return new Promise((resolve, reject) => {
this.server = http.createServer((req, res) => {
this.handleRequest(req, res, webhookPath).catch((err) => {
console.error("[voice-call] Webhook error:", err);
res.statusCode = 500;
res.end("Internal Server Error");
});
});
// Handle WebSocket upgrades for media streams
if (this.mediaStreamHandler) {
this.server.on("upgrade", (request, socket, head) => {
const url = new URL(
request.url || "/",
`http://${request.headers.host}`,
);
if (url.pathname === streamPath) {
console.log("[voice-call] WebSocket upgrade for media stream");
this.mediaStreamHandler?.handleUpgrade(request, socket, head);
} else {
socket.destroy();
}
});
}
this.server.on("error", reject);
this.server.listen(port, bind, () => {
const url = `http://${bind}:${port}${webhookPath}`;
console.log(`[voice-call] Webhook server listening on ${url}`);
if (this.mediaStreamHandler) {
console.log(
`[voice-call] Media stream WebSocket on ws://${bind}:${port}${streamPath}`,
);
}
resolve(url);
});
});
}
/**
* Stop the webhook server.
*/
async stop(): Promise<void> {
return new Promise((resolve) => {
if (this.server) {
this.server.close(() => {
this.server = null;
resolve();
});
} else {
resolve();
}
});
}
/**
* Handle incoming HTTP request.
*/
private async handleRequest(
req: http.IncomingMessage,
res: http.ServerResponse,
webhookPath: string,
): Promise<void> {
const url = new URL(req.url || "/", `http://${req.headers.host}`);
// Check path
if (!url.pathname.startsWith(webhookPath)) {
res.statusCode = 404;
res.end("Not Found");
return;
}
// Only accept POST
if (req.method !== "POST") {
res.statusCode = 405;
res.end("Method Not Allowed");
return;
}
// Read body
const body = await this.readBody(req);
// Build webhook context
const ctx: WebhookContext = {
headers: req.headers as Record<string, string | string[] | undefined>,
rawBody: body,
url: `http://${req.headers.host}${req.url}`,
method: "POST",
query: Object.fromEntries(url.searchParams),
};
// Verify signature
const verification = this.provider.verifyWebhook(ctx);
if (!verification.ok) {
console.warn(
`[voice-call] Webhook verification failed: ${verification.reason}`,
);
res.statusCode = 401;
res.end("Unauthorized");
return;
}
// Parse events
const result = this.provider.parseWebhookEvent(ctx);
// Process each event
for (const event of result.events) {
try {
this.manager.processEvent(event);
} catch (err) {
console.error(
`[voice-call] Error processing event ${event.type}:`,
err,
);
}
}
// Send response
res.statusCode = result.statusCode || 200;
if (result.providerResponseHeaders) {
for (const [key, value] of Object.entries(
result.providerResponseHeaders,
)) {
res.setHeader(key, value);
}
}
res.end(result.providerResponseBody || "OK");
}
/**
* Read request body as string.
*/
private readBody(req: http.IncomingMessage): Promise<string> {
return new Promise((resolve, reject) => {
const chunks: Buffer[] = [];
req.on("data", (chunk) => chunks.push(chunk));
req.on("end", () => resolve(Buffer.concat(chunks).toString("utf-8")));
req.on("error", reject);
});
}
/**
* Handle auto-response for inbound calls using the agent system.
* Supports tool calling for richer voice interactions.
*/
private async handleInboundResponse(
callId: string,
userMessage: string,
): Promise<void> {
console.log(
`[voice-call] Auto-responding to inbound call ${callId}: "${userMessage}"`,
);
// Get call context for conversation history
const call = this.manager.getCall(callId);
if (!call) {
console.warn(`[voice-call] Call ${callId} not found for auto-response`);
return;
}
if (!this.coreConfig) {
console.warn("[voice-call] Core config missing; skipping auto-response");
return;
}
try {
const { generateVoiceResponse } = await import("./response-generator.js");
const result = await generateVoiceResponse({
voiceConfig: this.config,
coreConfig: this.coreConfig,
callId,
from: call.from,
transcript: call.transcript,
userMessage,
});
if (result.error) {
console.error(
`[voice-call] Response generation error: ${result.error}`,
);
return;
}
if (result.text) {
console.log(`[voice-call] AI response: "${result.text}"`);
await this.manager.speak(callId, result.text);
}
} catch (err) {
console.error(`[voice-call] Auto-response error:`, err);
}
}
}
/**
* Resolve the current machine's Tailscale DNS name.
*/
export type TailscaleSelfInfo = {
dnsName: string | null;
nodeId: string | null;
};
/**
* Run a tailscale command with timeout, collecting stdout.
*/
function runTailscaleCommand(
args: string[],
timeoutMs = 2500,
): Promise<{ code: number; stdout: string }> {
return new Promise((resolve) => {
const proc = spawn("tailscale", args, {
stdio: ["ignore", "pipe", "pipe"],
});
let stdout = "";
proc.stdout.on("data", (data) => {
stdout += data;
});
const timer = setTimeout(() => {
proc.kill("SIGKILL");
resolve({ code: -1, stdout: "" });
}, timeoutMs);
proc.on("close", (code) => {
clearTimeout(timer);
resolve({ code: code ?? -1, stdout });
});
});
}
export async function getTailscaleSelfInfo(): Promise<TailscaleSelfInfo | null> {
const { code, stdout } = await runTailscaleCommand(["status", "--json"]);
if (code !== 0) return null;
try {
const status = JSON.parse(stdout);
return {
dnsName: status.Self?.DNSName?.replace(/\.$/, "") || null,
nodeId: status.Self?.ID || null,
};
} catch {
return null;
}
}
export async function getTailscaleDnsName(): Promise<string | null> {
const info = await getTailscaleSelfInfo();
return info?.dnsName ?? null;
}
export async function setupTailscaleExposureRoute(opts: {
mode: "serve" | "funnel";
path: string;
localUrl: string;
}): Promise<string | null> {
const dnsName = await getTailscaleDnsName();
if (!dnsName) {
console.warn("[voice-call] Could not get Tailscale DNS name");
return null;
}
const { code } = await runTailscaleCommand([
opts.mode,
"--bg",
"--yes",
"--set-path",
opts.path,
opts.localUrl,
]);
if (code === 0) {
const publicUrl = `https://${dnsName}${opts.path}`;
console.log(`[voice-call] Tailscale ${opts.mode} active: ${publicUrl}`);
return publicUrl;
}
console.warn(`[voice-call] Tailscale ${opts.mode} failed`);
return null;
}
export async function cleanupTailscaleExposureRoute(opts: {
mode: "serve" | "funnel";
path: string;
}): Promise<void> {
await runTailscaleCommand([opts.mode, "off", opts.path]);
}
/**
* Setup Tailscale serve/funnel for the webhook server.
* This is a helper that shells out to `tailscale serve` or `tailscale funnel`.
*/
export async function setupTailscaleExposure(
config: VoiceCallConfig,
): Promise<string | null> {
if (config.tailscale.mode === "off") {
return null;
}
const mode = config.tailscale.mode === "funnel" ? "funnel" : "serve";
// Include the path suffix so tailscale forwards to the correct endpoint
// (tailscale strips the mount path prefix when proxying)
const localUrl = `http://127.0.0.1:${config.serve.port}${config.serve.path}`;
return setupTailscaleExposureRoute({
mode,
path: config.tailscale.path,
localUrl,
});
}
/**
* Cleanup Tailscale serve/funnel.
*/
export async function cleanupTailscaleExposure(
config: VoiceCallConfig,
): Promise<void> {
if (config.tailscale.mode === "off") {
return;
}
const mode = config.tailscale.mode === "funnel" ? "funnel" : "serve";
await cleanupTailscaleExposureRoute({ mode, path: config.tailscale.path });
}
+3 -1
View File
@@ -1,6 +1,6 @@
{
"name": "clawdbot",
"version": "2026.1.11-1",
"version": "2026.1.11-4",
"description": "WhatsApp gateway CLI (Baileys web) with Pi RPC agent",
"type": "module",
"main": "dist/index.js",
@@ -27,6 +27,7 @@
"dist/media/**",
"dist/msteams/**",
"dist/process/**",
"dist/plugins/**",
"dist/sessions/**",
"dist/providers/**",
"dist/signal/**",
@@ -159,6 +160,7 @@
"json5": "^2.2.3",
"long": "5.3.2",
"markdown-it": "^14.1.0",
"node-llama-cpp": "3.14.5",
"osc-progress": "^0.2.0",
"playwright-core": "1.57.0",
"proper-lockfile": "^4.1.2",
+1216 -6
View File
File diff suppressed because it is too large. Load diff
+8 -2
View File
@@ -1,7 +1,13 @@
FROM ubuntu:24.04
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
RUN set -eux; \
for attempt in 1 2 3; do \
if apt-get update -o Acquire::Retries=3; then break; fi; \
echo "apt-get update failed (attempt ${attempt})" >&2; \
if [ "${attempt}" -eq 3 ]; then exit 1; fi; \
sleep 3; \
done; \
apt-get -o Acquire::Retries=3 install -y --no-install-recommends \
bash \
ca-certificates \
curl \
+8 -2
View File
@@ -1,7 +1,13 @@
FROM node:22-bookworm-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
RUN set -eux; \
for attempt in 1 2 3; do \
if apt-get update -o Acquire::Retries=3; then break; fi; \
echo "apt-get update failed (attempt ${attempt})" >&2; \
if [ "${attempt}" -eq 3 ]; then exit 1; fi; \
sleep 3; \
done; \
apt-get -o Acquire::Retries=3 install -y --no-install-recommends \
bash \
ca-certificates \
curl \
+14
View File
@@ -25,6 +25,18 @@ function getRepoRoot() {
return path.resolve(here, "..");
}
function ensureExecutable(targetPath) {
if (process.platform === "win32") return;
if (!fs.existsSync(targetPath)) return;
try {
const mode = fs.statSync(targetPath).mode & 0o777;
if (mode & 0o100) return;
fs.chmodSync(targetPath, 0o755);
} catch (err) {
console.warn(`[postinstall] chmod failed: ${err}`);
}
}
function extractPackageName(key) {
if (key.startsWith("@")) {
const idx = key.indexOf("@", 1);
@@ -213,6 +225,8 @@ function main() {
const repoRoot = getRepoRoot();
process.chdir(repoRoot);
ensureExecutable(path.join(repoRoot, "dist", "entry.js"));
if (!shouldApplyPnpmPatchedDependenciesFallback()) {
return;
}
-26
View File
@@ -1,26 +0,0 @@
---
name: qmd
description: Local search/indexing CLI (BM25 + vectors + rerank) with MCP mode.
homepage: https://tobi.lutke.com
metadata: {"clawdbot":{"emoji":"📝","requires":{"bins":["qmd"]},"install":[{"id":"node","kind":"node","package":"https://github.com/tobi/qmd","bins":["qmd"],"label":"Install qmd (node)"}]}}
---
# qmd
Use `qmd` to index local files and search them.
Indexing
- Add collection: `qmd collection add /path --name docs --mask "**/*.md"`
- Update index: `qmd update`
- Status: `qmd status`
Search
- BM25: `qmd search "query"`
- Vector: `qmd vsearch "query"`
- Hybrid: `qmd query "query"`
- Get doc: `qmd get docs/path.md:10 -l 40`
Notes
- Embeddings/rerank use Ollama at `OLLAMA_URL` (default `http://localhost:11434`).
- Index lives under `~/.cache/qmd` by default.
- MCP mode: `qmd mcp`.
+12 -10
View File
@@ -6,27 +6,29 @@ metadata: {"clawdbot":{"emoji":"📞","skillKey":"voice-call","requires":{"confi
# Voice Call
Use the voice-call plugin to start or inspect calls (Twilio or log fallback).
Use the voice-call plugin to start or inspect calls (Twilio, Telnyx, or mock).
## CLI
```bash
clawdbot voicecall start --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall status --sid CAxxxxxxxx
clawdbot voicecall call --to "+15555550123" --message "Hello from Clawdbot"
clawdbot voicecall status --call-id <id>
```
## Tool
Use `voice_call` for agent-initiated calls.
Parameters:
- `mode` ("call" | "status", optional, default call)
- `to` (string): phone number / target (required for call)
- `sid` (string): call SID (required for status)
- `message` (string, optional): optional intro or instruction
Actions:
- `initiate_call` (message, to?, mode?)
- `continue_call` (callId, message)
- `speak_to_user` (callId, message)
- `end_call` (callId)
- `get_status` (callId)
Notes:
- Requires the voice-call plugin to be enabled.
- Plugin config lives under `plugins.entries.voice-call.config`.
- Twilio config: `provider: "twilio"` + `twilio.accountSid/authToken/from` (statusCallbackUrl/twimlUrl optional).
- Dev fallback: `provider: "log"` (no network).
- Twilio config: `provider: "twilio"` + `twilio.accountSid/authToken` + `fromNumber`.
- Telnyx config: `provider: "telnyx"` + `telnyx.apiKey/connectionId` + `fromNumber`.
- Dev fallback: `provider: "mock"` (no network).
+2
View File
@@ -22,6 +22,7 @@ type ResolvedAgentConfig = {
workspace?: string;
agentDir?: string;
model?: string;
memorySearch?: AgentEntry["memorySearch"];
humanDelay?: AgentEntry["humanDelay"];
identity?: AgentEntry["identity"];
groupChat?: AgentEntry["groupChat"];
@@ -95,6 +96,7 @@ export function resolveAgentConfig(
typeof entry.workspace === "string" ? entry.workspace : undefined,
agentDir: typeof entry.agentDir === "string" ? entry.agentDir : undefined,
model: typeof entry.model === "string" ? entry.model : undefined,
memorySearch: entry.memorySearch,
humanDelay: entry.humanDelay,
identity: entry.identity,
groupChat: entry.groupChat,
@@ -687,6 +687,91 @@ describe("subagents", () => {
});
});
it("sessions_spawn applies default subagent model from defaults config", async () => {
resetSubagentRegistryForTests();
callGatewayMock.mockReset();
configOverride = {
session: { mainKey: "main", scope: "per-sender" },
agents: { defaults: { subagents: { model: "minimax/MiniMax-M2.1" } } },
};
const calls: Array<{ method?: string; params?: unknown }> = [];
callGatewayMock.mockImplementation(async (opts: unknown) => {
const request = opts as { method?: string; params?: unknown };
calls.push(request);
if (request.method === "sessions.patch") {
return { ok: true };
}
if (request.method === "agent") {
return { runId: "run-default-model", status: "accepted" };
}
return {};
});
const tool = createClawdbotTools({
agentSessionKey: "agent:main:main",
agentProvider: "discord",
}).find((candidate) => candidate.name === "sessions_spawn");
if (!tool) throw new Error("missing sessions_spawn tool");
const result = await tool.execute("call-default-model", {
task: "do thing",
});
expect(result.details).toMatchObject({
status: "accepted",
modelApplied: true,
});
const patchCall = calls.find((call) => call.method === "sessions.patch");
expect(patchCall?.params).toMatchObject({
model: "minimax/MiniMax-M2.1",
});
});
it("sessions_spawn prefers per-agent subagent model over defaults", async () => {
resetSubagentRegistryForTests();
callGatewayMock.mockReset();
configOverride = {
session: { mainKey: "main", scope: "per-sender" },
agents: {
defaults: { subagents: { model: "minimax/MiniMax-M2.1" } },
list: [{ id: "research", subagents: { model: "opencode/claude" } }],
},
};
const calls: Array<{ method?: string; params?: unknown }> = [];
callGatewayMock.mockImplementation(async (opts: unknown) => {
const request = opts as { method?: string; params?: unknown };
calls.push(request);
if (request.method === "sessions.patch") {
return { ok: true };
}
if (request.method === "agent") {
return { runId: "run-agent-model", status: "accepted" };
}
return {};
});
const tool = createClawdbotTools({
agentSessionKey: "agent:research:main",
agentProvider: "discord",
}).find((candidate) => candidate.name === "sessions_spawn");
if (!tool) throw new Error("missing sessions_spawn tool");
const result = await tool.execute("call-agent-model", {
task: "do thing",
});
expect(result.details).toMatchObject({
status: "accepted",
modelApplied: true,
});
const patchCall = calls.find((call) => call.method === "sessions.patch");
expect(patchCall?.params).toMatchObject({
model: "opencode/claude",
});
});
it("sessions_spawn skips invalid model overrides and continues", async () => {
resetSubagentRegistryForTests();
callGatewayMock.mockReset();
+21 -2
View File
@@ -9,6 +9,10 @@ import type { AnyAgentTool } from "./tools/common.js";
import { createCronTool } from "./tools/cron-tool.js";
import { createGatewayTool } from "./tools/gateway-tool.js";
import { createImageTool } from "./tools/image-tool.js";
import {
createMemoryGetTool,
createMemorySearchTool,
} from "./tools/memory-tool.js";
import { createMessageTool } from "./tools/message-tool.js";
import { createNodesTool } from "./tools/nodes-tool.js";
import { createSessionStatusTool } from "./tools/session-status-tool.js";
@@ -27,6 +31,7 @@ export function createClawdbotTools(options?: {
agentProvider?: GatewayMessageProvider;
agentAccountId?: string;
agentDir?: string;
sandboxRoot?: string;
workspaceDir?: string;
sandboxed?: boolean;
config?: ClawdbotConfig;
@@ -39,9 +44,20 @@ export function createClawdbotTools(options?: {
/** Mutable ref to track if a reply was sent (for "first" mode). */
hasRepliedRef?: { value: boolean };
}): AnyAgentTool[] {
const imageTool = createImageTool({
const imageTool = options?.agentDir?.trim()
? createImageTool({
config: options?.config,
agentDir: options.agentDir,
sandboxRoot: options?.sandboxRoot,
})
: null;
const memorySearchTool = createMemorySearchTool({
config: options?.config,
agentDir: options?.agentDir,
agentSessionKey: options?.agentSessionKey,
});
const memoryGetTool = createMemoryGetTool({
config: options?.config,
agentSessionKey: options?.agentSessionKey,
});
const tools: AnyAgentTool[] = [
createBrowserTool({
@@ -89,6 +105,9 @@ export function createClawdbotTools(options?: {
agentSessionKey: options?.agentSessionKey,
config: options?.config,
}),
...(memorySearchTool && memoryGetTool
? [memorySearchTool, memoryGetTool]
: []),
...(imageTool ? [imageTool] : []),
];
+56
View File
@@ -0,0 +1,56 @@
import { describe, expect, it } from "vitest";
import { resolveMemorySearchConfig } from "./memory-search.js";
describe("memory search config", () => {
it("returns null when disabled", () => {
const cfg = {
agents: {
defaults: {
memorySearch: { enabled: true },
},
list: [
{
id: "main",
default: true,
memorySearch: { enabled: false },
},
],
},
};
const resolved = resolveMemorySearchConfig(cfg, "main");
expect(resolved).toBeNull();
});
it("merges defaults and overrides", () => {
const cfg = {
agents: {
defaults: {
memorySearch: {
provider: "openai",
model: "text-embedding-3-small",
chunking: { tokens: 500, overlap: 100 },
query: { maxResults: 4, minScore: 0.2 },
},
},
list: [
{
id: "main",
default: true,
memorySearch: {
chunking: { tokens: 320 },
query: { maxResults: 8 },
},
},
],
},
};
const resolved = resolveMemorySearchConfig(cfg, "main");
expect(resolved?.provider).toBe("openai");
expect(resolved?.model).toBe("text-embedding-3-small");
expect(resolved?.chunking.tokens).toBe(320);
expect(resolved?.chunking.overlap).toBe(100);
expect(resolved?.query.maxResults).toBe(8);
expect(resolved?.query.minScore).toBe(0.2);
});
});
+134
View File
@@ -0,0 +1,134 @@
import os from "node:os";
import path from "node:path";
import type { ClawdbotConfig, MemorySearchConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { resolveUserPath } from "../utils.js";
import { resolveAgentConfig } from "./agent-scope.js";
export type ResolvedMemorySearchConfig = {
enabled: boolean;
provider: "openai" | "local";
fallback: "openai" | "none";
model: string;
local: {
modelPath?: string;
modelCacheDir?: string;
};
store: {
driver: "sqlite";
path: string;
};
chunking: {
tokens: number;
overlap: number;
};
sync: {
onSessionStart: boolean;
onSearch: boolean;
watch: boolean;
watchDebounceMs: number;
intervalMinutes: number;
};
query: {
maxResults: number;
minScore: number;
};
};
const DEFAULT_MODEL = "text-embedding-3-small";
const DEFAULT_CHUNK_TOKENS = 400;
const DEFAULT_CHUNK_OVERLAP = 80;
const DEFAULT_WATCH_DEBOUNCE_MS = 1500;
const DEFAULT_MAX_RESULTS = 6;
const DEFAULT_MIN_SCORE = 0.35;
function resolveStorePath(agentId: string, raw?: string): string {
const stateDir = resolveStateDir(process.env, os.homedir);
const fallback = path.join(stateDir, "memory", `${agentId}.sqlite`);
if (!raw) return fallback;
const withToken = raw.includes("{agentId}")
? raw.replaceAll("{agentId}", agentId)
: raw;
return resolveUserPath(withToken);
}
function mergeConfig(
defaults: MemorySearchConfig | undefined,
overrides: MemorySearchConfig | undefined,
agentId: string,
): ResolvedMemorySearchConfig {
const enabled = overrides?.enabled ?? defaults?.enabled ?? true;
const provider = overrides?.provider ?? defaults?.provider ?? "openai";
const fallback = overrides?.fallback ?? defaults?.fallback ?? "openai";
const model = overrides?.model ?? defaults?.model ?? DEFAULT_MODEL;
const local = {
modelPath: overrides?.local?.modelPath ?? defaults?.local?.modelPath,
modelCacheDir:
overrides?.local?.modelCacheDir ?? defaults?.local?.modelCacheDir,
};
const store = {
driver: overrides?.store?.driver ?? defaults?.store?.driver ?? "sqlite",
path: resolveStorePath(
agentId,
overrides?.store?.path ?? defaults?.store?.path,
),
};
const chunking = {
tokens:
overrides?.chunking?.tokens ??
defaults?.chunking?.tokens ??
DEFAULT_CHUNK_TOKENS,
overlap:
overrides?.chunking?.overlap ??
defaults?.chunking?.overlap ??
DEFAULT_CHUNK_OVERLAP,
};
const sync = {
onSessionStart:
overrides?.sync?.onSessionStart ?? defaults?.sync?.onSessionStart ?? true,
onSearch: overrides?.sync?.onSearch ?? defaults?.sync?.onSearch ?? true,
watch: overrides?.sync?.watch ?? defaults?.sync?.watch ?? true,
watchDebounceMs:
overrides?.sync?.watchDebounceMs ??
defaults?.sync?.watchDebounceMs ??
DEFAULT_WATCH_DEBOUNCE_MS,
intervalMinutes:
overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0,
};
const query = {
maxResults:
overrides?.query?.maxResults ??
defaults?.query?.maxResults ??
DEFAULT_MAX_RESULTS,
minScore:
overrides?.query?.minScore ??
defaults?.query?.minScore ??
DEFAULT_MIN_SCORE,
};
const overlap = Math.max(0, Math.min(chunking.overlap, chunking.tokens - 1));
const minScore = Math.max(0, Math.min(1, query.minScore));
return {
enabled,
provider,
fallback,
model,
local,
store,
chunking: { tokens: Math.max(1, chunking.tokens), overlap },
sync,
query: { ...query, minScore },
};
}
export function resolveMemorySearchConfig(
cfg: ClawdbotConfig,
agentId: string,
): ResolvedMemorySearchConfig | null {
const defaults = cfg.agents?.defaults?.memorySearch;
const overrides = resolveAgentConfig(cfg, agentId)?.memorySearch;
const resolved = mergeConfig(defaults, overrides, agentId);
if (!resolved.enabled) return null;
return resolved;
}
+115
View File
@@ -0,0 +1,115 @@
type MinimaxBaseResp = {
status_code?: number;
status_msg?: string;
};
function coerceApiHost(params: {
apiHost?: string;
modelBaseUrl?: string;
env?: NodeJS.ProcessEnv;
}): string {
const env = params.env ?? process.env;
const raw =
params.apiHost?.trim() ||
env.MINIMAX_API_HOST?.trim() ||
params.modelBaseUrl?.trim() ||
"https://api.minimax.io";
try {
const url = new URL(raw);
return url.origin;
} catch {}
try {
const url = new URL(`https://${raw}`);
return url.origin;
} catch {
return "https://api.minimax.io";
}
}
function isRecord(value: unknown): value is Record<string, unknown> {
return Boolean(value && typeof value === "object" && !Array.isArray(value));
}
function pickString(rec: Record<string, unknown>, key: string): string {
const v = rec[key];
return typeof v === "string" ? v : "";
}
export async function minimaxUnderstandImage(params: {
apiKey: string;
prompt: string;
imageDataUrl: string;
apiHost?: string;
modelBaseUrl?: string;
}): Promise<string> {
const apiKey = params.apiKey.trim();
if (!apiKey) throw new Error("MiniMax VLM: apiKey required");
const prompt = params.prompt.trim();
if (!prompt) throw new Error("MiniMax VLM: prompt required");
const imageDataUrl = params.imageDataUrl.trim();
if (!imageDataUrl) throw new Error("MiniMax VLM: imageDataUrl required");
if (!/^data:image\/(png|jpeg|webp);base64,/i.test(imageDataUrl)) {
throw new Error(
"MiniMax VLM: imageDataUrl must be a base64 data:image/(png|jpeg|webp) URL",
);
}
const host = coerceApiHost({
apiHost: params.apiHost,
modelBaseUrl: params.modelBaseUrl,
});
const url = new URL("/v1/coding_plan/vlm", host).toString();
const res = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"MM-API-Source": "Clawdbot",
},
body: JSON.stringify({
prompt,
image_url: imageDataUrl,
}),
});
const traceId = res.headers.get("Trace-Id") ?? "";
if (!res.ok) {
const body = await res.text().catch(() => "");
const trace = traceId ? ` Trace-Id: ${traceId}` : "";
throw new Error(
`MiniMax VLM request failed (${res.status} ${res.statusText}).${trace}${
body ? ` Body: ${body.slice(0, 400)}` : ""
}`,
);
}
const json = (await res.json().catch(() => null)) as unknown;
if (!isRecord(json)) {
const trace = traceId ? ` Trace-Id: ${traceId}` : "";
throw new Error(`MiniMax VLM response was not JSON.${trace}`);
}
const baseResp = isRecord(json.base_resp)
? (json.base_resp as MinimaxBaseResp)
: {};
const code =
typeof baseResp.status_code === "number" ? baseResp.status_code : -1;
if (code !== 0) {
const msg = (baseResp.status_msg ?? "").trim();
const trace = traceId ? ` Trace-Id: ${traceId}` : "";
throw new Error(
`MiniMax VLM API error (${code})${msg ? `: ${msg}` : ""}.${trace}`,
);
}
const content = pickString(json, "content").trim();
if (!content) {
const trace = traceId ? ` Trace-Id: ${traceId}` : "";
throw new Error(`MiniMax VLM returned no content.${trace}`);
}
return content;
}
+206
View File
@@ -0,0 +1,206 @@
import type { ClawdbotConfig } from "../config/config.js";
import {
ensureAuthProfileStore,
listProfilesForProvider,
} from "./auth-profiles.js";
import { resolveEnvApiKey } from "./model-auth.js";
type ModelsConfig = NonNullable<ClawdbotConfig["models"]>;
export type ProviderConfig = NonNullable<ModelsConfig["providers"]>[string];
const MINIMAX_API_BASE_URL = "https://api.minimax.io/anthropic";
const MINIMAX_DEFAULT_MODEL_ID = "MiniMax-M2.1";
const MINIMAX_DEFAULT_VISION_MODEL_ID = "MiniMax-VL-01";
const MINIMAX_DEFAULT_CONTEXT_WINDOW = 200000;
const MINIMAX_DEFAULT_MAX_TOKENS = 8192;
// Pricing: MiniMax doesn't publish public rates. Override in models.json for accurate costs.
const MINIMAX_API_COST = {
input: 15,
output: 60,
cacheRead: 2,
cacheWrite: 10,
};
const MOONSHOT_BASE_URL = "https://api.moonshot.ai/v1";
const MOONSHOT_DEFAULT_MODEL_ID = "kimi-k2-0905-preview";
const MOONSHOT_DEFAULT_CONTEXT_WINDOW = 256000;
const MOONSHOT_DEFAULT_MAX_TOKENS = 8192;
const MOONSHOT_DEFAULT_COST = {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
};
function normalizeApiKeyConfig(value: string): string {
const trimmed = value.trim();
const match = /^\$\{([A-Z0-9_]+)\}$/.exec(trimmed);
return match?.[1] ?? trimmed;
}
function resolveEnvApiKeyVarName(provider: string): string | undefined {
const resolved = resolveEnvApiKey(provider);
if (!resolved) return undefined;
const match = /^(?:env: |shell env: )([A-Z0-9_]+)$/.exec(resolved.source);
return match ? match[1] : undefined;
}
function resolveApiKeyFromProfiles(params: {
provider: string;
store: ReturnType<typeof ensureAuthProfileStore>;
}): string | undefined {
const ids = listProfilesForProvider(params.store, params.provider);
for (const id of ids) {
const cred = params.store.profiles[id];
if (!cred) continue;
if (cred.type === "api_key") return cred.key;
if (cred.type === "token") return cred.token;
}
return undefined;
}
export function normalizeGoogleModelId(id: string): string {
if (id === "gemini-3-pro") return "gemini-3-pro-preview";
if (id === "gemini-3-flash") return "gemini-3-flash-preview";
return id;
}
function normalizeGoogleProvider(provider: ProviderConfig): ProviderConfig {
let mutated = false;
const models = provider.models.map((model) => {
const nextId = normalizeGoogleModelId(model.id);
if (nextId === model.id) return model;
mutated = true;
return { ...model, id: nextId };
});
return mutated ? { ...provider, models } : provider;
}
export function normalizeProviders(params: {
providers: ModelsConfig["providers"];
agentDir: string;
}): ModelsConfig["providers"] {
const { providers } = params;
if (!providers) return providers;
const authStore = ensureAuthProfileStore(params.agentDir, {
allowKeychainPrompt: false,
});
let mutated = false;
const next: Record<string, ProviderConfig> = {};
for (const [key, provider] of Object.entries(providers)) {
const normalizedKey = key.trim();
let normalizedProvider = provider;
// Fix common misconfig: apiKey set to "${ENV_VAR}" instead of "ENV_VAR".
if (
normalizedProvider.apiKey &&
normalizeApiKeyConfig(normalizedProvider.apiKey) !==
normalizedProvider.apiKey
) {
mutated = true;
normalizedProvider = {
...normalizedProvider,
apiKey: normalizeApiKeyConfig(normalizedProvider.apiKey),
};
}
// If a provider defines models, pi's ModelRegistry requires apiKey to be set.
// Fill it from the environment or auth profiles when possible.
const hasModels =
Array.isArray(normalizedProvider.models) &&
normalizedProvider.models.length > 0;
if (hasModels && !normalizedProvider.apiKey?.trim()) {
const fromEnv = resolveEnvApiKeyVarName(normalizedKey);
const fromProfiles = resolveApiKeyFromProfiles({
provider: normalizedKey,
store: authStore,
});
const apiKey = fromEnv ?? fromProfiles;
if (apiKey?.trim()) {
mutated = true;
normalizedProvider = { ...normalizedProvider, apiKey };
}
}
if (normalizedKey === "google") {
const googleNormalized = normalizeGoogleProvider(normalizedProvider);
if (googleNormalized !== normalizedProvider) mutated = true;
normalizedProvider = googleNormalized;
}
next[key] = normalizedProvider;
}
return mutated ? next : providers;
}
function buildMinimaxProvider(): ProviderConfig {
return {
baseUrl: MINIMAX_API_BASE_URL,
api: "anthropic-messages",
models: [
{
id: MINIMAX_DEFAULT_MODEL_ID,
name: "MiniMax M2.1",
reasoning: false,
input: ["text"],
cost: MINIMAX_API_COST,
contextWindow: MINIMAX_DEFAULT_CONTEXT_WINDOW,
maxTokens: MINIMAX_DEFAULT_MAX_TOKENS,
},
{
id: MINIMAX_DEFAULT_VISION_MODEL_ID,
name: "MiniMax VL 01",
reasoning: false,
input: ["text", "image"],
cost: MINIMAX_API_COST,
contextWindow: MINIMAX_DEFAULT_CONTEXT_WINDOW,
maxTokens: MINIMAX_DEFAULT_MAX_TOKENS,
},
],
};
}
function buildMoonshotProvider(): ProviderConfig {
return {
baseUrl: MOONSHOT_BASE_URL,
api: "openai-completions",
models: [
{
id: MOONSHOT_DEFAULT_MODEL_ID,
name: "Kimi K2 0905 Preview",
reasoning: false,
input: ["text"],
cost: MOONSHOT_DEFAULT_COST,
contextWindow: MOONSHOT_DEFAULT_CONTEXT_WINDOW,
maxTokens: MOONSHOT_DEFAULT_MAX_TOKENS,
},
],
};
}
export function resolveImplicitProviders(params: {
agentDir: string;
}): ModelsConfig["providers"] {
const providers: Record<string, ProviderConfig> = {};
const authStore = ensureAuthProfileStore(params.agentDir, {
allowKeychainPrompt: false,
});
const minimaxKey =
resolveEnvApiKeyVarName("minimax") ??
resolveApiKeyFromProfiles({ provider: "minimax", store: authStore });
if (minimaxKey) {
providers.minimax = { ...buildMinimaxProvider(), apiKey: minimaxKey };
}
const moonshotKey =
resolveEnvApiKeyVarName("moonshot") ??
resolveApiKeyFromProfiles({ provider: "moonshot", store: authStore });
if (moonshotKey) {
providers.moonshot = { ...buildMoonshotProvider(), apiKey: moonshotKey };
}
return providers;
}
+438
View File
@@ -34,6 +34,356 @@ const MODELS_CONFIG: ClawdbotConfig = {
};
describe("models config", () => {
it("auto-injects github-copilot provider when token is present", async () => {
await withTempHome(async (home) => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
process.env.COPILOT_GITHUB_TOKEN = "gh-token";
try {
vi.resetModules();
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL:
"https://api.individual.githubcopilot.com",
resolveCopilotApiToken: vi.fn().mockResolvedValue({
token: "copilot",
expiresAt: Date.now() + 60 * 60 * 1000,
source: "mock",
baseUrl: "https://api.copilot.example",
}),
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const agentDir = path.join(home, "agent-default-base-url");
await ensureClawdbotModelsJson({ models: { providers: {} } }, agentDir);
const raw = await fs.readFile(
path.join(agentDir, "models.json"),
"utf8",
);
const parsed = JSON.parse(raw) as {
providers: Record<string, { baseUrl?: string; models?: unknown[] }>;
};
expect(parsed.providers["github-copilot"]?.baseUrl).toBe(
"https://api.copilot.example",
);
expect(parsed.providers["github-copilot"]?.models?.length ?? 0).toBe(0);
} finally {
process.env.COPILOT_GITHUB_TOKEN = previous;
}
});
});
it("prefers COPILOT_GITHUB_TOKEN over GH_TOKEN and GITHUB_TOKEN", async () => {
await withTempHome(async () => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
const previousGh = process.env.GH_TOKEN;
const previousGithub = process.env.GITHUB_TOKEN;
process.env.COPILOT_GITHUB_TOKEN = "copilot-token";
process.env.GH_TOKEN = "gh-token";
process.env.GITHUB_TOKEN = "github-token";
try {
vi.resetModules();
const resolveCopilotApiToken = vi.fn().mockResolvedValue({
token: "copilot",
expiresAt: Date.now() + 60 * 60 * 1000,
source: "mock",
baseUrl: "https://api.copilot.example",
});
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL:
"https://api.individual.githubcopilot.com",
resolveCopilotApiToken,
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
await ensureClawdbotModelsJson({ models: { providers: {} } });
expect(resolveCopilotApiToken).toHaveBeenCalledWith(
expect.objectContaining({ githubToken: "copilot-token" }),
);
} finally {
process.env.COPILOT_GITHUB_TOKEN = previous;
process.env.GH_TOKEN = previousGh;
process.env.GITHUB_TOKEN = previousGithub;
}
});
});
it("uses the first github-copilot profile when env tokens are missing", async () => {
await withTempHome(async (home) => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
const previousGh = process.env.GH_TOKEN;
const previousGithub = process.env.GITHUB_TOKEN;
delete process.env.COPILOT_GITHUB_TOKEN;
delete process.env.GH_TOKEN;
delete process.env.GITHUB_TOKEN;
try {
vi.resetModules();
const agentDir = path.join(home, "agent-profiles");
await fs.mkdir(agentDir, { recursive: true });
await fs.writeFile(
path.join(agentDir, "auth-profiles.json"),
JSON.stringify(
{
version: 1,
profiles: {
"github-copilot:alpha": {
type: "token",
provider: "github-copilot",
token: "alpha-token",
},
"github-copilot:beta": {
type: "token",
provider: "github-copilot",
token: "beta-token",
},
},
},
null,
2,
),
);
const resolveCopilotApiToken = vi.fn().mockResolvedValue({
token: "copilot",
expiresAt: Date.now() + 60 * 60 * 1000,
source: "mock",
baseUrl: "https://api.copilot.example",
});
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL:
"https://api.individual.githubcopilot.com",
resolveCopilotApiToken,
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
await ensureClawdbotModelsJson({ models: { providers: {} } }, agentDir);
expect(resolveCopilotApiToken).toHaveBeenCalledWith(
expect.objectContaining({ githubToken: "alpha-token" }),
);
} finally {
if (previous === undefined) delete process.env.COPILOT_GITHUB_TOKEN;
else process.env.COPILOT_GITHUB_TOKEN = previous;
if (previousGh === undefined) delete process.env.GH_TOKEN;
else process.env.GH_TOKEN = previousGh;
if (previousGithub === undefined) delete process.env.GITHUB_TOKEN;
else process.env.GITHUB_TOKEN = previousGithub;
}
});
});
it("does not override explicit github-copilot provider config", async () => {
await withTempHome(async () => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
process.env.COPILOT_GITHUB_TOKEN = "gh-token";
try {
vi.resetModules();
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL:
"https://api.individual.githubcopilot.com",
resolveCopilotApiToken: vi.fn().mockResolvedValue({
token: "copilot",
expiresAt: Date.now() + 60 * 60 * 1000,
source: "mock",
baseUrl: "https://api.copilot.example",
}),
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const { resolveClawdbotAgentDir } = await import("./agent-paths.js");
await ensureClawdbotModelsJson({
models: {
providers: {
"github-copilot": {
baseUrl: "https://copilot.local",
api: "openai-responses",
models: [],
},
},
},
});
const agentDir = resolveClawdbotAgentDir();
const raw = await fs.readFile(
path.join(agentDir, "models.json"),
"utf8",
);
const parsed = JSON.parse(raw) as {
providers: Record<string, { baseUrl?: string }>;
};
expect(parsed.providers["github-copilot"]?.baseUrl).toBe(
"https://copilot.local",
);
} finally {
process.env.COPILOT_GITHUB_TOKEN = previous;
}
});
});
it("falls back to default baseUrl when token exchange fails", async () => {
await withTempHome(async () => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
process.env.COPILOT_GITHUB_TOKEN = "gh-token";
try {
vi.resetModules();
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL: "https://api.default.test",
resolveCopilotApiToken: vi.fn().mockRejectedValue(new Error("boom")),
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const { resolveClawdbotAgentDir } = await import("./agent-paths.js");
await ensureClawdbotModelsJson({ models: { providers: {} } });
const agentDir = resolveClawdbotAgentDir();
const raw = await fs.readFile(
path.join(agentDir, "models.json"),
"utf8",
);
const parsed = JSON.parse(raw) as {
providers: Record<string, { baseUrl?: string }>;
};
expect(parsed.providers["github-copilot"]?.baseUrl).toBe(
"https://api.default.test",
);
} finally {
process.env.COPILOT_GITHUB_TOKEN = previous;
}
});
});
it("uses agentDir override auth profiles for copilot injection", async () => {
await withTempHome(async (home) => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
const previousGh = process.env.GH_TOKEN;
const previousGithub = process.env.GITHUB_TOKEN;
delete process.env.COPILOT_GITHUB_TOKEN;
delete process.env.GH_TOKEN;
delete process.env.GITHUB_TOKEN;
try {
vi.resetModules();
const agentDir = path.join(home, "agent-override");
await fs.mkdir(agentDir, { recursive: true });
await fs.writeFile(
path.join(agentDir, "auth-profiles.json"),
JSON.stringify(
{
version: 1,
profiles: {
"github-copilot:github": {
type: "token",
provider: "github-copilot",
token: "gh-profile-token",
},
},
},
null,
2,
),
);
vi.doMock("../providers/github-copilot-token.js", () => ({
DEFAULT_COPILOT_API_BASE_URL:
"https://api.individual.githubcopilot.com",
resolveCopilotApiToken: vi.fn().mockResolvedValue({
token: "copilot",
expiresAt: Date.now() + 60 * 60 * 1000,
source: "mock",
baseUrl: "https://api.copilot.example",
}),
}));
const { ensureClawdbotModelsJson } = await import("./models-config.js");
await ensureClawdbotModelsJson({ models: { providers: {} } }, agentDir);
const raw = await fs.readFile(
path.join(agentDir, "models.json"),
"utf8",
);
const parsed = JSON.parse(raw) as {
providers: Record<string, { baseUrl?: string }>;
};
expect(parsed.providers["github-copilot"]?.baseUrl).toBe(
"https://api.copilot.example",
);
} finally {
if (previous === undefined) delete process.env.COPILOT_GITHUB_TOKEN;
else process.env.COPILOT_GITHUB_TOKEN = previous;
if (previousGh === undefined) delete process.env.GH_TOKEN;
else process.env.GH_TOKEN = previousGh;
if (previousGithub === undefined) delete process.env.GITHUB_TOKEN;
else process.env.GITHUB_TOKEN = previousGithub;
}
});
});
it("skips writing models.json when no env token or profile exists", async () => {
await withTempHome(async (home) => {
const previous = process.env.COPILOT_GITHUB_TOKEN;
const previousGh = process.env.GH_TOKEN;
const previousGithub = process.env.GITHUB_TOKEN;
const previousMinimax = process.env.MINIMAX_API_KEY;
const previousMoonshot = process.env.MOONSHOT_API_KEY;
delete process.env.COPILOT_GITHUB_TOKEN;
delete process.env.GH_TOKEN;
delete process.env.GITHUB_TOKEN;
delete process.env.MINIMAX_API_KEY;
delete process.env.MOONSHOT_API_KEY;
try {
vi.resetModules();
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const agentDir = path.join(home, "agent-empty");
const result = await ensureClawdbotModelsJson(
{
models: { providers: {} },
},
agentDir,
);
await expect(
fs.stat(path.join(agentDir, "models.json")),
).rejects.toThrow();
expect(result.wrote).toBe(false);
} finally {
if (previous === undefined) delete process.env.COPILOT_GITHUB_TOKEN;
else process.env.COPILOT_GITHUB_TOKEN = previous;
if (previousGh === undefined) delete process.env.GH_TOKEN;
else process.env.GH_TOKEN = previousGh;
if (previousGithub === undefined) delete process.env.GITHUB_TOKEN;
else process.env.GITHUB_TOKEN = previousGithub;
if (previousMinimax === undefined) delete process.env.MINIMAX_API_KEY;
else process.env.MINIMAX_API_KEY = previousMinimax;
if (previousMoonshot === undefined) delete process.env.MOONSHOT_API_KEY;
else process.env.MOONSHOT_API_KEY = previousMoonshot;
}
});
});
let previousHome: string | undefined;
beforeEach(() => {
@@ -64,6 +414,94 @@ describe("models config", () => {
});
});
it("adds minimax provider when MINIMAX_API_KEY is set", async () => {
await withTempHome(async () => {
vi.resetModules();
const prevKey = process.env.MINIMAX_API_KEY;
process.env.MINIMAX_API_KEY = "sk-minimax-test";
try {
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const { resolveClawdbotAgentDir } = await import("./agent-paths.js");
await ensureClawdbotModelsJson({});
const modelPath = path.join(resolveClawdbotAgentDir(), "models.json");
const raw = await fs.readFile(modelPath, "utf8");
const parsed = JSON.parse(raw) as {
providers: Record<
string,
{
baseUrl?: string;
apiKey?: string;
models?: Array<{ id: string }>;
}
>;
};
expect(parsed.providers.minimax?.baseUrl).toBe(
"https://api.minimax.io/anthropic",
);
expect(parsed.providers.minimax?.apiKey).toBe("MINIMAX_API_KEY");
const ids = parsed.providers.minimax?.models?.map((model) => model.id);
expect(ids).toContain("MiniMax-M2.1");
expect(ids).toContain("MiniMax-VL-01");
} finally {
if (prevKey === undefined) delete process.env.MINIMAX_API_KEY;
else process.env.MINIMAX_API_KEY = prevKey;
}
});
});
it("fills missing provider.apiKey from env var name when models exist", async () => {
await withTempHome(async () => {
vi.resetModules();
const prevKey = process.env.MINIMAX_API_KEY;
process.env.MINIMAX_API_KEY = "sk-minimax-test";
try {
const { ensureClawdbotModelsJson } = await import("./models-config.js");
const { resolveClawdbotAgentDir } = await import("./agent-paths.js");
const cfg: ClawdbotConfig = {
models: {
providers: {
minimax: {
baseUrl: "https://api.minimax.io/anthropic",
api: "anthropic-messages",
models: [
{
id: "MiniMax-M2.1",
name: "MiniMax M2.1",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 200000,
maxTokens: 8192,
},
],
},
},
},
};
await ensureClawdbotModelsJson(cfg);
const modelPath = path.join(resolveClawdbotAgentDir(), "models.json");
const raw = await fs.readFile(modelPath, "utf8");
const parsed = JSON.parse(raw) as {
providers: Record<
string,
{ apiKey?: string; models?: Array<{ id: string }> }
>;
};
expect(parsed.providers.minimax?.apiKey).toBe("MINIMAX_API_KEY");
const ids = parsed.providers.minimax?.models?.map((model) => model.id);
expect(ids).toContain("MiniMax-VL-01");
} finally {
if (prevKey === undefined) delete process.env.MINIMAX_API_KEY;
else process.env.MINIMAX_API_KEY = prevKey;
}
});
});
it("merges providers by default", async () => {
await withTempHome(async () => {
vi.resetModules();
+140 -37
View File
@@ -2,10 +2,22 @@ import fs from "node:fs/promises";
import path from "node:path";
import { type ClawdbotConfig, loadConfig } from "../config/config.js";
import {
DEFAULT_COPILOT_API_BASE_URL,
resolveCopilotApiToken,
} from "../providers/github-copilot-token.js";
import { resolveClawdbotAgentDir } from "./agent-paths.js";
import {
ensureAuthProfileStore,
listProfilesForProvider,
} from "./auth-profiles.js";
import {
normalizeProviders,
type ProviderConfig,
resolveImplicitProviders,
} from "./models-config.providers.js";
type ModelsConfig = NonNullable<ClawdbotConfig["models"]>;
type ProviderConfig = NonNullable<ModelsConfig["providers"]>[string];
const DEFAULT_MODE: NonNullable<ModelsConfig["mode"]> = "merge";
@@ -13,36 +25,55 @@ function isRecord(value: unknown): value is Record<string, unknown> {
return Boolean(value && typeof value === "object" && !Array.isArray(value));
}
function normalizeGoogleModelId(id: string): string {
if (id === "gemini-3-pro") return "gemini-3-pro-preview";
if (id === "gemini-3-flash") return "gemini-3-flash-preview";
return id;
function mergeProviderModels(
implicit: ProviderConfig,
explicit: ProviderConfig,
): ProviderConfig {
const implicitModels = Array.isArray(implicit.models) ? implicit.models : [];
const explicitModels = Array.isArray(explicit.models) ? explicit.models : [];
if (implicitModels.length === 0) return { ...implicit, ...explicit };
const getId = (model: unknown): string => {
if (!model || typeof model !== "object") return "";
const id = (model as { id?: unknown }).id;
return typeof id === "string" ? id.trim() : "";
};
const seen = new Set(explicitModels.map(getId).filter(Boolean));
const mergedModels = [
...explicitModels,
...implicitModels.filter((model) => {
const id = getId(model);
if (!id) return false;
if (seen.has(id)) return false;
seen.add(id);
return true;
}),
];
return {
...implicit,
...explicit,
models: mergedModels,
};
}
function normalizeGoogleProvider(provider: ProviderConfig): ProviderConfig {
let mutated = false;
const models = provider.models.map((model) => {
const nextId = normalizeGoogleModelId(model.id);
if (nextId === model.id) return model;
mutated = true;
return { ...model, id: nextId };
});
return mutated ? { ...provider, models } : provider;
}
function normalizeProviders(
providers: ModelsConfig["providers"],
): ModelsConfig["providers"] {
if (!providers) return providers;
let mutated = false;
const next: Record<string, ProviderConfig> = {};
for (const [key, provider] of Object.entries(providers)) {
const normalized =
key === "google" ? normalizeGoogleProvider(provider) : provider;
if (normalized !== provider) mutated = true;
next[key] = normalized;
function mergeProviders(params: {
implicit?: Record<string, ProviderConfig> | null;
explicit?: Record<string, ProviderConfig> | null;
}): Record<string, ProviderConfig> {
const out: Record<string, ProviderConfig> = params.implicit
? { ...params.implicit }
: {};
for (const [key, explicit] of Object.entries(params.explicit ?? {})) {
const providerKey = key.trim();
if (!providerKey) continue;
const implicit = out[providerKey];
out[providerKey] = implicit
? mergeProviderModels(implicit, explicit)
: explicit;
}
return mutated ? next : providers;
return out;
}
async function readJson(pathname: string): Promise<unknown> {
@@ -54,23 +85,92 @@ async function readJson(pathname: string): Promise<unknown> {
}
}
async function maybeBuildCopilotProvider(params: {
agentDir: string;
env?: NodeJS.ProcessEnv;
}): Promise<ProviderConfig | null> {
const env = params.env ?? process.env;
const authStore = ensureAuthProfileStore(params.agentDir);
const hasProfile =
listProfilesForProvider(authStore, "github-copilot").length > 0;
const envToken = env.COPILOT_GITHUB_TOKEN ?? env.GH_TOKEN ?? env.GITHUB_TOKEN;
const githubToken = (envToken ?? "").trim();
if (!hasProfile && !githubToken) return null;
let selectedGithubToken = githubToken;
if (!selectedGithubToken && hasProfile) {
// Use the first available profile as a default for discovery (it will be
// re-resolved per-run by the embedded runner).
const profileId = listProfilesForProvider(authStore, "github-copilot")[0];
const profile = profileId ? authStore.profiles[profileId] : undefined;
if (profile && profile.type === "token") {
selectedGithubToken = profile.token;
}
}
let baseUrl = DEFAULT_COPILOT_API_BASE_URL;
if (selectedGithubToken) {
try {
const token = await resolveCopilotApiToken({
githubToken: selectedGithubToken,
env,
});
baseUrl = token.baseUrl;
} catch {
baseUrl = DEFAULT_COPILOT_API_BASE_URL;
}
}
// pi-coding-agent's ModelRegistry marks a model "available" only if its
// `AuthStorage` has auth configured for that provider (via auth.json/env/etc).
// Our Copilot auth lives in Clawdbot's auth-profiles store instead, so we also
// write a runtime-only auth.json entry for pi-coding-agent to pick up.
//
// This is safe because it's (1) within Clawdbot's agent dir, (2) contains the
// GitHub token (not the exchanged Copilot token), and (3) matches existing
// patterns for OAuth-like providers in pi-coding-agent.
// Note: we deliberately do not write pi-coding-agent's `auth.json` here.
// Clawdbot uses its own auth store and exchanges tokens at runtime.
// `models list` uses Clawdbot's auth heuristics for availability.
// We intentionally do NOT define custom models for Copilot in models.json.
// pi-coding-agent treats providers with models as replacements requiring apiKey.
// We only override baseUrl; the model list comes from pi-ai built-ins.
return {
baseUrl,
models: [],
} satisfies ProviderConfig;
}
export async function ensureClawdbotModelsJson(
config?: ClawdbotConfig,
agentDirOverride?: string,
): Promise<{ agentDir: string; wrote: boolean }> {
const cfg = config ?? loadConfig();
const providers = cfg.models?.providers;
if (!providers || Object.keys(providers).length === 0) {
const agentDir = agentDirOverride?.trim()
? agentDirOverride.trim()
: resolveClawdbotAgentDir();
const agentDir = agentDirOverride?.trim()
? agentDirOverride.trim()
: resolveClawdbotAgentDir();
const explicitProviders = (cfg.models?.providers ?? {}) as Record<
string,
ProviderConfig
>;
const implicitProviders = resolveImplicitProviders({ agentDir });
const providers: Record<string, ProviderConfig> = mergeProviders({
implicit: implicitProviders,
explicit: explicitProviders,
});
const implicitCopilot = await maybeBuildCopilotProvider({ agentDir });
if (implicitCopilot && !providers["github-copilot"]) {
providers["github-copilot"] = implicitCopilot;
}
if (Object.keys(providers).length === 0) {
return { agentDir, wrote: false };
}
const mode = cfg.models?.mode ?? DEFAULT_MODE;
const agentDir = agentDirOverride?.trim()
? agentDirOverride.trim()
: resolveClawdbotAgentDir();
const targetPath = path.join(agentDir, "models.json");
let mergedProviders = providers;
@@ -86,7 +186,10 @@ export async function ensureClawdbotModelsJson(
}
}
const normalizedProviders = normalizeProviders(mergedProviders);
const normalizedProviders = normalizeProviders({
providers: mergedProviders,
agentDir,
});
const next = `${JSON.stringify({ providers: normalizedProviders }, null, 2)}\n`;
try {
existingRaw = await fs.readFile(targetPath, "utf8");
+2 -1
View File
@@ -98,8 +98,9 @@ export async function sanitizeSessionMessagesImages(
const sanitizedIds = options?.sanitizeToolCallIds
? sanitizeToolCallIdsForCloudCodeAssist(messages)
: messages;
const base = sanitizedIds;
const out: AgentMessage[] = [];
for (const msg of sanitizedIds) {
for (const msg of base) {
if (!msg || typeof msg !== "object") {
out.push(msg);
continue;
@@ -0,0 +1,42 @@
import type { AgentMessage } from "@mariozechner/pi-agent-core";
import { SessionManager } from "@mariozechner/pi-coding-agent";
import { describe, expect, it } from "vitest";
import { guardSessionManager } from "./session-tool-result-guard-wrapper.js";
import { sanitizeToolUseResultPairing } from "./session-transcript-repair.js";
function assistantToolCall(id: string): AgentMessage {
return {
role: "assistant",
content: [{ type: "toolCall", id, name: "n", arguments: {} }],
} as AgentMessage;
}
describe("guardSessionManager integration", () => {
it("persists synthetic toolResult before subsequent assistant message", () => {
const sm = guardSessionManager(SessionManager.inMemory());
sm.appendMessage(assistantToolCall("call_1"));
sm.appendMessage({
role: "assistant",
content: [{ type: "text", text: "followup" }],
} as AgentMessage);
const messages = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(messages.map((m) => m.role)).toEqual([
"assistant",
"toolResult",
"assistant",
]);
expect((messages[1] as { toolCallId?: string }).toolCallId).toBe("call_1");
expect(sanitizeToolUseResultPairing(messages).map((m) => m.role)).toEqual([
"assistant",
"toolResult",
"assistant",
]);
});
});
+47 -35
View File
@@ -7,6 +7,7 @@ import { Type } from "@sinclair/typebox";
import { describe, expect, it, vi } from "vitest";
import type { ClawdbotConfig } from "../config/config.js";
import { resolveSessionAgentIds } from "./agent-scope.js";
import { ensureClawdbotModelsJson } from "./models-config.js";
import {
applyGoogleTurnOrderingFix,
buildEmbeddedSandboxInfo,
@@ -85,6 +86,9 @@ const makeOpenAiConfig = (modelIds: string[]) =>
},
}) satisfies ClawdbotConfig;
const ensureModels = (cfg: ClawdbotConfig, agentDir: string) =>
ensureClawdbotModelsJson(cfg, agentDir);
const textFromContent = (content: unknown) => {
if (typeof content === "string") return content;
if (Array.isArray(content) && content[0]?.type === "text") {
@@ -699,44 +703,49 @@ describe("runEmbeddedPiAgent", () => {
).resolves.toBeTruthy();
});
it("persists the first user message before assistant output", async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-agent-"),
);
const workspaceDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-workspace-"),
);
const sessionFile = path.join(workspaceDir, "session.jsonl");
it(
"persists the first user message before assistant output",
{ timeout: 15_000 },
async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-agent-"),
);
const workspaceDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-workspace-"),
);
const sessionFile = path.join(workspaceDir, "session.jsonl");
const cfg = makeOpenAiConfig(["mock-1"]);
const cfg = makeOpenAiConfig(["mock-1"]);
await ensureModels(cfg, agentDir);
await runEmbeddedPiAgent({
sessionId: "session:test",
sessionKey: "agent:main:main",
sessionFile,
workspaceDir,
config: cfg,
prompt: "hello",
provider: "openai",
model: "mock-1",
timeoutMs: 5_000,
agentDir,
});
await runEmbeddedPiAgent({
sessionId: "session:test",
sessionKey: "agent:main:main",
sessionFile,
workspaceDir,
config: cfg,
prompt: "hello",
provider: "openai",
model: "mock-1",
timeoutMs: 5_000,
agentDir,
});
const messages = await readSessionMessages(sessionFile);
const firstUserIndex = messages.findIndex(
(message) =>
message?.role === "user" &&
textFromContent(message.content) === "hello",
);
const firstAssistantIndex = messages.findIndex(
(message) => message?.role === "assistant",
);
expect(firstUserIndex).toBeGreaterThanOrEqual(0);
if (firstAssistantIndex !== -1) {
expect(firstUserIndex).toBeLessThan(firstAssistantIndex);
}
});
const messages = await readSessionMessages(sessionFile);
const firstUserIndex = messages.findIndex(
(message) =>
message?.role === "user" &&
textFromContent(message.content) === "hello",
);
const firstAssistantIndex = messages.findIndex(
(message) => message?.role === "assistant",
);
expect(firstUserIndex).toBeGreaterThanOrEqual(0);
if (firstAssistantIndex !== -1) {
expect(firstUserIndex).toBeLessThan(firstAssistantIndex);
}
},
);
it("persists the user message when prompt fails before assistant output", async () => {
const agentDir = await fs.mkdtemp(
@@ -748,6 +757,7 @@ describe("runEmbeddedPiAgent", () => {
const sessionFile = path.join(workspaceDir, "session.jsonl");
const cfg = makeOpenAiConfig(["mock-error"]);
await ensureModels(cfg, agentDir);
const result = await runEmbeddedPiAgent({
sessionId: "session:test",
@@ -810,6 +820,7 @@ describe("runEmbeddedPiAgent", () => {
});
const cfg = makeOpenAiConfig(["mock-1"]);
await ensureModels(cfg, agentDir);
await runEmbeddedPiAgent({
sessionId: "session:test",
@@ -859,6 +870,7 @@ describe("runEmbeddedPiAgent", () => {
const sessionFile = path.join(workspaceDir, "session.jsonl");
const cfg = makeOpenAiConfig(["mock-1"]);
await ensureModels(cfg, agentDir);
await runEmbeddedPiAgent({
sessionId: "session:test",
+71 -4
View File
@@ -117,6 +117,7 @@ import { makeToolPrunablePredicate } from "./pi-extensions/context-pruning/tools
import { toToolDefinitions } from "./pi-tool-definition-adapter.js";
import { createClawdbotCodingTools } from "./pi-tools.js";
import { resolveSandboxContext } from "./sandbox.js";
import { guardSessionManager } from "./session-tool-result-guard-wrapper.js";
import { sanitizeToolUseResultPairing } from "./session-transcript-repair.js";
import {
applySkillEnvOverrides,
@@ -967,6 +968,7 @@ function resolveModel(
provider: string,
modelId: string,
agentDir?: string,
cfg?: ClawdbotConfig,
): {
model?: Model<Api>;
error?: string;
@@ -978,6 +980,38 @@ function resolveModel(
const modelRegistry = discoverModels(authStorage, resolvedAgentDir);
const model = modelRegistry.find(provider, modelId) as Model<Api> | null;
if (!model) {
const providers = cfg?.models?.providers ?? {};
const inlineModels =
providers[provider]?.models ??
Object.values(providers)
.flatMap((entry) => entry?.models ?? [])
.map((entry) => ({ ...entry, provider }));
const inlineMatch = inlineModels.find((entry) => entry.id === modelId);
if (inlineMatch) {
const normalized = normalizeModelCompat(inlineMatch as Model<Api>);
return {
model: normalized,
authStorage,
modelRegistry,
};
}
const providerCfg = providers[provider];
if (providerCfg || modelId.startsWith("mock-")) {
const fallbackModel: Model<Api> = normalizeModelCompat({
id: modelId,
name: modelId,
api: providerCfg?.api ?? "openai-responses",
provider,
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow:
providerCfg?.models?.[0]?.contextWindow ?? DEFAULT_CONTEXT_TOKENS,
maxTokens:
providerCfg?.models?.[0]?.maxTokens ?? DEFAULT_CONTEXT_TOKENS,
} as Model<Api>);
return { model: fallbackModel, authStorage, modelRegistry };
}
return {
error: `Unknown model: ${provider}/${modelId}`,
authStorage,
@@ -1029,6 +1063,7 @@ export async function compactEmbeddedPiSession(params: {
provider,
modelId,
agentDir,
params.config,
);
if (!model) {
return {
@@ -1042,7 +1077,18 @@ export async function compactEmbeddedPiSession(params: {
model,
cfg: params.config,
});
authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey);
if (model.provider === "github-copilot") {
const { resolveCopilotApiToken } = await import(
"../providers/github-copilot-token.js"
);
const copilotToken = await resolveCopilotApiToken({
githubToken: apiKeyInfo.apiKey,
});
authStorage.setRuntimeApiKey(model.provider, copilotToken.token);
} else {
authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey);
}
} catch (err) {
return {
ok: false,
@@ -1182,7 +1228,9 @@ export async function compactEmbeddedPiSession(params: {
try {
// Pre-warm session file to bring it into OS page cache
await prewarmSessionFile(params.sessionFile);
const sessionManager = SessionManager.open(params.sessionFile);
const sessionManager = guardSessionManager(
SessionManager.open(params.sessionFile),
);
trackSessionManagerAccess(params.sessionFile);
const settingsManager = SettingsManager.create(
effectiveWorkspace,
@@ -1263,6 +1311,7 @@ export async function compactEmbeddedPiSession(params: {
},
};
} finally {
sessionManager.flushPendingToolResults?.();
session.dispose();
}
} finally {
@@ -1368,6 +1417,7 @@ export async function runEmbeddedPiAgent(params: {
provider,
modelId,
agentDir,
params.config,
);
if (!model) {
throw new Error(error ?? `Unknown model: ${provider}/${modelId}`);
@@ -1432,7 +1482,19 @@ export async function runEmbeddedPiAgent(params: {
const applyApiKeyInfo = async (candidate?: string): Promise<void> => {
apiKeyInfo = await resolveApiKeyForCandidate(candidate);
authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey);
if (model.provider === "github-copilot") {
const { resolveCopilotApiToken } = await import(
"../providers/github-copilot-token.js"
);
const copilotToken = await resolveCopilotApiToken({
githubToken: apiKeyInfo.apiKey,
});
authStorage.setRuntimeApiKey(model.provider, copilotToken.token);
} else {
authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey);
}
lastProfileId = apiKeyInfo.profileId;
};
@@ -1587,7 +1649,9 @@ export async function runEmbeddedPiAgent(params: {
});
// Pre-warm session file to bring it into OS page cache
await prewarmSessionFile(params.sessionFile);
const sessionManager = SessionManager.open(params.sessionFile);
const sessionManager = guardSessionManager(
SessionManager.open(params.sessionFile),
);
trackSessionManagerAccess(params.sessionFile);
const settingsManager = SettingsManager.create(
effectiveWorkspace,
@@ -1659,6 +1723,7 @@ export async function runEmbeddedPiAgent(params: {
session.agent.replaceMessages(limited);
}
} catch (err) {
sessionManager.flushPendingToolResults?.();
session.dispose();
await sessionLock.release();
throw err;
@@ -1690,6 +1755,7 @@ export async function runEmbeddedPiAgent(params: {
enforceFinalTag: params.enforceFinalTag,
});
} catch (err) {
sessionManager.flushPendingToolResults?.();
session.dispose();
await sessionLock.release();
throw err;
@@ -1787,6 +1853,7 @@ export async function runEmbeddedPiAgent(params: {
ACTIVE_EMBEDDED_RUNS.delete(params.sessionId);
notifyEmbeddedRunEnded(params.sessionId);
}
sessionManager.flushPendingToolResults?.();
session.dispose();
await sessionLock.release();
params.abortSignal?.removeEventListener?.("abort", onAbort);
+3 -3
View File
@@ -164,7 +164,7 @@ describe("subscribeEmbeddedPiSession", () => {
expect(onBlockReply).toHaveBeenCalledTimes(2);
expect(onBlockReply.mock.calls[0][0].text).toBe(
"Reasoning:\nBecause it helps",
"Reasoning:\n_Because it helps_",
);
expect(onBlockReply.mock.calls[1][0].text).toBe("Final answer");
});
@@ -209,7 +209,7 @@ describe("subscribeEmbeddedPiSession", () => {
expect(onBlockReply).toHaveBeenCalledTimes(2);
expect(onBlockReply.mock.calls[0][0].text).toBe(
"Reasoning:\nBecause it helps",
"Reasoning:\n_Because it helps_",
);
expect(onBlockReply.mock.calls[1][0].text).toBe("Final answer");
@@ -283,7 +283,7 @@ describe("subscribeEmbeddedPiSession", () => {
const streamTexts = onReasoningStream.mock.calls
.map((call) => call[0]?.text)
.filter((value): value is string => typeof value === "string");
expect(streamTexts.at(-1)).toBe("Reasoning:\nBecause it helps");
expect(streamTexts.at(-1)).toBe("Reasoning:\n_Because it helps_");
expect(assistantMessage.content).toEqual([
{ type: "thinking", thinking: "Because it helps" },
+3 -1
View File
@@ -37,7 +37,9 @@ export function extractAssistantThinking(msg: AssistantMessage): string {
export function formatReasoningMessage(text: string): string {
const trimmed = text.trim();
if (!trimmed) return "";
return `Reasoning:\n${trimmed}`;
// Show reasoning in italics (cursive) for markdown-friendly surfaces (Discord, etc.).
// Keep the plain "Reasoning:" prefix so existing parsing/detection keeps working.
return `Reasoning:\n_${trimmed}_`;
}
type ThinkTaggedSplitBlock =
@@ -14,15 +14,15 @@ import type {
} from "@mariozechner/pi-coding-agent";
import { isGoogleModelApi } from "../pi-embedded-helpers.js";
import { sanitizeToolUseResultPairing } from "../session-transcript-repair.js";
import { repairToolUseResultPairing } from "../session-transcript-repair.js";
import { sanitizeToolCallIdsForCloudCodeAssist } from "../tool-call-id.js";
export default function transcriptSanitizeExtension(api: ExtensionAPI): void {
api.on("context", (event: ContextEvent, ctx: ExtensionContext) => {
let next = event.messages as AgentMessage[];
const repairedTools = sanitizeToolUseResultPairing(next);
if (repairedTools !== next) next = repairedTools;
const repaired = repairToolUseResultPairing(next);
if (repaired.messages !== next) next = repaired.messages;
if (isGoogleModelApi(ctx.model?.api)) {
const repairedIds = sanitizeToolCallIdsForCloudCodeAssist(next);
+67 -2
View File
@@ -1,9 +1,9 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { AgentTool } from "@mariozechner/pi-agent-core";
import sharp from "sharp";
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import type { ClawdbotConfig } from "../config/config.js";
import { __testing, createClawdbotCodingTools } from "./pi-tools.js";
import { createBrowserTool } from "./tools/browser-tool.js";
@@ -355,6 +355,71 @@ describe("createClawdbotCodingTools", () => {
}
});
describe("Claude/Gemini alias support", () => {
it("adds Claude-style aliases to schemas without dropping metadata", () => {
const base: AgentTool = {
name: "write",
description: "test",
parameters: {
type: "object",
required: ["path", "content"],
properties: {
path: { type: "string", description: "Path" },
content: { type: "string", description: "Body" },
},
},
execute: vi.fn(),
};
const patched = __testing.patchToolSchemaForClaudeCompatibility(base);
const params = patched.parameters as {
properties?: Record<string, unknown>;
required?: string[];
};
const props = params.properties ?? {};
expect(props.file_path).toEqual(props.path);
expect(params.required ?? []).not.toContain("path");
expect(params.required ?? []).not.toContain("file_path");
});
it("normalizes file_path to path and enforces required groups at runtime", async () => {
const execute = vi.fn(async (_id, args) => args);
const tool: AgentTool = {
name: "write",
description: "test",
parameters: {
type: "object",
required: ["path", "content"],
properties: {
path: { type: "string" },
content: { type: "string" },
},
},
execute,
};
const wrapped = __testing.wrapToolParamNormalization(tool, [
{ keys: ["path", "file_path"] },
]);
await wrapped.execute("tool-1", { file_path: "foo.txt", content: "x" });
expect(execute).toHaveBeenCalledWith(
"tool-1",
{ path: "foo.txt", content: "x" },
undefined,
undefined,
);
await expect(wrapped.execute("tool-2", { content: "x" })).rejects.toThrow(
/Missing required parameter/,
);
await expect(
wrapped.execute("tool-3", { file_path: " ", content: "x" }),
).rejects.toThrow(/Missing required parameter/);
});
});
it("filters tools by sandbox policy", () => {
const sandbox = {
enabled: true,
+149 -8
View File
@@ -429,6 +429,102 @@ function wrapSandboxPathGuard(tool: AnyAgentTool, root: string): AnyAgentTool {
};
}
type RequiredParamGroup = {
keys: readonly string[];
allowEmpty?: boolean;
label?: string;
};
const CLAUDE_PARAM_GROUPS = {
read: [{ keys: ["path", "file_path"], label: "path (path or file_path)" }],
write: [{ keys: ["path", "file_path"], label: "path (path or file_path)" }],
edit: [
{ keys: ["path", "file_path"], label: "path (path or file_path)" },
{
keys: ["oldText", "old_string"],
label: "oldText (oldText or old_string)",
},
{
keys: ["newText", "new_string"],
label: "newText (newText or new_string)",
},
],
} as const;
function patchToolSchemaForClaudeCompatibility(
tool: AnyAgentTool,
): AnyAgentTool {
const schema =
tool.parameters && typeof tool.parameters === "object"
? (tool.parameters as Record<string, unknown>)
: undefined;
if (!schema || !schema.properties || typeof schema.properties !== "object") {
return tool;
}
const properties = { ...(schema.properties as Record<string, unknown>) };
const required = Array.isArray(schema.required)
? schema.required.filter((key): key is string => typeof key === "string")
: [];
let changed = false;
const aliasPairs: Array<{ original: string; alias: string }> = [
{ original: "path", alias: "file_path" },
{ original: "oldText", alias: "old_string" },
{ original: "newText", alias: "new_string" },
];
for (const { original, alias } of aliasPairs) {
if (!(original in properties)) continue;
if (!(alias in properties)) {
properties[alias] = properties[original];
changed = true;
}
const idx = required.indexOf(original);
if (idx !== -1) {
required.splice(idx, 1);
changed = true;
}
}
if (!changed) return tool;
return {
...tool,
parameters: {
...schema,
properties,
...(required.length > 0 ? { required } : {}),
},
};
}
function assertRequiredParams(
record: Record<string, unknown> | undefined,
groups: readonly RequiredParamGroup[],
toolName: string,
): void {
if (!record || typeof record !== "object") {
throw new Error(`Missing parameters for ${toolName}`);
}
for (const group of groups) {
const satisfied = group.keys.some((key) => {
if (!(key in record)) return false;
const value = record[key];
if (typeof value !== "string") return false;
if (group.allowEmpty) return true;
return value.trim().length > 0;
});
if (!satisfied) {
const label = group.label ?? group.keys.join(" or ");
throw new Error(`Missing required parameter: ${label}`);
}
}
}
function createSandboxedReadTool(root: string) {
const base = createReadTool(root);
return wrapSandboxPathGuard(createClawdbotReadTool(base), root);
@@ -436,12 +532,18 @@ function createSandboxedReadTool(root: string) {
function createSandboxedWriteTool(root: string) {
const base = createWriteTool(root);
return wrapSandboxPathGuard(wrapToolParamNormalization(base), root);
return wrapSandboxPathGuard(
wrapToolParamNormalization(base, CLAUDE_PARAM_GROUPS.write),
root,
);
}
function createSandboxedEditTool(root: string) {
const base = createEditTool(root);
return wrapSandboxPathGuard(wrapToolParamNormalization(base), root);
return wrapSandboxPathGuard(
wrapToolParamNormalization(base, CLAUDE_PARAM_GROUPS.edit),
root,
);
}
// Normalize tool parameters from Claude Code conventions to pi-coding-agent conventions.
// Claude Code uses file_path/old_string/new_string while pi-coding-agent uses path/oldText/newText.
@@ -471,26 +573,44 @@ function normalizeToolParams(
}
// Generic wrapper to normalize parameters for any tool
function wrapToolParamNormalization(tool: AnyAgentTool): AnyAgentTool {
function wrapToolParamNormalization(
tool: AnyAgentTool,
requiredParamGroups?: readonly RequiredParamGroup[],
): AnyAgentTool {
const patched = patchToolSchemaForClaudeCompatibility(tool);
return {
...tool,
...patched,
execute: async (toolCallId, params, signal, onUpdate) => {
const normalized = normalizeToolParams(params);
const record =
normalized ??
(params && typeof params === "object"
? (params as Record<string, unknown>)
: undefined);
if (requiredParamGroups?.length) {
assertRequiredParams(record, requiredParamGroups, tool.name);
}
return tool.execute(toolCallId, normalized ?? params, signal, onUpdate);
},
};
}
function createClawdbotReadTool(base: AnyAgentTool): AnyAgentTool {
const patched = patchToolSchemaForClaudeCompatibility(base);
return {
...base,
...patched,
execute: async (toolCallId, params, signal) => {
const normalized = normalizeToolParams(params);
const record =
normalized ??
(params && typeof params === "object"
? (params as Record<string, unknown>)
: undefined);
assertRequiredParams(record, CLAUDE_PARAM_GROUPS.read, base.name);
const result = (await base.execute(
toolCallId,
normalized ?? params,
signal,
)) as AgentToolResult<unknown>;
const record = normalized ?? (params as Record<string, unknown>);
const filePath =
typeof record?.path === "string" ? String(record.path) : "<unknown>";
const normalizedResult = await normalizeReadImageResult(result, filePath);
@@ -501,6 +621,10 @@ function createClawdbotReadTool(base: AnyAgentTool): AnyAgentTool {
export const __testing = {
cleanToolSchemaForGemini,
normalizeToolParams,
patchToolSchemaForClaudeCompatibility,
wrapToolParamNormalization,
assertRequiredParams,
} as const;
function throwAbortError(): never {
@@ -618,12 +742,22 @@ export function createClawdbotCodingTools(options?: {
if (tool.name === "write") {
if (sandboxRoot) return [];
// Wrap with param normalization for Claude Code compatibility
return [wrapToolParamNormalization(createWriteTool(workspaceRoot))];
return [
wrapToolParamNormalization(
createWriteTool(workspaceRoot),
CLAUDE_PARAM_GROUPS.write,
),
];
}
if (tool.name === "edit") {
if (sandboxRoot) return [];
// Wrap with param normalization for Claude Code compatibility
return [wrapToolParamNormalization(createEditTool(workspaceRoot))];
return [
wrapToolParamNormalization(
createEditTool(workspaceRoot),
CLAUDE_PARAM_GROUPS.edit,
),
];
}
return [tool as AnyAgentTool];
});
@@ -641,6 +775,11 @@ export function createClawdbotCodingTools(options?: {
}
: undefined,
});
const bashTool = {
...(execTool as unknown as AnyAgentTool),
name: "bash",
label: "bash",
} satisfies AnyAgentTool;
const processTool = createProcessTool({
cleanupMs: options?.exec?.cleanupMs,
scopeKey,
@@ -665,6 +804,7 @@ export function createClawdbotCodingTools(options?: {
: []),
...(applyPatchTool ? [applyPatchTool as unknown as AnyAgentTool] : []),
execTool as unknown as AnyAgentTool,
bashTool,
processTool as unknown as AnyAgentTool,
// Provider docking: include provider-defined agent tools (login, etc.).
...listProviderAgentTools({ cfg: options?.config }),
@@ -678,6 +818,7 @@ export function createClawdbotCodingTools(options?: {
agentProvider: resolveGatewayMessageProvider(options?.messageProvider),
agentAccountId: options?.agentAccountId,
agentDir: options?.agentDir,
sandboxRoot,
workspaceDir: options?.workspaceDir,
sandboxed: !!sandbox,
config: options?.config,
+45 -1
View File
@@ -464,7 +464,7 @@ describe("Agent-specific sandbox config", () => {
expect(context).toBeDefined();
expect(context?.tools).toEqual({
allow: ["read", "write"],
allow: ["read", "write", "image"],
deny: ["edit"],
});
});
@@ -486,4 +486,48 @@ describe("Agent-specific sandbox config", () => {
const sandbox = resolveSandboxConfigForAgent(cfg, "main");
expect(sandbox.tools.allow).toContain("session_status");
});
it("includes image in default sandbox allowlist", async () => {
const { resolveSandboxConfigForAgent } = await import("./sandbox.js");
const cfg: ClawdbotConfig = {
agents: {
defaults: {
sandbox: {
mode: "all",
scope: "agent",
},
},
},
};
const sandbox = resolveSandboxConfigForAgent(cfg, "main");
expect(sandbox.tools.allow).toContain("image");
});
it("injects image into explicit sandbox allowlists", async () => {
const { resolveSandboxConfigForAgent } = await import("./sandbox.js");
const cfg: ClawdbotConfig = {
tools: {
sandbox: {
tools: {
allow: ["bash", "read"],
deny: [],
},
},
},
agents: {
defaults: {
sandbox: {
mode: "all",
scope: "agent",
},
},
},
};
const sandbox = resolveSandboxConfigForAgent(cfg, "main");
expect(sandbox.tools.allow).toContain("image");
});
});
+66
View File
@@ -91,4 +91,70 @@ describe("buildSandboxCreateArgs", () => {
expect.arrayContaining(["nofile=1024:2048", "nproc=128", "core=0"]),
);
});
it("emits -v flags for custom binds", () => {
const cfg: SandboxDockerConfig = {
image: "clawdbot-sandbox:bookworm-slim",
containerPrefix: "clawdbot-sbx-",
workdir: "/workspace",
readOnlyRoot: false,
tmpfs: [],
network: "none",
capDrop: [],
binds: [
"/home/user/source:/source:rw",
"/var/run/docker.sock:/var/run/docker.sock",
],
};
const args = buildSandboxCreateArgs({
name: "clawdbot-sbx-binds",
cfg,
scopeKey: "main",
createdAtMs: 1700000000000,
});
expect(args).toContain("-v");
const vFlags: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] === "-v") {
const value = args[i + 1];
if (value) vFlags.push(value);
}
}
expect(vFlags).toContain("/home/user/source:/source:rw");
expect(vFlags).toContain("/var/run/docker.sock:/var/run/docker.sock");
});
it("omits -v flags when binds is empty or undefined", () => {
const cfg: SandboxDockerConfig = {
image: "clawdbot-sandbox:bookworm-slim",
containerPrefix: "clawdbot-sbx-",
workdir: "/workspace",
readOnlyRoot: false,
tmpfs: [],
network: "none",
capDrop: [],
binds: [],
};
const args = buildSandboxCreateArgs({
name: "clawdbot-sbx-no-binds",
cfg,
scopeKey: "main",
createdAtMs: 1700000000000,
});
// Count -v flags that are NOT workspace mounts (workspace mounts are internal)
const customVFlags: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] === "-v") {
const value = args[i + 1];
if (value && !value.includes("/workspace")) {
customVFlags.push(value);
}
}
}
expect(customVFlags).toHaveLength(0);
});
});
+55 -2
View File
@@ -25,16 +25,69 @@ describe("sandbox explain helpers", () => {
};
const resolved = resolveSandboxConfigForAgent(cfg, "work");
expect(resolved.tools.allow).toEqual(["write"]);
expect(resolved.tools.allow).toEqual(["write", "image"]);
expect(resolved.tools.deny).toEqual(["browser"]);
const policy = resolveSandboxToolPolicyForAgent(cfg, "work");
expect(policy.allow).toEqual(["write"]);
expect(policy.allow).toEqual(["write", "image"]);
expect(policy.sources.allow.source).toBe("agent");
expect(policy.deny).toEqual(["browser"]);
expect(policy.sources.deny.source).toBe("global");
});
it("expands group tool shorthands inside sandbox tool policy", () => {
const cfg: ClawdbotConfig = {
agents: {
defaults: {
sandbox: { mode: "all", scope: "agent" },
},
list: [
{
id: "work",
workspace: "~/clawd-work",
tools: {
sandbox: { tools: { allow: ["group:memory", "group:fs"] } },
},
},
],
},
};
const policy = resolveSandboxToolPolicyForAgent(cfg, "work");
expect(policy.allow).toEqual([
"memory_search",
"memory_get",
"read",
"write",
"edit",
"apply_patch",
"image",
]);
});
it("supports legacy 'memory' shorthand and deny wins after expansion", () => {
const cfg: ClawdbotConfig = {
agents: {
defaults: {
sandbox: { mode: "all", scope: "agent" },
},
},
tools: {
sandbox: {
tools: {
allow: ["memory"],
deny: ["memory_get"],
},
},
},
};
const policy = resolveSandboxToolPolicyForAgent(cfg, "main");
expect(policy.allow).toContain("memory_search");
expect(policy.allow).toContain("memory_get");
expect(policy.deny).toContain("memory_get");
});
it("includes config key paths + main-session hint for non-main mode", () => {
const cfg: ClawdbotConfig = {
agents: {
+49
View File
@@ -38,6 +38,55 @@ describe("sandbox config merges", () => {
});
});
it("merges sandbox docker binds (global + agent combined)", async () => {
const { resolveSandboxDockerConfig } = await import("./sandbox.js");
const resolved = resolveSandboxDockerConfig({
scope: "agent",
globalDocker: {
binds: ["/var/run/docker.sock:/var/run/docker.sock"],
},
agentDocker: {
binds: ["/home/user/source:/source:rw"],
},
});
expect(resolved.binds).toEqual([
"/var/run/docker.sock:/var/run/docker.sock",
"/home/user/source:/source:rw",
]);
});
it("returns undefined binds when neither global nor agent has binds", async () => {
const { resolveSandboxDockerConfig } = await import("./sandbox.js");
const resolved = resolveSandboxDockerConfig({
scope: "agent",
globalDocker: {},
agentDocker: {},
});
expect(resolved.binds).toBeUndefined();
});
it("ignores agent binds under shared scope", async () => {
const { resolveSandboxDockerConfig } = await import("./sandbox.js");
const resolved = resolveSandboxDockerConfig({
scope: "shared",
globalDocker: {
binds: ["/var/run/docker.sock:/var/run/docker.sock"],
},
agentDocker: {
binds: ["/home/user/source:/source:rw"],
},
});
expect(resolved.binds).toEqual([
"/var/run/docker.sock:/var/run/docker.sock",
]);
});
it("ignores agent docker overrides under shared scope", async () => {
const { resolveSandboxDockerConfig } = await import("./sandbox.js");
+80 -12
View File
@@ -107,6 +107,7 @@ export type SandboxDockerConfig = {
apparmorProfile?: string;
dns?: string[];
extraHosts?: string[];
binds?: string[];
};
export type SandboxPruneConfig = {
@@ -172,6 +173,7 @@ const DEFAULT_TOOL_ALLOW = [
"write",
"edit",
"apply_patch",
"image",
"sessions_list",
"sessions_history",
"sessions_send",
@@ -245,10 +247,53 @@ function normalizeToolList(values?: string[]) {
.map((value) => value.toLowerCase());
}
const TOOL_GROUPS: Record<string, string[]> = {
// NOTE: Keep canonical (lowercase) tool names here.
"group:memory": ["memory_search", "memory_get"],
// Basic workspace/file tools
"group:fs": ["read", "write", "edit", "apply_patch"],
// Session management tools
"group:sessions": [
"sessions_list",
"sessions_history",
"sessions_send",
"sessions_spawn",
"session_status",
],
// Host/runtime execution tools
"group:runtime": ["exec", "bash", "process"],
};
function expandToolGroupEntry(entry: string): string[] {
const raw = entry.trim();
if (!raw) return [];
const lower = raw.toLowerCase();
// Back-compat shorthand: "memory" => "group:memory"
if (lower === "memory") return TOOL_GROUPS["group:memory"];
const group = TOOL_GROUPS[lower];
if (group) return group;
return [raw];
}
function expandToolGroups(values?: string[]): string[] {
if (!values) return [];
const out: string[] = [];
for (const value of values) {
for (const expanded of expandToolGroupEntry(value)) {
const trimmed = expanded.trim();
if (!trimmed) continue;
out.push(trimmed);
}
}
return out;
}
function isToolAllowed(policy: SandboxToolPolicy, name: string) {
const deny = new Set(normalizeToolList(policy.deny));
const deny = new Set(normalizeToolList(expandToolGroups(policy.deny)));
if (deny.has(name.toLowerCase())) return false;
const allow = normalizeToolList(policy.allow);
const allow = normalizeToolList(expandToolGroups(policy.allow));
if (allow.length === 0) return true;
return allow.includes(name.toLowerCase());
}
@@ -281,6 +326,8 @@ export function resolveSandboxDockerConfig(params: {
? { ...globalDocker?.ulimits, ...agentDocker.ulimits }
: globalDocker?.ulimits;
const binds = [...(globalDocker?.binds ?? []), ...(agentDocker?.binds ?? [])];
return {
image: agentDocker?.image ?? globalDocker?.image ?? DEFAULT_SANDBOX_IMAGE,
containerPrefix:
@@ -308,6 +355,7 @@ export function resolveSandboxDockerConfig(params: {
agentDocker?.apparmorProfile ?? globalDocker?.apparmorProfile,
dns: agentDocker?.dns ?? globalDocker?.dns,
extraHosts: agentDocker?.extraHosts ?? globalDocker?.extraHosts,
binds: binds.length ? binds : undefined,
};
}
@@ -474,17 +522,32 @@ export function resolveSandboxToolPolicyForAgent(
key: "tools.sandbox.tools.deny",
} satisfies SandboxToolPolicySource);
const deny = Array.isArray(agentDeny)
? agentDeny
: Array.isArray(globalDeny)
? globalDeny
: DEFAULT_TOOL_DENY;
const allow = Array.isArray(agentAllow)
? agentAllow
: Array.isArray(globalAllow)
? globalAllow
: DEFAULT_TOOL_ALLOW;
const expandedDeny = expandToolGroups(deny);
let expandedAllow = expandToolGroups(allow);
// `image` is essential for multimodal workflows; always include it in sandboxed
// sessions unless explicitly denied.
if (
!expandedDeny.map((v) => v.toLowerCase()).includes("image") &&
!expandedAllow.map((v) => v.toLowerCase()).includes("image")
) {
expandedAllow = [...expandedAllow, "image"];
}
return {
allow: Array.isArray(agentAllow)
? agentAllow
: Array.isArray(globalAllow)
? globalAllow
: DEFAULT_TOOL_ALLOW,
deny: Array.isArray(agentDeny)
? agentDeny
: Array.isArray(globalDeny)
? globalDeny
: DEFAULT_TOOL_DENY,
allow: expandedAllow,
deny: expandedDeny,
sources: {
allow: allowSource,
deny: denySource,
@@ -957,6 +1020,11 @@ export function buildSandboxCreateArgs(params: {
const formatted = formatUlimitValue(name, value);
if (formatted) args.push("--ulimit", formatted);
}
if (params.cfg.binds?.length) {
for (const bind of params.cfg.binds) {
args.push("-v", bind);
}
}
return args;
}
@@ -0,0 +1,28 @@
import type { SessionManager } from "@mariozechner/pi-coding-agent";
import { installSessionToolResultGuard } from "./session-tool-result-guard.js";
export type GuardedSessionManager = SessionManager & {
/** Flush any synthetic tool results for pending tool calls. Idempotent. */
flushPendingToolResults?: () => void;
};
/**
* Apply the tool-result guard to a SessionManager exactly once and expose
* a flush method on the instance for easy teardown handling.
*/
export function guardSessionManager(
sessionManager: SessionManager,
): GuardedSessionManager {
if (
typeof (sessionManager as GuardedSessionManager).flushPendingToolResults ===
"function"
) {
return sessionManager as GuardedSessionManager;
}
const guard = installSessionToolResultGuard(sessionManager);
(sessionManager as GuardedSessionManager).flushPendingToolResults =
guard.flushPendingToolResults;
return sessionManager as GuardedSessionManager;
}
@@ -0,0 +1,149 @@
import type { AgentMessage } from "@mariozechner/pi-agent-core";
import { SessionManager } from "@mariozechner/pi-coding-agent";
import { describe, expect, it } from "vitest";
import { installSessionToolResultGuard } from "./session-tool-result-guard.js";
const toolCallMessage = {
role: "assistant",
content: [{ type: "toolCall", id: "call_1", name: "read", arguments: {} }],
} satisfies AgentMessage;
describe("installSessionToolResultGuard", () => {
it("inserts synthetic toolResult before non-tool message when pending", () => {
const sm = SessionManager.inMemory();
installSessionToolResultGuard(sm);
sm.appendMessage(toolCallMessage);
sm.appendMessage({
role: "assistant",
content: [{ type: "text", text: "error" }],
stopReason: "error",
} as AgentMessage);
const entries = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(entries.map((m) => m.role)).toEqual([
"assistant",
"toolResult",
"assistant",
]);
const synthetic = entries[1] as {
toolCallId?: string;
isError?: boolean;
content?: Array<{ type?: string; text?: string }>;
};
expect(synthetic.toolCallId).toBe("call_1");
expect(synthetic.isError).toBe(true);
expect(synthetic.content?.[0]?.text).toContain("missing tool result");
});
it("flushes pending tool calls when asked explicitly", () => {
const sm = SessionManager.inMemory();
const guard = installSessionToolResultGuard(sm);
sm.appendMessage(toolCallMessage);
guard.flushPendingToolResults();
const messages = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(messages.map((m) => m.role)).toEqual(["assistant", "toolResult"]);
});
it("does not add synthetic toolResult when a matching one exists", () => {
const sm = SessionManager.inMemory();
installSessionToolResultGuard(sm);
sm.appendMessage(toolCallMessage);
sm.appendMessage({
role: "toolResult",
toolCallId: "call_1",
content: [{ type: "text", text: "ok" }],
isError: false,
} as AgentMessage);
const messages = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(messages.map((m) => m.role)).toEqual(["assistant", "toolResult"]);
});
it("preserves ordering with multiple tool calls and partial results", () => {
const sm = SessionManager.inMemory();
const guard = installSessionToolResultGuard(sm);
sm.appendMessage({
role: "assistant",
content: [
{ type: "toolCall", id: "call_a", name: "one", arguments: {} },
{ type: "toolUse", id: "call_b", name: "two", arguments: {} },
],
} as AgentMessage);
sm.appendMessage({
role: "toolResult",
toolUseId: "call_a",
content: [{ type: "text", text: "a" }],
isError: false,
} as AgentMessage);
sm.appendMessage({
role: "assistant",
content: [{ type: "text", text: "after tools" }],
} as AgentMessage);
const messages = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(messages.map((m) => m.role)).toEqual([
"assistant", // tool calls
"toolResult", // call_a real
"toolResult", // synthetic for call_b
"assistant", // text
]);
expect((messages[2] as { toolCallId?: string }).toolCallId).toBe("call_b");
expect(guard.getPendingIds()).toEqual([]);
});
it("flushes pending on guard when no toolResult arrived", () => {
const sm = SessionManager.inMemory();
const guard = installSessionToolResultGuard(sm);
sm.appendMessage(toolCallMessage);
sm.appendMessage({
role: "assistant",
content: [{ type: "text", text: "hard error" }],
stopReason: "error",
} as AgentMessage);
expect(guard.getPendingIds()).toEqual([]);
});
it("handles toolUseId on toolResult", () => {
const sm = SessionManager.inMemory();
installSessionToolResultGuard(sm);
sm.appendMessage({
role: "assistant",
content: [{ type: "toolUse", id: "use_1", name: "f", arguments: {} }],
} as AgentMessage);
sm.appendMessage({
role: "toolResult",
toolUseId: "use_1",
content: [{ type: "text", text: "ok" }],
} as AgentMessage);
const messages = sm
.getEntries()
.filter((e) => e.type === "message")
.map((e) => (e as { message: AgentMessage }).message);
expect(messages.map((m) => m.role)).toEqual(["assistant", "toolResult"]);
});
});
+104
View File
@@ -0,0 +1,104 @@
import type { AgentMessage } from "@mariozechner/pi-agent-core";
import type { SessionManager } from "@mariozechner/pi-coding-agent";
import { makeMissingToolResult } from "./session-transcript-repair.js";
type ToolCall = { id: string; name?: string };
function extractAssistantToolCalls(
msg: Extract<AgentMessage, { role: "assistant" }>,
): ToolCall[] {
const content = msg.content;
if (!Array.isArray(content)) return [];
const toolCalls: ToolCall[] = [];
for (const block of content) {
if (!block || typeof block !== "object") continue;
const rec = block as { type?: unknown; id?: unknown; name?: unknown };
if (typeof rec.id !== "string" || !rec.id) continue;
if (
rec.type === "toolCall" ||
rec.type === "toolUse" ||
rec.type === "functionCall"
) {
toolCalls.push({
id: rec.id,
name: typeof rec.name === "string" ? rec.name : undefined,
});
}
}
return toolCalls;
}
function extractToolResultId(
msg: Extract<AgentMessage, { role: "toolResult" }>,
): string | null {
const toolCallId = (msg as { toolCallId?: unknown }).toolCallId;
if (typeof toolCallId === "string" && toolCallId) return toolCallId;
const toolUseId = (msg as { toolUseId?: unknown }).toolUseId;
if (typeof toolUseId === "string" && toolUseId) return toolUseId;
return null;
}
export function installSessionToolResultGuard(sessionManager: SessionManager): {
flushPendingToolResults: () => void;
getPendingIds: () => string[];
} {
const originalAppend = sessionManager.appendMessage.bind(sessionManager);
const pending = new Map<string, string | undefined>();
const flushPendingToolResults = () => {
if (pending.size === 0) return;
for (const [id, name] of pending.entries()) {
originalAppend(makeMissingToolResult({ toolCallId: id, toolName: name }));
}
pending.clear();
};
const guardedAppend = (message: AgentMessage) => {
const role = (message as { role?: unknown }).role;
if (role === "toolResult") {
const id = extractToolResultId(
message as Extract<AgentMessage, { role: "toolResult" }>,
);
if (id) pending.delete(id);
return originalAppend(message as never);
}
const toolCalls =
role === "assistant"
? extractAssistantToolCalls(
message as Extract<AgentMessage, { role: "assistant" }>,
)
: [];
// If previous tool calls are still pending, flush before non-tool results.
if (pending.size > 0 && (toolCalls.length === 0 || role !== "assistant")) {
flushPendingToolResults();
}
// If new tool calls arrive while older ones are pending, flush the old ones first.
if (pending.size > 0 && toolCalls.length > 0) {
flushPendingToolResults();
}
const result = originalAppend(message as never);
if (toolCalls.length > 0) {
for (const call of toolCalls) {
pending.set(call.id, call.name);
}
}
return result;
};
// Monkey-patch appendMessage with our guarded version.
sessionManager.appendMessage =
guardedAppend as SessionManager["appendMessage"];
return {
flushPendingToolResults,
getPendingIds: () => Array.from(pending.keys()),
};
}
+64 -8
View File
@@ -60,9 +60,25 @@ function makeMissingToolResult(params: {
} as Extract<AgentMessage, { role: "toolResult" }>;
}
export { makeMissingToolResult };
export function sanitizeToolUseResultPairing(
messages: AgentMessage[],
): AgentMessage[] {
return repairToolUseResultPairing(messages).messages;
}
export type ToolUseRepairReport = {
messages: AgentMessage[];
added: Array<Extract<AgentMessage, { role: "toolResult" }>>;
droppedDuplicateCount: number;
droppedOrphanCount: number;
moved: boolean;
};
export function repairToolUseResultPairing(
messages: AgentMessage[],
): ToolUseRepairReport {
// Anthropic (and Cloud Code Assist) reject transcripts where assistant tool calls are not
// immediately followed by matching tool results. Session files can end up with results
// displaced (e.g. after user turns) or duplicated. Repair by:
@@ -70,13 +86,22 @@ export function sanitizeToolUseResultPairing(
// - inserting synthetic error toolResults for missing ids
// - dropping duplicate toolResults for the same id (anywhere in the transcript)
const out: AgentMessage[] = [];
const added: Array<Extract<AgentMessage, { role: "toolResult" }>> = [];
const seenToolResultIds = new Set<string>();
let droppedDuplicateCount = 0;
let droppedOrphanCount = 0;
let moved = false;
let changed = false;
const pushToolResult = (
msg: Extract<AgentMessage, { role: "toolResult" }>,
) => {
const id = extractToolResultId(msg);
if (id && seenToolResultIds.has(id)) return;
if (id && seenToolResultIds.has(id)) {
droppedDuplicateCount += 1;
changed = true;
return;
}
if (id) seenToolResultIds.add(id);
out.push(msg);
};
@@ -93,7 +118,12 @@ export function sanitizeToolUseResultPairing(
// Tool results must only appear directly after the matching assistant tool call turn.
// Any "free-floating" toolResult entries in session history can make strict providers
// (Anthropic-compatible APIs, MiniMax, Cloud Code Assist) reject the entire request.
if (role !== "toolResult") out.push(msg);
if (role !== "toolResult") {
out.push(msg);
} else {
droppedOrphanCount += 1;
changed = true;
}
continue;
}
@@ -131,6 +161,8 @@ export function sanitizeToolUseResultPairing(
const id = extractToolResultId(toolResult);
if (id && toolCallIds.has(id)) {
if (seenToolResultIds.has(id)) {
droppedDuplicateCount += 1;
changed = true;
continue;
}
if (!spanResultsById.has(id)) {
@@ -141,17 +173,34 @@ export function sanitizeToolUseResultPairing(
}
// Drop tool results that don't match the current assistant tool calls.
if (nextRole !== "toolResult") remainder.push(next);
if (nextRole !== "toolResult") {
remainder.push(next);
} else {
droppedOrphanCount += 1;
changed = true;
}
}
out.push(msg);
if (spanResultsById.size > 0 && remainder.length > 0) {
moved = true;
changed = true;
}
for (const call of toolCalls) {
const existing = spanResultsById.get(call.id);
pushToolResult(
existing ??
makeMissingToolResult({ toolCallId: call.id, toolName: call.name }),
);
if (existing) {
pushToolResult(existing);
} else {
const missing = makeMissingToolResult({
toolCallId: call.id,
toolName: call.name,
});
added.push(missing);
changed = true;
pushToolResult(missing);
}
}
for (const rem of remainder) {
@@ -164,5 +213,12 @@ export function sanitizeToolUseResultPairing(
i = j - 1;
}
return out;
const changedOrMoved = changed || moved;
return {
messages: changedOrMoved ? out : messages,
added,
droppedDuplicateCount,
droppedOrphanCount,
moved: changedOrMoved,
};
}
+10
View File
@@ -222,6 +222,16 @@
"title": "Session Status",
"detailKeys": ["sessionKey", "model"]
},
"memory_search": {
"emoji": "🧠",
"title": "Memory Search",
"detailKeys": ["query"]
},
"memory_get": {
"emoji": "📓",
"title": "Memory Get",
"detailKeys": ["path", "from", "lines"]
},
"whatsapp_login": {
"emoji": "🟢",
"title": "WhatsApp Login",
+4 -1
View File
@@ -125,6 +125,7 @@ const BrowserToolSchema = Type.Object({
compact: Type.Optional(Type.Boolean()),
depth: Type.Optional(Type.Number()),
selector: Type.Optional(Type.String()),
frame: Type.Optional(Type.String()),
fullPage: Type.Optional(Type.Boolean()),
ref: Type.Optional(Type.String()),
element: Type.Optional(Type.String()),
@@ -354,16 +355,18 @@ export function createBrowserTool(opts?: {
typeof params.selector === "string"
? params.selector.trim()
: undefined;
const frame =
typeof params.frame === "string" ? params.frame.trim() : undefined;
const snapshot = await browserSnapshot(baseUrl, {
format,
targetId,
limit,
...(resolvedMaxChars ? { maxChars: resolvedMaxChars } : {}),
...(resolvedMaxChars ? { maxChars: resolvedMaxChars } : {}),
interactive,
compact,
depth,
selector,
frame,
profile,
});
if (snapshot.format === "ai") {
+408
View File
@@ -0,0 +1,408 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { ClawdbotConfig } from "../../config/config.js";
import {
__testing,
createImageTool,
resolveImageModelConfigForTool,
} from "./image-tool.js";
async function writeAuthProfiles(agentDir: string, profiles: unknown) {
await fs.mkdir(agentDir, { recursive: true });
await fs.writeFile(
path.join(agentDir, "auth-profiles.json"),
`${JSON.stringify(profiles, null, 2)}\n`,
"utf8",
);
}
describe("image tool implicit imageModel config", () => {
const priorFetch = global.fetch;
beforeEach(() => {
vi.stubEnv("OPENAI_API_KEY", "");
vi.stubEnv("ANTHROPIC_API_KEY", "");
vi.stubEnv("ANTHROPIC_OAUTH_TOKEN", "");
vi.stubEnv("MINIMAX_API_KEY", "");
// Avoid implicit Copilot provider discovery hitting the network in tests.
vi.stubEnv("COPILOT_GITHUB_TOKEN", "");
vi.stubEnv("GH_TOKEN", "");
vi.stubEnv("GITHUB_TOKEN", "");
});
afterEach(() => {
vi.unstubAllEnvs();
// @ts-expect-error global fetch cleanup
global.fetch = priorFetch;
});
it("stays disabled without auth when no pairing is possible", async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-"),
);
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "openai/gpt-5.2" } } },
};
expect(resolveImageModelConfigForTool({ cfg, agentDir })).toBeNull();
expect(createImageTool({ config: cfg, agentDir })).toBeNull();
});
it("pairs minimax primary with MiniMax-VL-01 (and fallbacks) when auth exists", async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-"),
);
vi.stubEnv("MINIMAX_API_KEY", "minimax-test");
vi.stubEnv("OPENAI_API_KEY", "openai-test");
vi.stubEnv("ANTHROPIC_API_KEY", "anthropic-test");
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "minimax/MiniMax-M2.1" } } },
};
expect(resolveImageModelConfigForTool({ cfg, agentDir })).toEqual({
primary: "minimax/MiniMax-VL-01",
fallbacks: ["openai/gpt-5-mini", "anthropic/claude-opus-4-5"],
});
expect(createImageTool({ config: cfg, agentDir })).not.toBeNull();
});
it("pairs a custom provider when it declares an image-capable model", async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-"),
);
await writeAuthProfiles(agentDir, {
version: 1,
profiles: {
"acme:default": { type: "api_key", provider: "acme", key: "sk-test" },
},
});
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "acme/text-1" } } },
models: {
providers: {
acme: {
models: [
{ id: "text-1", input: ["text"] },
{ id: "vision-1", input: ["text", "image"] },
],
},
},
},
};
expect(resolveImageModelConfigForTool({ cfg, agentDir })).toEqual({
primary: "acme/vision-1",
});
expect(createImageTool({ config: cfg, agentDir })).not.toBeNull();
});
it("prefers explicit agents.defaults.imageModel", async () => {
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-"),
);
const cfg: ClawdbotConfig = {
agents: {
defaults: {
model: { primary: "minimax/MiniMax-M2.1" },
imageModel: { primary: "openai/gpt-5-mini" },
},
},
};
expect(resolveImageModelConfigForTool({ cfg, agentDir })).toEqual({
primary: "openai/gpt-5-mini",
});
});
it("sandboxes image paths like the read tool", async () => {
const stateDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-sandbox-"),
);
const agentDir = path.join(stateDir, "agent");
const sandboxRoot = path.join(stateDir, "sandbox");
await fs.mkdir(agentDir, { recursive: true });
await fs.mkdir(sandboxRoot, { recursive: true });
await fs.writeFile(path.join(sandboxRoot, "img.png"), "fake", "utf8");
vi.stubEnv("OPENAI_API_KEY", "openai-test");
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "minimax/MiniMax-M2.1" } } },
};
const tool = createImageTool({ config: cfg, agentDir, sandboxRoot });
expect(tool).not.toBeNull();
if (!tool) throw new Error("expected image tool");
await expect(
tool.execute("t1", { image: "https://example.com/a.png" }),
).rejects.toThrow(/Sandboxed image tool does not allow remote URLs/i);
await expect(
tool.execute("t2", { image: "../escape.png" }),
).rejects.toThrow(/escapes sandbox root/i);
});
it("rewrites inbound absolute paths into sandbox media/inbound", async () => {
const stateDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-image-sandbox-"),
);
const agentDir = path.join(stateDir, "agent");
const sandboxRoot = path.join(stateDir, "sandbox");
await fs.mkdir(agentDir, { recursive: true });
await fs.mkdir(path.join(sandboxRoot, "media", "inbound"), {
recursive: true,
});
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
await fs.writeFile(
path.join(sandboxRoot, "media", "inbound", "photo.png"),
Buffer.from(pngB64, "base64"),
);
const fetch = vi.fn().mockResolvedValue({
ok: true,
status: 200,
statusText: "OK",
headers: new Headers(),
json: async () => ({
content: "ok",
base_resp: { status_code: 0, status_msg: "" },
}),
});
// @ts-expect-error partial global
global.fetch = fetch;
vi.stubEnv("MINIMAX_API_KEY", "minimax-test");
const cfg: ClawdbotConfig = {
agents: {
defaults: {
model: { primary: "minimax/MiniMax-M2.1" },
imageModel: { primary: "minimax/MiniMax-VL-01" },
},
},
};
const tool = createImageTool({ config: cfg, agentDir, sandboxRoot });
expect(tool).not.toBeNull();
if (!tool) throw new Error("expected image tool");
const res = await tool.execute("t1", {
prompt: "Describe the image.",
image: "@/Users/steipete/.clawdbot/media/inbound/photo.png",
});
expect(fetch).toHaveBeenCalledTimes(1);
expect((res.details as { rewrittenFrom?: string }).rewrittenFrom).toContain(
"photo.png",
);
});
});
describe("image tool data URL support", () => {
it("decodes base64 image data URLs", () => {
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const out = __testing.decodeDataUrl(`data:image/png;base64,${pngB64}`);
expect(out.kind).toBe("image");
expect(out.mimeType).toBe("image/png");
expect(out.buffer.length).toBeGreaterThan(0);
});
it("rejects non-image data URLs", () => {
expect(() =>
__testing.decodeDataUrl("data:text/plain;base64,SGVsbG8="),
).toThrow(/Unsupported data URL type/i);
});
});
describe("image tool MiniMax VLM routing", () => {
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const priorFetch = global.fetch;
beforeEach(() => {
vi.stubEnv("MINIMAX_API_KEY", "");
vi.stubEnv("COPILOT_GITHUB_TOKEN", "");
vi.stubEnv("GH_TOKEN", "");
vi.stubEnv("GITHUB_TOKEN", "");
});
afterEach(() => {
vi.unstubAllEnvs();
// @ts-expect-error global fetch cleanup
global.fetch = priorFetch;
});
it("calls /v1/coding_plan/vlm for minimax image models", async () => {
const fetch = vi.fn().mockResolvedValue({
ok: true,
status: 200,
statusText: "OK",
headers: new Headers(),
json: async () => ({
content: "ok",
base_resp: { status_code: 0, status_msg: "" },
}),
});
// @ts-expect-error partial global
global.fetch = fetch;
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-minimax-vlm-"),
);
vi.stubEnv("MINIMAX_API_KEY", "minimax-test");
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "minimax/MiniMax-M2.1" } } },
};
const tool = createImageTool({ config: cfg, agentDir });
expect(tool).not.toBeNull();
if (!tool) throw new Error("expected image tool");
const res = await tool.execute("t1", {
prompt: "Describe the image.",
image: `data:image/png;base64,${pngB64}`,
});
expect(fetch).toHaveBeenCalledTimes(1);
const [url, init] = fetch.mock.calls[0];
expect(String(url)).toBe("https://api.minimax.io/v1/coding_plan/vlm");
expect(init?.method).toBe("POST");
expect(
String((init?.headers as Record<string, string>)?.Authorization),
).toBe("Bearer minimax-test");
expect(String(init?.body)).toContain('"prompt":"Describe the image."');
expect(String(init?.body)).toContain('"image_url":"data:image/png;base64,');
const text = res.content?.find((b) => b.type === "text")?.text ?? "";
expect(text).toBe("ok");
});
it("surfaces MiniMax API errors from /v1/coding_plan/vlm", async () => {
const fetch = vi.fn().mockResolvedValue({
ok: true,
status: 200,
statusText: "OK",
headers: new Headers(),
json: async () => ({
content: "",
base_resp: { status_code: 1004, status_msg: "bad key" },
}),
});
// @ts-expect-error partial global
global.fetch = fetch;
const agentDir = await fs.mkdtemp(
path.join(os.tmpdir(), "clawdbot-minimax-vlm-"),
);
vi.stubEnv("MINIMAX_API_KEY", "minimax-test");
const cfg: ClawdbotConfig = {
agents: { defaults: { model: { primary: "minimax/MiniMax-M2.1" } } },
};
const tool = createImageTool({ config: cfg, agentDir });
expect(tool).not.toBeNull();
if (!tool) throw new Error("expected image tool");
await expect(
tool.execute("t1", {
prompt: "Describe the image.",
image: `data:image/png;base64,${pngB64}`,
}),
).rejects.toThrow(/MiniMax VLM API error/i);
});
});
describe("image tool response validation", () => {
it("rejects image-model responses with no final text", () => {
expect(() =>
__testing.coerceImageAssistantText({
provider: "openai",
model: "gpt-5-mini",
message: {
role: "assistant",
api: "openai-responses",
provider: "openai",
model: "gpt-5-mini",
stopReason: "stop",
timestamp: Date.now(),
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
total: 0,
},
},
content: [{ type: "thinking", thinking: "hmm" }],
},
}),
).toThrow(/returned no text/i);
});
it("surfaces provider errors from image-model responses", () => {
expect(() =>
__testing.coerceImageAssistantText({
provider: "openai",
model: "gpt-5-mini",
message: {
role: "assistant",
api: "openai-responses",
provider: "openai",
model: "gpt-5-mini",
stopReason: "error",
errorMessage: "boom",
timestamp: Date.now(),
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
total: 0,
},
},
content: [],
},
}),
).toThrow(/boom/i);
});
it("returns trimmed text from image-model responses", () => {
const text = __testing.coerceImageAssistantText({
provider: "anthropic",
model: "claude-opus-4-5",
message: {
role: "assistant",
api: "anthropic-messages",
provider: "anthropic",
model: "claude-opus-4-5",
stopReason: "stop",
timestamp: Date.now(),
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
total: 0,
},
},
content: [{ type: "text", text: " hello " }],
},
});
expect(text).toBe("hello");
});
});
+347 -23
View File
@@ -1,3 +1,6 @@
import fs from "node:fs/promises";
import path from "node:path";
import {
type Api,
type AssistantMessage,
@@ -14,15 +17,76 @@ import { Type } from "@sinclair/typebox";
import type { ClawdbotConfig } from "../../config/config.js";
import { resolveUserPath } from "../../utils.js";
import { loadWebMedia } from "../../web/media.js";
import { getApiKeyForModel } from "../model-auth.js";
import {
ensureAuthProfileStore,
listProfilesForProvider,
} from "../auth-profiles.js";
import { DEFAULT_MODEL, DEFAULT_PROVIDER } from "../defaults.js";
import { minimaxUnderstandImage } from "../minimax-vlm.js";
import { getApiKeyForModel, resolveEnvApiKey } from "../model-auth.js";
import { runWithImageModelFallback } from "../model-fallback.js";
import { parseModelRef } from "../model-selection.js";
import { ensureClawdbotModelsJson } from "../models-config.js";
import { extractAssistantText } from "../pi-embedded-utils.js";
import { assertSandboxPath } from "../sandbox-paths.js";
import type { AnyAgentTool } from "./common.js";
const DEFAULT_PROMPT = "Describe the image.";
function ensureImageToolConfigured(cfg?: ClawdbotConfig): boolean {
type ImageModelConfig = { primary?: string; fallbacks?: string[] };
function decodeDataUrl(dataUrl: string): {
buffer: Buffer;
mimeType: string;
kind: "image";
} {
const trimmed = dataUrl.trim();
const match = /^data:([^;,]+);base64,([a-z0-9+/=\r\n]+)$/i.exec(trimmed);
if (!match) throw new Error("Invalid data URL (expected base64 data: URL).");
const mimeType = (match[1] ?? "").trim().toLowerCase();
if (!mimeType.startsWith("image/")) {
throw new Error(`Unsupported data URL type: ${mimeType || "unknown"}`);
}
const b64 = (match[2] ?? "").trim();
const buffer = Buffer.from(b64, "base64");
if (buffer.length === 0) {
throw new Error("Invalid data URL: empty payload.");
}
return { buffer, mimeType, kind: "image" };
}
export const __testing = {
decodeDataUrl,
coerceImageAssistantText,
} as const;
function coerceImageAssistantText(params: {
message: AssistantMessage;
provider: string;
model: string;
}): string {
const stop = params.message.stopReason;
const errorMessage = params.message.errorMessage?.trim();
if (stop === "error" || stop === "aborted") {
throw new Error(
errorMessage
? `Image model failed (${params.provider}/${params.model}): ${errorMessage}`
: `Image model failed (${params.provider}/${params.model})`,
);
}
if (errorMessage) {
throw new Error(
`Image model failed (${params.provider}/${params.model}): ${errorMessage}`,
);
}
const text = extractAssistantText(params.message);
if (text.trim()) return text.trim();
throw new Error(
`Image model returned no text (${params.provider}/${params.model}).`,
);
}
function coerceImageModelConfig(cfg?: ClawdbotConfig): ImageModelConfig {
const imageModel = cfg?.agents?.defaults?.imageModel as
| { primary?: string; fallbacks?: string[] }
| string
@@ -31,7 +95,150 @@ function ensureImageToolConfigured(cfg?: ClawdbotConfig): boolean {
typeof imageModel === "string" ? imageModel.trim() : imageModel?.primary;
const fallbacks =
typeof imageModel === "object" ? (imageModel?.fallbacks ?? []) : [];
return Boolean(primary?.trim() || fallbacks.length > 0);
return {
...(primary?.trim() ? { primary: primary.trim() } : {}),
...(fallbacks.length > 0 ? { fallbacks } : {}),
};
}
function resolveProviderVisionModelFromConfig(params: {
cfg?: ClawdbotConfig;
provider: string;
}): string | null {
const providerCfg = params.cfg?.models?.providers?.[
params.provider
] as unknown as
| { models?: Array<{ id?: string; input?: string[] }> }
| undefined;
const models = providerCfg?.models ?? [];
const preferMinimaxVl =
params.provider === "minimax"
? models.find(
(m) =>
(m?.id ?? "").trim() === "MiniMax-VL-01" &&
Array.isArray(m?.input) &&
m.input.includes("image"),
)
: null;
const picked =
preferMinimaxVl ??
models.find(
(m) => Boolean((m?.id ?? "").trim()) && m.input?.includes("image"),
);
const id = (picked?.id ?? "").trim();
return id ? `${params.provider}/${id}` : null;
}
function resolveDefaultModelRef(cfg?: ClawdbotConfig): {
provider: string;
model: string;
} {
const modelConfig = cfg?.agents?.defaults?.model as
| { primary?: string }
| string
| undefined;
const raw =
typeof modelConfig === "string"
? modelConfig.trim()
: modelConfig?.primary?.trim();
const parsed =
parseModelRef(raw ?? "", DEFAULT_PROVIDER) ??
({ provider: DEFAULT_PROVIDER, model: DEFAULT_MODEL } as const);
return { provider: parsed.provider, model: parsed.model };
}
function hasAuthForProvider(params: {
provider: string;
agentDir: string;
}): boolean {
if (resolveEnvApiKey(params.provider)?.apiKey) return true;
const store = ensureAuthProfileStore(params.agentDir, {
allowKeychainPrompt: false,
});
return listProfilesForProvider(store, params.provider).length > 0;
}
/**
* Resolve the effective image model config for the `image` tool.
*
* - Prefer explicit config (`agents.defaults.imageModel`).
* - Otherwise, try to "pair" the primary model with an image-capable model:
* - same provider (best effort)
* - fall back to OpenAI/Anthropic when available
*/
export function resolveImageModelConfigForTool(params: {
cfg?: ClawdbotConfig;
agentDir: string;
}): ImageModelConfig | null {
const explicit = coerceImageModelConfig(params.cfg);
if (explicit.primary?.trim() || (explicit.fallbacks?.length ?? 0) > 0) {
return explicit;
}
const primary = resolveDefaultModelRef(params.cfg);
const openaiOk = hasAuthForProvider({
provider: "openai",
agentDir: params.agentDir,
});
const anthropicOk = hasAuthForProvider({
provider: "anthropic",
agentDir: params.agentDir,
});
const fallbacks: string[] = [];
const addFallback = (modelRef: string | null) => {
const ref = (modelRef ?? "").trim();
if (!ref) return;
if (fallbacks.includes(ref)) return;
fallbacks.push(ref);
};
const providerVisionFromConfig = resolveProviderVisionModelFromConfig({
cfg: params.cfg,
provider: primary.provider,
});
const providerOk = hasAuthForProvider({
provider: primary.provider,
agentDir: params.agentDir,
});
let preferred: string | null = null;
// MiniMax users: always try the canonical vision model first when auth exists.
if (primary.provider === "minimax" && providerOk) {
preferred = "minimax/MiniMax-VL-01";
} else if (providerOk && providerVisionFromConfig) {
preferred = providerVisionFromConfig;
} else if (primary.provider === "openai" && openaiOk) {
preferred = "openai/gpt-5-mini";
} else if (primary.provider === "anthropic" && anthropicOk) {
preferred = "anthropic/claude-opus-4-5";
}
if (preferred?.trim()) {
if (openaiOk) addFallback("openai/gpt-5-mini");
if (anthropicOk) addFallback("anthropic/claude-opus-4-5");
// Don't duplicate primary in fallbacks.
const pruned = fallbacks.filter((ref) => ref !== preferred);
return {
primary: preferred,
...(pruned.length > 0 ? { fallbacks: pruned } : {}),
};
}
// Cross-provider fallback when we can't pair with the primary provider.
if (openaiOk) {
if (anthropicOk) addFallback("anthropic/claude-opus-4-5");
return {
primary: "openai/gpt-5-mini",
...(fallbacks.length ? { fallbacks } : {}),
};
}
if (anthropicOk) {
return { primary: "anthropic/claude-opus-4-5" };
}
return null;
}
function pickMaxBytes(
@@ -75,20 +282,71 @@ function buildImageContext(
};
}
async function resolveSandboxedImagePath(params: {
sandboxRoot: string;
imagePath: string;
}): Promise<{ resolved: string; rewrittenFrom?: string }> {
const normalize = (p: string) =>
p.startsWith("file://") ? p.slice("file://".length) : p;
const filePath = normalize(params.imagePath);
try {
const out = await assertSandboxPath({
filePath,
cwd: params.sandboxRoot,
root: params.sandboxRoot,
});
return { resolved: out.resolved };
} catch (err) {
const name = path.basename(filePath);
const candidateRel = path.join("media", "inbound", name);
const candidateAbs = path.join(params.sandboxRoot, candidateRel);
try {
await fs.stat(candidateAbs);
} catch {
throw err;
}
const out = await assertSandboxPath({
filePath: candidateRel,
cwd: params.sandboxRoot,
root: params.sandboxRoot,
});
return { resolved: out.resolved, rewrittenFrom: filePath };
}
}
async function runImagePrompt(params: {
cfg?: ClawdbotConfig;
agentDir: string;
imageModelConfig: ImageModelConfig;
modelOverride?: string;
prompt: string;
base64: string;
mimeType: string;
}): Promise<{ text: string; provider: string; model: string }> {
await ensureClawdbotModelsJson(params.cfg, params.agentDir);
}): Promise<{
text: string;
provider: string;
model: string;
attempts: Array<{ provider: string; model: string; error: string }>;
}> {
const effectiveCfg: ClawdbotConfig | undefined = params.cfg
? {
...params.cfg,
agents: {
...params.cfg.agents,
defaults: {
...params.cfg.agents?.defaults,
imageModel: params.imageModelConfig,
},
},
}
: undefined;
await ensureClawdbotModelsJson(effectiveCfg, params.agentDir);
const authStorage = discoverAuthStorage(params.agentDir);
const modelRegistry = discoverModels(authStorage, params.agentDir);
const result = await runWithImageModelFallback({
cfg: params.cfg,
cfg: effectiveCfg,
modelOverride: params.modelOverride,
run: async (provider, modelId) => {
const model = modelRegistry.find(provider, modelId) as Model<Api> | null;
@@ -102,10 +360,22 @@ async function runImagePrompt(params: {
}
const apiKeyInfo = await getApiKeyForModel({
model,
cfg: params.cfg,
cfg: effectiveCfg,
agentDir: params.agentDir,
});
authStorage.setRuntimeApiKey(model.provider, apiKeyInfo.apiKey);
const imageDataUrl = `data:${params.mimeType};base64,${params.base64}`;
if (model.provider === "minimax") {
const text = await minimaxUnderstandImage({
apiKey: apiKeyInfo.apiKey,
prompt: params.prompt,
imageDataUrl,
modelBaseUrl: model.baseUrl,
});
return { text, provider: model.provider, model: model.id };
}
const context = buildImageContext(
params.prompt,
params.base64,
@@ -114,29 +384,46 @@ async function runImagePrompt(params: {
const message = (await complete(model, context, {
apiKey: apiKeyInfo.apiKey,
maxTokens: 512,
temperature: 0,
})) as AssistantMessage;
return message;
const text = coerceImageAssistantText({
message,
provider: model.provider,
model: model.id,
});
return { text, provider: model.provider, model: model.id };
},
});
const text = extractAssistantText(result.result);
return {
text: text || "(no text returned)",
provider: result.provider,
model: result.model,
text: result.result.text,
provider: result.result.provider,
model: result.result.model,
attempts: result.attempts.map((attempt) => ({
provider: attempt.provider,
model: attempt.model,
error: attempt.error,
})),
};
}
export function createImageTool(options?: {
config?: ClawdbotConfig;
agentDir?: string;
sandboxRoot?: string;
}): AnyAgentTool | null {
if (!ensureImageToolConfigured(options?.config)) return null;
const agentDir = options?.agentDir;
if (!agentDir?.trim()) {
throw new Error("createImageTool requires agentDir when enabled");
const agentDir = options?.agentDir?.trim();
if (!agentDir) {
const explicit = coerceImageModelConfig(options?.config);
if (explicit.primary?.trim() || (explicit.fallbacks?.length ?? 0) > 0) {
throw new Error("createImageTool requires agentDir when enabled");
}
return null;
}
const imageModelConfig = resolveImageModelConfigForTool({
cfg: options?.config,
agentDir,
});
if (!imageModelConfig) return null;
return {
label: "Image",
name: "image",
@@ -153,8 +440,11 @@ export function createImageTool(options?: {
args && typeof args === "object"
? (args as Record<string, unknown>)
: {};
const imageRaw =
const imageRawInput =
typeof record.image === "string" ? record.image.trim() : "";
const imageRaw = imageRawInput.startsWith("@")
? imageRawInput.slice(1).trim()
: imageRawInput;
if (!imageRaw) throw new Error("image required");
const promptRaw =
typeof record.prompt === "string" && record.prompt.trim()
@@ -168,19 +458,49 @@ export function createImageTool(options?: {
typeof record.maxBytesMb === "number" ? record.maxBytesMb : undefined;
const maxBytes = pickMaxBytes(options?.config, maxBytesMb);
const resolvedImage = imageRaw.startsWith("~")
? resolveUserPath(imageRaw)
: imageRaw;
const media = await loadWebMedia(resolvedImage, maxBytes);
const sandboxRoot = options?.sandboxRoot?.trim();
const isUrl = /^https?:\/\//i.test(imageRaw);
if (sandboxRoot && isUrl) {
throw new Error("Sandboxed image tool does not allow remote URLs.");
}
const isDataUrl = /^data:/i.test(imageRaw);
const resolvedImage = (() => {
if (sandboxRoot) return imageRaw;
if (imageRaw.startsWith("~")) return resolveUserPath(imageRaw);
return imageRaw;
})();
const resolvedPathInfo: { resolved: string; rewrittenFrom?: string } =
isDataUrl
? { resolved: "" }
: sandboxRoot
? await resolveSandboxedImagePath({
sandboxRoot,
imagePath: resolvedImage,
})
: {
resolved: resolvedImage.startsWith("file://")
? resolvedImage.slice("file://".length)
: resolvedImage,
};
const resolvedPath = isDataUrl ? null : resolvedPathInfo.resolved;
const media = isDataUrl
? decodeDataUrl(resolvedImage)
: await loadWebMedia(resolvedPath ?? resolvedImage, maxBytes);
if (media.kind !== "image") {
throw new Error(`Unsupported media type: ${media.kind}`);
}
const mimeType = media.contentType ?? "image/png";
const mimeType =
("contentType" in media && media.contentType) ||
("mimeType" in media && media.mimeType) ||
"image/png";
const base64 = media.buffer.toString("base64");
const result = await runImagePrompt({
cfg: options?.config,
agentDir,
imageModelConfig,
modelOverride,
prompt: promptRaw,
base64,
@@ -191,6 +511,10 @@ export function createImageTool(options?: {
details: {
model: `${result.provider}/${result.model}`,
image: resolvedImage,
...(resolvedPathInfo.rewrittenFrom
? { rewrittenFrom: resolvedPathInfo.rewrittenFrom }
: {}),
attempts: result.attempts,
},
};
},
+101
View File
@@ -0,0 +1,101 @@
import { Type } from "@sinclair/typebox";
import type { ClawdbotConfig } from "../../config/config.js";
import { getMemorySearchManager } from "../../memory/index.js";
import { resolveSessionAgentId } from "../agent-scope.js";
import { resolveMemorySearchConfig } from "../memory-search.js";
import type { AnyAgentTool } from "./common.js";
import { jsonResult, readNumberParam, readStringParam } from "./common.js";
const MemorySearchSchema = Type.Object({
query: Type.String(),
maxResults: Type.Optional(Type.Number()),
minScore: Type.Optional(Type.Number()),
});
const MemoryGetSchema = Type.Object({
path: Type.String(),
from: Type.Optional(Type.Number()),
lines: Type.Optional(Type.Number()),
});
export function createMemorySearchTool(options: {
config?: ClawdbotConfig;
agentSessionKey?: string;
}): AnyAgentTool | null {
const cfg = options.config;
if (!cfg) return null;
const agentId = resolveSessionAgentId({
sessionKey: options.agentSessionKey,
config: cfg,
});
if (!resolveMemorySearchConfig(cfg, agentId)) return null;
return {
label: "Memory Search",
name: "memory_search",
description:
"Search agent memory files (MEMORY.md + memory/*.md) using semantic vectors.",
parameters: MemorySearchSchema,
execute: async (_toolCallId, params) => {
const query = readStringParam(params, "query", { required: true });
const maxResults = readNumberParam(params, "maxResults");
const minScore = readNumberParam(params, "minScore");
const { manager, error } = await getMemorySearchManager({
cfg,
agentId,
});
if (!manager) {
return jsonResult({ results: [], disabled: true, error });
}
const results = await manager.search(query, {
maxResults,
minScore,
sessionKey: options.agentSessionKey,
});
const status = manager.status();
return jsonResult({
results,
provider: status.provider,
model: status.model,
fallback: status.fallback,
});
},
};
}
export function createMemoryGetTool(options: {
config?: ClawdbotConfig;
agentSessionKey?: string;
}): AnyAgentTool | null {
const cfg = options.config;
if (!cfg) return null;
const agentId = resolveSessionAgentId({
sessionKey: options.agentSessionKey,
config: cfg,
});
if (!resolveMemorySearchConfig(cfg, agentId)) return null;
return {
label: "Memory Get",
name: "memory_get",
description: "Read a memory file by path (workspace-relative).",
parameters: MemoryGetSchema,
execute: async (_toolCallId, params) => {
const relPath = readStringParam(params, "path", { required: true });
const from = readNumberParam(params, "from", { integer: true });
const lines = readNumberParam(params, "lines", { integer: true });
const { manager, error } = await getMemorySearchManager({
cfg,
agentId,
});
if (!manager) {
return jsonResult({ path: relPath, text: "", disabled: true, error });
}
const result = await manager.readFile({
relPath,
from: from ?? undefined,
lines: lines ?? undefined,
});
return jsonResult(result);
},
};
}
+20 -4
View File
@@ -35,6 +35,17 @@ const SessionsSpawnToolSchema = Type.Object({
),
});
function normalizeModelSelection(value: unknown): string | undefined {
if (typeof value === "string") {
const trimmed = value.trim();
return trimmed || undefined;
}
if (!value || typeof value !== "object") return undefined;
const primary = (value as { primary?: unknown }).primary;
if (typeof primary === "string" && primary.trim()) return primary.trim();
return undefined;
}
export function createSessionsSpawnTool(opts?: {
agentSessionKey?: string;
agentProvider?: GatewayMessageProvider;
@@ -51,7 +62,7 @@ export function createSessionsSpawnTool(opts?: {
const task = readStringParam(params, "task", { required: true });
const label = typeof params.label === "string" ? params.label.trim() : "";
const requestedAgentId = readStringParam(params, "agentId");
const model = readStringParam(params, "model");
const modelOverride = readStringParam(params, "model");
const cleanup =
params.cleanup === "keep" || params.cleanup === "delete"
? (params.cleanup as "keep" | "delete")
@@ -129,11 +140,16 @@ export function createSessionsSpawnTool(opts?: {
}
const childSessionKey = `agent:${targetAgentId}:subagent:${crypto.randomUUID()}`;
const shouldPatchSpawnedBy = opts?.sandboxed === true;
if (model) {
const targetAgentConfig = resolveAgentConfig(cfg, targetAgentId);
const resolvedModel =
normalizeModelSelection(modelOverride) ??
normalizeModelSelection(targetAgentConfig?.subagents?.model) ??
normalizeModelSelection(cfg.agents?.defaults?.subagents?.model);
if (resolvedModel) {
try {
await callGateway({
method: "sessions.patch",
params: { key: childSessionKey, model },
params: { key: childSessionKey, model: resolvedModel },
timeoutMs: 10_000,
});
modelApplied = true;
@@ -218,7 +234,7 @@ export function createSessionsSpawnTool(opts?: {
status: "accepted",
childSessionKey,
runId: childRunId,
modelApplied: model ? modelApplied : undefined,
modelApplied: resolvedModel ? modelApplied : undefined,
warning: modelWarning,
});
},
Loaded 100 of 194 files, more files were not shown because too many files have changed in this diff. Show more