Compare commits

...
Author SHA1 Message Date
Peter Steinberger 492c37a171 fix: harden message aborts + bluebubbles dm create (#1751) (thanks @tyler6204) 2026-01-25 10:02:41 +00:00
Tyler Yust 738e596c71 fix(bluebubbles): preserve friendly display names in formatTargetDisplay 2026-01-25 09:47:57 +00:00
Tyler Yust bea6b3eb5b fix: prevent cross-context decoration on direct message tool sends
Two fixes:

1. Cross-context decoration (e.g., '[from +19257864429]' prefix) was being
   added to ALL messages sent to a different target, even when the agent
   was just composing a new message via the message tool. This decoration
   should only be applied when forwarding/relaying messages between chats.

   Fix: Added skipCrossContextDecoration flag to ChannelThreadingToolContext.
   The message tool now sets this flag to true, so direct sends don't get
   decorated. The buildCrossContextDecoration function checks this flag
   and returns null when set.

2. Aborted requests were still completing because the abort signal wasn't
   being passed through the message tool execution chain.

   Fix: Added abortSignal propagation from message tool → runMessageAction →
   executeSendAction → sendMessage → deliverOutboundPayloads. Added abort
   checks at key points in the chain to fail fast when aborted.

Files changed:
- src/channels/plugins/types.core.ts: Added skipCrossContextDecoration field
- src/infra/outbound/outbound-policy.ts: Check skip flag before decorating
- src/agents/tools/message-tool.ts: Set skip flag, accept and pass abort signal
- src/infra/outbound/message-action-runner.ts: Pass abort signal through
- src/infra/outbound/outbound-send-service.ts: Check and pass abort signal
- src/infra/outbound/message.ts: Pass abort signal to delivery
2026-01-25 09:47:57 +00:00
Tyler Yust 00e00460c1 fix(bluebubbles): hide internal routing metadata in cross-context markers
When sending cross-context messages via BlueBubbles, the origin marker was
exposing internal chat_guid routing info like '[from bluebubbles:chat_guid:any;-;+19257864429]'.

This adds a formatTargetDisplay() function to the BlueBubbles plugin that:
- Extracts phone numbers from chat_guid formats (iMessage;-;+1234567890 -> +1234567890)
- Normalizes handles for clean display
- Avoids returning raw chat_guid formats containing internal routing metadata

Now cross-context markers show clean identifiers like '[from +19257864429]' instead
of exposing internal routing details to recipients.
2026-01-25 09:47:57 +00:00
Tyler Yust 64c6698ea4 feat(bluebubbles): auto-create new DM chats when sending to unknown phone numbers
When sending to a phone number that doesn't have an existing chat,
instead of failing with 'chatGuid not found', now automatically creates
a new chat using the /api/v1/chat/new endpoint.

- Added createNewChatWithMessage() helper function
- When resolveChatGuidForTarget returns null for a handle target,
  uses the new chat endpoint with addresses array and message
- Includes helpful error message if Private API isn't enabled
- Only applies to handle targets (phone numbers), not group chats
2026-01-25 09:47:57 +00:00
Tyler Yust f997a3f395 fix(bluebubbles): prevent message routing to group chats when targeting phone numbers
When sending a message to a phone number like +12622102921, the
resolveChatGuidForTarget function was finding and returning a GROUP
CHAT containing that phone number instead of a direct DM chat.

The bug was in the participantMatch fallback logic which matched ANY
chat containing the phone number as a participant, including groups.

This fix adds a check to ensure participantMatch only considers DM
chats (identified by ';-;' separator in the chat GUID). Group chats
(identified by ';+;' separator) are now explicitly excluded from
handle-based matching.

If a phone number only exists in a group chat (no direct DM exists),
the function now correctly returns null, which causes the send to
fail with a clear error rather than accidentally messaging a group.

Added test case to verify this behavior.
2026-01-25 09:47:57 +00:00
Tyler Yust 626a37f1ce fix(bluebubbles): prefer DM resolution + hide routing markers 2026-01-25 09:47:57 +00:00
Peter Steinberger 83f92e34af refactor: align voice-call TTS with core config 2026-01-25 09:29:57 +00:00
Vignesh Natarajan 9366cbc7db Docs: add Discord MESSAGE_CONTENT intent step to Railway guide 2026-01-25 01:26:53 -08:00
Peter Steinberger d4f895d8f2 fix: move gateway lock to temp dir 2026-01-25 09:21:46 +00:00
Vignesh Natarajan f08c34a73f Docs: fix Railway deploy URL and add PORT variable 2026-01-25 01:18:12 -08:00
zhixianandzhixian 6a9301c27d feat(tts): support custom OpenAI-compatible TTS endpoints (#1701)
* feat(tts): support custom OpenAI-compatible TTS endpoints

Add OPENAI_TTS_BASE_URL environment variable to allow using self-hosted
or third-party OpenAI-compatible TTS services like Kokoro, LocalAI, or
OpenedAI-Speech.

Changes:
- Add OPENAI_TTS_BASE_URL env var (defaults to OpenAI official API)
- Relax model/voice validation when using custom endpoints
- Add tts-1 and tts-1-hd to the model allowlist

This enables users to:
- Use local TTS for privacy and cost savings
- Use models with better non-English language support (Chinese, Japanese)
- Reduce latency with local inference

Example usage:
  OPENAI_TTS_BASE_URL=http://localhost:8880/v1

Tested with Kokoro-FastAPI.

* fix: strip trailing slashes from OPENAI_TTS_BASE_URL

Address review feedback: normalize the base URL by removing trailing
slashes to prevent double-slash paths like /v1//audio/speech which
cause 404 errors on some OpenAI-compatible servers.

* style: format code with oxfmt

* test: update tests for expanded OpenAI TTS model list

- Accept tts-1 and tts-1-hd as valid models
- Update OPENAI_TTS_MODELS length expectation to 3

---------

Co-authored-by: zhixian <zhixian@bunker.local>
2026-01-25 08:04:20 +00:00
36 changed files with 1217 additions and 113 deletions

No files matched your search

+2
View File
@@ -24,6 +24,7 @@ Docs: https://docs.clawd.bot
- Telegram: add verbose raw-update logging for inbound Telegram updates. (#1597) Thanks @rohannagpal.
### Fixes
- BlueBubbles: route phone-number targets to DMs, avoid leaking routing IDs, and auto-create missing DMs (Private API required). (#1751) Thanks @tyler6204. https://docs.clawd.bot/channels/bluebubbles
- BlueBubbles: keep part-index GUIDs in reply tags when short IDs are missing.
- Web UI: hide internal `message_id` hints in chat bubbles.
- Web UI: show Stop button during active runs, swap back to New session when idle. (#1664) Thanks @ndbroadbent.
@@ -42,6 +43,7 @@ Docs: https://docs.clawd.bot
- Gateway: reduce log noise for late invokes + remote node probes; debounce skills refresh. (#1607) Thanks @petter-b.
- Gateway: clarify Control UI/WebChat auth error hints for missing tokens. (#1690)
- Gateway: listen on IPv6 loopback when bound to 127.0.0.1 so localhost webhooks work.
- Gateway: store lock files in the temp directory to avoid stale locks on persistent volumes. (#1676)
- macOS: default direct-transport `ws://` URLs to port 18789; document `gateway.remote.transport`. (#1603) Thanks @ngutman.
- Voice Call: return stream TwiML for outbound conversation calls on initial Twilio webhook. (#1634)
- Google Chat: tighten email allowlist matching, typing cleanup, media caps, and onboarding/docs/tests. (#1635) Thanks @iHildy.
+1
View File
@@ -213,6 +213,7 @@ Prefer `chat_guid` for stable routing:
- `chat_id:123`
- `chat_identifier:...`
- Direct handles: `+15555550123`, `user@example.com`
- If a direct handle does not have an existing DM chat, Clawdbot will create one via `POST /api/v1/chat/new`. This requires the BlueBubbles Private API to be enabled.
## Security
- Webhook requests are authenticated by comparing `guid`/`password` query params or headers against `channels.bluebubbles.password`. Requests from `localhost` are also accepted.
+16
View File
@@ -67,6 +67,22 @@ Plugins can register:
Plugins run **in‑process** with the Gateway, so treat them as trusted code.
Tool authoring guide: [Plugin agent tools](/plugins/agent-tools).
## Runtime helpers
Plugins can access selected core helpers via `api.runtime`. For telephony TTS:
```ts
const result = await api.runtime.tts.textToSpeechTelephony({
text: "Hello from Clawdbot",
cfg: api.config,
});
```
Notes:
- Uses core `messages.tts` configuration (OpenAI or ElevenLabs).
- Returns PCM audio buffer + sample rate. Plugins must resample/encode for providers.
- Edge TTS is not supported for telephony.
## Discovery & precedence
Clawdbot scans, in order:
+81
View File
@@ -104,6 +104,87 @@ Notes:
- `mock` is a local dev provider (no network calls).
- `skipSignatureVerification` is for local testing only.
## TTS for calls
Voice Call uses the core `messages.tts` configuration (OpenAI or ElevenLabs) for
streaming speech on calls. You can override it under the plugin config with the
**same shape** — it deep‑merges with `messages.tts`.
```json5
{
tts: {
provider: "elevenlabs",
elevenlabs: {
voiceId: "pMsXgVXv3BLzUgSXRplE",
modelId: "eleven_multilingual_v2"
}
}
}
```
Notes:
- **Edge TTS is ignored for voice calls** (telephony audio needs PCM; Edge output is unreliable).
- Core TTS is used when Twilio media streaming is enabled; otherwise calls fall back to provider native voices.
### More examples
Use core TTS only (no override):
```json5
{
messages: {
tts: {
provider: "openai",
openai: { voice: "alloy" }
}
}
}
```
Override to ElevenLabs just for calls (keep core default elsewhere):
```json5
{
plugins: {
entries: {
"voice-call": {
config: {
tts: {
provider: "elevenlabs",
elevenlabs: {
apiKey: "elevenlabs_key",
voiceId: "pMsXgVXv3BLzUgSXRplE",
modelId: "eleven_multilingual_v2"
}
}
}
}
}
}
}
```
Override only the OpenAI model for calls (deep‑merge example):
```json5
{
plugins: {
entries: {
"voice-call": {
config: {
tts: {
openai: {
model: "gpt-4o-mini-tts",
voice: "marin"
}
}
}
}
}
}
}
```
## Inbound calls
Inbound policy defaults to `disabled`. To enable inbound calls, set:
+5 -3
View File
@@ -16,7 +16,7 @@ and you configure everything via the `/setup` web wizard.
## One-click deploy
<a href="https://railway.com/deploy/clawdbot-railway-template" target="_blank" rel="noreferrer">Deploy on Railway</a>
<a href="https://railway.app/new/template?template=https://github.com/vignesh07/clawdbot-railway-template" target="_blank" rel="noreferrer">Deploy on Railway</a>
After deploy, find your public URL in **Railway → your service → Settings → Domains**.
@@ -55,6 +55,7 @@ Attach a volume mounted at:
Set these variables on the service:
- `SETUP_PASSWORD` (required)
- `PORT=8080` (required — must match the port in Public Networking)
- `CLAWDBOT_STATE_DIR=/data/.clawdbot` (recommended)
- `CLAWDBOT_WORKSPACE_DIR=/data/workspace` (recommended)
- `CLAWDBOT_GATEWAY_TOKEN` (recommended; treat as an admin secret)
@@ -82,8 +83,9 @@ If Telegram DMs are set to pairing, the setup wizard can approve the pairing cod
1) Go to https://discord.com/developers/applications
2) **New Application** → choose a name
3) **Bot** → **Add Bot**
4) Copy the **Bot Token** and paste into `/setup`
5) Invite the bot to your server (OAuth2 URL Generator; scopes: `bot`, `applications.commands`)
4) **Enable MESSAGE CONTENT INTENT** under Bot → Privileged Gateway Intents (required or the bot will crash on startup)
5) Copy the **Bot Token** and paste into `/setup`
6) Invite the bot to your server (OAuth2 URL Generator; scopes: `bot`, `applications.commands`)
## Backups & migration
+54
View File
@@ -25,9 +25,11 @@ import { resolveBlueBubblesMessageId } from "./monitor.js";
import { probeBlueBubbles, type BlueBubblesProbe } from "./probe.js";
import { sendMessageBlueBubbles } from "./send.js";
import {
extractHandleFromChatGuid,
looksLikeBlueBubblesTargetId,
normalizeBlueBubblesHandle,
normalizeBlueBubblesMessagingTarget,
parseBlueBubblesTarget,
} from "./targets.js";
import { bluebubblesMessageActions } from "./actions.js";
import { monitorBlueBubblesProvider, resolveWebhookPathFromConfig } from "./monitor.js";
@@ -148,6 +150,58 @@ export const bluebubblesPlugin: ChannelPlugin<ResolvedBlueBubblesAccount> = {
looksLikeId: looksLikeBlueBubblesTargetId,
hint: "<handle|chat_guid:GUID|chat_id:ID|chat_identifier:ID>",
},
formatTargetDisplay: ({ target, display }) => {
const shouldParseDisplay = (value: string): boolean => {
if (looksLikeBlueBubblesTargetId(value)) return true;
return /^(bluebubbles:|chat_guid:|chat_id:|chat_identifier:)/i.test(value);
};
// Helper to extract a clean handle from any BlueBubbles target format
const extractCleanDisplay = (value: string | undefined): string | null => {
const trimmed = value?.trim();
if (!trimmed) return null;
try {
const parsed = parseBlueBubblesTarget(trimmed);
if (parsed.kind === "chat_guid") {
const handle = extractHandleFromChatGuid(parsed.chatGuid);
if (handle) return handle;
}
if (parsed.kind === "handle") {
return normalizeBlueBubblesHandle(parsed.to);
}
} catch {
// Fall through
}
// Strip common prefixes and try raw extraction
const stripped = trimmed
.replace(/^bluebubbles:/i, "")
.replace(/^chat_guid:/i, "")
.replace(/^chat_id:/i, "")
.replace(/^chat_identifier:/i, "");
const handle = extractHandleFromChatGuid(stripped);
if (handle) return handle;
// Don't return raw chat_guid formats - they contain internal routing info
if (stripped.includes(";-;") || stripped.includes(";+;")) return null;
return stripped;
};
// Try to get a clean display from the display parameter first
const trimmedDisplay = display?.trim();
if (trimmedDisplay) {
if (!shouldParseDisplay(trimmedDisplay)) {
return trimmedDisplay;
}
const cleanDisplay = extractCleanDisplay(trimmedDisplay);
if (cleanDisplay) return cleanDisplay;
}
// Fall back to extracting from target
const cleanTarget = extractCleanDisplay(target);
if (cleanTarget) return cleanTarget;
// Last resort: return display or target as-is
return display?.trim() || target?.trim() || "";
},
},
setup: {
resolveAccountId: ({ accountId }) => normalizeAccountId(accountId),
+94 -2
View File
@@ -187,6 +187,47 @@ describe("send", () => {
expect(result).toBe("iMessage;-;+15551234567");
});
it("returns null when handle only exists in group chat (not DM)", async () => {
// This is the critical fix: if a phone number only exists as a participant in a group chat
// (no direct DM chat), we should NOT send to that group. Return null instead.
mockFetch
.mockResolvedValueOnce({
ok: true,
json: () =>
Promise.resolve({
data: [
{
guid: "iMessage;+;group-the-council",
participants: [
{ address: "+12622102921" },
{ address: "+15550001111" },
{ address: "+15550002222" },
],
},
],
}),
})
// Empty second page to stop pagination
.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ data: [] }),
});
const target: BlueBubblesSendTarget = {
kind: "handle",
address: "+12622102921",
service: "imessage",
};
const result = await resolveChatGuidForTarget({
baseUrl: "http://localhost:1234",
password: "test",
target,
});
// Should return null, NOT the group chat GUID
expect(result).toBeNull();
});
it("returns null when chat not found", async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
@@ -344,14 +385,14 @@ describe("send", () => {
).rejects.toThrow("password is required");
});
it("throws when chatGuid cannot be resolved", async () => {
it("throws when chatGuid cannot be resolved for non-handle targets", async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ data: [] }),
});
await expect(
sendMessageBlueBubbles("+15559999999", "Hello", {
sendMessageBlueBubbles("chat_id:999", "Hello", {
serverUrl: "http://localhost:1234",
password: "test",
}),
@@ -398,6 +439,57 @@ describe("send", () => {
expect(body.method).toBeUndefined();
});
it("creates a new chat when handle target is missing", async () => {
mockFetch
.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ data: [] }),
})
.mockResolvedValueOnce({
ok: true,
text: () =>
Promise.resolve(
JSON.stringify({
data: { guid: "new-msg-guid" },
}),
),
});
const result = await sendMessageBlueBubbles("+15550009999", "Hello new chat", {
serverUrl: "http://localhost:1234",
password: "test",
});
expect(result.messageId).toBe("new-msg-guid");
expect(mockFetch).toHaveBeenCalledTimes(2);
const createCall = mockFetch.mock.calls[1];
expect(createCall[0]).toContain("/api/v1/chat/new");
const body = JSON.parse(createCall[1].body);
expect(body.addresses).toEqual(["+15550009999"]);
expect(body.message).toBe("Hello new chat");
});
it("throws when creating a new chat requires Private API", async () => {
mockFetch
.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ data: [] }),
})
.mockResolvedValueOnce({
ok: false,
status: 403,
text: () => Promise.resolve("Private API not enabled"),
});
await expect(
sendMessageBlueBubbles("+15550008888", "Hello", {
serverUrl: "http://localhost:1234",
password: "test",
}),
).rejects.toThrow("Private API must be enabled");
});
it("uses private-api when reply metadata is present", async () => {
mockFetch
.mockResolvedValueOnce({
+71 -5
View File
@@ -257,11 +257,17 @@ export async function resolveChatGuidForTarget(params: {
return guid;
}
if (!participantMatch && guid) {
const participants = extractParticipantAddresses(chat).map((entry) =>
normalizeBlueBubblesHandle(entry),
);
if (participants.includes(normalizedHandle)) {
participantMatch = guid;
// Only consider DM chats (`;-;` separator) as participant matches.
// Group chats (`;+;` separator) should never match when searching by handle/phone.
// This prevents routing "send to +1234567890" to a group chat that contains that number.
const isDmChat = guid.includes(";-;");
if (isDmChat) {
const participants = extractParticipantAddresses(chat).map((entry) =>
normalizeBlueBubblesHandle(entry),
);
if (participants.includes(normalizedHandle)) {
participantMatch = guid;
}
}
}
}
@@ -270,6 +276,55 @@ export async function resolveChatGuidForTarget(params: {
return participantMatch;
}
/**
* Creates a new chat (DM) and optionally sends an initial message.
* Requires Private API to be enabled in BlueBubbles.
*/
async function createNewChatWithMessage(params: {
baseUrl: string;
password: string;
address: string;
message: string;
timeoutMs?: number;
}): Promise<BlueBubblesSendResult> {
const url = buildBlueBubblesApiUrl({
baseUrl: params.baseUrl,
path: "/api/v1/chat/new",
password: params.password,
});
const payload = {
addresses: [params.address],
message: params.message,
};
const res = await blueBubblesFetchWithTimeout(
url,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
},
params.timeoutMs,
);
if (!res.ok) {
const errorText = await res.text();
// Check for Private API not enabled error
if (res.status === 400 || res.status === 403 || errorText.toLowerCase().includes("private api")) {
throw new Error(
`BlueBubbles send failed: Cannot create new chat - Private API must be enabled. Original error: ${errorText || res.status}`,
);
}
throw new Error(`BlueBubbles create chat failed (${res.status}): ${errorText || "unknown"}`);
}
const body = await res.text();
if (!body) return { messageId: "ok" };
try {
const parsed = JSON.parse(body) as unknown;
return { messageId: extractMessageId(parsed) };
} catch {
return { messageId: "ok" };
}
}
export async function sendMessageBlueBubbles(
to: string,
text: string,
@@ -297,6 +352,17 @@ export async function sendMessageBlueBubbles(
target,
});
if (!chatGuid) {
// If target is a phone number/handle and no existing chat found,
// auto-create a new DM chat using the /api/v1/chat/new endpoint
if (target.kind === "handle") {
return createNewChatWithMessage({
baseUrl,
password,
address: target.address,
message: trimmedText,
timeoutMs: opts.timeoutMs,
});
}
throw new Error(
"BlueBubbles send failed: chatGuid not found for target. Use a chat_guid target or ensure the chat exists.",
);
+7
View File
@@ -1,5 +1,12 @@
# Changelog
## 2026.1.24
### Changes
- Breaking: voice-call TTS now uses core `messages.tts` (plugin TTS config deep‑merges with core).
- Telephony TTS supports OpenAI + ElevenLabs; Edge TTS is ignored for calls.
- Removed legacy `tts.model`/`tts.voice`/`tts.instructions` plugin fields.
## 2026.1.23
### Changes
+21
View File
@@ -75,6 +75,27 @@ Notes:
- Twilio/Telnyx/Plivo require a **publicly reachable** webhook URL.
- `mock` is a local dev provider (no network calls).
## TTS for calls
Voice Call uses the core `messages.tts` configuration (OpenAI or ElevenLabs) for
streaming speech on calls. You can override it under the plugin config with the
same shape — overrides deep-merge with `messages.tts`.
```json5
{
tts: {
provider: "openai",
openai: {
voice: "alloy"
}
}
}
```
Notes:
- Edge TTS is ignored for voice calls (telephony audio needs PCM; Edge output is unreliable).
- Core TTS is used when Twilio media streaming is enabled; otherwise calls fall back to provider native voices.
## CLI
```bash
+207 -11
View File
@@ -99,16 +99,39 @@
"label": "Media Stream Path",
"advanced": true
},
"tts.model": {
"label": "TTS Model",
"tts.provider": {
"label": "TTS Provider Override",
"help": "Deep-merges with messages.tts (Edge is ignored for calls).",
"advanced": true
},
"tts.voice": {
"label": "TTS Voice",
"tts.openai.model": {
"label": "OpenAI TTS Model",
"advanced": true
},
"tts.instructions": {
"label": "TTS Instructions",
"tts.openai.voice": {
"label": "OpenAI TTS Voice",
"advanced": true
},
"tts.openai.apiKey": {
"label": "OpenAI API Key",
"sensitive": true,
"advanced": true
},
"tts.elevenlabs.modelId": {
"label": "ElevenLabs Model ID",
"advanced": true
},
"tts.elevenlabs.voiceId": {
"label": "ElevenLabs Voice ID",
"advanced": true
},
"tts.elevenlabs.apiKey": {
"label": "ElevenLabs API Key",
"sensitive": true,
"advanced": true
},
"tts.elevenlabs.baseUrl": {
"label": "ElevenLabs Base URL",
"advanced": true
},
"publicUrl": {
@@ -370,20 +393,193 @@
"type": "object",
"additionalProperties": false,
"properties": {
"auto": {
"type": "string",
"enum": [
"off",
"always",
"inbound",
"tagged"
]
},
"enabled": {
"type": "boolean"
},
"mode": {
"type": "string",
"enum": [
"final",
"all"
]
},
"provider": {
"type": "string",
"enum": [
"openai"
"openai",
"elevenlabs",
"edge"
]
},
"model": {
"summaryModel": {
"type": "string"
},
"voice": {
"modelOverrides": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
},
"allowText": {
"type": "boolean"
},
"allowProvider": {
"type": "boolean"
},
"allowVoice": {
"type": "boolean"
},
"allowModelId": {
"type": "boolean"
},
"allowVoiceSettings": {
"type": "boolean"
},
"allowNormalization": {
"type": "boolean"
},
"allowSeed": {
"type": "boolean"
}
}
},
"elevenlabs": {
"type": "object",
"additionalProperties": false,
"properties": {
"apiKey": {
"type": "string"
},
"baseUrl": {
"type": "string"
},
"voiceId": {
"type": "string"
},
"modelId": {
"type": "string"
},
"seed": {
"type": "integer",
"minimum": 0,
"maximum": 4294967295
},
"applyTextNormalization": {
"type": "string",
"enum": [
"auto",
"on",
"off"
]
},
"languageCode": {
"type": "string"
},
"voiceSettings": {
"type": "object",
"additionalProperties": false,
"properties": {
"stability": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"similarityBoost": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"style": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"useSpeakerBoost": {
"type": "boolean"
},
"speed": {
"type": "number",
"minimum": 0.5,
"maximum": 2
}
}
}
}
},
"openai": {
"type": "object",
"additionalProperties": false,
"properties": {
"apiKey": {
"type": "string"
},
"model": {
"type": "string"
},
"voice": {
"type": "string"
}
}
},
"edge": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
},
"voice": {
"type": "string"
},
"lang": {
"type": "string"
},
"outputFormat": {
"type": "string"
},
"pitch": {
"type": "string"
},
"rate": {
"type": "string"
},
"volume": {
"type": "string"
},
"saveSubtitles": {
"type": "boolean"
},
"proxy": {
"type": "string"
},
"timeoutMs": {
"type": "integer",
"minimum": 1000,
"maximum": 120000
}
}
},
"prefsPath": {
"type": "string"
},
"instructions": {
"type": "string"
"maxTextLength": {
"type": "integer",
"minimum": 1
},
"timeoutMs": {
"type": "integer",
"minimum": 1000,
"maximum": 120000
}
}
},
+21 -3
View File
@@ -74,9 +74,26 @@ const voiceCallConfigSchema = {
},
"streaming.sttModel": { label: "Realtime STT Model", advanced: true },
"streaming.streamPath": { label: "Media Stream Path", advanced: true },
"tts.model": { label: "TTS Model", advanced: true },
"tts.voice": { label: "TTS Voice", advanced: true },
"tts.instructions": { label: "TTS Instructions", advanced: true },
"tts.provider": {
label: "TTS Provider Override",
help: "Deep-merges with messages.tts (Edge is ignored for calls).",
advanced: true,
},
"tts.openai.model": { label: "OpenAI TTS Model", advanced: true },
"tts.openai.voice": { label: "OpenAI TTS Voice", advanced: true },
"tts.openai.apiKey": {
label: "OpenAI API Key",
sensitive: true,
advanced: true,
},
"tts.elevenlabs.modelId": { label: "ElevenLabs Model ID", advanced: true },
"tts.elevenlabs.voiceId": { label: "ElevenLabs Voice ID", advanced: true },
"tts.elevenlabs.apiKey": {
label: "ElevenLabs API Key",
sensitive: true,
advanced: true,
},
"tts.elevenlabs.baseUrl": { label: "ElevenLabs Base URL", advanced: true },
publicUrl: { label: "Public Webhook URL", advanced: true },
skipSignatureVerification: {
label: "Skip Signature Verification",
@@ -161,6 +178,7 @@ const voiceCallPlugin = {
runtimePromise = createVoiceCallRuntime({
config: cfg,
coreConfig: api.config as CoreConfig,
ttsRuntime: api.runtime.tts,
logger: api.logger,
});
}
+73 -22
View File
@@ -82,31 +82,82 @@ export const SttConfigSchema = z
.default({ provider: "openai", model: "whisper-1" });
export type SttConfig = z.infer<typeof SttConfigSchema>;
export const TtsProviderSchema = z.enum(["openai", "elevenlabs", "edge"]);
export const TtsModeSchema = z.enum(["final", "all"]);
export const TtsAutoSchema = z.enum(["off", "always", "inbound", "tagged"]);
export const TtsConfigSchema = z
.object({
/** TTS provider (currently only OpenAI supported) */
provider: z.literal("openai").default("openai"),
/**
* TTS model to use:
* - gpt-4o-mini-tts: newest, supports instructions for tone/style control (recommended)
* - tts-1: lower latency
* - tts-1-hd: higher quality
*/
model: z.string().min(1).default("gpt-4o-mini-tts"),
/**
* Voice ID. For best quality, use marin or cedar.
* All voices: alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar
*/
voice: z.string().min(1).default("coral"),
/**
* Instructions for speech style (only works with gpt-4o-mini-tts).
* Examples: "Speak in a cheerful tone", "Talk like a sympathetic customer service agent"
*/
instructions: z.string().optional(),
auto: TtsAutoSchema.optional(),
enabled: z.boolean().optional(),
mode: TtsModeSchema.optional(),
provider: TtsProviderSchema.optional(),
summaryModel: z.string().optional(),
modelOverrides: z
.object({
enabled: z.boolean().optional(),
allowText: z.boolean().optional(),
allowProvider: z.boolean().optional(),
allowVoice: z.boolean().optional(),
allowModelId: z.boolean().optional(),
allowVoiceSettings: z.boolean().optional(),
allowNormalization: z.boolean().optional(),
allowSeed: z.boolean().optional(),
})
.strict()
.optional(),
elevenlabs: z
.object({
apiKey: z.string().optional(),
baseUrl: z.string().optional(),
voiceId: z.string().optional(),
modelId: z.string().optional(),
seed: z.number().int().min(0).max(4294967295).optional(),
applyTextNormalization: z.enum(["auto", "on", "off"]).optional(),
languageCode: z.string().optional(),
voiceSettings: z
.object({
stability: z.number().min(0).max(1).optional(),
similarityBoost: z.number().min(0).max(1).optional(),
style: z.number().min(0).max(1).optional(),
useSpeakerBoost: z.boolean().optional(),
speed: z.number().min(0.5).max(2).optional(),
})
.strict()
.optional(),
})
.strict()
.optional(),
openai: z
.object({
apiKey: z.string().optional(),
model: z.string().optional(),
voice: z.string().optional(),
})
.strict()
.optional(),
edge: z
.object({
enabled: z.boolean().optional(),
voice: z.string().optional(),
lang: z.string().optional(),
outputFormat: z.string().optional(),
pitch: z.string().optional(),
rate: z.string().optional(),
volume: z.string().optional(),
saveSubtitles: z.boolean().optional(),
proxy: z.string().optional(),
timeoutMs: z.number().int().min(1000).max(120000).optional(),
})
.strict()
.optional(),
prefsPath: z.string().optional(),
maxTextLength: z.number().int().min(1).optional(),
timeoutMs: z.number().int().min(1000).max(120000).optional(),
})
.strict()
.default({ provider: "openai", model: "gpt-4o-mini-tts", voice: "coral" });
export type TtsConfig = z.infer<typeof TtsConfigSchema>;
.optional();
export type VoiceCallTtsConfig = z.infer<typeof TtsConfigSchema>;
// -----------------------------------------------------------------------------
// Webhook Server Configuration
@@ -307,7 +358,7 @@ export const VoiceCallConfigSchema = z
/** STT configuration */
stt: SttConfigSchema,
/** TTS configuration */
/** TTS override (deep-merges with core messages.tts) */
tts: TtsConfigSchema,
/** Store path for call logs */
+6
View File
@@ -2,10 +2,16 @@ import fs from "node:fs";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import type { VoiceCallTtsConfig } from "./config.js";
export type CoreConfig = {
session?: {
store?: string;
};
messages?: {
tts?: VoiceCallTtsConfig;
};
[key: string]: unknown;
};
type CoreAgentDeps = {
+4 -2
View File
@@ -143,7 +143,7 @@ export class CallManager {
// For notify mode with a message, use inline TwiML with <Say>
let inlineTwiml: string | undefined;
if (mode === "notify" && initialMessage) {
const pollyVoice = mapVoiceToPolly(this.config.tts.voice);
const pollyVoice = mapVoiceToPolly(this.config.tts?.openai?.voice);
inlineTwiml = this.generateNotifyTwiml(initialMessage, pollyVoice);
console.log(
`[voice-call] Using inline TwiML for notify mode (voice: ${pollyVoice})`,
@@ -210,11 +210,13 @@ export class CallManager {
this.addTranscriptEntry(call, "bot", text);
// Play TTS
const voice =
this.provider?.name === "twilio" ? this.config.tts?.openai?.voice : undefined;
await this.provider.playTts({
callId,
providerCallId: call.providerCallId,
text,
voice: this.config.tts.voice,
voice,
});
return { success: true };
@@ -68,7 +68,7 @@ export async function initiateCall(
// For notify mode with a message, use inline TwiML with <Say>.
let inlineTwiml: string | undefined;
if (mode === "notify" && initialMessage) {
const pollyVoice = mapVoiceToPolly(ctx.config.tts.voice);
const pollyVoice = mapVoiceToPolly(ctx.config.tts?.openai?.voice);
inlineTwiml = generateNotifyTwiml(initialMessage, pollyVoice);
console.log(`[voice-call] Using inline TwiML for notify mode (voice: ${pollyVoice})`);
}
@@ -120,11 +120,13 @@ export async function speak(
addTranscriptEntry(call, "bot", text);
const voice =
ctx.provider?.name === "twilio" ? ctx.config.tts?.openai?.voice : undefined;
await ctx.provider.playTts({
callId,
providerCallId: call.providerCallId,
text,
voice: ctx.config.tts.voice,
voice,
});
return { success: true };
@@ -244,4 +246,3 @@ export async function endCall(
return { success: false, error: err instanceof Error ? err.message : String(err) };
}
}
+14 -14
View File
@@ -15,9 +15,9 @@ import type {
WebhookVerificationResult,
} from "../types.js";
import { escapeXml, mapVoiceToPolly } from "../voice-mapping.js";
import { chunkAudio } from "../telephony-audio.js";
import type { TelephonyTtsProvider } from "../telephony-tts.js";
import type { VoiceCallProvider } from "./base.js";
import type { OpenAITTSProvider } from "./tts-openai.js";
import { chunkAudio } from "./tts-openai.js";
import { twilioApiRequest } from "./twilio/api.js";
import { verifyTwilioProviderWebhook } from "./twilio/webhook.js";
@@ -53,8 +53,8 @@ export class TwilioProvider implements VoiceCallProvider {
/** Current public webhook URL (set when tunnel starts or from config) */
private currentPublicUrl: string | null = null;
/** Optional OpenAI TTS provider for streaming TTS */
private ttsProvider: OpenAITTSProvider | null = null;
/** Optional telephony TTS provider for streaming TTS */
private ttsProvider: TelephonyTtsProvider | null = null;
/** Optional media stream handler for sending audio */
private mediaStreamHandler: MediaStreamHandler | null = null;
@@ -119,7 +119,7 @@ export class TwilioProvider implements VoiceCallProvider {
return this.currentPublicUrl;
}
setTTSProvider(provider: OpenAITTSProvider): void {
setTTSProvider(provider: TelephonyTtsProvider): void {
this.ttsProvider = provider;
}
@@ -454,13 +454,13 @@ export class TwilioProvider implements VoiceCallProvider {
* Play TTS audio via Twilio.
*
* Two modes:
* 1. OpenAI TTS + Media Streams: If TTS provider and media stream are available,
* generates audio via OpenAI and streams it through WebSocket (preferred).
* 1. Core TTS + Media Streams: If TTS provider and media stream are available,
* generates audio via core TTS and streams it through WebSocket (preferred).
* 2. TwiML <Say>: Falls back to Twilio's native TTS with Polly voices.
* Note: This may not work on all Twilio accounts.
*/
async playTts(input: PlayTtsInput): Promise<void> {
// Try OpenAI TTS via media stream first (if configured)
// Try telephony TTS via media stream first (if configured)
const streamSid = this.callStreamMap.get(input.providerCallId);
if (this.ttsProvider && this.mediaStreamHandler && streamSid) {
try {
@@ -468,7 +468,7 @@ export class TwilioProvider implements VoiceCallProvider {
return;
} catch (err) {
console.warn(
`[voice-call] OpenAI TTS failed, falling back to Twilio <Say>:`,
`[voice-call] Telephony TTS failed, falling back to Twilio <Say>:`,
err instanceof Error ? err.message : err,
);
// Fall through to TwiML <Say> fallback
@@ -484,7 +484,7 @@ export class TwilioProvider implements VoiceCallProvider {
}
console.warn(
"[voice-call] Using TwiML <Say> fallback - OpenAI TTS not configured or media stream not active",
"[voice-call] Using TwiML <Say> fallback - telephony TTS not configured or media stream not active",
);
const pollyVoice = mapVoiceToPolly(input.voice);
@@ -502,8 +502,8 @@ export class TwilioProvider implements VoiceCallProvider {
}
/**
* Play TTS via OpenAI and Twilio Media Streams.
* Generates audio with OpenAI TTS, converts to mu-law, and streams via WebSocket.
* Play TTS via core TTS and Twilio Media Streams.
* Generates audio with core TTS, converts to mu-law, and streams via WebSocket.
* Uses a jitter buffer to smooth out timing variations.
*/
private async playTtsViaStream(
@@ -514,8 +514,8 @@ export class TwilioProvider implements VoiceCallProvider {
throw new Error("TTS provider and media stream handler required");
}
// Generate audio with OpenAI TTS (returns mu-law at 8kHz)
const muLawAudio = await this.ttsProvider.synthesizeForTwilio(text);
// Generate audio with core TTS (returns mu-law at 8kHz)
const muLawAudio = await this.ttsProvider.synthesizeForTelephony(text);
// Stream audio in 20ms chunks (160 bytes at 8kHz mu-law)
const CHUNK_SIZE = 160;
+12 -13
View File
@@ -6,8 +6,9 @@ import type { VoiceCallProvider } from "./providers/base.js";
import { MockProvider } from "./providers/mock.js";
import { PlivoProvider } from "./providers/plivo.js";
import { TelnyxProvider } from "./providers/telnyx.js";
import { OpenAITTSProvider } from "./providers/tts-openai.js";
import { TwilioProvider } from "./providers/twilio.js";
import type { TelephonyTtsRuntime } from "./telephony-tts.js";
import { createTelephonyTtsProvider } from "./telephony-tts.js";
import { startTunnel, type TunnelResult } from "./tunnel.js";
import {
cleanupTailscaleExposure,
@@ -81,9 +82,10 @@ function resolveProvider(config: VoiceCallConfig): VoiceCallProvider {
export async function createVoiceCallRuntime(params: {
config: VoiceCallConfig;
coreConfig: CoreConfig;
ttsRuntime?: TelephonyTtsRuntime;
logger?: Logger;
}): Promise<VoiceCallRuntime> {
const { config, coreConfig, logger } = params;
const { config, coreConfig, ttsRuntime, logger } = params;
const log = logger ?? {
info: console.log,
warn: console.warn,
@@ -149,27 +151,24 @@ export async function createVoiceCallRuntime(params: {
if (provider.name === "twilio" && config.streaming?.enabled) {
const twilioProvider = provider as TwilioProvider;
const openaiApiKey =
config.streaming.openaiApiKey || process.env.OPENAI_API_KEY;
if (openaiApiKey) {
if (ttsRuntime?.textToSpeechTelephony) {
try {
const ttsProvider = new OpenAITTSProvider({
apiKey: openaiApiKey,
voice: config.tts.voice,
model: config.tts.model,
instructions: config.tts.instructions,
const ttsProvider = createTelephonyTtsProvider({
coreConfig,
ttsOverride: config.tts,
runtime: ttsRuntime,
});
twilioProvider.setTTSProvider(ttsProvider);
log.info("[voice-call] OpenAI TTS provider configured");
log.info("[voice-call] Telephony TTS provider configured");
} catch (err) {
log.warn(
`[voice-call] Failed to initialize OpenAI TTS: ${
`[voice-call] Failed to initialize telephony TTS: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
} else {
log.warn("[voice-call] OpenAI TTS key missing; streaming TTS disabled");
log.warn("[voice-call] Telephony TTS unavailable; streaming TTS disabled");
}
const mediaHandler = webhookServer.getMediaStreamHandler();
@@ -0,0 +1,88 @@
const TELEPHONY_SAMPLE_RATE = 8000;
function clamp16(value: number): number {
return Math.max(-32768, Math.min(32767, value));
}
/**
* Resample 16-bit PCM (little-endian mono) to 8kHz using linear interpolation.
*/
export function resamplePcmTo8k(input: Buffer, inputSampleRate: number): Buffer {
if (inputSampleRate === TELEPHONY_SAMPLE_RATE) return input;
const inputSamples = Math.floor(input.length / 2);
if (inputSamples === 0) return Buffer.alloc(0);
const ratio = inputSampleRate / TELEPHONY_SAMPLE_RATE;
const outputSamples = Math.floor(inputSamples / ratio);
const output = Buffer.alloc(outputSamples * 2);
for (let i = 0; i < outputSamples; i++) {
const srcPos = i * ratio;
const srcIndex = Math.floor(srcPos);
const frac = srcPos - srcIndex;
const s0 = input.readInt16LE(srcIndex * 2);
const s1Index = Math.min(srcIndex + 1, inputSamples - 1);
const s1 = input.readInt16LE(s1Index * 2);
const sample = Math.round(s0 + frac * (s1 - s0));
output.writeInt16LE(clamp16(sample), i * 2);
}
return output;
}
/**
* Convert 16-bit PCM to 8-bit mu-law (G.711).
*/
export function pcmToMulaw(pcm: Buffer): Buffer {
const samples = Math.floor(pcm.length / 2);
const mulaw = Buffer.alloc(samples);
for (let i = 0; i < samples; i++) {
const sample = pcm.readInt16LE(i * 2);
mulaw[i] = linearToMulaw(sample);
}
return mulaw;
}
export function convertPcmToMulaw8k(
pcm: Buffer,
inputSampleRate: number,
): Buffer {
const pcm8k = resamplePcmTo8k(pcm, inputSampleRate);
return pcmToMulaw(pcm8k);
}
/**
* Chunk audio buffer into 20ms frames for streaming (8kHz mono mu-law).
*/
export function chunkAudio(
audio: Buffer,
chunkSize = 160,
): Generator<Buffer, void, unknown> {
return (function* () {
for (let i = 0; i < audio.length; i += chunkSize) {
yield audio.subarray(i, Math.min(i + chunkSize, audio.length));
}
})();
}
function linearToMulaw(sample: number): number {
const BIAS = 132;
const CLIP = 32635;
const sign = sample < 0 ? 0x80 : 0;
if (sample < 0) sample = -sample;
if (sample > CLIP) sample = CLIP;
sample += BIAS;
let exponent = 7;
for (let expMask = 0x4000; (sample & expMask) === 0 && exponent > 0; exponent--) {
expMask >>= 1;
}
const mantissa = (sample >> (exponent + 3)) & 0x0f;
return ~(sign | (exponent << 4) | mantissa) & 0xff;
}
@@ -0,0 +1,95 @@
import type { CoreConfig } from "./core-bridge.js";
import type { VoiceCallTtsConfig } from "./config.js";
import { convertPcmToMulaw8k } from "./telephony-audio.js";
export type TelephonyTtsRuntime = {
textToSpeechTelephony: (params: {
text: string;
cfg: CoreConfig;
prefsPath?: string;
}) => Promise<{
success: boolean;
audioBuffer?: Buffer;
sampleRate?: number;
provider?: string;
error?: string;
}>;
};
export type TelephonyTtsProvider = {
synthesizeForTelephony: (text: string) => Promise<Buffer>;
};
export function createTelephonyTtsProvider(params: {
coreConfig: CoreConfig;
ttsOverride?: VoiceCallTtsConfig;
runtime: TelephonyTtsRuntime;
}): TelephonyTtsProvider {
const { coreConfig, ttsOverride, runtime } = params;
const mergedConfig = applyTtsOverride(coreConfig, ttsOverride);
return {
synthesizeForTelephony: async (text: string) => {
const result = await runtime.textToSpeechTelephony({
text,
cfg: mergedConfig,
});
if (!result.success || !result.audioBuffer || !result.sampleRate) {
throw new Error(result.error ?? "TTS conversion failed");
}
return convertPcmToMulaw8k(result.audioBuffer, result.sampleRate);
},
};
}
function applyTtsOverride(
coreConfig: CoreConfig,
override?: VoiceCallTtsConfig,
): CoreConfig {
if (!override) return coreConfig;
const base = coreConfig.messages?.tts;
const merged = mergeTtsConfig(base, override);
if (!merged) return coreConfig;
return {
...coreConfig,
messages: {
...(coreConfig.messages ?? {}),
tts: merged,
},
};
}
function mergeTtsConfig(
base?: VoiceCallTtsConfig,
override?: VoiceCallTtsConfig,
): VoiceCallTtsConfig | undefined {
if (!base && !override) return undefined;
if (!override) return base;
if (!base) return override;
return deepMerge(base, override);
}
function deepMerge<T>(base: T, override: T): T {
if (!isPlainObject(base) || !isPlainObject(override)) {
return override;
}
const result: Record<string, unknown> = { ...base };
for (const [key, value] of Object.entries(override)) {
if (value === undefined) continue;
const existing = (base as Record<string, unknown>)[key];
if (isPlainObject(existing) && isPlainObject(value)) {
result[key] = deepMerge(existing, value);
} else {
result[key] = value;
}
}
return result as T;
}
function isPlainObject(value: unknown): value is Record<string, unknown> {
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
}
+11 -1
View File
@@ -333,7 +333,13 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
name: "message",
description,
parameters: schema,
execute: async (_toolCallId, args) => {
execute: async (_toolCallId, args, signal) => {
// Check if already aborted before doing any work
if (signal?.aborted) {
const err = new Error("Message send aborted");
err.name = "AbortError";
throw err;
}
const params = args as Record<string, unknown>;
const cfg = options?.config ?? loadConfig();
const action = readStringParam(params, "action", {
@@ -366,6 +372,9 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
currentThreadTs: options?.currentThreadTs,
replyToMode: options?.replyToMode,
hasRepliedRef: options?.hasRepliedRef,
// Direct tool invocations should not add cross-context decoration.
// The agent is composing a message, not forwarding from another chat.
skipCrossContextDecoration: true,
}
: undefined;
@@ -379,6 +388,7 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
agentId: options?.agentSessionKey
? resolveSessionAgentId({ sessionKey: options.agentSessionKey, config: cfg })
: undefined,
abortSignal: signal,
});
const toolResult = getToolResult(result);
+6
View File
@@ -240,6 +240,12 @@ export type ChannelThreadingToolContext = {
currentThreadTs?: string;
replyToMode?: "off" | "first" | "all";
hasRepliedRef?: { value: boolean };
/**
* When true, skip cross-context decoration (e.g., "[from X]" prefix).
* Use this for direct tool invocations where the agent is composing a new message,
* not forwarding/relaying a message from another conversation.
*/
skipCrossContextDecoration?: boolean;
};
export type ChannelMessagingAdapter = {
+11
View File
@@ -59,6 +59,17 @@ export const CONFIG_PATH_CLAWDBOT = resolveConfigPath();
export const DEFAULT_GATEWAY_PORT = 18789;
/**
* Gateway lock directory (ephemeral).
* Default: os.tmpdir()/clawdbot-<uid> (uid suffix when available).
*/
export function resolveGatewayLockDir(tmpdir: () => string = os.tmpdir): string {
const base = tmpdir();
const uid = typeof process.getuid === "function" ? process.getuid() : undefined;
const suffix = uid != null ? `clawdbot-${uid}` : "clawdbot";
return path.join(base, suffix);
}
const OAUTH_FILENAME = "oauth.json";
/**
+4 -2
View File
@@ -7,12 +7,13 @@ import path from "node:path";
import { describe, expect, it, vi } from "vitest";
import { acquireGatewayLock, GatewayLockError } from "./gateway-lock.js";
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
async function makeEnv() {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-gateway-lock-"));
const configPath = path.join(dir, "clawdbot.json");
await fs.writeFile(configPath, "{}", "utf8");
await fs.mkdir(resolveGatewayLockDir(), { recursive: true });
return {
env: {
...process.env,
@@ -29,7 +30,8 @@ function resolveLockPath(env: NodeJS.ProcessEnv) {
const stateDir = resolveStateDir(env);
const configPath = resolveConfigPath(env, stateDir);
const hash = createHash("sha1").update(configPath).digest("hex").slice(0, 8);
return { lockPath: path.join(stateDir, `gateway.${hash}.lock`), configPath };
const lockDir = resolveGatewayLockDir();
return { lockPath: path.join(lockDir, `gateway.${hash}.lock`), configPath };
}
function makeProcStat(pid: number, startTime: number) {
+3 -2
View File
@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
import fsSync from "node:fs";
import path from "node:path";
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
const DEFAULT_TIMEOUT_MS = 5000;
const DEFAULT_POLL_INTERVAL_MS = 100;
@@ -150,7 +150,8 @@ function resolveGatewayLockPath(env: NodeJS.ProcessEnv) {
const stateDir = resolveStateDir(env);
const configPath = resolveConfigPath(env, stateDir);
const hash = createHash("sha1").update(configPath).digest("hex").slice(0, 8);
const lockPath = path.join(stateDir, `gateway.${hash}.lock`);
const lockDir = resolveGatewayLockDir();
const lockPath = path.join(lockDir, `gateway.${hash}.lock`);
return { lockPath, configPath };
}
@@ -321,6 +321,44 @@ describe("runMessageAction context isolation", () => {
}),
).rejects.toThrow(/Cross-context messaging denied/);
});
it("aborts send when abortSignal is already aborted", async () => {
const controller = new AbortController();
controller.abort();
await expect(
runMessageAction({
cfg: slackConfig,
action: "send",
params: {
channel: "slack",
target: "#C12345678",
message: "hi",
},
dryRun: true,
abortSignal: controller.signal,
}),
).rejects.toMatchObject({ name: "AbortError" });
});
it("aborts broadcast when abortSignal is already aborted", async () => {
const controller = new AbortController();
controller.abort();
await expect(
runMessageAction({
cfg: slackConfig,
action: "broadcast",
params: {
targets: ["channel:C12345678"],
channel: "slack",
message: "hi",
},
dryRun: true,
abortSignal: controller.signal,
}),
).rejects.toMatchObject({ name: "AbortError" });
});
});
describe("runMessageAction sendAttachment hydration", () => {
+37 -3
View File
@@ -64,6 +64,7 @@ export type RunMessageActionParams = {
sessionKey?: string;
agentId?: string;
dryRun?: boolean;
abortSignal?: AbortSignal;
};
export type MessageActionRunResult =
@@ -507,6 +508,7 @@ type ResolvedActionContext = {
input: RunMessageActionParams;
agentId?: string;
resolvedTarget?: ResolvedMessagingTarget;
abortSignal?: AbortSignal;
};
function resolveGateway(input: RunMessageActionParams): MessageActionRunnerGateway | undefined {
if (!input.gateway) return undefined;
@@ -524,6 +526,7 @@ async function handleBroadcastAction(
input: RunMessageActionParams,
params: Record<string, unknown>,
): Promise<MessageActionRunResult> {
throwIfAborted(input.abortSignal);
const broadcastEnabled = input.cfg.tools?.message?.broadcast?.enabled !== false;
if (!broadcastEnabled) {
throw new Error("Broadcast is disabled. Set tools.message.broadcast.enabled to true.");
@@ -548,8 +551,11 @@ async function handleBroadcastAction(
error?: string;
result?: MessageSendResult;
}> = [];
const isAbortError = (err: unknown): boolean => err instanceof Error && err.name === "AbortError";
for (const targetChannel of targetChannels) {
throwIfAborted(input.abortSignal);
for (const target of rawTargets) {
throwIfAborted(input.abortSignal);
try {
const resolved = await resolveChannelTarget({
cfg: input.cfg,
@@ -573,6 +579,7 @@ async function handleBroadcastAction(
result: sendResult.kind === "send" ? sendResult.sendResult : undefined,
});
} catch (err) {
if (isAbortError(err)) throw err;
results.push({
channel: targetChannel,
to: target,
@@ -592,8 +599,28 @@ async function handleBroadcastAction(
};
}
function throwIfAborted(abortSignal?: AbortSignal): void {
if (abortSignal?.aborted) {
const err = new Error("Message send aborted");
err.name = "AbortError";
throw err;
}
}
async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
const { cfg, params, channel, accountId, dryRun, gateway, input, agentId, resolvedTarget } = ctx;
const {
cfg,
params,
channel,
accountId,
dryRun,
gateway,
input,
agentId,
resolvedTarget,
abortSignal,
} = ctx;
throwIfAborted(abortSignal);
const action: ChannelMessageActionName = "send";
const to = readStringParam(params, "to", { required: true });
// Support media, path, and filePath parameters for attachments
@@ -676,6 +703,7 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
}
const mirrorMediaUrls =
mergedMediaUrls.length > 0 ? mergedMediaUrls : mediaUrl ? [mediaUrl] : undefined;
throwIfAborted(abortSignal);
const send = await executeSendAction({
ctx: {
cfg,
@@ -695,6 +723,7 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
mediaUrls: mirrorMediaUrls,
}
: undefined,
abortSignal,
},
to,
message,
@@ -718,7 +747,8 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
}
async function handlePollAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
const { cfg, params, channel, accountId, dryRun, gateway, input } = ctx;
const { cfg, params, channel, accountId, dryRun, gateway, input, abortSignal } = ctx;
throwIfAborted(abortSignal);
const action: ChannelMessageActionName = "poll";
const to = readStringParam(params, "to", { required: true });
const question = readStringParam(params, "pollQuestion", {
@@ -777,7 +807,8 @@ async function handlePollAction(ctx: ResolvedActionContext): Promise<MessageActi
}
async function handlePluginAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
const { cfg, params, channel, accountId, dryRun, gateway, input } = ctx;
const { cfg, params, channel, accountId, dryRun, gateway, input, abortSignal } = ctx;
throwIfAborted(abortSignal);
const action = input.action as Exclude<ChannelMessageActionName, "send" | "poll" | "broadcast">;
if (dryRun) {
return {
@@ -930,6 +961,7 @@ export async function runMessageAction(
input,
agentId: resolvedAgentId,
resolvedTarget,
abortSignal: input.abortSignal,
});
}
@@ -942,6 +974,7 @@ export async function runMessageAction(
dryRun,
gateway,
input,
abortSignal: input.abortSignal,
});
}
@@ -953,5 +986,6 @@ export async function runMessageAction(
dryRun,
gateway,
input,
abortSignal: input.abortSignal,
});
}
+2
View File
@@ -50,6 +50,7 @@ type MessageSendParams = {
text?: string;
mediaUrls?: string[];
};
abortSignal?: AbortSignal;
};
export type MessageSendResult = {
@@ -167,6 +168,7 @@ export async function sendMessage(params: MessageSendParams): Promise<MessageSen
gifPlayback: params.gifPlayback,
deps: params.deps,
bestEffort: params.bestEffort,
abortSignal: params.abortSignal,
mirror: params.mirror
? {
...params.mirror,
+3 -1
View File
@@ -119,6 +119,8 @@ export async function buildCrossContextDecoration(params: {
accountId?: string | null;
}): Promise<CrossContextDecoration | null> {
if (!params.toolContext?.currentChannelId) return null;
// Skip decoration for direct tool sends (agent composing, not forwarding)
if (params.toolContext.skipCrossContextDecoration) return null;
if (!isCrossContextTarget(params)) return null;
const markerConfig = params.cfg.tools?.message?.crossContext?.marker;
@@ -131,11 +133,11 @@ export async function buildCrossContextDecoration(params: {
targetId: params.toolContext.currentChannelId,
accountId: params.accountId ?? undefined,
})) ?? params.toolContext.currentChannelId;
// Don't force group formatting here; currentChannelId can be a DM or a group.
const originLabel = formatTargetDisplay({
channel: params.channel,
target: params.toolContext.currentChannelId,
display: currentName,
kind: "group",
});
const prefixTemplate = markerConfig?.prefix ?? "[from {channel}] ";
const suffixTemplate = markerConfig?.suffix ?? "";
@@ -32,6 +32,7 @@ export type OutboundSendContext = {
text?: string;
mediaUrls?: string[];
};
abortSignal?: AbortSignal;
};
function extractToolPayload(result: AgentToolResult<unknown>): unknown {
@@ -56,6 +57,14 @@ function extractToolPayload(result: AgentToolResult<unknown>): unknown {
return result.content ?? result;
}
function throwIfAborted(abortSignal?: AbortSignal): void {
if (abortSignal?.aborted) {
const err = new Error("Message send aborted");
err.name = "AbortError";
throw err;
}
}
export async function executeSendAction(params: {
ctx: OutboundSendContext;
to: string;
@@ -70,6 +79,7 @@ export async function executeSendAction(params: {
toolResult?: AgentToolResult<unknown>;
sendResult?: MessageSendResult;
}> {
throwIfAborted(params.ctx.abortSignal);
if (!params.ctx.dryRun) {
const handled = await dispatchChannelMessageAction({
channel: params.ctx.channel,
@@ -103,6 +113,7 @@ export async function executeSendAction(params: {
}
}
throwIfAborted(params.ctx.abortSignal);
const result: MessageSendResult = await sendMessage({
cfg: params.ctx.cfg,
to: params.to,
@@ -117,6 +128,7 @@ export async function executeSendAction(params: {
deps: params.ctx.deps,
gateway: params.ctx.gateway,
mirror: params.ctx.mirror,
abortSignal: params.ctx.abortSignal,
});
return {
+70 -17
View File
@@ -100,7 +100,12 @@ export function formatTargetDisplay(params: {
if (!trimmedTarget) return trimmedTarget;
if (trimmedTarget.startsWith("#") || trimmedTarget.startsWith("@")) return trimmedTarget;
const withoutPrefix = trimmedTarget.replace(/^telegram:/i, "");
const channelPrefix = `${params.channel}:`;
const withoutProvider = trimmedTarget.toLowerCase().startsWith(channelPrefix)
? trimmedTarget.slice(channelPrefix.length)
: trimmedTarget;
const withoutPrefix = withoutProvider.replace(/^telegram:/i, "");
if (/^channel:/i.test(withoutPrefix)) {
return `#${withoutPrefix.replace(/^channel:/i, "")}`;
}
@@ -119,14 +124,23 @@ function preserveTargetCase(channel: ChannelId, raw: string, normalized: string)
return trimmed;
}
function detectTargetKind(raw: string, preferred?: TargetResolveKind): TargetResolveKind {
function detectTargetKind(
channel: ChannelId,
raw: string,
preferred?: TargetResolveKind,
): TargetResolveKind {
if (preferred) return preferred;
const trimmed = raw.trim();
if (!trimmed) return "group";
if (trimmed.startsWith("@") || /^<@!?/.test(trimmed) || /^user:/i.test(trimmed)) return "user";
if (trimmed.startsWith("#") || /^channel:/i.test(trimmed)) {
return "group";
if (trimmed.startsWith("#") || /^channel:/i.test(trimmed)) return "group";
// For some channels (e.g., BlueBubbles/iMessage), bare phone numbers are almost always DM targets.
if ((channel === "bluebubbles" || channel === "imessage") && /^\+?\d{6,}$/.test(trimmed)) {
return "user";
}
return "group";
}
@@ -282,7 +296,7 @@ export async function resolveMessagingTarget(params: {
const plugin = getChannelPlugin(params.channel);
const providerLabel = plugin?.meta?.label ?? params.channel;
const hint = plugin?.messaging?.targetResolver?.hint;
const kind = detectTargetKind(raw, params.preferredKind);
const kind = detectTargetKind(params.channel, raw, params.preferredKind);
const normalized = normalizeTargetForProvider(params.channel, raw) ?? raw;
const looksLikeTargetId = (): boolean => {
const trimmed = raw.trim();
@@ -291,7 +305,12 @@ export async function resolveMessagingTarget(params: {
if (lookup) return lookup(trimmed, normalized);
if (/^(channel|group|user):/i.test(trimmed)) return true;
if (/^[@#]/.test(trimmed)) return true;
if (/^\+?\d{6,}$/.test(trimmed)) return true;
if (/^\+?\d{6,}$/.test(trimmed)) {
// BlueBubbles/iMessage phone numbers should usually resolve via the directory to a DM chat,
// otherwise the provider may pick an existing group containing that handle.
if (params.channel === "bluebubbles" || params.channel === "imessage") return false;
return true;
}
if (trimmed.includes("@thread")) return true;
if (/^(conversation|user):/i.test(trimmed)) return true;
return false;
@@ -353,6 +372,24 @@ export async function resolveMessagingTarget(params: {
candidates: match.entries,
};
}
// For iMessage-style channels, allow sending directly to the normalized handle
// even if the directory doesn't contain an entry yet.
if (
(params.channel === "bluebubbles" || params.channel === "imessage") &&
/^\+?\d{6,}$/.test(query)
) {
const directTarget = preserveTargetCase(params.channel, raw, normalized);
return {
ok: true,
target: {
to: directTarget,
kind,
display: stripTargetPrefixes(raw),
source: "normalized",
},
};
}
return {
ok: false,
error: unknownTargetError(providerLabel, raw, hint),
@@ -367,16 +404,32 @@ export async function lookupDirectoryDisplay(params: {
runtime?: RuntimeEnv;
}): Promise<string | undefined> {
const normalized = normalizeTargetForProvider(params.channel, params.targetId) ?? params.targetId;
const candidates = await getDirectoryEntries({
cfg: params.cfg,
channel: params.channel,
accountId: params.accountId,
kind: "group",
runtime: params.runtime,
preferLiveOnMiss: false,
});
const entry = candidates.find(
(candidate) => normalizeDirectoryEntryId(params.channel, candidate) === normalized,
);
// Targets can resolve to either peers (DMs) or groups. Try both.
const [groups, users] = await Promise.all([
getDirectoryEntries({
cfg: params.cfg,
channel: params.channel,
accountId: params.accountId,
kind: "group",
runtime: params.runtime,
preferLiveOnMiss: false,
}),
getDirectoryEntries({
cfg: params.cfg,
channel: params.channel,
accountId: params.accountId,
kind: "user",
runtime: params.runtime,
preferLiveOnMiss: false,
}),
]);
const findMatch = (candidates: ChannelDirectoryEntry[]) =>
candidates.find(
(candidate) => normalizeDirectoryEntryId(params.channel, candidate) === normalized,
);
const entry = findMatch(groups) ?? findMatch(users);
return entry?.name ?? entry?.handle ?? undefined;
}
+4
View File
@@ -124,6 +124,7 @@ import { startWebLoginWithQr, waitForWebLogin } from "../../web/login-qr.js";
import { sendMessageWhatsApp, sendPollWhatsApp } from "../../web/outbound.js";
import { registerMemoryCli } from "../../cli/memory-cli.js";
import { formatNativeDependencyHint } from "./native-deps.js";
import { textToSpeechTelephony } from "../../tts/tts.js";
import type { PluginRuntime } from "./types.js";
@@ -162,6 +163,9 @@ export function createPluginRuntime(): PluginRuntime {
getImageMetadata,
resizeToJpeg,
},
tts: {
textToSpeechTelephony,
},
tools: {
createMemoryGetTool,
createMemorySearchTool,
+4
View File
@@ -16,6 +16,7 @@ type UpsertChannelPairingRequest =
typeof import("../../pairing/pairing-store.js").upsertChannelPairingRequest;
type FetchRemoteMedia = typeof import("../../media/fetch.js").fetchRemoteMedia;
type SaveMediaBuffer = typeof import("../../media/store.js").saveMediaBuffer;
type TextToSpeechTelephony = typeof import("../../tts/tts.js").textToSpeechTelephony;
type BuildMentionRegexes = typeof import("../../auto-reply/reply/mentions.js").buildMentionRegexes;
type MatchesMentionPatterns =
typeof import("../../auto-reply/reply/mentions.js").matchesMentionPatterns;
@@ -173,6 +174,9 @@ export type PluginRuntime = {
getImageMetadata: GetImageMetadata;
resizeToJpeg: ResizeToJpeg;
};
tts: {
textToSpeechTelephony: TextToSpeechTelephony;
};
tools: {
createMemoryGetTool: CreateMemoryGetTool;
createMemorySearchTool: CreateMemorySearchTool;
+2
View File
@@ -43,6 +43,7 @@ function setup(config: Record<string, unknown>): Registered {
source: "test",
config: {},
pluginConfig: config,
runtime: { tts: { textToSpeechTelephony: vi.fn() } },
logger: noopLogger,
registerGatewayMethod: (method, handler) => methods.set(method, handler),
registerTool: (tool) => tools.push(tool),
@@ -142,6 +143,7 @@ describe("voice-call plugin", () => {
source: "test",
config: {},
pluginConfig: { provider: "mock" },
runtime: { tts: { textToSpeechTelephony: vi.fn() } },
logger: noopLogger,
registerGatewayMethod: () => {},
registerTool: () => {},
+8 -6
View File
@@ -109,13 +109,13 @@ describe("tts", () => {
});
describe("isValidOpenAIModel", () => {
it("accepts gpt-4o-mini-tts model", () => {
it("accepts supported models", () => {
expect(isValidOpenAIModel("gpt-4o-mini-tts")).toBe(true);
expect(isValidOpenAIModel("tts-1")).toBe(true);
expect(isValidOpenAIModel("tts-1-hd")).toBe(true);
});
it("rejects other models", () => {
expect(isValidOpenAIModel("tts-1")).toBe(false);
expect(isValidOpenAIModel("tts-1-hd")).toBe(false);
it("rejects unsupported models", () => {
expect(isValidOpenAIModel("invalid")).toBe(false);
expect(isValidOpenAIModel("")).toBe(false);
expect(isValidOpenAIModel("gpt-4")).toBe(false);
@@ -123,9 +123,11 @@ describe("tts", () => {
});
describe("OPENAI_TTS_MODELS", () => {
it("contains only gpt-4o-mini-tts", () => {
it("contains supported models", () => {
expect(OPENAI_TTS_MODELS).toContain("gpt-4o-mini-tts");
expect(OPENAI_TTS_MODELS).toHaveLength(1);
expect(OPENAI_TTS_MODELS).toContain("tts-1");
expect(OPENAI_TTS_MODELS).toContain("tts-1-hd");
expect(OPENAI_TTS_MODELS).toHaveLength(3);
});
it("is a non-empty array", () => {
+126 -3
View File
@@ -76,6 +76,11 @@ const DEFAULT_OUTPUT = {
voiceCompatible: false,
};
const TELEPHONY_OUTPUT = {
openai: { format: "pcm" as const, sampleRate: 24000 },
elevenlabs: { format: "pcm_22050", sampleRate: 22050 },
};
const TTS_AUTO_MODES = new Set<TtsAutoMode>(["off", "always", "inbound", "tagged"]);
export type ResolvedTtsConfig = {
@@ -180,6 +185,16 @@ export type TtsResult = {
voiceCompatible?: boolean;
};
export type TtsTelephonyResult = {
success: boolean;
audioBuffer?: Buffer;
error?: string;
latencyMs?: number;
provider?: string;
outputFormat?: string;
sampleRate?: number;
};
type TtsStatusEntry = {
timestamp: number;
success: boolean;
@@ -736,7 +751,17 @@ function parseTtsDirectives(
};
}
export const OPENAI_TTS_MODELS = ["gpt-4o-mini-tts"] as const;
export const OPENAI_TTS_MODELS = ["gpt-4o-mini-tts", "tts-1", "tts-1-hd"] as const;
/**
* Custom OpenAI-compatible TTS endpoint.
* When set, model/voice validation is relaxed to allow non-OpenAI models.
* Example: OPENAI_TTS_BASE_URL=http://localhost:8880/v1
*/
const OPENAI_TTS_BASE_URL = (
process.env.OPENAI_TTS_BASE_URL?.trim() || "https://api.openai.com/v1"
).replace(/\/+$/, "");
const isCustomOpenAIEndpoint = OPENAI_TTS_BASE_URL !== "https://api.openai.com/v1";
export const OPENAI_TTS_VOICES = [
"alloy",
"ash",
@@ -752,10 +777,14 @@ export const OPENAI_TTS_VOICES = [
type OpenAiTtsVoice = (typeof OPENAI_TTS_VOICES)[number];
function isValidOpenAIModel(model: string): boolean {
// Allow any model when using custom endpoint (e.g., Kokoro, LocalAI)
if (isCustomOpenAIEndpoint) return true;
return OPENAI_TTS_MODELS.includes(model as (typeof OPENAI_TTS_MODELS)[number]);
}
function isValidOpenAIVoice(voice: string): voice is OpenAiTtsVoice {
// Allow any voice when using custom endpoint (e.g., Kokoro Chinese voices)
if (isCustomOpenAIEndpoint) return true;
return OPENAI_TTS_VOICES.includes(voice as OpenAiTtsVoice);
}
@@ -966,7 +995,7 @@ async function openaiTTS(params: {
apiKey: string;
model: string;
voice: string;
responseFormat: "mp3" | "opus";
responseFormat: "mp3" | "opus" | "pcm";
timeoutMs: number;
}): Promise<Buffer> {
const { text, apiKey, model, voice, responseFormat, timeoutMs } = params;
@@ -982,7 +1011,7 @@ async function openaiTTS(params: {
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch("https://api.openai.com/v1/audio/speech", {
const response = await fetch(`${OPENAI_TTS_BASE_URL}/audio/speech`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
@@ -1210,6 +1239,100 @@ export async function textToSpeech(params: {
};
}
export async function textToSpeechTelephony(params: {
text: string;
cfg: ClawdbotConfig;
prefsPath?: string;
}): Promise<TtsTelephonyResult> {
const config = resolveTtsConfig(params.cfg);
const prefsPath = params.prefsPath ?? resolveTtsPrefsPath(config);
if (params.text.length > config.maxTextLength) {
return {
success: false,
error: `Text too long (${params.text.length} chars, max ${config.maxTextLength})`,
};
}
const userProvider = getTtsProvider(config, prefsPath);
const providers = resolveTtsProviderOrder(userProvider);
let lastError: string | undefined;
for (const provider of providers) {
const providerStart = Date.now();
try {
if (provider === "edge") {
lastError = "edge: unsupported for telephony";
continue;
}
const apiKey = resolveTtsApiKey(config, provider);
if (!apiKey) {
lastError = `No API key for ${provider}`;
continue;
}
if (provider === "elevenlabs") {
const output = TELEPHONY_OUTPUT.elevenlabs;
const audioBuffer = await elevenLabsTTS({
text: params.text,
apiKey,
baseUrl: config.elevenlabs.baseUrl,
voiceId: config.elevenlabs.voiceId,
modelId: config.elevenlabs.modelId,
outputFormat: output.format,
seed: config.elevenlabs.seed,
applyTextNormalization: config.elevenlabs.applyTextNormalization,
languageCode: config.elevenlabs.languageCode,
voiceSettings: config.elevenlabs.voiceSettings,
timeoutMs: config.timeoutMs,
});
return {
success: true,
audioBuffer,
latencyMs: Date.now() - providerStart,
provider,
outputFormat: output.format,
sampleRate: output.sampleRate,
};
}
const output = TELEPHONY_OUTPUT.openai;
const audioBuffer = await openaiTTS({
text: params.text,
apiKey,
model: config.openai.model,
voice: config.openai.voice,
responseFormat: output.format,
timeoutMs: config.timeoutMs,
});
return {
success: true,
audioBuffer,
latencyMs: Date.now() - providerStart,
provider,
outputFormat: output.format,
sampleRate: output.sampleRate,
};
} catch (err) {
const error = err as Error;
if (error.name === "AbortError") {
lastError = `${provider}: request timed out`;
} else {
lastError = `${provider}: ${error.message}`;
}
}
}
return {
success: false,
error: `TTS conversion failed: ${lastError || "no providers available"}`,
};
}
export async function maybeApplyTtsToPayload(params: {
payload: ReplyPayload;
cfg: ClawdbotConfig;