Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
492c37a171 | ||
|
|
738e596c71 | ||
|
|
bea6b3eb5b | ||
|
|
00e00460c1 | ||
|
|
64c6698ea4 | ||
|
|
f997a3f395 | ||
|
|
626a37f1ce | ||
|
|
83f92e34af | ||
|
|
9366cbc7db | ||
|
|
d4f895d8f2 | ||
|
|
f08c34a73f | ||
|
|
6a9301c27d |
No files matched your search
@@ -24,6 +24,7 @@ Docs: https://docs.clawd.bot
|
||||
- Telegram: add verbose raw-update logging for inbound Telegram updates. (#1597) Thanks @rohannagpal.
|
||||
|
||||
### Fixes
|
||||
- BlueBubbles: route phone-number targets to DMs, avoid leaking routing IDs, and auto-create missing DMs (Private API required). (#1751) Thanks @tyler6204. https://docs.clawd.bot/channels/bluebubbles
|
||||
- BlueBubbles: keep part-index GUIDs in reply tags when short IDs are missing.
|
||||
- Web UI: hide internal `message_id` hints in chat bubbles.
|
||||
- Web UI: show Stop button during active runs, swap back to New session when idle. (#1664) Thanks @ndbroadbent.
|
||||
@@ -42,6 +43,7 @@ Docs: https://docs.clawd.bot
|
||||
- Gateway: reduce log noise for late invokes + remote node probes; debounce skills refresh. (#1607) Thanks @petter-b.
|
||||
- Gateway: clarify Control UI/WebChat auth error hints for missing tokens. (#1690)
|
||||
- Gateway: listen on IPv6 loopback when bound to 127.0.0.1 so localhost webhooks work.
|
||||
- Gateway: store lock files in the temp directory to avoid stale locks on persistent volumes. (#1676)
|
||||
- macOS: default direct-transport `ws://` URLs to port 18789; document `gateway.remote.transport`. (#1603) Thanks @ngutman.
|
||||
- Voice Call: return stream TwiML for outbound conversation calls on initial Twilio webhook. (#1634)
|
||||
- Google Chat: tighten email allowlist matching, typing cleanup, media caps, and onboarding/docs/tests. (#1635) Thanks @iHildy.
|
||||
|
||||
@@ -213,6 +213,7 @@ Prefer `chat_guid` for stable routing:
|
||||
- `chat_id:123`
|
||||
- `chat_identifier:...`
|
||||
- Direct handles: `+15555550123`, `user@example.com`
|
||||
- If a direct handle does not have an existing DM chat, Clawdbot will create one via `POST /api/v1/chat/new`. This requires the BlueBubbles Private API to be enabled.
|
||||
|
||||
## Security
|
||||
- Webhook requests are authenticated by comparing `guid`/`password` query params or headers against `channels.bluebubbles.password`. Requests from `localhost` are also accepted.
|
||||
|
||||
@@ -67,6 +67,22 @@ Plugins can register:
|
||||
Plugins run **in‑process** with the Gateway, so treat them as trusted code.
|
||||
Tool authoring guide: [Plugin agent tools](/plugins/agent-tools).
|
||||
|
||||
## Runtime helpers
|
||||
|
||||
Plugins can access selected core helpers via `api.runtime`. For telephony TTS:
|
||||
|
||||
```ts
|
||||
const result = await api.runtime.tts.textToSpeechTelephony({
|
||||
text: "Hello from Clawdbot",
|
||||
cfg: api.config,
|
||||
});
|
||||
```
|
||||
|
||||
Notes:
|
||||
- Uses core `messages.tts` configuration (OpenAI or ElevenLabs).
|
||||
- Returns PCM audio buffer + sample rate. Plugins must resample/encode for providers.
|
||||
- Edge TTS is not supported for telephony.
|
||||
|
||||
## Discovery & precedence
|
||||
|
||||
Clawdbot scans, in order:
|
||||
|
||||
@@ -104,6 +104,87 @@ Notes:
|
||||
- `mock` is a local dev provider (no network calls).
|
||||
- `skipSignatureVerification` is for local testing only.
|
||||
|
||||
## TTS for calls
|
||||
|
||||
Voice Call uses the core `messages.tts` configuration (OpenAI or ElevenLabs) for
|
||||
streaming speech on calls. You can override it under the plugin config with the
|
||||
**same shape** — it deep‑merges with `messages.tts`.
|
||||
|
||||
```json5
|
||||
{
|
||||
tts: {
|
||||
provider: "elevenlabs",
|
||||
elevenlabs: {
|
||||
voiceId: "pMsXgVXv3BLzUgSXRplE",
|
||||
modelId: "eleven_multilingual_v2"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- **Edge TTS is ignored for voice calls** (telephony audio needs PCM; Edge output is unreliable).
|
||||
- Core TTS is used when Twilio media streaming is enabled; otherwise calls fall back to provider native voices.
|
||||
|
||||
### More examples
|
||||
|
||||
Use core TTS only (no override):
|
||||
|
||||
```json5
|
||||
{
|
||||
messages: {
|
||||
tts: {
|
||||
provider: "openai",
|
||||
openai: { voice: "alloy" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Override to ElevenLabs just for calls (keep core default elsewhere):
|
||||
|
||||
```json5
|
||||
{
|
||||
plugins: {
|
||||
entries: {
|
||||
"voice-call": {
|
||||
config: {
|
||||
tts: {
|
||||
provider: "elevenlabs",
|
||||
elevenlabs: {
|
||||
apiKey: "elevenlabs_key",
|
||||
voiceId: "pMsXgVXv3BLzUgSXRplE",
|
||||
modelId: "eleven_multilingual_v2"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Override only the OpenAI model for calls (deep‑merge example):
|
||||
|
||||
```json5
|
||||
{
|
||||
plugins: {
|
||||
entries: {
|
||||
"voice-call": {
|
||||
config: {
|
||||
tts: {
|
||||
openai: {
|
||||
model: "gpt-4o-mini-tts",
|
||||
voice: "marin"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Inbound calls
|
||||
|
||||
Inbound policy defaults to `disabled`. To enable inbound calls, set:
|
||||
|
||||
+5
-3
@@ -16,7 +16,7 @@ and you configure everything via the `/setup` web wizard.
|
||||
|
||||
## One-click deploy
|
||||
|
||||
<a href="https://railway.com/deploy/clawdbot-railway-template" target="_blank" rel="noreferrer">Deploy on Railway</a>
|
||||
<a href="https://railway.app/new/template?template=https://github.com/vignesh07/clawdbot-railway-template" target="_blank" rel="noreferrer">Deploy on Railway</a>
|
||||
|
||||
After deploy, find your public URL in **Railway → your service → Settings → Domains**.
|
||||
|
||||
@@ -55,6 +55,7 @@ Attach a volume mounted at:
|
||||
Set these variables on the service:
|
||||
|
||||
- `SETUP_PASSWORD` (required)
|
||||
- `PORT=8080` (required — must match the port in Public Networking)
|
||||
- `CLAWDBOT_STATE_DIR=/data/.clawdbot` (recommended)
|
||||
- `CLAWDBOT_WORKSPACE_DIR=/data/workspace` (recommended)
|
||||
- `CLAWDBOT_GATEWAY_TOKEN` (recommended; treat as an admin secret)
|
||||
@@ -82,8 +83,9 @@ If Telegram DMs are set to pairing, the setup wizard can approve the pairing cod
|
||||
1) Go to https://discord.com/developers/applications
|
||||
2) **New Application** → choose a name
|
||||
3) **Bot** → **Add Bot**
|
||||
4) Copy the **Bot Token** and paste into `/setup`
|
||||
5) Invite the bot to your server (OAuth2 URL Generator; scopes: `bot`, `applications.commands`)
|
||||
4) **Enable MESSAGE CONTENT INTENT** under Bot → Privileged Gateway Intents (required or the bot will crash on startup)
|
||||
5) Copy the **Bot Token** and paste into `/setup`
|
||||
6) Invite the bot to your server (OAuth2 URL Generator; scopes: `bot`, `applications.commands`)
|
||||
|
||||
## Backups & migration
|
||||
|
||||
|
||||
@@ -25,9 +25,11 @@ import { resolveBlueBubblesMessageId } from "./monitor.js";
|
||||
import { probeBlueBubbles, type BlueBubblesProbe } from "./probe.js";
|
||||
import { sendMessageBlueBubbles } from "./send.js";
|
||||
import {
|
||||
extractHandleFromChatGuid,
|
||||
looksLikeBlueBubblesTargetId,
|
||||
normalizeBlueBubblesHandle,
|
||||
normalizeBlueBubblesMessagingTarget,
|
||||
parseBlueBubblesTarget,
|
||||
} from "./targets.js";
|
||||
import { bluebubblesMessageActions } from "./actions.js";
|
||||
import { monitorBlueBubblesProvider, resolveWebhookPathFromConfig } from "./monitor.js";
|
||||
@@ -148,6 +150,58 @@ export const bluebubblesPlugin: ChannelPlugin<ResolvedBlueBubblesAccount> = {
|
||||
looksLikeId: looksLikeBlueBubblesTargetId,
|
||||
hint: "<handle|chat_guid:GUID|chat_id:ID|chat_identifier:ID>",
|
||||
},
|
||||
formatTargetDisplay: ({ target, display }) => {
|
||||
const shouldParseDisplay = (value: string): boolean => {
|
||||
if (looksLikeBlueBubblesTargetId(value)) return true;
|
||||
return /^(bluebubbles:|chat_guid:|chat_id:|chat_identifier:)/i.test(value);
|
||||
};
|
||||
|
||||
// Helper to extract a clean handle from any BlueBubbles target format
|
||||
const extractCleanDisplay = (value: string | undefined): string | null => {
|
||||
const trimmed = value?.trim();
|
||||
if (!trimmed) return null;
|
||||
try {
|
||||
const parsed = parseBlueBubblesTarget(trimmed);
|
||||
if (parsed.kind === "chat_guid") {
|
||||
const handle = extractHandleFromChatGuid(parsed.chatGuid);
|
||||
if (handle) return handle;
|
||||
}
|
||||
if (parsed.kind === "handle") {
|
||||
return normalizeBlueBubblesHandle(parsed.to);
|
||||
}
|
||||
} catch {
|
||||
// Fall through
|
||||
}
|
||||
// Strip common prefixes and try raw extraction
|
||||
const stripped = trimmed
|
||||
.replace(/^bluebubbles:/i, "")
|
||||
.replace(/^chat_guid:/i, "")
|
||||
.replace(/^chat_id:/i, "")
|
||||
.replace(/^chat_identifier:/i, "");
|
||||
const handle = extractHandleFromChatGuid(stripped);
|
||||
if (handle) return handle;
|
||||
// Don't return raw chat_guid formats - they contain internal routing info
|
||||
if (stripped.includes(";-;") || stripped.includes(";+;")) return null;
|
||||
return stripped;
|
||||
};
|
||||
|
||||
// Try to get a clean display from the display parameter first
|
||||
const trimmedDisplay = display?.trim();
|
||||
if (trimmedDisplay) {
|
||||
if (!shouldParseDisplay(trimmedDisplay)) {
|
||||
return trimmedDisplay;
|
||||
}
|
||||
const cleanDisplay = extractCleanDisplay(trimmedDisplay);
|
||||
if (cleanDisplay) return cleanDisplay;
|
||||
}
|
||||
|
||||
// Fall back to extracting from target
|
||||
const cleanTarget = extractCleanDisplay(target);
|
||||
if (cleanTarget) return cleanTarget;
|
||||
|
||||
// Last resort: return display or target as-is
|
||||
return display?.trim() || target?.trim() || "";
|
||||
},
|
||||
},
|
||||
setup: {
|
||||
resolveAccountId: ({ accountId }) => normalizeAccountId(accountId),
|
||||
|
||||
@@ -187,6 +187,47 @@ describe("send", () => {
|
||||
expect(result).toBe("iMessage;-;+15551234567");
|
||||
});
|
||||
|
||||
it("returns null when handle only exists in group chat (not DM)", async () => {
|
||||
// This is the critical fix: if a phone number only exists as a participant in a group chat
|
||||
// (no direct DM chat), we should NOT send to that group. Return null instead.
|
||||
mockFetch
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: () =>
|
||||
Promise.resolve({
|
||||
data: [
|
||||
{
|
||||
guid: "iMessage;+;group-the-council",
|
||||
participants: [
|
||||
{ address: "+12622102921" },
|
||||
{ address: "+15550001111" },
|
||||
{ address: "+15550002222" },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
})
|
||||
// Empty second page to stop pagination
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: () => Promise.resolve({ data: [] }),
|
||||
});
|
||||
|
||||
const target: BlueBubblesSendTarget = {
|
||||
kind: "handle",
|
||||
address: "+12622102921",
|
||||
service: "imessage",
|
||||
};
|
||||
const result = await resolveChatGuidForTarget({
|
||||
baseUrl: "http://localhost:1234",
|
||||
password: "test",
|
||||
target,
|
||||
});
|
||||
|
||||
// Should return null, NOT the group chat GUID
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null when chat not found", async () => {
|
||||
mockFetch.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
@@ -344,14 +385,14 @@ describe("send", () => {
|
||||
).rejects.toThrow("password is required");
|
||||
});
|
||||
|
||||
it("throws when chatGuid cannot be resolved", async () => {
|
||||
it("throws when chatGuid cannot be resolved for non-handle targets", async () => {
|
||||
mockFetch.mockResolvedValue({
|
||||
ok: true,
|
||||
json: () => Promise.resolve({ data: [] }),
|
||||
});
|
||||
|
||||
await expect(
|
||||
sendMessageBlueBubbles("+15559999999", "Hello", {
|
||||
sendMessageBlueBubbles("chat_id:999", "Hello", {
|
||||
serverUrl: "http://localhost:1234",
|
||||
password: "test",
|
||||
}),
|
||||
@@ -398,6 +439,57 @@ describe("send", () => {
|
||||
expect(body.method).toBeUndefined();
|
||||
});
|
||||
|
||||
it("creates a new chat when handle target is missing", async () => {
|
||||
mockFetch
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: () => Promise.resolve({ data: [] }),
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
text: () =>
|
||||
Promise.resolve(
|
||||
JSON.stringify({
|
||||
data: { guid: "new-msg-guid" },
|
||||
}),
|
||||
),
|
||||
});
|
||||
|
||||
const result = await sendMessageBlueBubbles("+15550009999", "Hello new chat", {
|
||||
serverUrl: "http://localhost:1234",
|
||||
password: "test",
|
||||
});
|
||||
|
||||
expect(result.messageId).toBe("new-msg-guid");
|
||||
expect(mockFetch).toHaveBeenCalledTimes(2);
|
||||
|
||||
const createCall = mockFetch.mock.calls[1];
|
||||
expect(createCall[0]).toContain("/api/v1/chat/new");
|
||||
const body = JSON.parse(createCall[1].body);
|
||||
expect(body.addresses).toEqual(["+15550009999"]);
|
||||
expect(body.message).toBe("Hello new chat");
|
||||
});
|
||||
|
||||
it("throws when creating a new chat requires Private API", async () => {
|
||||
mockFetch
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: () => Promise.resolve({ data: [] }),
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
status: 403,
|
||||
text: () => Promise.resolve("Private API not enabled"),
|
||||
});
|
||||
|
||||
await expect(
|
||||
sendMessageBlueBubbles("+15550008888", "Hello", {
|
||||
serverUrl: "http://localhost:1234",
|
||||
password: "test",
|
||||
}),
|
||||
).rejects.toThrow("Private API must be enabled");
|
||||
});
|
||||
|
||||
it("uses private-api when reply metadata is present", async () => {
|
||||
mockFetch
|
||||
.mockResolvedValueOnce({
|
||||
|
||||
@@ -257,11 +257,17 @@ export async function resolveChatGuidForTarget(params: {
|
||||
return guid;
|
||||
}
|
||||
if (!participantMatch && guid) {
|
||||
const participants = extractParticipantAddresses(chat).map((entry) =>
|
||||
normalizeBlueBubblesHandle(entry),
|
||||
);
|
||||
if (participants.includes(normalizedHandle)) {
|
||||
participantMatch = guid;
|
||||
// Only consider DM chats (`;-;` separator) as participant matches.
|
||||
// Group chats (`;+;` separator) should never match when searching by handle/phone.
|
||||
// This prevents routing "send to +1234567890" to a group chat that contains that number.
|
||||
const isDmChat = guid.includes(";-;");
|
||||
if (isDmChat) {
|
||||
const participants = extractParticipantAddresses(chat).map((entry) =>
|
||||
normalizeBlueBubblesHandle(entry),
|
||||
);
|
||||
if (participants.includes(normalizedHandle)) {
|
||||
participantMatch = guid;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -270,6 +276,55 @@ export async function resolveChatGuidForTarget(params: {
|
||||
return participantMatch;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a new chat (DM) and optionally sends an initial message.
|
||||
* Requires Private API to be enabled in BlueBubbles.
|
||||
*/
|
||||
async function createNewChatWithMessage(params: {
|
||||
baseUrl: string;
|
||||
password: string;
|
||||
address: string;
|
||||
message: string;
|
||||
timeoutMs?: number;
|
||||
}): Promise<BlueBubblesSendResult> {
|
||||
const url = buildBlueBubblesApiUrl({
|
||||
baseUrl: params.baseUrl,
|
||||
path: "/api/v1/chat/new",
|
||||
password: params.password,
|
||||
});
|
||||
const payload = {
|
||||
addresses: [params.address],
|
||||
message: params.message,
|
||||
};
|
||||
const res = await blueBubblesFetchWithTimeout(
|
||||
url,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(payload),
|
||||
},
|
||||
params.timeoutMs,
|
||||
);
|
||||
if (!res.ok) {
|
||||
const errorText = await res.text();
|
||||
// Check for Private API not enabled error
|
||||
if (res.status === 400 || res.status === 403 || errorText.toLowerCase().includes("private api")) {
|
||||
throw new Error(
|
||||
`BlueBubbles send failed: Cannot create new chat - Private API must be enabled. Original error: ${errorText || res.status}`,
|
||||
);
|
||||
}
|
||||
throw new Error(`BlueBubbles create chat failed (${res.status}): ${errorText || "unknown"}`);
|
||||
}
|
||||
const body = await res.text();
|
||||
if (!body) return { messageId: "ok" };
|
||||
try {
|
||||
const parsed = JSON.parse(body) as unknown;
|
||||
return { messageId: extractMessageId(parsed) };
|
||||
} catch {
|
||||
return { messageId: "ok" };
|
||||
}
|
||||
}
|
||||
|
||||
export async function sendMessageBlueBubbles(
|
||||
to: string,
|
||||
text: string,
|
||||
@@ -297,6 +352,17 @@ export async function sendMessageBlueBubbles(
|
||||
target,
|
||||
});
|
||||
if (!chatGuid) {
|
||||
// If target is a phone number/handle and no existing chat found,
|
||||
// auto-create a new DM chat using the /api/v1/chat/new endpoint
|
||||
if (target.kind === "handle") {
|
||||
return createNewChatWithMessage({
|
||||
baseUrl,
|
||||
password,
|
||||
address: target.address,
|
||||
message: trimmedText,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
});
|
||||
}
|
||||
throw new Error(
|
||||
"BlueBubbles send failed: chatGuid not found for target. Use a chat_guid target or ensure the chat exists.",
|
||||
);
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# Changelog
|
||||
|
||||
## 2026.1.24
|
||||
|
||||
### Changes
|
||||
- Breaking: voice-call TTS now uses core `messages.tts` (plugin TTS config deep‑merges with core).
|
||||
- Telephony TTS supports OpenAI + ElevenLabs; Edge TTS is ignored for calls.
|
||||
- Removed legacy `tts.model`/`tts.voice`/`tts.instructions` plugin fields.
|
||||
|
||||
## 2026.1.23
|
||||
|
||||
### Changes
|
||||
|
||||
@@ -75,6 +75,27 @@ Notes:
|
||||
- Twilio/Telnyx/Plivo require a **publicly reachable** webhook URL.
|
||||
- `mock` is a local dev provider (no network calls).
|
||||
|
||||
## TTS for calls
|
||||
|
||||
Voice Call uses the core `messages.tts` configuration (OpenAI or ElevenLabs) for
|
||||
streaming speech on calls. You can override it under the plugin config with the
|
||||
same shape — overrides deep-merge with `messages.tts`.
|
||||
|
||||
```json5
|
||||
{
|
||||
tts: {
|
||||
provider: "openai",
|
||||
openai: {
|
||||
voice: "alloy"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- Edge TTS is ignored for voice calls (telephony audio needs PCM; Edge output is unreliable).
|
||||
- Core TTS is used when Twilio media streaming is enabled; otherwise calls fall back to provider native voices.
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
|
||||
@@ -99,16 +99,39 @@
|
||||
"label": "Media Stream Path",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.model": {
|
||||
"label": "TTS Model",
|
||||
"tts.provider": {
|
||||
"label": "TTS Provider Override",
|
||||
"help": "Deep-merges with messages.tts (Edge is ignored for calls).",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.voice": {
|
||||
"label": "TTS Voice",
|
||||
"tts.openai.model": {
|
||||
"label": "OpenAI TTS Model",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.instructions": {
|
||||
"label": "TTS Instructions",
|
||||
"tts.openai.voice": {
|
||||
"label": "OpenAI TTS Voice",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.openai.apiKey": {
|
||||
"label": "OpenAI API Key",
|
||||
"sensitive": true,
|
||||
"advanced": true
|
||||
},
|
||||
"tts.elevenlabs.modelId": {
|
||||
"label": "ElevenLabs Model ID",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.elevenlabs.voiceId": {
|
||||
"label": "ElevenLabs Voice ID",
|
||||
"advanced": true
|
||||
},
|
||||
"tts.elevenlabs.apiKey": {
|
||||
"label": "ElevenLabs API Key",
|
||||
"sensitive": true,
|
||||
"advanced": true
|
||||
},
|
||||
"tts.elevenlabs.baseUrl": {
|
||||
"label": "ElevenLabs Base URL",
|
||||
"advanced": true
|
||||
},
|
||||
"publicUrl": {
|
||||
@@ -370,20 +393,193 @@
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"auto": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"off",
|
||||
"always",
|
||||
"inbound",
|
||||
"tagged"
|
||||
]
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"final",
|
||||
"all"
|
||||
]
|
||||
},
|
||||
"provider": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"openai"
|
||||
"openai",
|
||||
"elevenlabs",
|
||||
"edge"
|
||||
]
|
||||
},
|
||||
"model": {
|
||||
"summaryModel": {
|
||||
"type": "string"
|
||||
},
|
||||
"voice": {
|
||||
"modelOverrides": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowText": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowProvider": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowVoice": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowModelId": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowVoiceSettings": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowNormalization": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"allowSeed": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"elevenlabs": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"apiKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"baseUrl": {
|
||||
"type": "string"
|
||||
},
|
||||
"voiceId": {
|
||||
"type": "string"
|
||||
},
|
||||
"modelId": {
|
||||
"type": "string"
|
||||
},
|
||||
"seed": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"applyTextNormalization": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"auto",
|
||||
"on",
|
||||
"off"
|
||||
]
|
||||
},
|
||||
"languageCode": {
|
||||
"type": "string"
|
||||
},
|
||||
"voiceSettings": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"stability": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"similarityBoost": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"style": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"useSpeakerBoost": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"speed": {
|
||||
"type": "number",
|
||||
"minimum": 0.5,
|
||||
"maximum": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"openai": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"apiKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"voice": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"edge": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"voice": {
|
||||
"type": "string"
|
||||
},
|
||||
"lang": {
|
||||
"type": "string"
|
||||
},
|
||||
"outputFormat": {
|
||||
"type": "string"
|
||||
},
|
||||
"pitch": {
|
||||
"type": "string"
|
||||
},
|
||||
"rate": {
|
||||
"type": "string"
|
||||
},
|
||||
"volume": {
|
||||
"type": "string"
|
||||
},
|
||||
"saveSubtitles": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"proxy": {
|
||||
"type": "string"
|
||||
},
|
||||
"timeoutMs": {
|
||||
"type": "integer",
|
||||
"minimum": 1000,
|
||||
"maximum": 120000
|
||||
}
|
||||
}
|
||||
},
|
||||
"prefsPath": {
|
||||
"type": "string"
|
||||
},
|
||||
"instructions": {
|
||||
"type": "string"
|
||||
"maxTextLength": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"timeoutMs": {
|
||||
"type": "integer",
|
||||
"minimum": 1000,
|
||||
"maximum": 120000
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -74,9 +74,26 @@ const voiceCallConfigSchema = {
|
||||
},
|
||||
"streaming.sttModel": { label: "Realtime STT Model", advanced: true },
|
||||
"streaming.streamPath": { label: "Media Stream Path", advanced: true },
|
||||
"tts.model": { label: "TTS Model", advanced: true },
|
||||
"tts.voice": { label: "TTS Voice", advanced: true },
|
||||
"tts.instructions": { label: "TTS Instructions", advanced: true },
|
||||
"tts.provider": {
|
||||
label: "TTS Provider Override",
|
||||
help: "Deep-merges with messages.tts (Edge is ignored for calls).",
|
||||
advanced: true,
|
||||
},
|
||||
"tts.openai.model": { label: "OpenAI TTS Model", advanced: true },
|
||||
"tts.openai.voice": { label: "OpenAI TTS Voice", advanced: true },
|
||||
"tts.openai.apiKey": {
|
||||
label: "OpenAI API Key",
|
||||
sensitive: true,
|
||||
advanced: true,
|
||||
},
|
||||
"tts.elevenlabs.modelId": { label: "ElevenLabs Model ID", advanced: true },
|
||||
"tts.elevenlabs.voiceId": { label: "ElevenLabs Voice ID", advanced: true },
|
||||
"tts.elevenlabs.apiKey": {
|
||||
label: "ElevenLabs API Key",
|
||||
sensitive: true,
|
||||
advanced: true,
|
||||
},
|
||||
"tts.elevenlabs.baseUrl": { label: "ElevenLabs Base URL", advanced: true },
|
||||
publicUrl: { label: "Public Webhook URL", advanced: true },
|
||||
skipSignatureVerification: {
|
||||
label: "Skip Signature Verification",
|
||||
@@ -161,6 +178,7 @@ const voiceCallPlugin = {
|
||||
runtimePromise = createVoiceCallRuntime({
|
||||
config: cfg,
|
||||
coreConfig: api.config as CoreConfig,
|
||||
ttsRuntime: api.runtime.tts,
|
||||
logger: api.logger,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -82,31 +82,82 @@ export const SttConfigSchema = z
|
||||
.default({ provider: "openai", model: "whisper-1" });
|
||||
export type SttConfig = z.infer<typeof SttConfigSchema>;
|
||||
|
||||
export const TtsProviderSchema = z.enum(["openai", "elevenlabs", "edge"]);
|
||||
export const TtsModeSchema = z.enum(["final", "all"]);
|
||||
export const TtsAutoSchema = z.enum(["off", "always", "inbound", "tagged"]);
|
||||
|
||||
export const TtsConfigSchema = z
|
||||
.object({
|
||||
/** TTS provider (currently only OpenAI supported) */
|
||||
provider: z.literal("openai").default("openai"),
|
||||
/**
|
||||
* TTS model to use:
|
||||
* - gpt-4o-mini-tts: newest, supports instructions for tone/style control (recommended)
|
||||
* - tts-1: lower latency
|
||||
* - tts-1-hd: higher quality
|
||||
*/
|
||||
model: z.string().min(1).default("gpt-4o-mini-tts"),
|
||||
/**
|
||||
* Voice ID. For best quality, use marin or cedar.
|
||||
* All voices: alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar
|
||||
*/
|
||||
voice: z.string().min(1).default("coral"),
|
||||
/**
|
||||
* Instructions for speech style (only works with gpt-4o-mini-tts).
|
||||
* Examples: "Speak in a cheerful tone", "Talk like a sympathetic customer service agent"
|
||||
*/
|
||||
instructions: z.string().optional(),
|
||||
auto: TtsAutoSchema.optional(),
|
||||
enabled: z.boolean().optional(),
|
||||
mode: TtsModeSchema.optional(),
|
||||
provider: TtsProviderSchema.optional(),
|
||||
summaryModel: z.string().optional(),
|
||||
modelOverrides: z
|
||||
.object({
|
||||
enabled: z.boolean().optional(),
|
||||
allowText: z.boolean().optional(),
|
||||
allowProvider: z.boolean().optional(),
|
||||
allowVoice: z.boolean().optional(),
|
||||
allowModelId: z.boolean().optional(),
|
||||
allowVoiceSettings: z.boolean().optional(),
|
||||
allowNormalization: z.boolean().optional(),
|
||||
allowSeed: z.boolean().optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
elevenlabs: z
|
||||
.object({
|
||||
apiKey: z.string().optional(),
|
||||
baseUrl: z.string().optional(),
|
||||
voiceId: z.string().optional(),
|
||||
modelId: z.string().optional(),
|
||||
seed: z.number().int().min(0).max(4294967295).optional(),
|
||||
applyTextNormalization: z.enum(["auto", "on", "off"]).optional(),
|
||||
languageCode: z.string().optional(),
|
||||
voiceSettings: z
|
||||
.object({
|
||||
stability: z.number().min(0).max(1).optional(),
|
||||
similarityBoost: z.number().min(0).max(1).optional(),
|
||||
style: z.number().min(0).max(1).optional(),
|
||||
useSpeakerBoost: z.boolean().optional(),
|
||||
speed: z.number().min(0.5).max(2).optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
openai: z
|
||||
.object({
|
||||
apiKey: z.string().optional(),
|
||||
model: z.string().optional(),
|
||||
voice: z.string().optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
edge: z
|
||||
.object({
|
||||
enabled: z.boolean().optional(),
|
||||
voice: z.string().optional(),
|
||||
lang: z.string().optional(),
|
||||
outputFormat: z.string().optional(),
|
||||
pitch: z.string().optional(),
|
||||
rate: z.string().optional(),
|
||||
volume: z.string().optional(),
|
||||
saveSubtitles: z.boolean().optional(),
|
||||
proxy: z.string().optional(),
|
||||
timeoutMs: z.number().int().min(1000).max(120000).optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
prefsPath: z.string().optional(),
|
||||
maxTextLength: z.number().int().min(1).optional(),
|
||||
timeoutMs: z.number().int().min(1000).max(120000).optional(),
|
||||
})
|
||||
.strict()
|
||||
.default({ provider: "openai", model: "gpt-4o-mini-tts", voice: "coral" });
|
||||
export type TtsConfig = z.infer<typeof TtsConfigSchema>;
|
||||
.optional();
|
||||
export type VoiceCallTtsConfig = z.infer<typeof TtsConfigSchema>;
|
||||
|
||||
// -----------------------------------------------------------------------------
|
||||
// Webhook Server Configuration
|
||||
@@ -307,7 +358,7 @@ export const VoiceCallConfigSchema = z
|
||||
/** STT configuration */
|
||||
stt: SttConfigSchema,
|
||||
|
||||
/** TTS configuration */
|
||||
/** TTS override (deep-merges with core messages.tts) */
|
||||
tts: TtsConfigSchema,
|
||||
|
||||
/** Store path for call logs */
|
||||
|
||||
@@ -2,10 +2,16 @@ import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
import type { VoiceCallTtsConfig } from "./config.js";
|
||||
|
||||
export type CoreConfig = {
|
||||
session?: {
|
||||
store?: string;
|
||||
};
|
||||
messages?: {
|
||||
tts?: VoiceCallTtsConfig;
|
||||
};
|
||||
[key: string]: unknown;
|
||||
};
|
||||
|
||||
type CoreAgentDeps = {
|
||||
|
||||
@@ -143,7 +143,7 @@ export class CallManager {
|
||||
// For notify mode with a message, use inline TwiML with <Say>
|
||||
let inlineTwiml: string | undefined;
|
||||
if (mode === "notify" && initialMessage) {
|
||||
const pollyVoice = mapVoiceToPolly(this.config.tts.voice);
|
||||
const pollyVoice = mapVoiceToPolly(this.config.tts?.openai?.voice);
|
||||
inlineTwiml = this.generateNotifyTwiml(initialMessage, pollyVoice);
|
||||
console.log(
|
||||
`[voice-call] Using inline TwiML for notify mode (voice: ${pollyVoice})`,
|
||||
@@ -210,11 +210,13 @@ export class CallManager {
|
||||
this.addTranscriptEntry(call, "bot", text);
|
||||
|
||||
// Play TTS
|
||||
const voice =
|
||||
this.provider?.name === "twilio" ? this.config.tts?.openai?.voice : undefined;
|
||||
await this.provider.playTts({
|
||||
callId,
|
||||
providerCallId: call.providerCallId,
|
||||
text,
|
||||
voice: this.config.tts.voice,
|
||||
voice,
|
||||
});
|
||||
|
||||
return { success: true };
|
||||
|
||||
@@ -68,7 +68,7 @@ export async function initiateCall(
|
||||
// For notify mode with a message, use inline TwiML with <Say>.
|
||||
let inlineTwiml: string | undefined;
|
||||
if (mode === "notify" && initialMessage) {
|
||||
const pollyVoice = mapVoiceToPolly(ctx.config.tts.voice);
|
||||
const pollyVoice = mapVoiceToPolly(ctx.config.tts?.openai?.voice);
|
||||
inlineTwiml = generateNotifyTwiml(initialMessage, pollyVoice);
|
||||
console.log(`[voice-call] Using inline TwiML for notify mode (voice: ${pollyVoice})`);
|
||||
}
|
||||
@@ -120,11 +120,13 @@ export async function speak(
|
||||
|
||||
addTranscriptEntry(call, "bot", text);
|
||||
|
||||
const voice =
|
||||
ctx.provider?.name === "twilio" ? ctx.config.tts?.openai?.voice : undefined;
|
||||
await ctx.provider.playTts({
|
||||
callId,
|
||||
providerCallId: call.providerCallId,
|
||||
text,
|
||||
voice: ctx.config.tts.voice,
|
||||
voice,
|
||||
});
|
||||
|
||||
return { success: true };
|
||||
@@ -244,4 +246,3 @@ export async function endCall(
|
||||
return { success: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,9 +15,9 @@ import type {
|
||||
WebhookVerificationResult,
|
||||
} from "../types.js";
|
||||
import { escapeXml, mapVoiceToPolly } from "../voice-mapping.js";
|
||||
import { chunkAudio } from "../telephony-audio.js";
|
||||
import type { TelephonyTtsProvider } from "../telephony-tts.js";
|
||||
import type { VoiceCallProvider } from "./base.js";
|
||||
import type { OpenAITTSProvider } from "./tts-openai.js";
|
||||
import { chunkAudio } from "./tts-openai.js";
|
||||
import { twilioApiRequest } from "./twilio/api.js";
|
||||
import { verifyTwilioProviderWebhook } from "./twilio/webhook.js";
|
||||
|
||||
@@ -53,8 +53,8 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
/** Current public webhook URL (set when tunnel starts or from config) */
|
||||
private currentPublicUrl: string | null = null;
|
||||
|
||||
/** Optional OpenAI TTS provider for streaming TTS */
|
||||
private ttsProvider: OpenAITTSProvider | null = null;
|
||||
/** Optional telephony TTS provider for streaming TTS */
|
||||
private ttsProvider: TelephonyTtsProvider | null = null;
|
||||
|
||||
/** Optional media stream handler for sending audio */
|
||||
private mediaStreamHandler: MediaStreamHandler | null = null;
|
||||
@@ -119,7 +119,7 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
return this.currentPublicUrl;
|
||||
}
|
||||
|
||||
setTTSProvider(provider: OpenAITTSProvider): void {
|
||||
setTTSProvider(provider: TelephonyTtsProvider): void {
|
||||
this.ttsProvider = provider;
|
||||
}
|
||||
|
||||
@@ -454,13 +454,13 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
* Play TTS audio via Twilio.
|
||||
*
|
||||
* Two modes:
|
||||
* 1. OpenAI TTS + Media Streams: If TTS provider and media stream are available,
|
||||
* generates audio via OpenAI and streams it through WebSocket (preferred).
|
||||
* 1. Core TTS + Media Streams: If TTS provider and media stream are available,
|
||||
* generates audio via core TTS and streams it through WebSocket (preferred).
|
||||
* 2. TwiML <Say>: Falls back to Twilio's native TTS with Polly voices.
|
||||
* Note: This may not work on all Twilio accounts.
|
||||
*/
|
||||
async playTts(input: PlayTtsInput): Promise<void> {
|
||||
// Try OpenAI TTS via media stream first (if configured)
|
||||
// Try telephony TTS via media stream first (if configured)
|
||||
const streamSid = this.callStreamMap.get(input.providerCallId);
|
||||
if (this.ttsProvider && this.mediaStreamHandler && streamSid) {
|
||||
try {
|
||||
@@ -468,7 +468,7 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
return;
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[voice-call] OpenAI TTS failed, falling back to Twilio <Say>:`,
|
||||
`[voice-call] Telephony TTS failed, falling back to Twilio <Say>:`,
|
||||
err instanceof Error ? err.message : err,
|
||||
);
|
||||
// Fall through to TwiML <Say> fallback
|
||||
@@ -484,7 +484,7 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
}
|
||||
|
||||
console.warn(
|
||||
"[voice-call] Using TwiML <Say> fallback - OpenAI TTS not configured or media stream not active",
|
||||
"[voice-call] Using TwiML <Say> fallback - telephony TTS not configured or media stream not active",
|
||||
);
|
||||
|
||||
const pollyVoice = mapVoiceToPolly(input.voice);
|
||||
@@ -502,8 +502,8 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
}
|
||||
|
||||
/**
|
||||
* Play TTS via OpenAI and Twilio Media Streams.
|
||||
* Generates audio with OpenAI TTS, converts to mu-law, and streams via WebSocket.
|
||||
* Play TTS via core TTS and Twilio Media Streams.
|
||||
* Generates audio with core TTS, converts to mu-law, and streams via WebSocket.
|
||||
* Uses a jitter buffer to smooth out timing variations.
|
||||
*/
|
||||
private async playTtsViaStream(
|
||||
@@ -514,8 +514,8 @@ export class TwilioProvider implements VoiceCallProvider {
|
||||
throw new Error("TTS provider and media stream handler required");
|
||||
}
|
||||
|
||||
// Generate audio with OpenAI TTS (returns mu-law at 8kHz)
|
||||
const muLawAudio = await this.ttsProvider.synthesizeForTwilio(text);
|
||||
// Generate audio with core TTS (returns mu-law at 8kHz)
|
||||
const muLawAudio = await this.ttsProvider.synthesizeForTelephony(text);
|
||||
|
||||
// Stream audio in 20ms chunks (160 bytes at 8kHz mu-law)
|
||||
const CHUNK_SIZE = 160;
|
||||
|
||||
@@ -6,8 +6,9 @@ import type { VoiceCallProvider } from "./providers/base.js";
|
||||
import { MockProvider } from "./providers/mock.js";
|
||||
import { PlivoProvider } from "./providers/plivo.js";
|
||||
import { TelnyxProvider } from "./providers/telnyx.js";
|
||||
import { OpenAITTSProvider } from "./providers/tts-openai.js";
|
||||
import { TwilioProvider } from "./providers/twilio.js";
|
||||
import type { TelephonyTtsRuntime } from "./telephony-tts.js";
|
||||
import { createTelephonyTtsProvider } from "./telephony-tts.js";
|
||||
import { startTunnel, type TunnelResult } from "./tunnel.js";
|
||||
import {
|
||||
cleanupTailscaleExposure,
|
||||
@@ -81,9 +82,10 @@ function resolveProvider(config: VoiceCallConfig): VoiceCallProvider {
|
||||
export async function createVoiceCallRuntime(params: {
|
||||
config: VoiceCallConfig;
|
||||
coreConfig: CoreConfig;
|
||||
ttsRuntime?: TelephonyTtsRuntime;
|
||||
logger?: Logger;
|
||||
}): Promise<VoiceCallRuntime> {
|
||||
const { config, coreConfig, logger } = params;
|
||||
const { config, coreConfig, ttsRuntime, logger } = params;
|
||||
const log = logger ?? {
|
||||
info: console.log,
|
||||
warn: console.warn,
|
||||
@@ -149,27 +151,24 @@ export async function createVoiceCallRuntime(params: {
|
||||
|
||||
if (provider.name === "twilio" && config.streaming?.enabled) {
|
||||
const twilioProvider = provider as TwilioProvider;
|
||||
const openaiApiKey =
|
||||
config.streaming.openaiApiKey || process.env.OPENAI_API_KEY;
|
||||
if (openaiApiKey) {
|
||||
if (ttsRuntime?.textToSpeechTelephony) {
|
||||
try {
|
||||
const ttsProvider = new OpenAITTSProvider({
|
||||
apiKey: openaiApiKey,
|
||||
voice: config.tts.voice,
|
||||
model: config.tts.model,
|
||||
instructions: config.tts.instructions,
|
||||
const ttsProvider = createTelephonyTtsProvider({
|
||||
coreConfig,
|
||||
ttsOverride: config.tts,
|
||||
runtime: ttsRuntime,
|
||||
});
|
||||
twilioProvider.setTTSProvider(ttsProvider);
|
||||
log.info("[voice-call] OpenAI TTS provider configured");
|
||||
log.info("[voice-call] Telephony TTS provider configured");
|
||||
} catch (err) {
|
||||
log.warn(
|
||||
`[voice-call] Failed to initialize OpenAI TTS: ${
|
||||
`[voice-call] Failed to initialize telephony TTS: ${
|
||||
err instanceof Error ? err.message : String(err)
|
||||
}`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
log.warn("[voice-call] OpenAI TTS key missing; streaming TTS disabled");
|
||||
log.warn("[voice-call] Telephony TTS unavailable; streaming TTS disabled");
|
||||
}
|
||||
|
||||
const mediaHandler = webhookServer.getMediaStreamHandler();
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
const TELEPHONY_SAMPLE_RATE = 8000;
|
||||
|
||||
function clamp16(value: number): number {
|
||||
return Math.max(-32768, Math.min(32767, value));
|
||||
}
|
||||
|
||||
/**
|
||||
* Resample 16-bit PCM (little-endian mono) to 8kHz using linear interpolation.
|
||||
*/
|
||||
export function resamplePcmTo8k(input: Buffer, inputSampleRate: number): Buffer {
|
||||
if (inputSampleRate === TELEPHONY_SAMPLE_RATE) return input;
|
||||
const inputSamples = Math.floor(input.length / 2);
|
||||
if (inputSamples === 0) return Buffer.alloc(0);
|
||||
|
||||
const ratio = inputSampleRate / TELEPHONY_SAMPLE_RATE;
|
||||
const outputSamples = Math.floor(inputSamples / ratio);
|
||||
const output = Buffer.alloc(outputSamples * 2);
|
||||
|
||||
for (let i = 0; i < outputSamples; i++) {
|
||||
const srcPos = i * ratio;
|
||||
const srcIndex = Math.floor(srcPos);
|
||||
const frac = srcPos - srcIndex;
|
||||
|
||||
const s0 = input.readInt16LE(srcIndex * 2);
|
||||
const s1Index = Math.min(srcIndex + 1, inputSamples - 1);
|
||||
const s1 = input.readInt16LE(s1Index * 2);
|
||||
|
||||
const sample = Math.round(s0 + frac * (s1 - s0));
|
||||
output.writeInt16LE(clamp16(sample), i * 2);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert 16-bit PCM to 8-bit mu-law (G.711).
|
||||
*/
|
||||
export function pcmToMulaw(pcm: Buffer): Buffer {
|
||||
const samples = Math.floor(pcm.length / 2);
|
||||
const mulaw = Buffer.alloc(samples);
|
||||
|
||||
for (let i = 0; i < samples; i++) {
|
||||
const sample = pcm.readInt16LE(i * 2);
|
||||
mulaw[i] = linearToMulaw(sample);
|
||||
}
|
||||
|
||||
return mulaw;
|
||||
}
|
||||
|
||||
export function convertPcmToMulaw8k(
|
||||
pcm: Buffer,
|
||||
inputSampleRate: number,
|
||||
): Buffer {
|
||||
const pcm8k = resamplePcmTo8k(pcm, inputSampleRate);
|
||||
return pcmToMulaw(pcm8k);
|
||||
}
|
||||
|
||||
/**
|
||||
* Chunk audio buffer into 20ms frames for streaming (8kHz mono mu-law).
|
||||
*/
|
||||
export function chunkAudio(
|
||||
audio: Buffer,
|
||||
chunkSize = 160,
|
||||
): Generator<Buffer, void, unknown> {
|
||||
return (function* () {
|
||||
for (let i = 0; i < audio.length; i += chunkSize) {
|
||||
yield audio.subarray(i, Math.min(i + chunkSize, audio.length));
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
function linearToMulaw(sample: number): number {
|
||||
const BIAS = 132;
|
||||
const CLIP = 32635;
|
||||
|
||||
const sign = sample < 0 ? 0x80 : 0;
|
||||
if (sample < 0) sample = -sample;
|
||||
if (sample > CLIP) sample = CLIP;
|
||||
|
||||
sample += BIAS;
|
||||
let exponent = 7;
|
||||
for (let expMask = 0x4000; (sample & expMask) === 0 && exponent > 0; exponent--) {
|
||||
expMask >>= 1;
|
||||
}
|
||||
|
||||
const mantissa = (sample >> (exponent + 3)) & 0x0f;
|
||||
return ~(sign | (exponent << 4) | mantissa) & 0xff;
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import type { CoreConfig } from "./core-bridge.js";
|
||||
import type { VoiceCallTtsConfig } from "./config.js";
|
||||
import { convertPcmToMulaw8k } from "./telephony-audio.js";
|
||||
|
||||
export type TelephonyTtsRuntime = {
|
||||
textToSpeechTelephony: (params: {
|
||||
text: string;
|
||||
cfg: CoreConfig;
|
||||
prefsPath?: string;
|
||||
}) => Promise<{
|
||||
success: boolean;
|
||||
audioBuffer?: Buffer;
|
||||
sampleRate?: number;
|
||||
provider?: string;
|
||||
error?: string;
|
||||
}>;
|
||||
};
|
||||
|
||||
export type TelephonyTtsProvider = {
|
||||
synthesizeForTelephony: (text: string) => Promise<Buffer>;
|
||||
};
|
||||
|
||||
export function createTelephonyTtsProvider(params: {
|
||||
coreConfig: CoreConfig;
|
||||
ttsOverride?: VoiceCallTtsConfig;
|
||||
runtime: TelephonyTtsRuntime;
|
||||
}): TelephonyTtsProvider {
|
||||
const { coreConfig, ttsOverride, runtime } = params;
|
||||
const mergedConfig = applyTtsOverride(coreConfig, ttsOverride);
|
||||
|
||||
return {
|
||||
synthesizeForTelephony: async (text: string) => {
|
||||
const result = await runtime.textToSpeechTelephony({
|
||||
text,
|
||||
cfg: mergedConfig,
|
||||
});
|
||||
|
||||
if (!result.success || !result.audioBuffer || !result.sampleRate) {
|
||||
throw new Error(result.error ?? "TTS conversion failed");
|
||||
}
|
||||
|
||||
return convertPcmToMulaw8k(result.audioBuffer, result.sampleRate);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function applyTtsOverride(
|
||||
coreConfig: CoreConfig,
|
||||
override?: VoiceCallTtsConfig,
|
||||
): CoreConfig {
|
||||
if (!override) return coreConfig;
|
||||
|
||||
const base = coreConfig.messages?.tts;
|
||||
const merged = mergeTtsConfig(base, override);
|
||||
if (!merged) return coreConfig;
|
||||
|
||||
return {
|
||||
...coreConfig,
|
||||
messages: {
|
||||
...(coreConfig.messages ?? {}),
|
||||
tts: merged,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function mergeTtsConfig(
|
||||
base?: VoiceCallTtsConfig,
|
||||
override?: VoiceCallTtsConfig,
|
||||
): VoiceCallTtsConfig | undefined {
|
||||
if (!base && !override) return undefined;
|
||||
if (!override) return base;
|
||||
if (!base) return override;
|
||||
return deepMerge(base, override);
|
||||
}
|
||||
|
||||
function deepMerge<T>(base: T, override: T): T {
|
||||
if (!isPlainObject(base) || !isPlainObject(override)) {
|
||||
return override;
|
||||
}
|
||||
const result: Record<string, unknown> = { ...base };
|
||||
for (const [key, value] of Object.entries(override)) {
|
||||
if (value === undefined) continue;
|
||||
const existing = (base as Record<string, unknown>)[key];
|
||||
if (isPlainObject(existing) && isPlainObject(value)) {
|
||||
result[key] = deepMerge(existing, value);
|
||||
} else {
|
||||
result[key] = value;
|
||||
}
|
||||
}
|
||||
return result as T;
|
||||
}
|
||||
|
||||
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
@@ -333,7 +333,13 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
|
||||
name: "message",
|
||||
description,
|
||||
parameters: schema,
|
||||
execute: async (_toolCallId, args) => {
|
||||
execute: async (_toolCallId, args, signal) => {
|
||||
// Check if already aborted before doing any work
|
||||
if (signal?.aborted) {
|
||||
const err = new Error("Message send aborted");
|
||||
err.name = "AbortError";
|
||||
throw err;
|
||||
}
|
||||
const params = args as Record<string, unknown>;
|
||||
const cfg = options?.config ?? loadConfig();
|
||||
const action = readStringParam(params, "action", {
|
||||
@@ -366,6 +372,9 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
|
||||
currentThreadTs: options?.currentThreadTs,
|
||||
replyToMode: options?.replyToMode,
|
||||
hasRepliedRef: options?.hasRepliedRef,
|
||||
// Direct tool invocations should not add cross-context decoration.
|
||||
// The agent is composing a message, not forwarding from another chat.
|
||||
skipCrossContextDecoration: true,
|
||||
}
|
||||
: undefined;
|
||||
|
||||
@@ -379,6 +388,7 @@ export function createMessageTool(options?: MessageToolOptions): AnyAgentTool {
|
||||
agentId: options?.agentSessionKey
|
||||
? resolveSessionAgentId({ sessionKey: options.agentSessionKey, config: cfg })
|
||||
: undefined,
|
||||
abortSignal: signal,
|
||||
});
|
||||
|
||||
const toolResult = getToolResult(result);
|
||||
|
||||
@@ -240,6 +240,12 @@ export type ChannelThreadingToolContext = {
|
||||
currentThreadTs?: string;
|
||||
replyToMode?: "off" | "first" | "all";
|
||||
hasRepliedRef?: { value: boolean };
|
||||
/**
|
||||
* When true, skip cross-context decoration (e.g., "[from X]" prefix).
|
||||
* Use this for direct tool invocations where the agent is composing a new message,
|
||||
* not forwarding/relaying a message from another conversation.
|
||||
*/
|
||||
skipCrossContextDecoration?: boolean;
|
||||
};
|
||||
|
||||
export type ChannelMessagingAdapter = {
|
||||
|
||||
@@ -59,6 +59,17 @@ export const CONFIG_PATH_CLAWDBOT = resolveConfigPath();
|
||||
|
||||
export const DEFAULT_GATEWAY_PORT = 18789;
|
||||
|
||||
/**
|
||||
* Gateway lock directory (ephemeral).
|
||||
* Default: os.tmpdir()/clawdbot-<uid> (uid suffix when available).
|
||||
*/
|
||||
export function resolveGatewayLockDir(tmpdir: () => string = os.tmpdir): string {
|
||||
const base = tmpdir();
|
||||
const uid = typeof process.getuid === "function" ? process.getuid() : undefined;
|
||||
const suffix = uid != null ? `clawdbot-${uid}` : "clawdbot";
|
||||
return path.join(base, suffix);
|
||||
}
|
||||
|
||||
const OAUTH_FILENAME = "oauth.json";
|
||||
|
||||
/**
|
||||
|
||||
@@ -7,12 +7,13 @@ import path from "node:path";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { acquireGatewayLock, GatewayLockError } from "./gateway-lock.js";
|
||||
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
|
||||
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
|
||||
|
||||
async function makeEnv() {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-gateway-lock-"));
|
||||
const configPath = path.join(dir, "clawdbot.json");
|
||||
await fs.writeFile(configPath, "{}", "utf8");
|
||||
await fs.mkdir(resolveGatewayLockDir(), { recursive: true });
|
||||
return {
|
||||
env: {
|
||||
...process.env,
|
||||
@@ -29,7 +30,8 @@ function resolveLockPath(env: NodeJS.ProcessEnv) {
|
||||
const stateDir = resolveStateDir(env);
|
||||
const configPath = resolveConfigPath(env, stateDir);
|
||||
const hash = createHash("sha1").update(configPath).digest("hex").slice(0, 8);
|
||||
return { lockPath: path.join(stateDir, `gateway.${hash}.lock`), configPath };
|
||||
const lockDir = resolveGatewayLockDir();
|
||||
return { lockPath: path.join(lockDir, `gateway.${hash}.lock`), configPath };
|
||||
}
|
||||
|
||||
function makeProcStat(pid: number, startTime: number) {
|
||||
|
||||
@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
|
||||
import fsSync from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
|
||||
import { resolveConfigPath, resolveGatewayLockDir, resolveStateDir } from "../config/paths.js";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 5000;
|
||||
const DEFAULT_POLL_INTERVAL_MS = 100;
|
||||
@@ -150,7 +150,8 @@ function resolveGatewayLockPath(env: NodeJS.ProcessEnv) {
|
||||
const stateDir = resolveStateDir(env);
|
||||
const configPath = resolveConfigPath(env, stateDir);
|
||||
const hash = createHash("sha1").update(configPath).digest("hex").slice(0, 8);
|
||||
const lockPath = path.join(stateDir, `gateway.${hash}.lock`);
|
||||
const lockDir = resolveGatewayLockDir();
|
||||
const lockPath = path.join(lockDir, `gateway.${hash}.lock`);
|
||||
return { lockPath, configPath };
|
||||
}
|
||||
|
||||
|
||||
@@ -321,6 +321,44 @@ describe("runMessageAction context isolation", () => {
|
||||
}),
|
||||
).rejects.toThrow(/Cross-context messaging denied/);
|
||||
});
|
||||
|
||||
it("aborts send when abortSignal is already aborted", async () => {
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
|
||||
await expect(
|
||||
runMessageAction({
|
||||
cfg: slackConfig,
|
||||
action: "send",
|
||||
params: {
|
||||
channel: "slack",
|
||||
target: "#C12345678",
|
||||
message: "hi",
|
||||
},
|
||||
dryRun: true,
|
||||
abortSignal: controller.signal,
|
||||
}),
|
||||
).rejects.toMatchObject({ name: "AbortError" });
|
||||
});
|
||||
|
||||
it("aborts broadcast when abortSignal is already aborted", async () => {
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
|
||||
await expect(
|
||||
runMessageAction({
|
||||
cfg: slackConfig,
|
||||
action: "broadcast",
|
||||
params: {
|
||||
targets: ["channel:C12345678"],
|
||||
channel: "slack",
|
||||
message: "hi",
|
||||
},
|
||||
dryRun: true,
|
||||
abortSignal: controller.signal,
|
||||
}),
|
||||
).rejects.toMatchObject({ name: "AbortError" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("runMessageAction sendAttachment hydration", () => {
|
||||
|
||||
@@ -64,6 +64,7 @@ export type RunMessageActionParams = {
|
||||
sessionKey?: string;
|
||||
agentId?: string;
|
||||
dryRun?: boolean;
|
||||
abortSignal?: AbortSignal;
|
||||
};
|
||||
|
||||
export type MessageActionRunResult =
|
||||
@@ -507,6 +508,7 @@ type ResolvedActionContext = {
|
||||
input: RunMessageActionParams;
|
||||
agentId?: string;
|
||||
resolvedTarget?: ResolvedMessagingTarget;
|
||||
abortSignal?: AbortSignal;
|
||||
};
|
||||
function resolveGateway(input: RunMessageActionParams): MessageActionRunnerGateway | undefined {
|
||||
if (!input.gateway) return undefined;
|
||||
@@ -524,6 +526,7 @@ async function handleBroadcastAction(
|
||||
input: RunMessageActionParams,
|
||||
params: Record<string, unknown>,
|
||||
): Promise<MessageActionRunResult> {
|
||||
throwIfAborted(input.abortSignal);
|
||||
const broadcastEnabled = input.cfg.tools?.message?.broadcast?.enabled !== false;
|
||||
if (!broadcastEnabled) {
|
||||
throw new Error("Broadcast is disabled. Set tools.message.broadcast.enabled to true.");
|
||||
@@ -548,8 +551,11 @@ async function handleBroadcastAction(
|
||||
error?: string;
|
||||
result?: MessageSendResult;
|
||||
}> = [];
|
||||
const isAbortError = (err: unknown): boolean => err instanceof Error && err.name === "AbortError";
|
||||
for (const targetChannel of targetChannels) {
|
||||
throwIfAborted(input.abortSignal);
|
||||
for (const target of rawTargets) {
|
||||
throwIfAborted(input.abortSignal);
|
||||
try {
|
||||
const resolved = await resolveChannelTarget({
|
||||
cfg: input.cfg,
|
||||
@@ -573,6 +579,7 @@ async function handleBroadcastAction(
|
||||
result: sendResult.kind === "send" ? sendResult.sendResult : undefined,
|
||||
});
|
||||
} catch (err) {
|
||||
if (isAbortError(err)) throw err;
|
||||
results.push({
|
||||
channel: targetChannel,
|
||||
to: target,
|
||||
@@ -592,8 +599,28 @@ async function handleBroadcastAction(
|
||||
};
|
||||
}
|
||||
|
||||
function throwIfAborted(abortSignal?: AbortSignal): void {
|
||||
if (abortSignal?.aborted) {
|
||||
const err = new Error("Message send aborted");
|
||||
err.name = "AbortError";
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
|
||||
const { cfg, params, channel, accountId, dryRun, gateway, input, agentId, resolvedTarget } = ctx;
|
||||
const {
|
||||
cfg,
|
||||
params,
|
||||
channel,
|
||||
accountId,
|
||||
dryRun,
|
||||
gateway,
|
||||
input,
|
||||
agentId,
|
||||
resolvedTarget,
|
||||
abortSignal,
|
||||
} = ctx;
|
||||
throwIfAborted(abortSignal);
|
||||
const action: ChannelMessageActionName = "send";
|
||||
const to = readStringParam(params, "to", { required: true });
|
||||
// Support media, path, and filePath parameters for attachments
|
||||
@@ -676,6 +703,7 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
|
||||
}
|
||||
const mirrorMediaUrls =
|
||||
mergedMediaUrls.length > 0 ? mergedMediaUrls : mediaUrl ? [mediaUrl] : undefined;
|
||||
throwIfAborted(abortSignal);
|
||||
const send = await executeSendAction({
|
||||
ctx: {
|
||||
cfg,
|
||||
@@ -695,6 +723,7 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
|
||||
mediaUrls: mirrorMediaUrls,
|
||||
}
|
||||
: undefined,
|
||||
abortSignal,
|
||||
},
|
||||
to,
|
||||
message,
|
||||
@@ -718,7 +747,8 @@ async function handleSendAction(ctx: ResolvedActionContext): Promise<MessageActi
|
||||
}
|
||||
|
||||
async function handlePollAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
|
||||
const { cfg, params, channel, accountId, dryRun, gateway, input } = ctx;
|
||||
const { cfg, params, channel, accountId, dryRun, gateway, input, abortSignal } = ctx;
|
||||
throwIfAborted(abortSignal);
|
||||
const action: ChannelMessageActionName = "poll";
|
||||
const to = readStringParam(params, "to", { required: true });
|
||||
const question = readStringParam(params, "pollQuestion", {
|
||||
@@ -777,7 +807,8 @@ async function handlePollAction(ctx: ResolvedActionContext): Promise<MessageActi
|
||||
}
|
||||
|
||||
async function handlePluginAction(ctx: ResolvedActionContext): Promise<MessageActionRunResult> {
|
||||
const { cfg, params, channel, accountId, dryRun, gateway, input } = ctx;
|
||||
const { cfg, params, channel, accountId, dryRun, gateway, input, abortSignal } = ctx;
|
||||
throwIfAborted(abortSignal);
|
||||
const action = input.action as Exclude<ChannelMessageActionName, "send" | "poll" | "broadcast">;
|
||||
if (dryRun) {
|
||||
return {
|
||||
@@ -930,6 +961,7 @@ export async function runMessageAction(
|
||||
input,
|
||||
agentId: resolvedAgentId,
|
||||
resolvedTarget,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -942,6 +974,7 @@ export async function runMessageAction(
|
||||
dryRun,
|
||||
gateway,
|
||||
input,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -953,5 +986,6 @@ export async function runMessageAction(
|
||||
dryRun,
|
||||
gateway,
|
||||
input,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
}
|
||||
@@ -50,6 +50,7 @@ type MessageSendParams = {
|
||||
text?: string;
|
||||
mediaUrls?: string[];
|
||||
};
|
||||
abortSignal?: AbortSignal;
|
||||
};
|
||||
|
||||
export type MessageSendResult = {
|
||||
@@ -167,6 +168,7 @@ export async function sendMessage(params: MessageSendParams): Promise<MessageSen
|
||||
gifPlayback: params.gifPlayback,
|
||||
deps: params.deps,
|
||||
bestEffort: params.bestEffort,
|
||||
abortSignal: params.abortSignal,
|
||||
mirror: params.mirror
|
||||
? {
|
||||
...params.mirror,
|
||||
|
||||
@@ -119,6 +119,8 @@ export async function buildCrossContextDecoration(params: {
|
||||
accountId?: string | null;
|
||||
}): Promise<CrossContextDecoration | null> {
|
||||
if (!params.toolContext?.currentChannelId) return null;
|
||||
// Skip decoration for direct tool sends (agent composing, not forwarding)
|
||||
if (params.toolContext.skipCrossContextDecoration) return null;
|
||||
if (!isCrossContextTarget(params)) return null;
|
||||
|
||||
const markerConfig = params.cfg.tools?.message?.crossContext?.marker;
|
||||
@@ -131,11 +133,11 @@ export async function buildCrossContextDecoration(params: {
|
||||
targetId: params.toolContext.currentChannelId,
|
||||
accountId: params.accountId ?? undefined,
|
||||
})) ?? params.toolContext.currentChannelId;
|
||||
// Don't force group formatting here; currentChannelId can be a DM or a group.
|
||||
const originLabel = formatTargetDisplay({
|
||||
channel: params.channel,
|
||||
target: params.toolContext.currentChannelId,
|
||||
display: currentName,
|
||||
kind: "group",
|
||||
});
|
||||
const prefixTemplate = markerConfig?.prefix ?? "[from {channel}] ";
|
||||
const suffixTemplate = markerConfig?.suffix ?? "";
|
||||
|
||||
@@ -32,6 +32,7 @@ export type OutboundSendContext = {
|
||||
text?: string;
|
||||
mediaUrls?: string[];
|
||||
};
|
||||
abortSignal?: AbortSignal;
|
||||
};
|
||||
|
||||
function extractToolPayload(result: AgentToolResult<unknown>): unknown {
|
||||
@@ -56,6 +57,14 @@ function extractToolPayload(result: AgentToolResult<unknown>): unknown {
|
||||
return result.content ?? result;
|
||||
}
|
||||
|
||||
function throwIfAborted(abortSignal?: AbortSignal): void {
|
||||
if (abortSignal?.aborted) {
|
||||
const err = new Error("Message send aborted");
|
||||
err.name = "AbortError";
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
export async function executeSendAction(params: {
|
||||
ctx: OutboundSendContext;
|
||||
to: string;
|
||||
@@ -70,6 +79,7 @@ export async function executeSendAction(params: {
|
||||
toolResult?: AgentToolResult<unknown>;
|
||||
sendResult?: MessageSendResult;
|
||||
}> {
|
||||
throwIfAborted(params.ctx.abortSignal);
|
||||
if (!params.ctx.dryRun) {
|
||||
const handled = await dispatchChannelMessageAction({
|
||||
channel: params.ctx.channel,
|
||||
@@ -103,6 +113,7 @@ export async function executeSendAction(params: {
|
||||
}
|
||||
}
|
||||
|
||||
throwIfAborted(params.ctx.abortSignal);
|
||||
const result: MessageSendResult = await sendMessage({
|
||||
cfg: params.ctx.cfg,
|
||||
to: params.to,
|
||||
@@ -117,6 +128,7 @@ export async function executeSendAction(params: {
|
||||
deps: params.ctx.deps,
|
||||
gateway: params.ctx.gateway,
|
||||
mirror: params.ctx.mirror,
|
||||
abortSignal: params.ctx.abortSignal,
|
||||
});
|
||||
|
||||
return {
|
||||
|
||||
@@ -100,7 +100,12 @@ export function formatTargetDisplay(params: {
|
||||
if (!trimmedTarget) return trimmedTarget;
|
||||
if (trimmedTarget.startsWith("#") || trimmedTarget.startsWith("@")) return trimmedTarget;
|
||||
|
||||
const withoutPrefix = trimmedTarget.replace(/^telegram:/i, "");
|
||||
const channelPrefix = `${params.channel}:`;
|
||||
const withoutProvider = trimmedTarget.toLowerCase().startsWith(channelPrefix)
|
||||
? trimmedTarget.slice(channelPrefix.length)
|
||||
: trimmedTarget;
|
||||
|
||||
const withoutPrefix = withoutProvider.replace(/^telegram:/i, "");
|
||||
if (/^channel:/i.test(withoutPrefix)) {
|
||||
return `#${withoutPrefix.replace(/^channel:/i, "")}`;
|
||||
}
|
||||
@@ -119,14 +124,23 @@ function preserveTargetCase(channel: ChannelId, raw: string, normalized: string)
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
function detectTargetKind(raw: string, preferred?: TargetResolveKind): TargetResolveKind {
|
||||
function detectTargetKind(
|
||||
channel: ChannelId,
|
||||
raw: string,
|
||||
preferred?: TargetResolveKind,
|
||||
): TargetResolveKind {
|
||||
if (preferred) return preferred;
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) return "group";
|
||||
|
||||
if (trimmed.startsWith("@") || /^<@!?/.test(trimmed) || /^user:/i.test(trimmed)) return "user";
|
||||
if (trimmed.startsWith("#") || /^channel:/i.test(trimmed)) {
|
||||
return "group";
|
||||
if (trimmed.startsWith("#") || /^channel:/i.test(trimmed)) return "group";
|
||||
|
||||
// For some channels (e.g., BlueBubbles/iMessage), bare phone numbers are almost always DM targets.
|
||||
if ((channel === "bluebubbles" || channel === "imessage") && /^\+?\d{6,}$/.test(trimmed)) {
|
||||
return "user";
|
||||
}
|
||||
|
||||
return "group";
|
||||
}
|
||||
|
||||
@@ -282,7 +296,7 @@ export async function resolveMessagingTarget(params: {
|
||||
const plugin = getChannelPlugin(params.channel);
|
||||
const providerLabel = plugin?.meta?.label ?? params.channel;
|
||||
const hint = plugin?.messaging?.targetResolver?.hint;
|
||||
const kind = detectTargetKind(raw, params.preferredKind);
|
||||
const kind = detectTargetKind(params.channel, raw, params.preferredKind);
|
||||
const normalized = normalizeTargetForProvider(params.channel, raw) ?? raw;
|
||||
const looksLikeTargetId = (): boolean => {
|
||||
const trimmed = raw.trim();
|
||||
@@ -291,7 +305,12 @@ export async function resolveMessagingTarget(params: {
|
||||
if (lookup) return lookup(trimmed, normalized);
|
||||
if (/^(channel|group|user):/i.test(trimmed)) return true;
|
||||
if (/^[@#]/.test(trimmed)) return true;
|
||||
if (/^\+?\d{6,}$/.test(trimmed)) return true;
|
||||
if (/^\+?\d{6,}$/.test(trimmed)) {
|
||||
// BlueBubbles/iMessage phone numbers should usually resolve via the directory to a DM chat,
|
||||
// otherwise the provider may pick an existing group containing that handle.
|
||||
if (params.channel === "bluebubbles" || params.channel === "imessage") return false;
|
||||
return true;
|
||||
}
|
||||
if (trimmed.includes("@thread")) return true;
|
||||
if (/^(conversation|user):/i.test(trimmed)) return true;
|
||||
return false;
|
||||
@@ -353,6 +372,24 @@ export async function resolveMessagingTarget(params: {
|
||||
candidates: match.entries,
|
||||
};
|
||||
}
|
||||
// For iMessage-style channels, allow sending directly to the normalized handle
|
||||
// even if the directory doesn't contain an entry yet.
|
||||
if (
|
||||
(params.channel === "bluebubbles" || params.channel === "imessage") &&
|
||||
/^\+?\d{6,}$/.test(query)
|
||||
) {
|
||||
const directTarget = preserveTargetCase(params.channel, raw, normalized);
|
||||
return {
|
||||
ok: true,
|
||||
target: {
|
||||
to: directTarget,
|
||||
kind,
|
||||
display: stripTargetPrefixes(raw),
|
||||
source: "normalized",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
ok: false,
|
||||
error: unknownTargetError(providerLabel, raw, hint),
|
||||
@@ -367,16 +404,32 @@ export async function lookupDirectoryDisplay(params: {
|
||||
runtime?: RuntimeEnv;
|
||||
}): Promise<string | undefined> {
|
||||
const normalized = normalizeTargetForProvider(params.channel, params.targetId) ?? params.targetId;
|
||||
const candidates = await getDirectoryEntries({
|
||||
cfg: params.cfg,
|
||||
channel: params.channel,
|
||||
accountId: params.accountId,
|
||||
kind: "group",
|
||||
runtime: params.runtime,
|
||||
preferLiveOnMiss: false,
|
||||
});
|
||||
const entry = candidates.find(
|
||||
(candidate) => normalizeDirectoryEntryId(params.channel, candidate) === normalized,
|
||||
);
|
||||
|
||||
// Targets can resolve to either peers (DMs) or groups. Try both.
|
||||
const [groups, users] = await Promise.all([
|
||||
getDirectoryEntries({
|
||||
cfg: params.cfg,
|
||||
channel: params.channel,
|
||||
accountId: params.accountId,
|
||||
kind: "group",
|
||||
runtime: params.runtime,
|
||||
preferLiveOnMiss: false,
|
||||
}),
|
||||
getDirectoryEntries({
|
||||
cfg: params.cfg,
|
||||
channel: params.channel,
|
||||
accountId: params.accountId,
|
||||
kind: "user",
|
||||
runtime: params.runtime,
|
||||
preferLiveOnMiss: false,
|
||||
}),
|
||||
]);
|
||||
|
||||
const findMatch = (candidates: ChannelDirectoryEntry[]) =>
|
||||
candidates.find(
|
||||
(candidate) => normalizeDirectoryEntryId(params.channel, candidate) === normalized,
|
||||
);
|
||||
|
||||
const entry = findMatch(groups) ?? findMatch(users);
|
||||
return entry?.name ?? entry?.handle ?? undefined;
|
||||
}
|
||||
@@ -124,6 +124,7 @@ import { startWebLoginWithQr, waitForWebLogin } from "../../web/login-qr.js";
|
||||
import { sendMessageWhatsApp, sendPollWhatsApp } from "../../web/outbound.js";
|
||||
import { registerMemoryCli } from "../../cli/memory-cli.js";
|
||||
import { formatNativeDependencyHint } from "./native-deps.js";
|
||||
import { textToSpeechTelephony } from "../../tts/tts.js";
|
||||
|
||||
import type { PluginRuntime } from "./types.js";
|
||||
|
||||
@@ -162,6 +163,9 @@ export function createPluginRuntime(): PluginRuntime {
|
||||
getImageMetadata,
|
||||
resizeToJpeg,
|
||||
},
|
||||
tts: {
|
||||
textToSpeechTelephony,
|
||||
},
|
||||
tools: {
|
||||
createMemoryGetTool,
|
||||
createMemorySearchTool,
|
||||
|
||||
@@ -16,6 +16,7 @@ type UpsertChannelPairingRequest =
|
||||
typeof import("../../pairing/pairing-store.js").upsertChannelPairingRequest;
|
||||
type FetchRemoteMedia = typeof import("../../media/fetch.js").fetchRemoteMedia;
|
||||
type SaveMediaBuffer = typeof import("../../media/store.js").saveMediaBuffer;
|
||||
type TextToSpeechTelephony = typeof import("../../tts/tts.js").textToSpeechTelephony;
|
||||
type BuildMentionRegexes = typeof import("../../auto-reply/reply/mentions.js").buildMentionRegexes;
|
||||
type MatchesMentionPatterns =
|
||||
typeof import("../../auto-reply/reply/mentions.js").matchesMentionPatterns;
|
||||
@@ -173,6 +174,9 @@ export type PluginRuntime = {
|
||||
getImageMetadata: GetImageMetadata;
|
||||
resizeToJpeg: ResizeToJpeg;
|
||||
};
|
||||
tts: {
|
||||
textToSpeechTelephony: TextToSpeechTelephony;
|
||||
};
|
||||
tools: {
|
||||
createMemoryGetTool: CreateMemoryGetTool;
|
||||
createMemorySearchTool: CreateMemorySearchTool;
|
||||
|
||||
@@ -43,6 +43,7 @@ function setup(config: Record<string, unknown>): Registered {
|
||||
source: "test",
|
||||
config: {},
|
||||
pluginConfig: config,
|
||||
runtime: { tts: { textToSpeechTelephony: vi.fn() } },
|
||||
logger: noopLogger,
|
||||
registerGatewayMethod: (method, handler) => methods.set(method, handler),
|
||||
registerTool: (tool) => tools.push(tool),
|
||||
@@ -142,6 +143,7 @@ describe("voice-call plugin", () => {
|
||||
source: "test",
|
||||
config: {},
|
||||
pluginConfig: { provider: "mock" },
|
||||
runtime: { tts: { textToSpeechTelephony: vi.fn() } },
|
||||
logger: noopLogger,
|
||||
registerGatewayMethod: () => {},
|
||||
registerTool: () => {},
|
||||
|
||||
+8
-6
@@ -109,13 +109,13 @@ describe("tts", () => {
|
||||
});
|
||||
|
||||
describe("isValidOpenAIModel", () => {
|
||||
it("accepts gpt-4o-mini-tts model", () => {
|
||||
it("accepts supported models", () => {
|
||||
expect(isValidOpenAIModel("gpt-4o-mini-tts")).toBe(true);
|
||||
expect(isValidOpenAIModel("tts-1")).toBe(true);
|
||||
expect(isValidOpenAIModel("tts-1-hd")).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects other models", () => {
|
||||
expect(isValidOpenAIModel("tts-1")).toBe(false);
|
||||
expect(isValidOpenAIModel("tts-1-hd")).toBe(false);
|
||||
it("rejects unsupported models", () => {
|
||||
expect(isValidOpenAIModel("invalid")).toBe(false);
|
||||
expect(isValidOpenAIModel("")).toBe(false);
|
||||
expect(isValidOpenAIModel("gpt-4")).toBe(false);
|
||||
@@ -123,9 +123,11 @@ describe("tts", () => {
|
||||
});
|
||||
|
||||
describe("OPENAI_TTS_MODELS", () => {
|
||||
it("contains only gpt-4o-mini-tts", () => {
|
||||
it("contains supported models", () => {
|
||||
expect(OPENAI_TTS_MODELS).toContain("gpt-4o-mini-tts");
|
||||
expect(OPENAI_TTS_MODELS).toHaveLength(1);
|
||||
expect(OPENAI_TTS_MODELS).toContain("tts-1");
|
||||
expect(OPENAI_TTS_MODELS).toContain("tts-1-hd");
|
||||
expect(OPENAI_TTS_MODELS).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("is a non-empty array", () => {
|
||||
|
||||
+126
-3
@@ -76,6 +76,11 @@ const DEFAULT_OUTPUT = {
|
||||
voiceCompatible: false,
|
||||
};
|
||||
|
||||
const TELEPHONY_OUTPUT = {
|
||||
openai: { format: "pcm" as const, sampleRate: 24000 },
|
||||
elevenlabs: { format: "pcm_22050", sampleRate: 22050 },
|
||||
};
|
||||
|
||||
const TTS_AUTO_MODES = new Set<TtsAutoMode>(["off", "always", "inbound", "tagged"]);
|
||||
|
||||
export type ResolvedTtsConfig = {
|
||||
@@ -180,6 +185,16 @@ export type TtsResult = {
|
||||
voiceCompatible?: boolean;
|
||||
};
|
||||
|
||||
export type TtsTelephonyResult = {
|
||||
success: boolean;
|
||||
audioBuffer?: Buffer;
|
||||
error?: string;
|
||||
latencyMs?: number;
|
||||
provider?: string;
|
||||
outputFormat?: string;
|
||||
sampleRate?: number;
|
||||
};
|
||||
|
||||
type TtsStatusEntry = {
|
||||
timestamp: number;
|
||||
success: boolean;
|
||||
@@ -736,7 +751,17 @@ function parseTtsDirectives(
|
||||
};
|
||||
}
|
||||
|
||||
export const OPENAI_TTS_MODELS = ["gpt-4o-mini-tts"] as const;
|
||||
export const OPENAI_TTS_MODELS = ["gpt-4o-mini-tts", "tts-1", "tts-1-hd"] as const;
|
||||
|
||||
/**
|
||||
* Custom OpenAI-compatible TTS endpoint.
|
||||
* When set, model/voice validation is relaxed to allow non-OpenAI models.
|
||||
* Example: OPENAI_TTS_BASE_URL=http://localhost:8880/v1
|
||||
*/
|
||||
const OPENAI_TTS_BASE_URL = (
|
||||
process.env.OPENAI_TTS_BASE_URL?.trim() || "https://api.openai.com/v1"
|
||||
).replace(/\/+$/, "");
|
||||
const isCustomOpenAIEndpoint = OPENAI_TTS_BASE_URL !== "https://api.openai.com/v1";
|
||||
export const OPENAI_TTS_VOICES = [
|
||||
"alloy",
|
||||
"ash",
|
||||
@@ -752,10 +777,14 @@ export const OPENAI_TTS_VOICES = [
|
||||
type OpenAiTtsVoice = (typeof OPENAI_TTS_VOICES)[number];
|
||||
|
||||
function isValidOpenAIModel(model: string): boolean {
|
||||
// Allow any model when using custom endpoint (e.g., Kokoro, LocalAI)
|
||||
if (isCustomOpenAIEndpoint) return true;
|
||||
return OPENAI_TTS_MODELS.includes(model as (typeof OPENAI_TTS_MODELS)[number]);
|
||||
}
|
||||
|
||||
function isValidOpenAIVoice(voice: string): voice is OpenAiTtsVoice {
|
||||
// Allow any voice when using custom endpoint (e.g., Kokoro Chinese voices)
|
||||
if (isCustomOpenAIEndpoint) return true;
|
||||
return OPENAI_TTS_VOICES.includes(voice as OpenAiTtsVoice);
|
||||
}
|
||||
|
||||
@@ -966,7 +995,7 @@ async function openaiTTS(params: {
|
||||
apiKey: string;
|
||||
model: string;
|
||||
voice: string;
|
||||
responseFormat: "mp3" | "opus";
|
||||
responseFormat: "mp3" | "opus" | "pcm";
|
||||
timeoutMs: number;
|
||||
}): Promise<Buffer> {
|
||||
const { text, apiKey, model, voice, responseFormat, timeoutMs } = params;
|
||||
@@ -982,7 +1011,7 @@ async function openaiTTS(params: {
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
const response = await fetch("https://api.openai.com/v1/audio/speech", {
|
||||
const response = await fetch(`${OPENAI_TTS_BASE_URL}/audio/speech`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${apiKey}`,
|
||||
@@ -1210,6 +1239,100 @@ export async function textToSpeech(params: {
|
||||
};
|
||||
}
|
||||
|
||||
export async function textToSpeechTelephony(params: {
|
||||
text: string;
|
||||
cfg: ClawdbotConfig;
|
||||
prefsPath?: string;
|
||||
}): Promise<TtsTelephonyResult> {
|
||||
const config = resolveTtsConfig(params.cfg);
|
||||
const prefsPath = params.prefsPath ?? resolveTtsPrefsPath(config);
|
||||
|
||||
if (params.text.length > config.maxTextLength) {
|
||||
return {
|
||||
success: false,
|
||||
error: `Text too long (${params.text.length} chars, max ${config.maxTextLength})`,
|
||||
};
|
||||
}
|
||||
|
||||
const userProvider = getTtsProvider(config, prefsPath);
|
||||
const providers = resolveTtsProviderOrder(userProvider);
|
||||
|
||||
let lastError: string | undefined;
|
||||
|
||||
for (const provider of providers) {
|
||||
const providerStart = Date.now();
|
||||
try {
|
||||
if (provider === "edge") {
|
||||
lastError = "edge: unsupported for telephony";
|
||||
continue;
|
||||
}
|
||||
|
||||
const apiKey = resolveTtsApiKey(config, provider);
|
||||
if (!apiKey) {
|
||||
lastError = `No API key for ${provider}`;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (provider === "elevenlabs") {
|
||||
const output = TELEPHONY_OUTPUT.elevenlabs;
|
||||
const audioBuffer = await elevenLabsTTS({
|
||||
text: params.text,
|
||||
apiKey,
|
||||
baseUrl: config.elevenlabs.baseUrl,
|
||||
voiceId: config.elevenlabs.voiceId,
|
||||
modelId: config.elevenlabs.modelId,
|
||||
outputFormat: output.format,
|
||||
seed: config.elevenlabs.seed,
|
||||
applyTextNormalization: config.elevenlabs.applyTextNormalization,
|
||||
languageCode: config.elevenlabs.languageCode,
|
||||
voiceSettings: config.elevenlabs.voiceSettings,
|
||||
timeoutMs: config.timeoutMs,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
audioBuffer,
|
||||
latencyMs: Date.now() - providerStart,
|
||||
provider,
|
||||
outputFormat: output.format,
|
||||
sampleRate: output.sampleRate,
|
||||
};
|
||||
}
|
||||
|
||||
const output = TELEPHONY_OUTPUT.openai;
|
||||
const audioBuffer = await openaiTTS({
|
||||
text: params.text,
|
||||
apiKey,
|
||||
model: config.openai.model,
|
||||
voice: config.openai.voice,
|
||||
responseFormat: output.format,
|
||||
timeoutMs: config.timeoutMs,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
audioBuffer,
|
||||
latencyMs: Date.now() - providerStart,
|
||||
provider,
|
||||
outputFormat: output.format,
|
||||
sampleRate: output.sampleRate,
|
||||
};
|
||||
} catch (err) {
|
||||
const error = err as Error;
|
||||
if (error.name === "AbortError") {
|
||||
lastError = `${provider}: request timed out`;
|
||||
} else {
|
||||
lastError = `${provider}: ${error.message}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: false,
|
||||
error: `TTS conversion failed: ${lastError || "no providers available"}`,
|
||||
};
|
||||
}
|
||||
|
||||
export async function maybeApplyTtsToPayload(params: {
|
||||
payload: ReplyPayload;
|
||||
cfg: ClawdbotConfig;
|
||||
|
||||
Reference in new issue
Block a user