Compare commits

..
Author SHA1 Message Date
Peter Steinberger b4b5b74854 fix: msteams probe scope (#1574) (thanks @Evizero) 2026-01-24 08:30:14 +00:00
Christof Salis 9c9e2ee6be fix(msteams): remove remaining /.default postfix
This fixes the msteams probe which otherwise incorrectly assumes teams is not working.

The @microsoft/agents-hosting SDK's MsalTokenProvider automatically appends /.default to all scope strings in its token acquisition methods (acquireAccessTokenViaSecret, acquireAccessTokenViaFIC, acquireAccessTokenViaWID, acquireTokenWithCertificate in msalTokenProvider.ts). This is consistent SDK behavior, not a recent change.

The current code is including .default in scope URLs, resulting in invalid double suffixes like https://graph.microsoft.com/.default/.default. I am not sure how the .default postfixed worked in the past for you if I am honest.

This was confirmed to cause Graph API authentication errors. Removing the .default suffix from our scope strings allows the SDK to append it correctly, resolving the issue. I confirmed it manually on my teams setup

Before: we pass .default -> SDK appends -> double .default (broken)
After: we pass base URL -> SDK appends -> single .default (works)
2026-01-24 08:27:30 +00:00
Peter Steinberger 8ea8801d06 fix: show tool error fallback for tool-only replies 2026-01-24 08:17:50 +00:00
Peter Steinberger c97bf23a4a fix: gate openai reasoning downgrade on model switches (#1562) (thanks @roshanasingh4) 2026-01-24 08:16:42 +00:00
Peter Steinberger 3fff943ba1 fix: harden gateway lock validation (#1572) (thanks @steipete) 2026-01-24 08:15:07 +00:00
Peter Steinberger 90685ef814 docs(fly): comprehensive deployment guide with real-world learnings
Based on actual Flawd deployment experience:
- Proper fly.toml configuration with all required settings
- Step-by-step guide following exe.dev doc format
- Troubleshooting section with common issues and fixes
- Config file creation via SSH
- Cost estimates
2026-01-24 08:15:07 +00:00
Peter Steinberger a8f2ac5411 docs(fly): add configuration guidance for bind mode, memory, and troubleshooting 2026-01-24 08:15:07 +00:00
Peter Steinberger dea96a2c3d fix: handle PID recycling in container gateway lock
In containers, PIDs can be recycled quickly after restarts. When a container
restarts, a different process might get the same PID as the previous gateway,
causing the lock check to incorrectly think the old gateway is still running.

This fix adds isGatewayProcess() which verifies on Linux that the PID actually
belongs to a clawdbot gateway by checking /proc/PID/cmdline. If the cmdline
doesn't contain 'clawdbot' or 'gateway', we assume the lock is stale.

Fixes gateway boot-loop in Docker/Fly.io deployments.
2026-01-24 08:15:07 +00:00
Peter Steinberger 90ae2f541c feat: add Fly.io deployment support
- Add fly.toml configuration for Fly.io deployment
- Add docs/platforms/fly.md with deployment guide
- Uses London (lhr) region by default
- Includes persistent volume for data storage
2026-01-24 08:15:07 +00:00
Peter Steinberger d9a467fe3b feat: move TTS into core (#1559) (thanks @Glucksberg) 2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 aef88cd9f1 test(telegram-tts): add unit tests for summarizeText function
- Export summarizeText in _test for testing
- Add 8 tests covering:
  - Successful summarization with metrics
  - OpenAI API call parameters verification
  - targetLength validation (min/max boundaries)
  - Error handling (API failures, empty responses)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 104d977d12 feat(telegram-tts): add latency logging, status tracking, and unit tests
- Add latency metrics to summarizeText and textToSpeech functions
- Add /tts_status command showing config and last attempt result
- Add /tts_summary command for feature flag control
- Fix atomic write to clean up temp file on rename failure
- Add timer.unref() to prevent blocking process shutdown
- Add unit tests for validation functions (13 tests)
- Update README with new commands and features

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 4b24753be7 feat(telegram-tts): add /tts_limit command and auto-summarization
- Add /tts_limit command to configure max text length (default 1500)
- Auto-summarize long texts with gpt-4o-mini before TTS conversion
- Add truncation safeguard if summary exceeds hard limit
- Validate targetLength parameter (100-10000)
- Use conservative max_tokens for multilingual text
- Add prompt injection defense with XML delimiters

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 df09e583aa feat(telegram-tts): add auto-TTS hook and provider switching
- Integrate message_sending hook into Telegram delivery path
- Send text first, then audio as voice message after
- Add /tts_provider command to switch between OpenAI and ElevenLabs
- Implement automatic fallback when primary provider fails
- Use gpt-4o-mini-tts as default OpenAI model
- Add hook integration to route-reply.ts for other channels

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 46e6546bb9 feat(telegram-tts): make extension self-contained with direct API calls
- Remove sag CLI dependency
- Add direct ElevenLabs API integration via fetch
- Add OpenAI TTS as alternative provider
- Support multi-provider configuration
- Add tts.providers RPC method
- Update config schema with OpenAI options
- Bump version to 0.2.0

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
GlucksbergandClaude Opus 4.5 5428c97685 feat(extensions): add telegram-tts extension for voice responses
Add a new extension that provides automatic text-to-speech for chat
responses using ElevenLabs API.

Features:
- `speak` tool for converting text to voice messages
- RPC methods: tts.status, tts.enable, tts.disable, tts.convert
- User preferences file for persistent TTS state
- Configurable voice ID, model, and max text length

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 08:00:44 +00:00
Roshan Singh 202d7af855 Fix OpenAI Responses transcript after model switch 2026-01-24 07:58:25 +00:00
Bradley PriestandPeter Steinberger 72020b37c3 fix(bird skill): gate brew install to macOS (#1569)
* fix(bird skill): gate brew install to macOS

* fix: gate bird brew install to macOS (#1569) (thanks @bradleypriest)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-24 07:53:29 +00:00
Peter Steinberger b051621bd4 fix: update changelog + clawtributors (#1571) (thanks @Takhoffman) 2026-01-24 07:47:35 +00:00
Tak hoffman ff52aec38e Agents: drop bash tool alias 2026-01-24 07:44:04 +00:00
37 changed files with 2450 additions and 28 deletions

No files matched your search

+7
View File
@@ -18,9 +18,15 @@ Docs: https://docs.clawd.bot
- Markdown: add per-channel table conversion (bullets for Signal/WhatsApp, code blocks elsewhere). (#1495) Thanks @odysseus0.
- Tlon: add Urbit channel plugin (DMs, group mentions, thread replies). (#1544) Thanks @wca4a.
- Channels: allow per-group tool allow/deny policies across built-in + plugin channels. (#1546) Thanks @adam91holt.
- TTS: move Telegram TTS into core with auto-replies, commands, and gateway methods. (#1559) Thanks @Glucksberg.
### Fixes
- Gateway: compare Linux process start time to avoid PID recycling lock loops; keep locks unless stale. (#1572) Thanks @steipete.
- Skills: gate bird Homebrew install to macOS. (#1569) Thanks @bradleypriest.
- Slack: honor open groupPolicy for unlisted channels in message + slash gating. (#1563) Thanks @itsjaydesu.
- Agents: show tool error fallback when the last assistant turn only invoked tools (prevents silent stops).
- Agents: ignore IDENTITY.md template placeholders when parsing identity to avoid placeholder replies. (#1556)
- Agents: drop orphaned OpenAI Responses reasoning blocks on model switches. (#1562) Thanks @roshanasingh4.
- Docker: update gateway command in docker-compose and Hetzner guide. (#1514)
- Sessions: reject array-backed session stores to prevent silent wipes. (#1469)
- Voice wake: auto-save wake words on blur/submit across iOS/Android and align limits with macOS.
@@ -55,6 +61,7 @@ Docs: https://docs.clawd.bot
- Agents: treat plugin-only tool allowlists as opt-ins; keep core tools enabled. (#1467)
- Exec approvals: persist allowlist entry ids to keep macOS allowlist rows stable. (#1521) Thanks @ngutman.
- MS Teams (plugin): remove `.default` suffix from Graph scopes to avoid double-appending. (#1507) Thanks @Evizero.
- MS Teams (plugin): remove `.default` suffix from Bot Framework probe scope to avoid double-appending. (#1574) Thanks @Evizero.
- Browser: keep extension relay tabs controllable when the extension reuses a session id after switching tabs. (#1160)
- Agents: warn and ignore tool allowlists that only reference unknown or unloaded plugin tools. (#1566)
+1
View File
@@ -1045,6 +1045,7 @@
"platforms/android",
"platforms/windows",
"platforms/linux",
"platforms/fly",
"platforms/hetzner",
"platforms/exe-dev"
]
+38
View File
@@ -1446,6 +1446,44 @@ active agent’s `identity.emoji` when set, otherwise `"👀"`. Set it to `""` t
`removeAckAfterReply` removes the bot’s ack reaction after a reply is sent
(Slack/Discord/Telegram only). Default: `false`.
#### `messages.tts`
Enable text-to-speech for outbound replies. When on, Clawdbot generates audio
using ElevenLabs or OpenAI and attaches it to responses. Telegram uses Opus
voice notes; other channels send MP3 audio.
```json5
{
messages: {
tts: {
enabled: true,
mode: "final", // final | all (include tool/block replies)
provider: "elevenlabs",
maxTextLength: 4000,
timeoutMs: 30000,
prefsPath: "~/.clawdbot/settings/tts.json",
elevenlabs: {
apiKey: "elevenlabs_api_key",
voiceId: "voice_id",
modelId: "eleven_multilingual_v2"
},
openai: {
apiKey: "openai_api_key",
model: "gpt-4o-mini-tts",
voice: "alloy"
}
}
}
}
```
Notes:
- `messages.tts.enabled` can be overridden by local user prefs (see `/tts_on`, `/tts_off`).
- `prefsPath` stores local overrides (enabled/provider/limit/summarize).
- `maxTextLength` is a hard cap for TTS input; summaries are truncated to fit.
- `/tts_limit` and `/tts_summary` control per-user summarization settings.
- `apiKey` values fall back to `ELEVENLABS_API_KEY`/`XI_API_KEY` and `OPENAI_API_KEY`.
### `talk`
Defaults for Talk mode (macOS/iOS/Android). Voice IDs fall back to `ELEVENLABS_VOICE_ID` or `SAG_VOICE_ID` when unset.
+267
View File
@@ -0,0 +1,267 @@
---
title: Fly.io
description: Deploy Clawdbot on Fly.io
---
# Fly.io Deployment
**Goal:** Clawdbot Gateway running on a [Fly.io](https://fly.io) machine with persistent storage, automatic HTTPS, and Discord/channel access.
## What you need
- [flyctl CLI](https://fly.io/docs/hands-on/install-flyctl/) installed
- Fly.io account (free tier works)
- Model auth: Anthropic API key (or other provider keys)
- Channel credentials: Discord bot token, Telegram token, etc.
## Beginner quick path
1. Clone repo → customize `fly.toml`
2. Create app + volume → set secrets
3. Deploy with `fly deploy`
4. SSH in to create config or use Control UI
## 1) Create the Fly app
```bash
# Clone the repo
git clone https://github.com/clawdbot/clawdbot.git
cd clawdbot
# Create a new Fly app (pick your own name)
fly apps create my-clawdbot
# Create a persistent volume (1GB is usually enough)
fly volumes create clawdbot_data --size 1 --region lhr
```
**Tip:** Choose a region close to you. Common options: `lhr` (London), `iad` (Virginia), `sjc` (San Jose).
## 2) Configure fly.toml
Edit `fly.toml` to match your app name and requirements:
```toml
app = "my-clawdbot" # Your app name
primary_region = "lhr"
[build]
dockerfile = "Dockerfile"
[env]
NODE_ENV = "production"
CLAWDBOT_PREFER_PNPM = "1"
CLAWDBOT_STATE_DIR = "/data"
NODE_OPTIONS = "--max-old-space-size=1536"
[processes]
app = "node dist/index.js gateway --allow-unconfigured --port 3000 --bind lan"
[http_service]
internal_port = 3000
force_https = true
auto_stop_machines = false
auto_start_machines = true
min_machines_running = 1
processes = ["app"]
[[vm]]
size = "shared-cpu-2x"
memory = "2048mb"
[mounts]
source = "clawdbot_data"
destination = "/data"
```
**Key settings:**
| Setting | Why |
|---------|-----|
| `--bind lan` | Binds to `0.0.0.0` so Fly's proxy can reach the gateway |
| `--allow-unconfigured` | Starts without a config file (you'll create one after) |
| `memory = "2048mb"` | 512MB is too small; 2GB recommended |
| `CLAWDBOT_STATE_DIR = "/data"` | Persists state on the volume |
## 3) Set secrets
```bash
# Required: Gateway token (for non-loopback binding)
fly secrets set CLAWDBOT_GATEWAY_TOKEN=$(openssl rand -hex 32)
# Model provider API keys
fly secrets set ANTHROPIC_API_KEY=sk-ant-...
# Optional: Other providers
fly secrets set OPENAI_API_KEY=sk-...
fly secrets set GOOGLE_API_KEY=...
# Channel tokens
fly secrets set DISCORD_BOT_TOKEN=MTQ...
```
**Notes:**
- Non-loopback binds (`--bind lan`) require `CLAWDBOT_GATEWAY_TOKEN` for security.
- Treat these tokens like passwords.
## 4) Deploy
```bash
fly deploy
```
First deploy builds the Docker image (~2-3 minutes). Subsequent deploys are faster.
After deployment, verify:
```bash
fly status
fly logs
```
You should see:
```
[gateway] listening on ws://0.0.0.0:3000 (PID xxx)
[discord] logged in to discord as xxx
```
## 5) Create config file
SSH into the machine to create a proper config:
```bash
fly ssh console
```
Create the config directory and file:
```bash
mkdir -p /data/.clawdbot
cat > /data/.clawdbot/clawdbot.json << 'EOF'
{
"agents": {
"defaults": {
"model": {
"primary": "anthropic/claude-opus-4-5"
},
"models": {
"anthropic/claude-opus-4-5": {},
"anthropic/claude-sonnet-4-5": {}
},
"maxConcurrent": 4
},
"list": [
{
"id": "main",
"default": true
}
]
},
"channels": {
"discord": {
"enabled": true
}
}
}
EOF
```
Restart to apply:
```bash
exit
fly machine restart <machine-id>
```
## 6) Access the Gateway
### Control UI
Open in browser:
```bash
fly open
```
Or visit `https://my-clawdbot.fly.dev/`
Paste your gateway token (the one from `CLAWDBOT_GATEWAY_TOKEN`) to authenticate.
### Logs
```bash
fly logs # Live logs
fly logs --no-tail # Recent logs
```
### SSH Console
```bash
fly ssh console
```
## Troubleshooting
### "App is not listening on expected address"
The gateway is binding to `127.0.0.1` instead of `0.0.0.0`.
**Fix:** Add `--bind lan` to your process command in `fly.toml`.
### OOM / Memory Issues
Container keeps restarting or getting killed.
**Fix:** Increase memory in `fly.toml`:
```toml
[[vm]]
memory = "2048mb"
```
### Gateway Lock Issues
Gateway refuses to start with "already running" errors.
This happens when the container restarts but the PID lock file persists on the volume.
**Fix:** Delete the lock file:
```bash
fly ssh console
rm /data/.clawdbot/run/gateway.*.lock
exit
fly machine restart <machine-id>
```
### Config Not Being Read
If using `--allow-unconfigured`, the gateway creates a minimal config. Your custom config at `/data/.clawdbot/clawdbot.json` should be read on restart.
Verify the config exists:
```bash
fly ssh console --command "cat /data/.clawdbot/clawdbot.json"
```
## Updates
```bash
# Pull latest changes
git pull
# Redeploy
fly deploy
# Check health
fly status
fly logs
```
## Notes
- Fly.io uses **x86 architecture** (not ARM)
- The Dockerfile is compatible with both architectures
- For WhatsApp/Telegram onboarding, use `fly ssh console`
- Persistent data lives on the volume at `/data`
## Cost
With the recommended config (`shared-cpu-2x`, 2GB RAM):
- ~$10-15/month depending on usage
- Free tier includes some allowance
See [Fly.io pricing](https://fly.io/docs/about/pricing/) for details.
+1
View File
@@ -23,6 +23,7 @@ Native companion apps for Windows are also planned; the Gateway is recommended v
## VPS & hosting
- Fly.io: [Fly.io](/platforms/fly)
- Hetzner (Docker): [Hetzner](/platforms/hetzner)
- exe.dev (VM + HTTPS proxy): [exe.dev](/platforms/exe-dev)
+1
View File
@@ -48,6 +48,7 @@ Implementation:
**OpenAI / OpenAI Codex**
- Image sanitization only.
- On model switch into OpenAI Responses/Codex, drop orphaned reasoning signatures (standalone reasoning items without a following content block).
- No tool call id sanitization.
- No tool result pairing repair.
- No turn validation or reordering.
+7
View File
@@ -67,6 +67,13 @@ Text + native (when enabled):
- `/config show|get|set|unset` (persist config to disk, owner-only; requires `commands.config: true`)
- `/debug show|set|unset|reset` (runtime overrides, owner-only; requires `commands.debug: true`)
- `/usage off|tokens|full|cost` (per-response usage footer or local cost summary)
- `/tts_on` (enable TTS replies)
- `/tts_off` (disable TTS replies)
- `/tts_provider [openai|elevenlabs]` (set or show TTS provider)
- `/tts_limit <chars>` (max chars before TTS summarization)
- `/tts_summary on|off` (toggle TTS auto-summary)
- `/tts_status` (show TTS status)
- `/audio <text>` (convert text to a TTS audio reply)
- `/stop`
- `/restart`
- `/dock-telegram` (alias: `/dock_telegram`) (switch replies to Telegram)
+1 -1
View File
@@ -65,7 +65,7 @@ export async function probeMSTeams(cfg?: MSTeamsConfig): Promise<ProbeMSTeamsRes
try {
const { sdk, authConfig } = await loadMSTeamsSdkWithAuth(creds);
const tokenProvider = new sdk.MsalTokenProvider(authConfig);
await tokenProvider.getAccessToken("https://api.botframework.com/.default");
await tokenProvider.getAccessToken("https://api.botframework.com");
let graph:
| {
ok: boolean;
+28
View File
@@ -0,0 +1,28 @@
# Clawdbot Fly.io deployment configuration
# See https://fly.io/docs/reference/configuration/
app = "clawdbot"
primary_region = "lhr" # London
[build]
dockerfile = "Dockerfile"
[env]
NODE_ENV = "production"
# Fly uses x86, but keep this for consistency
CLAWDBOT_PREFER_PNPM = "1"
[http_service]
internal_port = 3000
force_https = true
auto_stop_machines = false # Keep running for persistent connections
auto_start_machines = true
min_machines_running = 1
[[vm]]
size = "shared-cpu-1x"
memory = "512mb"
[mounts]
source = "clawdbot_data"
destination = "/data"
+2
View File
@@ -393,6 +393,8 @@ importers:
extensions/telegram: {}
extensions/telegram-tts: {}
extensions/tlon:
dependencies:
'@urbit/aura':
+1 -1
View File
@@ -2,7 +2,7 @@
name: bird
description: X/Twitter CLI for reading, searching, posting, and engagement via cookies.
homepage: https://bird.fast
metadata: {"clawdbot":{"emoji":"🐦","requires":{"bins":["bird"]},"install":[{"id":"brew","kind":"brew","formula":"steipete/tap/bird","bins":["bird"],"label":"Install bird (brew)"},{"id":"npm","kind":"node","package":"@steipete/bird","bins":["bird"],"label":"Install bird (npm)"}]}}
metadata: {"clawdbot":{"emoji":"🐦","requires":{"bins":["bird"]},"install":[{"id":"brew","kind":"brew","formula":"steipete/tap/bird","bins":["bird"],"label":"Install bird (brew)","os":["darwin"]},{"id":"npm","kind":"node","package":"@steipete/bird","bins":["bird"],"label":"Install bird (npm)"}]}}
---
# bird 🐦
+5
View File
@@ -17,6 +17,7 @@ import { createSessionsListTool } from "./tools/sessions-list-tool.js";
import { createSessionsSendTool } from "./tools/sessions-send-tool.js";
import { createSessionsSpawnTool } from "./tools/sessions-spawn-tool.js";
import { createWebFetchTool, createWebSearchTool } from "./tools/web-tools.js";
import { createTtsTool } from "./tools/tts-tool.js";
export function createClawdbotTools(options?: {
browserControlUrl?: string;
@@ -96,6 +97,10 @@ export function createClawdbotTools(options?: {
replyToMode: options?.replyToMode,
hasRepliedRef: options?.hasRepliedRef,
}),
createTtsTool({
agentChannel: options?.agentChannel,
config: options?.config,
}),
createGatewayTool({
agentSessionKey: options?.agentSessionKey,
config: options?.config,
@@ -0,0 +1,74 @@
import { describe, expect, it } from "vitest";
import { downgradeOpenAIReasoningBlocks } from "./pi-embedded-helpers.js";
describe("downgradeOpenAIReasoningBlocks", () => {
it("keeps reasoning signatures when followed by content", () => {
const input = [
{
role: "assistant",
content: [
{
type: "thinking",
thinking: "internal reasoning",
thinkingSignature: JSON.stringify({ id: "rs_123", type: "reasoning" }),
},
{ type: "text", text: "answer" },
],
},
];
expect(downgradeOpenAIReasoningBlocks(input as any)).toEqual(input);
});
it("drops orphaned reasoning blocks without following content", () => {
const input = [
{
role: "assistant",
content: [
{
type: "thinking",
thinkingSignature: JSON.stringify({ id: "rs_abc", type: "reasoning" }),
},
],
},
{ role: "user", content: "next" },
];
expect(downgradeOpenAIReasoningBlocks(input as any)).toEqual([
{ role: "user", content: "next" },
]);
});
it("drops object-form orphaned signatures", () => {
const input = [
{
role: "assistant",
content: [
{
type: "thinking",
thinkingSignature: { id: "rs_obj", type: "reasoning" },
},
],
},
];
expect(downgradeOpenAIReasoningBlocks(input as any)).toEqual([]);
});
it("keeps non-reasoning thinking signatures", () => {
const input = [
{
role: "assistant",
content: [
{
type: "thinking",
thinking: "t",
thinkingSignature: "reasoning_content",
},
],
},
];
expect(downgradeOpenAIReasoningBlocks(input as any)).toEqual(input);
});
});
+2
View File
@@ -31,6 +31,8 @@ export {
parseImageDimensionError,
} from "./pi-embedded-helpers/errors.js";
export { isGoogleModelApi, sanitizeGoogleTurnOrdering } from "./pi-embedded-helpers/google.js";
export { downgradeOpenAIReasoningBlocks } from "./pi-embedded-helpers/openai.js";
export {
isEmptyAssistantMessageContent,
sanitizeSessionMessagesImages,
+118
View File
@@ -0,0 +1,118 @@
import type { AgentMessage } from "@mariozechner/pi-agent-core";
type OpenAIThinkingBlock = {
type?: unknown;
thinking?: unknown;
thinkingSignature?: unknown;
};
type OpenAIReasoningSignature = {
id: string;
type: string;
};
function parseOpenAIReasoningSignature(value: unknown): OpenAIReasoningSignature | null {
if (!value) return null;
let candidate: { id?: unknown; type?: unknown } | null = null;
if (typeof value === "string") {
const trimmed = value.trim();
if (!trimmed.startsWith("{") || !trimmed.endsWith("}")) return null;
try {
candidate = JSON.parse(trimmed) as { id?: unknown; type?: unknown };
} catch {
return null;
}
} else if (typeof value === "object") {
candidate = value as { id?: unknown; type?: unknown };
}
if (!candidate) return null;
const id = typeof candidate.id === "string" ? candidate.id : "";
const type = typeof candidate.type === "string" ? candidate.type : "";
if (!id.startsWith("rs_")) return null;
if (type === "reasoning" || type.startsWith("reasoning.")) {
return { id, type };
}
return null;
}
function hasFollowingNonThinkingBlock(
content: Extract<AgentMessage, { role: "assistant" }>["content"],
index: number,
): boolean {
for (let i = index + 1; i < content.length; i++) {
const block = content[i];
if (!block || typeof block !== "object") return true;
if ((block as { type?: unknown }).type !== "thinking") return true;
}
return false;
}
/**
* OpenAI Responses API can reject transcripts that contain a standalone `reasoning` item id
* without the required following item.
*
* Clawdbot persists provider-specific reasoning metadata in `thinkingSignature`; if that metadata
* is incomplete, drop the block to keep history usable.
*/
export function downgradeOpenAIReasoningBlocks(messages: AgentMessage[]): AgentMessage[] {
const out: AgentMessage[] = [];
for (const msg of messages) {
if (!msg || typeof msg !== "object") {
out.push(msg);
continue;
}
const role = (msg as { role?: unknown }).role;
if (role !== "assistant") {
out.push(msg);
continue;
}
const assistantMsg = msg as Extract<AgentMessage, { role: "assistant" }>;
if (!Array.isArray(assistantMsg.content)) {
out.push(msg);
continue;
}
let changed = false;
type AssistantContentBlock = (typeof assistantMsg.content)[number];
const nextContent: AssistantContentBlock[] = [];
for (let i = 0; i < assistantMsg.content.length; i++) {
const block = assistantMsg.content[i];
if (!block || typeof block !== "object") {
nextContent.push(block as AssistantContentBlock);
continue;
}
const record = block as OpenAIThinkingBlock;
if (record.type !== "thinking") {
nextContent.push(block as AssistantContentBlock);
continue;
}
const signature = parseOpenAIReasoningSignature(record.thinkingSignature);
if (!signature) {
nextContent.push(block as AssistantContentBlock);
continue;
}
if (hasFollowingNonThinkingBlock(assistantMsg.content, i)) {
nextContent.push(block as AssistantContentBlock);
continue;
}
changed = true;
}
if (!changed) {
out.push(msg);
continue;
}
if (nextContent.length === 0) {
continue;
}
out.push({ ...assistantMsg, content: nextContent } as AgentMessage);
}
return out;
}
@@ -161,4 +161,92 @@ describe("sanitizeSessionHistory", () => {
expect(result).toHaveLength(1);
expect(result[0]?.role).toBe("assistant");
});
it("does not downgrade openai reasoning when the model has not changed", async () => {
const sessionEntries: Array<{ type: string; customType: string; data: unknown }> = [
{
type: "custom",
customType: "model-snapshot",
data: {
timestamp: Date.now(),
provider: "openai",
modelApi: "openai-responses",
modelId: "gpt-5.2-codex",
},
},
];
const sessionManager = {
getEntries: vi.fn(() => sessionEntries),
appendCustomEntry: vi.fn((customType: string, data: unknown) => {
sessionEntries.push({ type: "custom", customType, data });
}),
} as unknown as SessionManager;
const messages: AgentMessage[] = [
{
role: "assistant",
content: [
{
type: "thinking",
thinking: "reasoning",
thinkingSignature: JSON.stringify({ id: "rs_test", type: "reasoning" }),
},
],
},
];
const result = await sanitizeSessionHistory({
messages,
modelApi: "openai-responses",
provider: "openai",
modelId: "gpt-5.2-codex",
sessionManager,
sessionId: "test-session",
});
expect(result).toEqual(messages);
});
it("downgrades openai reasoning only when the model changes", async () => {
const sessionEntries: Array<{ type: string; customType: string; data: unknown }> = [
{
type: "custom",
customType: "model-snapshot",
data: {
timestamp: Date.now(),
provider: "anthropic",
modelApi: "anthropic-messages",
modelId: "claude-3-7",
},
},
];
const sessionManager = {
getEntries: vi.fn(() => sessionEntries),
appendCustomEntry: vi.fn((customType: string, data: unknown) => {
sessionEntries.push({ type: "custom", customType, data });
}),
} as unknown as SessionManager;
const messages: AgentMessage[] = [
{
role: "assistant",
content: [
{
type: "thinking",
thinking: "reasoning",
thinkingSignature: { id: "rs_test", type: "reasoning" },
},
],
},
];
const result = await sanitizeSessionHistory({
messages,
modelApi: "openai-responses",
provider: "openai",
modelId: "gpt-5.2-codex",
sessionManager,
sessionId: "test-session",
});
expect(result).toEqual([]);
});
});
+73 -3
View File
@@ -6,6 +6,7 @@ import type { SessionManager } from "@mariozechner/pi-coding-agent";
import { registerUnhandledRejectionHandler } from "../../infra/unhandled-rejections.js";
import {
downgradeOpenAIReasoningBlocks,
isCompactionFailureError,
isGoogleModelApi,
sanitizeGoogleTurnOrdering,
@@ -211,7 +212,50 @@ registerUnhandledRejectionHandler((reason) => {
return true;
});
type CustomEntryLike = { type?: unknown; customType?: unknown };
type CustomEntryLike = { type?: unknown; customType?: unknown; data?: unknown };
type ModelSnapshotEntry = {
timestamp: number;
provider?: string;
modelApi?: string | null;
modelId?: string;
};
const MODEL_SNAPSHOT_CUSTOM_TYPE = "model-snapshot";
function readLastModelSnapshot(sessionManager: SessionManager): ModelSnapshotEntry | null {
try {
const entries = sessionManager.getEntries();
for (let i = entries.length - 1; i >= 0; i--) {
const entry = entries[i] as CustomEntryLike;
if (entry?.type !== "custom" || entry?.customType !== MODEL_SNAPSHOT_CUSTOM_TYPE) continue;
const data = entry?.data as ModelSnapshotEntry | undefined;
if (data && typeof data === "object") {
return data;
}
}
} catch {
return null;
}
return null;
}
function appendModelSnapshot(sessionManager: SessionManager, data: ModelSnapshotEntry): void {
try {
sessionManager.appendCustomEntry(MODEL_SNAPSHOT_CUSTOM_TYPE, data);
} catch {
// ignore persistence failures
}
}
function isSameModelSnapshot(a: ModelSnapshotEntry, b: ModelSnapshotEntry): boolean {
const normalize = (value?: string | null) => value ?? "";
return (
normalize(a.provider) === normalize(b.provider) &&
normalize(a.modelApi) === normalize(b.modelApi) &&
normalize(a.modelId) === normalize(b.modelId)
);
}
function hasGoogleTurnOrderingMarker(sessionManager: SessionManager): boolean {
try {
@@ -292,12 +336,38 @@ export async function sanitizeSessionHistory(params: {
? sanitizeToolUseResultPairing(sanitizedThinking)
: sanitizedThinking;
const isOpenAIResponsesApi =
params.modelApi === "openai-responses" || params.modelApi === "openai-codex-responses";
const hasSnapshot = Boolean(params.provider || params.modelApi || params.modelId);
const priorSnapshot = hasSnapshot ? readLastModelSnapshot(params.sessionManager) : null;
const modelChanged = priorSnapshot
? !isSameModelSnapshot(priorSnapshot, {
timestamp: 0,
provider: params.provider,
modelApi: params.modelApi,
modelId: params.modelId,
})
: false;
const sanitizedOpenAI =
isOpenAIResponsesApi && modelChanged
? downgradeOpenAIReasoningBlocks(repairedTools)
: repairedTools;
if (hasSnapshot && (!priorSnapshot || modelChanged)) {
appendModelSnapshot(params.sessionManager, {
timestamp: Date.now(),
provider: params.provider,
modelApi: params.modelApi,
modelId: params.modelId,
});
}
if (!policy.applyGoogleTurnOrdering) {
return repairedTools;
return sanitizedOpenAI;
}
return applyGoogleTurnOrderingFix({
messages: repairedTools,
messages: sanitizedOpenAI,
modelApi: params.modelApi,
sessionManager: params.sessionManager,
sessionId: params.sessionId,
@@ -148,6 +148,35 @@ describe("buildEmbeddedRunPayloads", () => {
expect(payloads[0]?.text).toBe("All good");
});
it("adds tool error fallback when the assistant only invoked tools", () => {
const payloads = buildEmbeddedRunPayloads({
assistantTexts: [],
toolMetas: [],
lastAssistant: {
stopReason: "toolUse",
content: [
{
type: "toolCall",
id: "toolu_01",
name: "exec",
arguments: { command: "echo hi" },
},
],
} as AssistantMessage,
lastToolError: { toolName: "exec", error: "Command exited with code 1" },
sessionKey: "session:telegram",
inlineToolResultsAllowed: false,
verboseLevel: "off",
reasoningLevel: "off",
toolResultFormat: "plain",
});
expect(payloads).toHaveLength(1);
expect(payloads[0]?.isError).toBe(true);
expect(payloads[0]?.text).toContain("Exec");
expect(payloads[0]?.text).toContain("code 1");
});
it("suppresses recoverable tool errors containing 'required'", () => {
const payloads = buildEmbeddedRunPayloads({
assistantTexts: [],
+10 -1
View File
@@ -169,7 +169,16 @@ export function buildEmbeddedRunPayloads(params: {
}
if (params.lastToolError) {
const hasUserFacingReply = replyItems.length > 0;
const lastAssistantHasToolCalls =
Array.isArray(params.lastAssistant?.content) &&
params.lastAssistant?.content.some((block) =>
block && typeof block === "object"
? (block as { type?: unknown }).type === "toolCall"
: false,
);
const lastAssistantWasToolUse = params.lastAssistant?.stopReason === "toolUse";
const hasUserFacingReply =
replyItems.length > 0 && !lastAssistantHasToolCalls && !lastAssistantWasToolUse;
// Check if this is a recoverable/internal tool error that shouldn't be shown to users
// when there's already a user-facing reply (the model should have retried).
const errorLower = (params.lastToolError.error ?? "").toLowerCase();
+60
View File
@@ -0,0 +1,60 @@
import { Type } from "@sinclair/typebox";
import { loadConfig } from "../../config/config.js";
import type { ClawdbotConfig } from "../../config/config.js";
import type { GatewayMessageChannel } from "../../utils/message-channel.js";
import { textToSpeech } from "../../tts/tts.js";
import type { AnyAgentTool } from "./common.js";
import { readStringParam } from "./common.js";
const TtsToolSchema = Type.Object({
text: Type.String({ description: "Text to convert to speech." }),
channel: Type.Optional(
Type.String({ description: "Optional channel id to pick output format (e.g. telegram)." }),
),
});
export function createTtsTool(opts?: {
config?: ClawdbotConfig;
agentChannel?: GatewayMessageChannel;
}): AnyAgentTool {
return {
label: "TTS",
name: "tts",
description:
"Convert text to speech and return a MEDIA: path. Use when the user requests audio or TTS is enabled. Copy the MEDIA line exactly.",
parameters: TtsToolSchema,
execute: async (_toolCallId, args) => {
const params = args as Record<string, unknown>;
const text = readStringParam(params, "text", { required: true });
const channel = readStringParam(params, "channel");
const cfg = opts?.config ?? loadConfig();
const result = await textToSpeech({
text,
cfg,
channel: channel ?? opts?.agentChannel,
});
if (result.success && result.audioPath) {
const lines: string[] = [];
// Tag Telegram Opus output as a voice bubble instead of a file attachment.
if (result.voiceCompatible) lines.push("[[audio_as_voice]]");
lines.push(`MEDIA:${result.audioPath}`);
return {
content: [{ type: "text", text: lines.join("\n") }],
details: { audioPath: result.audioPath, provider: result.provider },
};
}
return {
content: [
{
type: "text",
text: result.error ?? "TTS conversion failed",
},
],
details: { error: result.error },
};
},
};
}
+75
View File
@@ -272,6 +272,81 @@ function buildChatCommands(): ChatCommandDefinition[] {
],
argsMenu: "auto",
}),
defineChatCommand({
key: "audio",
nativeName: "audio",
description: "Convert text to a TTS audio reply.",
textAlias: "/audio",
args: [
{
name: "text",
description: "Text to speak",
type: "string",
captureRemaining: true,
},
],
}),
defineChatCommand({
key: "tts_on",
nativeName: "tts_on",
description: "Enable text-to-speech for replies.",
textAlias: "/tts_on",
}),
defineChatCommand({
key: "tts_off",
nativeName: "tts_off",
description: "Disable text-to-speech for replies.",
textAlias: "/tts_off",
}),
defineChatCommand({
key: "tts_provider",
nativeName: "tts_provider",
description: "Set or show the TTS provider.",
textAlias: "/tts_provider",
args: [
{
name: "provider",
description: "openai or elevenlabs",
type: "string",
choices: ["openai", "elevenlabs"],
},
],
argsMenu: "auto",
}),
defineChatCommand({
key: "tts_limit",
nativeName: "tts_limit",
description: "Set or show the max TTS text length.",
textAlias: "/tts_limit",
args: [
{
name: "maxLength",
description: "Max chars before summarizing",
type: "number",
},
],
}),
defineChatCommand({
key: "tts_summary",
nativeName: "tts_summary",
description: "Enable or disable TTS auto-summary.",
textAlias: "/tts_summary",
args: [
{
name: "mode",
description: "on or off",
type: "string",
choices: ["on", "off"],
},
],
argsMenu: "auto",
}),
defineChatCommand({
key: "tts_status",
nativeName: "tts_status",
description: "Show TTS status and last attempt.",
textAlias: "/tts_status",
}),
defineChatCommand({
key: "stop",
nativeName: "stop",
+2
View File
@@ -16,6 +16,7 @@ import {
import { handleAllowlistCommand } from "./commands-allowlist.js";
import { handleSubagentsCommand } from "./commands-subagents.js";
import { handleModelsCommand } from "./commands-models.js";
import { handleTtsCommands } from "./commands-tts.js";
import {
handleAbortTrigger,
handleActivationCommand,
@@ -39,6 +40,7 @@ const HANDLERS: CommandHandler[] = [
handleSendPolicyCommand,
handleUsageCommand,
handleRestartCommand,
handleTtsCommands,
handleHelpCommand,
handleCommandsListCommand,
handleStatusCommand,
+214
View File
@@ -0,0 +1,214 @@
import { logVerbose } from "../../globals.js";
import type { ReplyPayload } from "../types.js";
import type { CommandHandler } from "./commands-types.js";
import {
getLastTtsAttempt,
getTtsMaxLength,
getTtsProvider,
isSummarizationEnabled,
isTtsEnabled,
resolveTtsApiKey,
resolveTtsConfig,
resolveTtsPrefsPath,
setLastTtsAttempt,
setSummarizationEnabled,
setTtsEnabled,
setTtsMaxLength,
setTtsProvider,
textToSpeech,
} from "../../tts/tts.js";
function parseCommandArg(normalized: string, command: string): string | null {
if (normalized === command) return "";
if (normalized.startsWith(`${command} `)) return normalized.slice(command.length).trim();
return null;
}
export const handleTtsCommands: CommandHandler = async (params, allowTextCommands) => {
if (!allowTextCommands) return null;
const normalized = params.command.commandBodyNormalized;
if (
!normalized.startsWith("/tts_") &&
normalized !== "/audio" &&
!normalized.startsWith("/audio ")
) {
return null;
}
if (!params.command.isAuthorizedSender) {
logVerbose(
`Ignoring TTS command from unauthorized sender: ${params.command.senderId || "<unknown>"}`,
);
return { shouldContinue: false };
}
const config = resolveTtsConfig(params.cfg);
const prefsPath = resolveTtsPrefsPath(config);
if (normalized === "/tts_on") {
setTtsEnabled(prefsPath, true);
return { shouldContinue: false, reply: { text: "🔊 TTS enabled." } };
}
if (normalized === "/tts_off") {
setTtsEnabled(prefsPath, false);
return { shouldContinue: false, reply: { text: "🔇 TTS disabled." } };
}
const audioArg = parseCommandArg(normalized, "/audio");
if (audioArg !== null) {
if (!audioArg.trim()) {
return { shouldContinue: false, reply: { text: "⚙️ Usage: /audio <text>" } };
}
const start = Date.now();
const result = await textToSpeech({
text: audioArg,
cfg: params.cfg,
channel: params.command.channel,
prefsPath,
});
if (result.success && result.audioPath) {
setLastTtsAttempt({
timestamp: Date.now(),
success: true,
textLength: audioArg.length,
summarized: false,
provider: result.provider,
latencyMs: result.latencyMs,
});
const payload: ReplyPayload = {
mediaUrl: result.audioPath,
audioAsVoice: result.voiceCompatible === true,
};
return { shouldContinue: false, reply: payload };
}
setLastTtsAttempt({
timestamp: Date.now(),
success: false,
textLength: audioArg.length,
summarized: false,
error: result.error,
latencyMs: Date.now() - start,
});
return {
shouldContinue: false,
reply: { text: `❌ Error generating audio: ${result.error ?? "unknown error"}` },
};
}
const providerArg = parseCommandArg(normalized, "/tts_provider");
if (providerArg !== null) {
const currentProvider = getTtsProvider(config, prefsPath);
if (!providerArg.trim()) {
const fallback = currentProvider === "openai" ? "elevenlabs" : "openai";
const hasOpenAI = Boolean(resolveTtsApiKey(config, "openai"));
const hasElevenLabs = Boolean(resolveTtsApiKey(config, "elevenlabs"));
return {
shouldContinue: false,
reply: {
text:
`🎙️ TTS provider\n` +
`Primary: ${currentProvider}\n` +
`Fallback: ${fallback}\n` +
`OpenAI key: ${hasOpenAI ? "✅" : "❌"}\n` +
`ElevenLabs key: ${hasElevenLabs ? "✅" : "❌"}\n` +
`Usage: /tts_provider openai | elevenlabs`,
},
};
}
const requested = providerArg.trim().toLowerCase();
if (requested !== "openai" && requested !== "elevenlabs") {
return {
shouldContinue: false,
reply: { text: "⚙️ Usage: /tts_provider openai | elevenlabs" },
};
}
setTtsProvider(prefsPath, requested);
const fallback = requested === "openai" ? "elevenlabs" : "openai";
return {
shouldContinue: false,
reply: { text: `✅ TTS provider set to ${requested} (fallback: ${fallback}).` },
};
}
const limitArg = parseCommandArg(normalized, "/tts_limit");
if (limitArg !== null) {
if (!limitArg.trim()) {
const currentLimit = getTtsMaxLength(prefsPath);
return {
shouldContinue: false,
reply: { text: `📏 TTS limit: ${currentLimit} characters.` },
};
}
const next = Number.parseInt(limitArg.trim(), 10);
if (!Number.isFinite(next) || next < 100 || next > 10_000) {
return {
shouldContinue: false,
reply: { text: "⚙️ Usage: /tts_limit <100-10000>" },
};
}
setTtsMaxLength(prefsPath, next);
return {
shouldContinue: false,
reply: { text: `✅ TTS limit set to ${next} characters.` },
};
}
const summaryArg = parseCommandArg(normalized, "/tts_summary");
if (summaryArg !== null) {
if (!summaryArg.trim()) {
const enabled = isSummarizationEnabled(prefsPath);
return {
shouldContinue: false,
reply: { text: `📝 TTS auto-summary: ${enabled ? "on" : "off"}.` },
};
}
const requested = summaryArg.trim().toLowerCase();
if (requested !== "on" && requested !== "off") {
return { shouldContinue: false, reply: { text: "⚙️ Usage: /tts_summary on|off" } };
}
setSummarizationEnabled(prefsPath, requested === "on");
return {
shouldContinue: false,
reply: {
text: requested === "on" ? "✅ TTS auto-summary enabled." : "❌ TTS auto-summary disabled.",
},
};
}
if (normalized === "/tts_status") {
const enabled = isTtsEnabled(config, prefsPath);
const provider = getTtsProvider(config, prefsPath);
const hasKey = Boolean(resolveTtsApiKey(config, provider));
const maxLength = getTtsMaxLength(prefsPath);
const summarize = isSummarizationEnabled(prefsPath);
const last = getLastTtsAttempt();
const lines = [
"📊 TTS status",
`State: ${enabled ? "✅ enabled" : "❌ disabled"}`,
`Provider: ${provider} (${hasKey ? "✅ key" : "❌ no key"})`,
`Text limit: ${maxLength} chars`,
`Auto-summary: ${summarize ? "on" : "off"}`,
];
if (last) {
const timeAgo = Math.round((Date.now() - last.timestamp) / 1000);
lines.push("");
lines.push(`Last attempt (${timeAgo}s ago): ${last.success ? "✅" : "❌"}`);
lines.push(`Text: ${last.textLength} chars${last.summarized ? " (summarized)" : ""}`);
if (last.success) {
lines.push(`Provider: ${last.provider ?? "unknown"}`);
lines.push(`Latency: ${last.latencyMs ?? 0}ms`);
} else if (last.error) {
lines.push(`Error: ${last.error}`);
}
}
return { shouldContinue: false, reply: { text: lines.join("\n") } };
}
return null;
};
+38 -16
View File
@@ -13,6 +13,7 @@ import { formatAbortReplyText, tryFastAbortFromMessage } from "./abort.js";
import { shouldSkipDuplicateInbound } from "./inbound-dedupe.js";
import type { ReplyDispatcher, ReplyDispatchKind } from "./reply-dispatcher.js";
import { isRoutableChannel, routeReply } from "./route-reply.js";
import { maybeApplyTtsToPayload } from "../../tts/tts.js";
export type DispatchFromConfigResult = {
queuedFinal: boolean;
@@ -91,6 +92,7 @@ export async function dispatchReplyFromConfig(params: {
const currentSurface = (ctx.Surface ?? ctx.Provider)?.toLowerCase();
const shouldRouteToOriginating =
isRoutableChannel(originatingChannel) && originatingTo && originatingChannel !== currentSurface;
const ttsChannel = shouldRouteToOriginating ? originatingChannel : currentSurface;
/**
* Helper to send a payload via route-reply (async).
@@ -164,22 +166,36 @@ export async function dispatchReplyFromConfig(params: {
{
...params.replyOptions,
onToolResult: (payload: ReplyPayload) => {
if (shouldRouteToOriginating) {
// Fire-and-forget for streaming tool results when routing.
void sendPayloadAsync(payload);
} else {
// Synchronous dispatch to preserve callback timing.
dispatcher.sendToolResult(payload);
}
const run = async () => {
const ttsPayload = await maybeApplyTtsToPayload({
payload,
cfg,
channel: ttsChannel,
kind: "tool",
});
if (shouldRouteToOriginating) {
await sendPayloadAsync(ttsPayload);
} else {
dispatcher.sendToolResult(ttsPayload);
}
};
return run();
},
onBlockReply: (payload: ReplyPayload, context) => {
if (shouldRouteToOriginating) {
// Await routed sends so upstream can enforce ordering/timeouts.
return sendPayloadAsync(payload, context?.abortSignal);
} else {
// Synchronous dispatch to preserve callback timing.
dispatcher.sendBlockReply(payload);
}
const run = async () => {
const ttsPayload = await maybeApplyTtsToPayload({
payload,
cfg,
channel: ttsChannel,
kind: "block",
});
if (shouldRouteToOriginating) {
await sendPayloadAsync(ttsPayload, context?.abortSignal);
} else {
dispatcher.sendBlockReply(ttsPayload);
}
};
return run();
},
},
cfg,
@@ -190,10 +206,16 @@ export async function dispatchReplyFromConfig(params: {
let queuedFinal = false;
let routedFinalCount = 0;
for (const reply of replies) {
const ttsReply = await maybeApplyTtsToPayload({
payload: reply,
cfg,
channel: ttsChannel,
kind: "final",
});
if (shouldRouteToOriginating && originatingChannel && originatingTo) {
// Route final reply to originating channel.
const result = await routeReply({
payload: reply,
payload: ttsReply,
channel: originatingChannel,
to: originatingTo,
sessionKey: ctx.SessionKey,
@@ -209,7 +231,7 @@ export async function dispatchReplyFromConfig(params: {
queuedFinal = result.ok || queuedFinal;
if (result.ok) routedFinalCount += 1;
} else {
queuedFinal = dispatcher.sendFinalReply(reply) || queuedFinal;
queuedFinal = dispatcher.sendFinalReply(ttsReply) || queuedFinal;
}
}
await dispatcher.waitForIdle();
+2 -2
View File
@@ -72,8 +72,8 @@ export async function routeReply(params: RouteReplyParams): Promise<RouteReplyRe
});
if (!normalized) return { ok: true };
const text = normalized.text ?? "";
const mediaUrls = (normalized.mediaUrls?.filter(Boolean) ?? []).length
let text = normalized.text ?? "";
let mediaUrls = (normalized.mediaUrls?.filter(Boolean) ?? []).length
? (normalized.mediaUrls?.filter(Boolean) as string[])
: normalized.mediaUrl
? [normalized.mediaUrl]
+3
View File
@@ -1,4 +1,5 @@
import type { QueueDropPolicy, QueueMode, QueueModeByProvider } from "./types.queue.js";
import type { TtsConfig } from "./types.tts.js";
export type GroupChatConfig = {
mentionPatterns?: string[];
@@ -81,6 +82,8 @@ export type MessagesConfig = {
ackReactionScope?: "group-mentions" | "group-all" | "direct" | "all";
/** Remove ack reaction after reply is sent (default: false). */
removeAckAfterReply?: boolean;
/** Text-to-speech settings for outbound replies. */
tts?: TtsConfig;
};
export type NativeCommandsSetting = boolean | "auto";
+1
View File
@@ -23,5 +23,6 @@ export * from "./types.signal.js";
export * from "./types.skills.js";
export * from "./types.slack.js";
export * from "./types.telegram.js";
export * from "./types.tts.js";
export * from "./types.tools.js";
export * from "./types.whatsapp.js";
+30
View File
@@ -0,0 +1,30 @@
export type TtsProvider = "elevenlabs" | "openai";
export type TtsMode = "final" | "all";
export type TtsConfig = {
/** Enable auto-TTS (can be overridden by local prefs). */
enabled?: boolean;
/** Apply TTS to final replies only or to all replies (tool/block/final). */
mode?: TtsMode;
/** Primary TTS provider (fallbacks are automatic). */
provider?: TtsProvider;
/** ElevenLabs configuration. */
elevenlabs?: {
apiKey?: string;
voiceId?: string;
modelId?: string;
};
/** OpenAI configuration. */
openai?: {
apiKey?: string;
model?: string;
voice?: string;
};
/** Optional path for local TTS user preferences JSON. */
prefsPath?: string;
/** Hard cap for text sent to TTS (chars). */
maxTextLength?: number;
/** API request timeout (ms). */
timeoutMs?: number;
};
+30
View File
@@ -155,6 +155,36 @@ export const MarkdownConfigSchema = z
.strict()
.optional();
export const TtsProviderSchema = z.enum(["elevenlabs", "openai"]);
export const TtsModeSchema = z.enum(["final", "all"]);
export const TtsConfigSchema = z
.object({
enabled: z.boolean().optional(),
mode: TtsModeSchema.optional(),
provider: TtsProviderSchema.optional(),
elevenlabs: z
.object({
apiKey: z.string().optional(),
voiceId: z.string().optional(),
modelId: z.string().optional(),
})
.strict()
.optional(),
openai: z
.object({
apiKey: z.string().optional(),
model: z.string().optional(),
voice: z.string().optional(),
})
.strict()
.optional(),
prefsPath: z.string().optional(),
maxTextLength: z.number().int().min(1).optional(),
timeoutMs: z.number().int().min(1000).max(120000).optional(),
})
.strict()
.optional();
export const HumanDelaySchema = z
.object({
mode: z.union([z.literal("off"), z.literal("natural"), z.literal("custom")]).optional(),
+2
View File
@@ -5,6 +5,7 @@ import {
InboundDebounceSchema,
NativeCommandsSettingSchema,
QueueSchema,
TtsConfigSchema,
} from "./zod-schema.core.js";
const SessionResetConfigSchema = z
@@ -90,6 +91,7 @@ export const MessagesSchema = z
ackReaction: z.string().optional(),
ackReactionScope: z.enum(["group-mentions", "group-all", "direct", "all"]).optional(),
removeAckAfterReply: z.boolean().optional(),
tts: TtsConfigSchema,
})
.strict()
.optional();
+6
View File
@@ -8,6 +8,12 @@ const BASE_METHODS = [
"status",
"usage.status",
"usage.cost",
"tts.status",
"tts.providers",
"tts.enable",
"tts.disable",
"tts.convert",
"tts.setProvider",
"config.get",
"config.set",
"config.apply",
+8
View File
@@ -17,6 +17,7 @@ import { sessionsHandlers } from "./server-methods/sessions.js";
import { skillsHandlers } from "./server-methods/skills.js";
import { systemHandlers } from "./server-methods/system.js";
import { talkHandlers } from "./server-methods/talk.js";
import { ttsHandlers } from "./server-methods/tts.js";
import type { GatewayRequestHandlers, GatewayRequestOptions } from "./server-methods/types.js";
import { updateHandlers } from "./server-methods/update.js";
import { usageHandlers } from "./server-methods/usage.js";
@@ -53,6 +54,8 @@ const READ_METHODS = new Set([
"status",
"usage.status",
"usage.cost",
"tts.status",
"tts.providers",
"models.list",
"agents.list",
"agent.identity.get",
@@ -75,6 +78,10 @@ const WRITE_METHODS = new Set([
"agent.wait",
"wake",
"talk.mode",
"tts.enable",
"tts.disable",
"tts.convert",
"tts.setProvider",
"voicewake.set",
"node.invoke",
"chat.send",
@@ -151,6 +158,7 @@ export const coreGatewayHandlers: GatewayRequestHandlers = {
...configHandlers,
...wizardHandlers,
...talkHandlers,
...ttsHandlers,
...skillsHandlers,
...sessionsHandlers,
...systemHandlers,
+138
View File
@@ -0,0 +1,138 @@
import { loadConfig } from "../../config/config.js";
import {
OPENAI_TTS_MODELS,
OPENAI_TTS_VOICES,
getTtsProvider,
isTtsEnabled,
resolveTtsApiKey,
resolveTtsConfig,
resolveTtsPrefsPath,
setTtsEnabled,
setTtsProvider,
textToSpeech,
} from "../../tts/tts.js";
import { ErrorCodes, errorShape } from "../protocol/index.js";
import { formatForLog } from "../ws-log.js";
import type { GatewayRequestHandlers } from "./types.js";
export const ttsHandlers: GatewayRequestHandlers = {
"tts.status": async ({ respond }) => {
try {
const cfg = loadConfig();
const config = resolveTtsConfig(cfg);
const prefsPath = resolveTtsPrefsPath(config);
const provider = getTtsProvider(config, prefsPath);
respond(true, {
enabled: isTtsEnabled(config, prefsPath),
provider,
fallbackProvider: provider === "openai" ? "elevenlabs" : "openai",
prefsPath,
hasOpenAIKey: Boolean(resolveTtsApiKey(config, "openai")),
hasElevenLabsKey: Boolean(resolveTtsApiKey(config, "elevenlabs")),
});
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
"tts.enable": async ({ respond }) => {
try {
const cfg = loadConfig();
const config = resolveTtsConfig(cfg);
const prefsPath = resolveTtsPrefsPath(config);
setTtsEnabled(prefsPath, true);
respond(true, { enabled: true });
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
"tts.disable": async ({ respond }) => {
try {
const cfg = loadConfig();
const config = resolveTtsConfig(cfg);
const prefsPath = resolveTtsPrefsPath(config);
setTtsEnabled(prefsPath, false);
respond(true, { enabled: false });
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
"tts.convert": async ({ params, respond }) => {
const text = typeof params.text === "string" ? params.text.trim() : "";
if (!text) {
respond(
false,
undefined,
errorShape(ErrorCodes.INVALID_REQUEST, "tts.convert requires text"),
);
return;
}
try {
const cfg = loadConfig();
const channel = typeof params.channel === "string" ? params.channel.trim() : undefined;
const result = await textToSpeech({ text, cfg, channel });
if (result.success && result.audioPath) {
respond(true, {
audioPath: result.audioPath,
provider: result.provider,
outputFormat: result.outputFormat,
voiceCompatible: result.voiceCompatible,
});
return;
}
respond(
false,
undefined,
errorShape(ErrorCodes.UNAVAILABLE, result.error ?? "TTS conversion failed"),
);
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
"tts.setProvider": async ({ params, respond }) => {
const provider = typeof params.provider === "string" ? params.provider.trim() : "";
if (provider !== "openai" && provider !== "elevenlabs") {
respond(
false,
undefined,
errorShape(ErrorCodes.INVALID_REQUEST, "Invalid provider. Use openai or elevenlabs."),
);
return;
}
try {
const cfg = loadConfig();
const config = resolveTtsConfig(cfg);
const prefsPath = resolveTtsPrefsPath(config);
setTtsProvider(prefsPath, provider);
respond(true, { provider });
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
"tts.providers": async ({ respond }) => {
try {
const cfg = loadConfig();
const config = resolveTtsConfig(cfg);
const prefsPath = resolveTtsPrefsPath(config);
respond(true, {
providers: [
{
id: "openai",
name: "OpenAI",
configured: Boolean(resolveTtsApiKey(config, "openai")),
models: [...OPENAI_TTS_MODELS],
voices: [...OPENAI_TTS_VOICES],
},
{
id: "elevenlabs",
name: "ElevenLabs",
configured: Boolean(resolveTtsApiKey(config, "elevenlabs")),
models: ["eleven_multilingual_v2", "eleven_turbo_v2_5", "eleven_monolingual_v1"],
},
],
active: getTtsProvider(config, prefsPath),
});
} catch (err) {
respond(false, undefined, errorShape(ErrorCodes.UNAVAILABLE, formatForLog(err)));
}
},
};
+133 -1
View File
@@ -1,10 +1,13 @@
import { createHash } from "node:crypto";
import fsSync from "node:fs";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import { acquireGatewayLock, GatewayLockError } from "./gateway-lock.js";
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
async function makeEnv() {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-gateway-lock-"));
@@ -22,6 +25,41 @@ async function makeEnv() {
};
}
function resolveLockPath(env: NodeJS.ProcessEnv) {
const stateDir = resolveStateDir(env);
const configPath = resolveConfigPath(env, stateDir);
const hash = createHash("sha1").update(configPath).digest("hex").slice(0, 8);
return { lockPath: path.join(stateDir, `gateway.${hash}.lock`), configPath };
}
function makeProcStat(pid: number, startTime: number) {
const fields = [
"R",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
"1",
String(startTime),
"1",
"1",
];
return `${pid} (node) ${fields.join(" ")}`;
}
describe("gateway lock", () => {
it("blocks concurrent acquisition until release", async () => {
const { env, cleanup } = await makeEnv();
@@ -52,4 +90,98 @@ describe("gateway lock", () => {
await lock2?.release();
await cleanup();
});
it("treats recycled linux pid as stale when start time mismatches", async () => {
const { env, cleanup } = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
const payload = {
pid: process.pid,
createdAt: new Date().toISOString(),
configPath,
startTime: 111,
};
await fs.writeFile(lockPath, JSON.stringify(payload), "utf8");
const readFileSync = fsSync.readFileSync;
const statValue = makeProcStat(process.pid, 222);
const spy = vi.spyOn(fsSync, "readFileSync").mockImplementation((filePath, encoding) => {
if (filePath === `/proc/${process.pid}/stat`) {
return statValue;
}
return readFileSync(filePath as never, encoding as never) as never;
});
const lock = await acquireGatewayLock({
env,
allowInTests: true,
timeoutMs: 200,
pollIntervalMs: 20,
platform: "linux",
});
expect(lock).not.toBeNull();
await lock?.release();
spy.mockRestore();
await cleanup();
});
it("keeps lock on linux when proc access fails unless stale", async () => {
const { env, cleanup } = await makeEnv();
const { lockPath, configPath } = resolveLockPath(env);
const payload = {
pid: process.pid,
createdAt: new Date().toISOString(),
configPath,
startTime: 111,
};
await fs.writeFile(lockPath, JSON.stringify(payload), "utf8");
const readFileSync = fsSync.readFileSync;
const spy = vi.spyOn(fsSync, "readFileSync").mockImplementation((filePath, encoding) => {
if (filePath === `/proc/${process.pid}/stat`) {
throw new Error("EACCES");
}
return readFileSync(filePath as never, encoding as never) as never;
});
await expect(
acquireGatewayLock({
env,
allowInTests: true,
timeoutMs: 120,
pollIntervalMs: 20,
staleMs: 10_000,
platform: "linux",
}),
).rejects.toBeInstanceOf(GatewayLockError);
spy.mockRestore();
const stalePayload = {
...payload,
createdAt: new Date(0).toISOString(),
};
await fs.writeFile(lockPath, JSON.stringify(stalePayload), "utf8");
const staleSpy = vi.spyOn(fsSync, "readFileSync").mockImplementation((filePath, encoding) => {
if (filePath === `/proc/${process.pid}/stat`) {
throw new Error("EACCES");
}
return readFileSync(filePath as never, encoding as never) as never;
});
const lock = await acquireGatewayLock({
env,
allowInTests: true,
timeoutMs: 200,
pollIntervalMs: 20,
staleMs: 1,
platform: "linux",
});
expect(lock).not.toBeNull();
await lock?.release();
staleSpy.mockRestore();
await cleanup();
});
});
+91 -3
View File
@@ -1,5 +1,6 @@
import { createHash } from "node:crypto";
import fs from "node:fs/promises";
import fsSync from "node:fs";
import path from "node:path";
import { resolveConfigPath, resolveStateDir } from "../config/paths.js";
@@ -12,6 +13,7 @@ type LockPayload = {
pid: number;
createdAt: string;
configPath: string;
startTime?: number;
};
export type GatewayLockHandle = {
@@ -26,6 +28,7 @@ export type GatewayLockOptions = {
pollIntervalMs?: number;
staleMs?: number;
allowInTests?: boolean;
platform?: NodeJS.Platform;
};
export class GatewayLockError extends Error {
@@ -38,6 +41,8 @@ export class GatewayLockError extends Error {
}
}
type LockOwnerStatus = "alive" | "dead" | "unknown";
function isAlive(pid: number): boolean {
if (!Number.isFinite(pid) || pid <= 0) return false;
try {
@@ -48,6 +53,80 @@ function isAlive(pid: number): boolean {
}
}
function normalizeProcArg(arg: string): string {
return arg.replaceAll("\\", "/").toLowerCase();
}
function parseProcCmdline(raw: string): string[] {
return raw
.split("\0")
.map((entry) => entry.trim())
.filter(Boolean);
}
function isGatewayArgv(args: string[]): boolean {
const normalized = args.map(normalizeProcArg);
if (!normalized.includes("gateway")) return false;
const entryCandidates = [
"dist/index.js",
"dist/index.mjs",
"dist/entry.js",
"dist/entry.mjs",
"scripts/run-node.mjs",
"src/index.ts",
];
if (normalized.some((arg) => entryCandidates.some((entry) => arg.endsWith(entry)))) {
return true;
}
const exe = normalized[0] ?? "";
return exe.endsWith("/clawdbot") || exe === "clawdbot";
}
function readLinuxCmdline(pid: number): string[] | null {
try {
const raw = fsSync.readFileSync(`/proc/${pid}/cmdline`, "utf8");
return parseProcCmdline(raw);
} catch {
return null;
}
}
function readLinuxStartTime(pid: number): number | null {
try {
const raw = fsSync.readFileSync(`/proc/${pid}/stat`, "utf8").trim();
const closeParen = raw.lastIndexOf(")");
if (closeParen < 0) return null;
const rest = raw.slice(closeParen + 1).trim();
const fields = rest.split(/\s+/);
const startTime = Number.parseInt(fields[19] ?? "", 10);
return Number.isFinite(startTime) ? startTime : null;
} catch {
return null;
}
}
function resolveGatewayOwnerStatus(
pid: number,
payload: LockPayload | null,
platform: NodeJS.Platform,
): LockOwnerStatus {
if (!isAlive(pid)) return "dead";
if (platform !== "linux") return "alive";
const payloadStartTime = payload?.startTime;
if (Number.isFinite(payloadStartTime)) {
const currentStartTime = readLinuxStartTime(pid);
if (currentStartTime == null) return "unknown";
return currentStartTime === payloadStartTime ? "alive" : "dead";
}
const args = readLinuxCmdline(pid);
if (!args) return "unknown";
return isGatewayArgv(args) ? "alive" : "dead";
}
async function readLockPayload(lockPath: string): Promise<LockPayload | null> {
try {
const raw = await fs.readFile(lockPath, "utf8");
@@ -55,10 +134,12 @@ async function readLockPayload(lockPath: string): Promise<LockPayload | null> {
if (typeof parsed.pid !== "number") return null;
if (typeof parsed.createdAt !== "string") return null;
if (typeof parsed.configPath !== "string") return null;
const startTime = typeof parsed.startTime === "number" ? parsed.startTime : undefined;
return {
pid: parsed.pid,
createdAt: parsed.createdAt,
configPath: parsed.configPath,
startTime,
};
} catch {
return null;
@@ -88,6 +169,7 @@ export async function acquireGatewayLock(
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const pollIntervalMs = opts.pollIntervalMs ?? DEFAULT_POLL_INTERVAL_MS;
const staleMs = opts.staleMs ?? DEFAULT_STALE_MS;
const platform = opts.platform ?? process.platform;
const { lockPath, configPath } = resolveGatewayLockPath(env);
await fs.mkdir(path.dirname(lockPath), { recursive: true });
@@ -97,11 +179,15 @@ export async function acquireGatewayLock(
while (Date.now() - startedAt < timeoutMs) {
try {
const handle = await fs.open(lockPath, "wx");
const startTime = platform === "linux" ? readLinuxStartTime(process.pid) : null;
const payload: LockPayload = {
pid: process.pid,
createdAt: new Date().toISOString(),
configPath,
};
if (typeof startTime === "number" && Number.isFinite(startTime)) {
payload.startTime = startTime;
}
await handle.writeFile(JSON.stringify(payload), "utf8");
return {
lockPath,
@@ -119,12 +205,14 @@ export async function acquireGatewayLock(
lastPayload = await readLockPayload(lockPath);
const ownerPid = lastPayload?.pid;
const ownerAlive = ownerPid ? isAlive(ownerPid) : false;
if (!ownerAlive && ownerPid) {
const ownerStatus = ownerPid
? resolveGatewayOwnerStatus(ownerPid, lastPayload, platform)
: "unknown";
if (ownerStatus === "dead" && ownerPid) {
await fs.rm(lockPath, { force: true });
continue;
}
if (!ownerAlive) {
if (ownerStatus !== "alive") {
let stale = false;
if (lastPayload?.createdAt) {
const createdAt = Date.parse(lastPayload.createdAt);
+234
View File
@@ -0,0 +1,234 @@
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import { _test } from "./tts.js";
const {
isValidVoiceId,
isValidOpenAIVoice,
isValidOpenAIModel,
OPENAI_TTS_MODELS,
OPENAI_TTS_VOICES,
summarizeText,
resolveOutputFormat,
} = _test;
describe("tts", () => {
describe("isValidVoiceId", () => {
it("accepts valid ElevenLabs voice IDs", () => {
expect(isValidVoiceId("pMsXgVXv3BLzUgSXRplE")).toBe(true);
expect(isValidVoiceId("21m00Tcm4TlvDq8ikWAM")).toBe(true);
expect(isValidVoiceId("EXAVITQu4vr4xnSDxMaL")).toBe(true);
});
it("accepts voice IDs of varying valid lengths", () => {
expect(isValidVoiceId("a1b2c3d4e5")).toBe(true);
expect(isValidVoiceId("a".repeat(40))).toBe(true);
});
it("rejects too short voice IDs", () => {
expect(isValidVoiceId("")).toBe(false);
expect(isValidVoiceId("abc")).toBe(false);
expect(isValidVoiceId("123456789")).toBe(false);
});
it("rejects too long voice IDs", () => {
expect(isValidVoiceId("a".repeat(41))).toBe(false);
expect(isValidVoiceId("a".repeat(100))).toBe(false);
});
it("rejects voice IDs with invalid characters", () => {
expect(isValidVoiceId("pMsXgVXv3BLz-gSXRplE")).toBe(false);
expect(isValidVoiceId("pMsXgVXv3BLz_gSXRplE")).toBe(false);
expect(isValidVoiceId("pMsXgVXv3BLz gSXRplE")).toBe(false);
expect(isValidVoiceId("../../../etc/passwd")).toBe(false);
expect(isValidVoiceId("voice?param=value")).toBe(false);
});
});
describe("isValidOpenAIVoice", () => {
it("accepts all valid OpenAI voices", () => {
for (const voice of OPENAI_TTS_VOICES) {
expect(isValidOpenAIVoice(voice)).toBe(true);
}
});
it("rejects invalid voice names", () => {
expect(isValidOpenAIVoice("invalid")).toBe(false);
expect(isValidOpenAIVoice("")).toBe(false);
expect(isValidOpenAIVoice("ALLOY")).toBe(false);
expect(isValidOpenAIVoice("alloy ")).toBe(false);
expect(isValidOpenAIVoice(" alloy")).toBe(false);
});
});
describe("isValidOpenAIModel", () => {
it("accepts gpt-4o-mini-tts model", () => {
expect(isValidOpenAIModel("gpt-4o-mini-tts")).toBe(true);
});
it("rejects other models", () => {
expect(isValidOpenAIModel("tts-1")).toBe(false);
expect(isValidOpenAIModel("tts-1-hd")).toBe(false);
expect(isValidOpenAIModel("invalid")).toBe(false);
expect(isValidOpenAIModel("")).toBe(false);
expect(isValidOpenAIModel("gpt-4")).toBe(false);
});
});
describe("OPENAI_TTS_MODELS", () => {
it("contains only gpt-4o-mini-tts", () => {
expect(OPENAI_TTS_MODELS).toContain("gpt-4o-mini-tts");
expect(OPENAI_TTS_MODELS).toHaveLength(1);
});
it("is a non-empty array", () => {
expect(Array.isArray(OPENAI_TTS_MODELS)).toBe(true);
expect(OPENAI_TTS_MODELS.length).toBeGreaterThan(0);
});
});
describe("resolveOutputFormat", () => {
it("uses Opus for Telegram", () => {
const output = resolveOutputFormat("telegram");
expect(output.openai).toBe("opus");
expect(output.elevenlabs).toBe("opus_48000_64");
expect(output.extension).toBe(".opus");
expect(output.voiceCompatible).toBe(true);
});
it("uses MP3 for other channels", () => {
const output = resolveOutputFormat("discord");
expect(output.openai).toBe("mp3");
expect(output.elevenlabs).toBe("mp3_44100_128");
expect(output.extension).toBe(".mp3");
expect(output.voiceCompatible).toBe(false);
});
});
describe("summarizeText", () => {
const mockApiKey = "test-api-key";
const originalFetch = globalThis.fetch;
beforeEach(() => {
vi.useFakeTimers({ shouldAdvanceTime: true });
});
afterEach(() => {
globalThis.fetch = originalFetch;
vi.useRealTimers();
});
it("summarizes text and returns result with metrics", async () => {
const mockSummary = "This is a summarized version of the text.";
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({
choices: [{ message: { content: mockSummary } }],
}),
});
const longText = "A".repeat(2000);
const result = await summarizeText(longText, 1500, mockApiKey, 30_000);
expect(result.summary).toBe(mockSummary);
expect(result.inputLength).toBe(2000);
expect(result.outputLength).toBe(mockSummary.length);
expect(result.latencyMs).toBeGreaterThanOrEqual(0);
expect(globalThis.fetch).toHaveBeenCalledTimes(1);
});
it("calls OpenAI API with correct parameters", async () => {
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({
choices: [{ message: { content: "Summary" } }],
}),
});
await summarizeText("Long text to summarize", 500, mockApiKey, 30_000);
expect(globalThis.fetch).toHaveBeenCalledWith(
"https://api.openai.com/v1/chat/completions",
expect.objectContaining({
method: "POST",
headers: {
Authorization: `Bearer ${mockApiKey}`,
"Content-Type": "application/json",
},
}),
);
const callArgs = (globalThis.fetch as ReturnType<typeof vi.fn>).mock.calls[0];
const body = JSON.parse(callArgs[1].body);
expect(body.model).toBe("gpt-4o-mini");
expect(body.temperature).toBe(0.3);
expect(body.max_tokens).toBe(250);
});
it("rejects targetLength below minimum (100)", async () => {
await expect(summarizeText("text", 99, mockApiKey, 30_000)).rejects.toThrow(
"Invalid targetLength: 99",
);
});
it("rejects targetLength above maximum (10000)", async () => {
await expect(summarizeText("text", 10001, mockApiKey, 30_000)).rejects.toThrow(
"Invalid targetLength: 10001",
);
});
it("accepts targetLength at boundaries", async () => {
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({
choices: [{ message: { content: "Summary" } }],
}),
});
await expect(summarizeText("text", 100, mockApiKey, 30_000)).resolves.toBeDefined();
await expect(summarizeText("text", 10000, mockApiKey, 30_000)).resolves.toBeDefined();
});
it("throws error when API returns non-ok response", async () => {
globalThis.fetch = vi.fn().mockResolvedValue({
ok: false,
status: 500,
});
await expect(summarizeText("text", 500, mockApiKey, 30_000)).rejects.toThrow(
"Summarization service unavailable",
);
});
it("throws error when no summary is returned", async () => {
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({
choices: [],
}),
});
await expect(summarizeText("text", 500, mockApiKey, 30_000)).rejects.toThrow(
"No summary returned",
);
});
it("throws error when summary content is empty", async () => {
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({
choices: [{ message: { content: " " } }],
}),
});
await expect(summarizeText("text", 500, mockApiKey, 30_000)).rejects.toThrow(
"No summary returned",
);
});
});
});
+630
View File
@@ -0,0 +1,630 @@
import {
existsSync,
mkdirSync,
readFileSync,
writeFileSync,
mkdtempSync,
rmSync,
renameSync,
unlinkSync,
} from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import type { ReplyPayload } from "../auto-reply/types.js";
import { normalizeChannelId } from "../channels/plugins/index.js";
import type { ChannelId } from "../channels/plugins/types.js";
import type { ClawdbotConfig } from "../config/config.js";
import type { TtsConfig, TtsMode, TtsProvider } from "../config/types.tts.js";
import { logVerbose } from "../globals.js";
import { CONFIG_DIR, resolveUserPath } from "../utils.js";
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_TTS_MAX_LENGTH = 1500;
const DEFAULT_TTS_SUMMARIZE = true;
const DEFAULT_MAX_TEXT_LENGTH = 4000;
const TEMP_FILE_CLEANUP_DELAY_MS = 5 * 60 * 1000; // 5 minutes
const DEFAULT_ELEVENLABS_VOICE_ID = "pMsXgVXv3BLzUgSXRplE";
const DEFAULT_ELEVENLABS_MODEL_ID = "eleven_multilingual_v2";
const DEFAULT_OPENAI_MODEL = "gpt-4o-mini-tts";
const DEFAULT_OPENAI_VOICE = "alloy";
const TELEGRAM_OUTPUT = {
openai: "opus" as const,
// ElevenLabs output formats use codec_sample_rate_bitrate naming.
// Opus @ 48kHz/64kbps is a good voice-note tradeoff for Telegram.
elevenlabs: "opus_48000_64",
extension: ".opus",
voiceCompatible: true,
};
const DEFAULT_OUTPUT = {
openai: "mp3" as const,
elevenlabs: "mp3_44100_128",
extension: ".mp3",
voiceCompatible: false,
};
export type ResolvedTtsConfig = {
enabled: boolean;
mode: TtsMode;
provider: TtsProvider;
elevenlabs: {
apiKey?: string;
voiceId: string;
modelId: string;
};
openai: {
apiKey?: string;
model: string;
voice: string;
};
prefsPath?: string;
maxTextLength: number;
timeoutMs: number;
};
type TtsUserPrefs = {
tts?: {
enabled?: boolean;
provider?: TtsProvider;
maxLength?: number;
summarize?: boolean;
};
};
export type TtsResult = {
success: boolean;
audioPath?: string;
error?: string;
latencyMs?: number;
provider?: string;
outputFormat?: string;
voiceCompatible?: boolean;
};
type TtsStatusEntry = {
timestamp: number;
success: boolean;
textLength: number;
summarized: boolean;
provider?: string;
latencyMs?: number;
error?: string;
};
let lastTtsAttempt: TtsStatusEntry | undefined;
export function resolveTtsConfig(cfg: ClawdbotConfig): ResolvedTtsConfig {
const raw: TtsConfig = cfg.messages?.tts ?? {};
return {
enabled: raw.enabled ?? false,
mode: raw.mode ?? "final",
provider: raw.provider ?? "elevenlabs",
elevenlabs: {
apiKey: raw.elevenlabs?.apiKey,
voiceId: raw.elevenlabs?.voiceId ?? DEFAULT_ELEVENLABS_VOICE_ID,
modelId: raw.elevenlabs?.modelId ?? DEFAULT_ELEVENLABS_MODEL_ID,
},
openai: {
apiKey: raw.openai?.apiKey,
model: raw.openai?.model ?? DEFAULT_OPENAI_MODEL,
voice: raw.openai?.voice ?? DEFAULT_OPENAI_VOICE,
},
prefsPath: raw.prefsPath,
maxTextLength: raw.maxTextLength ?? DEFAULT_MAX_TEXT_LENGTH,
timeoutMs: raw.timeoutMs ?? DEFAULT_TIMEOUT_MS,
};
}
export function resolveTtsPrefsPath(config: ResolvedTtsConfig): string {
if (config.prefsPath?.trim()) return resolveUserPath(config.prefsPath.trim());
const envPath = process.env.CLAWDBOT_TTS_PREFS?.trim();
if (envPath) return resolveUserPath(envPath);
return path.join(CONFIG_DIR, "settings", "tts.json");
}
function readPrefs(prefsPath: string): TtsUserPrefs {
try {
if (!existsSync(prefsPath)) return {};
return JSON.parse(readFileSync(prefsPath, "utf8")) as TtsUserPrefs;
} catch {
return {};
}
}
function atomicWriteFileSync(filePath: string, content: string): void {
const tmpPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).slice(2)}`;
writeFileSync(tmpPath, content);
try {
renameSync(tmpPath, filePath);
} catch (err) {
try {
unlinkSync(tmpPath);
} catch {
// ignore
}
throw err;
}
}
function updatePrefs(prefsPath: string, update: (prefs: TtsUserPrefs) => void): void {
const prefs = readPrefs(prefsPath);
update(prefs);
mkdirSync(path.dirname(prefsPath), { recursive: true });
atomicWriteFileSync(prefsPath, JSON.stringify(prefs, null, 2));
}
export function isTtsEnabled(config: ResolvedTtsConfig, prefsPath: string): boolean {
const prefs = readPrefs(prefsPath);
if (prefs.tts?.enabled !== undefined) return prefs.tts.enabled === true;
return config.enabled;
}
export function setTtsEnabled(prefsPath: string, enabled: boolean): void {
updatePrefs(prefsPath, (prefs) => {
prefs.tts = { ...prefs.tts, enabled };
});
}
export function getTtsProvider(config: ResolvedTtsConfig, prefsPath: string): TtsProvider {
const prefs = readPrefs(prefsPath);
return prefs.tts?.provider ?? config.provider;
}
export function setTtsProvider(prefsPath: string, provider: TtsProvider): void {
updatePrefs(prefsPath, (prefs) => {
prefs.tts = { ...prefs.tts, provider };
});
}
export function getTtsMaxLength(prefsPath: string): number {
const prefs = readPrefs(prefsPath);
return prefs.tts?.maxLength ?? DEFAULT_TTS_MAX_LENGTH;
}
export function setTtsMaxLength(prefsPath: string, maxLength: number): void {
updatePrefs(prefsPath, (prefs) => {
prefs.tts = { ...prefs.tts, maxLength };
});
}
export function isSummarizationEnabled(prefsPath: string): boolean {
const prefs = readPrefs(prefsPath);
return prefs.tts?.summarize ?? DEFAULT_TTS_SUMMARIZE;
}
export function setSummarizationEnabled(prefsPath: string, enabled: boolean): void {
updatePrefs(prefsPath, (prefs) => {
prefs.tts = { ...prefs.tts, summarize: enabled };
});
}
export function getLastTtsAttempt(): TtsStatusEntry | undefined {
return lastTtsAttempt;
}
export function setLastTtsAttempt(entry: TtsStatusEntry | undefined): void {
lastTtsAttempt = entry;
}
function resolveOutputFormat(channelId?: string | null) {
if (channelId === "telegram") return TELEGRAM_OUTPUT;
return DEFAULT_OUTPUT;
}
function resolveChannelId(channel: string | undefined): ChannelId | null {
return channel ? normalizeChannelId(channel) : null;
}
export function resolveTtsApiKey(
config: ResolvedTtsConfig,
provider: TtsProvider,
): string | undefined {
if (provider === "elevenlabs") {
return config.elevenlabs.apiKey || process.env.ELEVENLABS_API_KEY || process.env.XI_API_KEY;
}
if (provider === "openai") {
return config.openai.apiKey || process.env.OPENAI_API_KEY;
}
return undefined;
}
function isValidVoiceId(voiceId: string): boolean {
return /^[a-zA-Z0-9]{10,40}$/.test(voiceId);
}
export const OPENAI_TTS_MODELS = ["gpt-4o-mini-tts"] as const;
export const OPENAI_TTS_VOICES = [
"alloy",
"ash",
"coral",
"echo",
"fable",
"onyx",
"nova",
"sage",
"shimmer",
] as const;
type OpenAiTtsVoice = (typeof OPENAI_TTS_VOICES)[number];
function isValidOpenAIModel(model: string): boolean {
return OPENAI_TTS_MODELS.includes(model as (typeof OPENAI_TTS_MODELS)[number]);
}
function isValidOpenAIVoice(voice: string): voice is OpenAiTtsVoice {
return OPENAI_TTS_VOICES.includes(voice as OpenAiTtsVoice);
}
type SummarizeResult = {
summary: string;
latencyMs: number;
inputLength: number;
outputLength: number;
};
async function summarizeText(
text: string,
targetLength: number,
apiKey: string,
timeoutMs: number,
): Promise<SummarizeResult> {
if (targetLength < 100 || targetLength > 10_000) {
throw new Error(`Invalid targetLength: ${targetLength}`);
}
const startTime = Date.now();
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "gpt-4o-mini",
messages: [
{
role: "system",
content: `You are an assistant that summarizes texts concisely while keeping the most important information. Summarize the text to approximately ${targetLength} characters. Maintain the original tone and style. Reply only with the summary, without additional explanations.`,
},
{
role: "user",
content: `<text_to_summarize>\n${text}\n</text_to_summarize>`,
},
],
max_tokens: Math.ceil(targetLength / 2),
temperature: 0.3,
}),
signal: controller.signal,
});
if (!response.ok) {
throw new Error("Summarization service unavailable");
}
const data = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const summary = data.choices?.[0]?.message?.content?.trim();
if (!summary) {
throw new Error("No summary returned");
}
return {
summary,
latencyMs: Date.now() - startTime,
inputLength: text.length,
outputLength: summary.length,
};
} finally {
clearTimeout(timeout);
}
}
function scheduleCleanup(tempDir: string, delayMs: number = TEMP_FILE_CLEANUP_DELAY_MS): void {
const timer = setTimeout(() => {
try {
rmSync(tempDir, { recursive: true, force: true });
} catch {
// ignore cleanup errors
}
}, delayMs);
timer.unref();
}
async function elevenLabsTTS(params: {
text: string;
apiKey: string;
voiceId: string;
modelId: string;
outputFormat: string;
timeoutMs: number;
}): Promise<Buffer> {
const { text, apiKey, voiceId, modelId, outputFormat, timeoutMs } = params;
if (!isValidVoiceId(voiceId)) {
throw new Error("Invalid voiceId format");
}
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const url = new URL(`https://api.elevenlabs.io/v1/text-to-speech/${voiceId}`);
if (outputFormat) {
url.searchParams.set("output_format", outputFormat);
}
const response = await fetch(url.toString(), {
method: "POST",
headers: {
"xi-api-key": apiKey,
"Content-Type": "application/json",
Accept: "audio/mpeg",
},
body: JSON.stringify({
text,
model_id: modelId,
voice_settings: {
stability: 0.5,
similarity_boost: 0.75,
style: 0.0,
use_speaker_boost: true,
},
}),
signal: controller.signal,
});
if (!response.ok) {
throw new Error(`ElevenLabs API error (${response.status})`);
}
return Buffer.from(await response.arrayBuffer());
} finally {
clearTimeout(timeout);
}
}
async function openaiTTS(params: {
text: string;
apiKey: string;
model: string;
voice: string;
responseFormat: "mp3" | "opus";
timeoutMs: number;
}): Promise<Buffer> {
const { text, apiKey, model, voice, responseFormat, timeoutMs } = params;
if (!isValidOpenAIModel(model)) {
throw new Error(`Invalid model: ${model}`);
}
if (!isValidOpenAIVoice(voice)) {
throw new Error(`Invalid voice: ${voice}`);
}
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch("https://api.openai.com/v1/audio/speech", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model,
input: text,
voice,
response_format: responseFormat,
}),
signal: controller.signal,
});
if (!response.ok) {
throw new Error(`OpenAI TTS API error (${response.status})`);
}
return Buffer.from(await response.arrayBuffer());
} finally {
clearTimeout(timeout);
}
}
export async function textToSpeech(params: {
text: string;
cfg: ClawdbotConfig;
prefsPath?: string;
channel?: string;
}): Promise<TtsResult> {
const config = resolveTtsConfig(params.cfg);
const prefsPath = params.prefsPath ?? resolveTtsPrefsPath(config);
const channelId = resolveChannelId(params.channel);
const output = resolveOutputFormat(channelId);
if (params.text.length > config.maxTextLength) {
return {
success: false,
error: `Text too long (${params.text.length} chars, max ${config.maxTextLength})`,
};
}
const userProvider = getTtsProvider(config, prefsPath);
const providers: TtsProvider[] = [
userProvider,
userProvider === "openai" ? "elevenlabs" : "openai",
];
let lastError: string | undefined;
for (const provider of providers) {
const apiKey = resolveTtsApiKey(config, provider);
if (!apiKey) {
lastError = `No API key for ${provider}`;
continue;
}
const providerStart = Date.now();
try {
let audioBuffer: Buffer;
if (provider === "elevenlabs") {
audioBuffer = await elevenLabsTTS({
text: params.text,
apiKey,
voiceId: config.elevenlabs.voiceId,
modelId: config.elevenlabs.modelId,
outputFormat: output.elevenlabs,
timeoutMs: config.timeoutMs,
});
} else {
audioBuffer = await openaiTTS({
text: params.text,
apiKey,
model: config.openai.model,
voice: config.openai.voice,
responseFormat: output.openai,
timeoutMs: config.timeoutMs,
});
}
const latencyMs = Date.now() - providerStart;
const tempDir = mkdtempSync(path.join(tmpdir(), "tts-"));
const audioPath = path.join(tempDir, `voice-${Date.now()}${output.extension}`);
writeFileSync(audioPath, audioBuffer);
scheduleCleanup(tempDir);
return {
success: true,
audioPath,
latencyMs,
provider,
outputFormat: provider === "openai" ? output.openai : output.elevenlabs,
voiceCompatible: output.voiceCompatible,
};
} catch (err) {
const error = err as Error;
if (error.name === "AbortError") {
lastError = `${provider}: request timed out`;
} else {
lastError = `${provider}: ${error.message}`;
}
}
}
return {
success: false,
error: `TTS conversion failed: ${lastError || "no providers available"}`,
};
}
export async function maybeApplyTtsToPayload(params: {
payload: ReplyPayload;
cfg: ClawdbotConfig;
channel?: string;
kind?: "tool" | "block" | "final";
}): Promise<ReplyPayload> {
const config = resolveTtsConfig(params.cfg);
const prefsPath = resolveTtsPrefsPath(config);
if (!isTtsEnabled(config, prefsPath)) return params.payload;
const mode = config.mode ?? "final";
if (mode === "final" && params.kind && params.kind !== "final") return params.payload;
const text = params.payload.text ?? "";
if (!text.trim()) return params.payload;
if (params.payload.mediaUrl || (params.payload.mediaUrls?.length ?? 0) > 0) return params.payload;
if (text.includes("MEDIA:")) return params.payload;
if (text.trim().length < 10) return params.payload;
const maxLength = getTtsMaxLength(prefsPath);
let textForAudio = text.trim();
let wasSummarized = false;
if (textForAudio.length > maxLength) {
if (!isSummarizationEnabled(prefsPath)) {
logVerbose(
`TTS: skipping long text (${textForAudio.length} > ${maxLength}), summarization disabled.`,
);
return params.payload;
}
const openaiKey = resolveTtsApiKey(config, "openai");
if (!openaiKey) {
logVerbose("TTS: skipping summarization - OpenAI key missing.");
return params.payload;
}
try {
const summary = await summarizeText(textForAudio, maxLength, openaiKey, config.timeoutMs);
textForAudio = summary.summary;
wasSummarized = true;
if (textForAudio.length > config.maxTextLength) {
logVerbose(
`TTS: summary exceeded hard limit (${textForAudio.length} > ${config.maxTextLength}); truncating.`,
);
textForAudio = `${textForAudio.slice(0, config.maxTextLength - 3)}...`;
}
} catch (err) {
const error = err as Error;
logVerbose(`TTS: summarization failed: ${error.message}`);
return params.payload;
}
}
const ttsStart = Date.now();
const result = await textToSpeech({
text: textForAudio,
cfg: params.cfg,
prefsPath,
channel: params.channel,
});
if (result.success && result.audioPath) {
lastTtsAttempt = {
timestamp: Date.now(),
success: true,
textLength: text.length,
summarized: wasSummarized,
provider: result.provider,
latencyMs: result.latencyMs,
};
const channelId = resolveChannelId(params.channel);
const shouldVoice = channelId === "telegram" && result.voiceCompatible === true;
return {
...params.payload,
mediaUrl: result.audioPath,
audioAsVoice: shouldVoice || params.payload.audioAsVoice,
};
}
lastTtsAttempt = {
timestamp: Date.now(),
success: false,
textLength: text.length,
summarized: wasSummarized,
error: result.error,
};
const latency = Date.now() - ttsStart;
logVerbose(`TTS: conversion failed after ${latency}ms (${result.error ?? "unknown"}).`);
return params.payload;
}
export const _test = {
isValidVoiceId,
isValidOpenAIVoice,
isValidOpenAIModel,
OPENAI_TTS_MODELS,
OPENAI_TTS_VOICES,
summarizeText,
resolveOutputFormat,
};