Compare commits

..
Author SHA1 Message Date
Peter Steinberger 1f9a96df8e fix: session store perms + codex prompt fallback (#1049) (thanks @YuriNachos)
chore: temp landpr 1049 staging
2026-01-17 00:07:13 +00:00
37 changed files with 313 additions and 1587 deletions

No files matched your search

+31 -43
View File
@@ -2,48 +2,6 @@
## 2026.1.16 (unreleased)
### Highlights
- Web search: add `country`/`language` parameters (schema + Brave API) and docs. (#1046) — thanks @YuriNachos.
- Plugins: add Zalo Personal plugin (`@clawdbot/zalouser`) and unify channel directory for plugins. (#1032) — thanks @suminhthanh.
- Models: add Vercel AI Gateway auth choice + onboarding updates. (#1016) — thanks @timolins.
- Sessions: add `session.identityLinks` for cross-platform DM session linking. (#1033) — thanks @thewilloftheshadow.
### Breaking
- **BREAKING:** Discord/Telegram channel tokens now prefer config over env (env is fallback only).
- **BREAKING:** Matrix channel credentials now prefer config over env (env is fallback only).
### Changes
- Tools: improve `web_fetch` extraction using Readability (with fallback).
- Tools: add Firecrawl fallback for `web_fetch` when configured.
- Tools: send Chrome-like headers by default for `web_fetch` to improve extraction on bot-sensitive sites.
- Tools: Firecrawl fallback now uses bot-circumvention + cache by default; remove basic HTML fallback when extraction fails.
- Directory: unify `clawdbot directory` across channels and plugin channels.
- UI: allow deleting sessions from the Control UI.
- Skills: add user-invocable skill commands and expanded skill command registration.
- Telegram: default reaction level to minimal and enable reaction notifications by default.
- Telegram: allow reply-chain messages to bypass mention gating in groups. (#1038) — thanks @adityashaw2.
- Cron: isolated cron jobs now start a fresh session id on every run to prevent context buildup.
- Docs: add `/help` hub, Node/npm PATH guide, and expand directory CLI docs.
### Fixes
- Messages: `/stop` now hard-aborts queued followups and sub-agent runs; suppress zero-count stop notes.
- Sessions: reset `compactionCount` on `/new` and `/reset`, and preserve `sessions.json` file mode (0600).
- Sessions: repair orphaned user turns before embedded prompts.
- Channels: treat replies to the bot as implicit mentions across supported channels.
- WhatsApp: scope self-chat response prefix; inject pending-only group history and clear after any processed message.
- Agents: drop unsigned Gemini tool calls and avoid JSON Schema `format` keyword collisions.
- Auth: inherit/merge sub-agent auth profiles from the main agent.
- Gateway: resolve local auth for security probe and validate gateway token/password file modes. (#1011, #1022) — thanks @ivanrvpereira, @kkarimi.
- Signal/iMessage: bound transport readiness waits to 30s with periodic logging. (#1014) — thanks @Szpadel.
- OpenAI image-gen: remove deprecated `response_format` and use URL downloads.
- CLI: auto-update global installs when installed via a package manager.
- Discord: truncate skill command descriptions to 100 chars for slash command limits. (#1018) — thanks @evalexpr.
- Security: bump `tar` to 7.5.3.
- Models: align ZAI thinking toggles.
- iMessage: fetch remote attachments for SSH wrappers and document `remoteHost`. (#1054) — thanks @tyler6204.
## 2026.1.15
### Highlights
- Plugins: add provider auth registry + `clawdbot models auth login` for plugin-driven OAuth/API key flows.
- Browser: improve remote CDP/Browserless support (auth passthrough, `wss` upgrade, timeouts, clearer errors).
@@ -53,6 +11,8 @@
### Breaking
- **BREAKING:** iOS minimum version is now 18.0 to support Textual markdown rendering in native chat. (#702)
- **BREAKING:** Microsoft Teams is now a plugin; install `@clawdbot/msteams` via `clawdbot plugins install @clawdbot/msteams`.
- **BREAKING:** Discord/Telegram channel tokens now prefer config over env (env is fallback only).
- **BREAKING:** Matrix channel credentials now prefer config over env (env is fallback only).
### Changes
- CLI: set process titles to `clawdbot-<command>` for clearer process listings.
@@ -60,25 +20,34 @@
- Telegram: scope inline buttons with allowlist default + callback gating in DMs/groups.
- Telegram: default reaction notifications to own.
- Tools: improve `web_fetch` extraction using Readability (with fallback).
- Channels: inject only pending (mention-gated) group history; clear history on any processed message.
- Heartbeat: tighten prompt guidance + suppress duplicate alerts for 24h. (#980) — thanks @voidserf.
- Repo: ignore local identity files to avoid accidental commits. (#1001) — thanks @gerardward2007.
- Sessions/Security: add `session.dmScope` for multi-user DM isolation and audit warnings. (#948) — thanks @Alphonse-arianee.
- Sessions: add `session.identityLinks` for cross-platform DM session linking. (#1033) — thanks @thewilloftheshadow.
- Plugins: add provider auth registry + `clawdbot models auth login` for plugin-driven OAuth/API key flows.
- Skills: add user-invocable skill commands (sanitized + unique), native skill registration gating, and an opt-out for model invocation.
- Onboarding: switch channels setup to a single-select loop with per-channel actions and disabled hints in the picker.
- TUI: show provider/model labels for the active session and default model.
- Heartbeat: add per-agent heartbeat configuration and multi-agent docs example.
- UI: show gateway auth guidance + doc link on unauthorized Control UI connections.
- Security: warn on weak model tiers (Haiku, below GPT-5, below Claude 4.5) in `clawdbot security audit`.
- Apps: store node auth tokens encrypted (Keychain/SecurePrefs).
- Cron: isolated cron jobs now start a fresh session id on every run to prevent context buildup.
- Daemon: share profile/state-dir resolution across service helpers and honor `CLAWDBOT_STATE_DIR` for Windows task scripts.
- Docs: clarify multi-gateway rescue bot guidance. (#969) — thanks @bjesuiter.
- Docs: add `/help` hub, Node/npm PATH sanity guide, and installer PATH warnings (for “installed but command not found” setups). (#861)
- Docs: clarify web_search country defaults to Brave’s region choice. (#1046) — thanks @YuriNachos.
- Agents: add Current Date & Time system prompt section with configurable time format (auto/12/24).
- Agents: default to no narration for routine tool calls. (#1008) — thanks @cpojer.
- Tools: normalize Slack/Discord message timestamps with `timestampMs`/`timestampUtc` while keeping raw provider fields.
- macOS: add `system.which` for prompt-free remote skill discovery (with gateway fallback to `system.run`).
- Docs: add Date & Time guide and update prompt/timezone configuration docs.
- Tools: fall back to local Codex prompt files for unknown slash commands (supports `$1`, `$2`, `$*`).
- Messages: debounce rapid inbound messages across channels with per-connector overrides. (#971) — thanks @juanpablodlc.
- Messages: allow media-only sends (CLI/tool) and show Telegram voice recording status for voice notes. (#957) — thanks @rdev.
- Auth/Status: keep auth profiles sticky per session (rotate on compaction/new), surface provider usage headers in `/status` and `clawdbot models status`, and update docs.
- Models: add Vercel AI Gateway auth choice + onboarding docs. (#1016) — thanks @timolins.
- CLI: add `--json` output for `clawdbot daemon` lifecycle/install commands.
- Memory: make `node-llama-cpp` an optional dependency (avoid Node 25 install failures) and improve local-embeddings fallback/errors.
- Browser: add `snapshot refs=aria` (Playwright aria-ref ids) for self-resolving refs across `snapshot` → `act`.
@@ -87,23 +56,33 @@
- Browser: increase remote CDP reachability timeouts + add `remoteCdpTimeoutMs`/`remoteCdpHandshakeTimeoutMs`.
- Browser: preserve auth/query tokens for remote CDP endpoints and pass Basic auth for CDP HTTP/WS. (#895) — thanks @mukhtharcm.
- Telegram: add bidirectional reaction support with configurable notifications and agent guidance. (#964) — thanks @bohdanpodvirnyi.
- Telegram: default reaction level to minimal for more delightful baseline reactions.
- Telegram: allow custom commands in the bot menu (merged with native; conflicts ignored). (#860) — thanks @nachoiacovino.
- Discord: allow allowlisted guilds without channel lists to receive messages when `groupPolicy="allowlist"`. — thanks @thewilloftheshadow.
- Discord: allow emoji/sticker uploads + channel actions in config defaults. (#870) — thanks @JDIVE.
- Channels/Plugins: add Zalo Personal plugin (`@clawdbot/zalouser`) via zca-cli. (#1032) — thanks @suminhthanh.
### Fixes
- Messages: make `/stop` clear queued followups and pending session lane work for a hard abort.
- Messages: make `/stop` abort active sub-agent runs spawned from the requester session and report how many were stopped.
- WhatsApp: default response prefix only for self-chat, using identity name when set.
- Signal/iMessage: bound transport readiness waits to 30s with periodic logging. (#1014) — thanks @Szpadel.
- Auth: merge main auth profiles into per-agent stores for sub-agents and document inheritance. (#1013) — thanks @marcmarg.
- Agents: avoid JSON Schema `format` collisions in tool params by renaming snapshot format fields. (#1013) — thanks @marcmarg.
- Fix: make `clawdbot update` auto-update global installs when installed via a package manager.
- Fix: list model picker entries as provider/model pairs for explicit selection. (#970) — thanks @mcinteerj.
- Fix: align OpenAI image-gen defaults with DALL-E 3 standard quality and document output formats. (#880) — thanks @mkbehr.
- Fix: persist `gateway.mode=local` after selecting Local run mode in `clawdbot configure`, even if no other sections are chosen.
- Sessions: keep session store writes private on disk even when `chmod` fails. (#1049) — thanks @YuriNachos.
- Daemon: fix profile-aware service label resolution (env-driven) and add coverage for launchd/systemd/schtasks. (#969) — thanks @bjesuiter.
- Agents: avoid false positives when logging unsupported Google tool schema keywords.
- Agents: skip Gemini history downgrades for google-antigravity to preserve tool calls. (#894) — thanks @mukhtharcm.
- Agents: drop unsigned Gemini tool calls (and matching results) from context to prevent tool-call markup leakage across models.
- Status: restore usage summary line for current provider when no OAuth profiles exist.
- Fix: guard model fallback against undefined provider/model values. (#954) — thanks @roshanasingh4.
- Fix: refactor session store updates, add chat.inject, and harden subagent cleanup flow. (#944) — thanks @tyler6204.
- Fix: clean up suspended CLI processes across backends. (#978) — thanks @Nachx639.
- Fix: support MiniMax coding plan usage responses with `model_remains`/`current_interval_*` payloads.
- Fix: honor message tool channel for duplicate suppression (prefer `NO_REPLY` after `message` tool sends). (#1053) — thanks @sashcatanzarite.
- Fix: suppress WhatsApp pairing replies for historical catch-up DMs on initial link. (#904)
- Browser: extension mode recovers when only one tab is attached (stale targetId fallback).
- Browser: fix `tab not found` for extension relay snapshots/actions when Playwright blocks `newCDPSession` (use the single available Page).
@@ -113,6 +92,15 @@
- Fix: normalize pairing CLI aliases, allow extension channels, and harden Zalo webhook payload parsing. (#991) — thanks @longmaba.
- Fix: allow local Tailscale Serve hostnames without treating tailnet clients as direct. (#885) — thanks @oswalpalash.
- Fix: reset sessions after role-ordering conflicts to recover from consecutive user turns. (#998)
- Fix: repair orphaned user turns before embedded prompts to avoid role-ordering lockouts. (#1026) — thanks @odrobnik.
- Dev: auto-sync Peekaboo submodule on install when clean to avoid dirty checkouts.
- Fix: keep background exec aborts from killing backgrounded sessions while honoring timeouts.
- Fix: use local auth for gateway security probe unless remote mode has a URL. (#1011) — thanks @ivanrvpereira.
- Discord: truncate skill command descriptions for slash command limits. (#1018) — thanks @evalexpr.
- macOS: resolve gateway token/password using config mode/remote URL, and warn when `launchctl setenv` overrides config. (#1022, #1021) — thanks @kkarimi.
- Telegram: allow reply-chain messages to bypass mention gating in groups. (#1038) — thanks @adityashaw2.
- Groups: treat replies to the bot as implicit mentions across supported channels.
- Security: bump dependency `tar` to 7.5.3 to fix GHSA-8qq5-rm4j-mr97.
## 2026.1.14-1
+2 -19
View File
@@ -108,26 +108,10 @@ If you want iMessage on another Mac, set `channels.imessage.cliPath` to a wrappe
Example wrapper:
```bash
#!/usr/bin/env bash
exec ssh -T gateway-host imsg "$@"
exec ssh -T mac-mini imsg "$@"
```
**Remote attachments:** When `cliPath` points to a remote host via SSH, attachment paths in the Messages database reference files on the remote machine. Clawdbot can automatically fetch these over SCP by setting `channels.imessage.remoteHost`:
```json5
{
channels: {
imessage: {
cliPath: "~/imsg-ssh", // SSH wrapper to remote Mac
remoteHost: "user@gateway-host", // for SCP file transfer
includeAttachments: true
}
}
}
```
If `remoteHost` is not set, Clawdbot attempts to auto-detect it by parsing the SSH command in your wrapper script. Explicit configuration is recommended for reliability.
Multi-account support: use `channels.imessage.accounts` with per-account config and optional `name`. See [`gateway/configuration`](/gateway/configuration#telegramaccounts--discordaccounts--slackaccounts--signalaccounts--imessageaccounts) for the shared pattern. Don't commit `~/.clawdbot/clawdbot.json` (it often contains tokens).
Multi-account support: use `channels.imessage.accounts` with per-account config and optional `name`. See [`gateway/configuration`](/gateway/configuration#telegramaccounts--discordaccounts--slackaccounts--signalaccounts--imessageaccounts) for the shared pattern. Don’t commit `~/.clawdbot/clawdbot.json` (it often contains tokens).
## Access control (DMs + groups)
DMs:
@@ -198,7 +182,6 @@ Provider options:
- `channels.imessage.enabled`: enable/disable channel startup.
- `channels.imessage.cliPath`: path to `imsg`.
- `channels.imessage.dbPath`: Messages DB path.
- `channels.imessage.remoteHost`: SSH host for SCP attachment transfer when `cliPath` points to a remote Mac (e.g., `user@gateway-host`). Auto-detected from SSH wrapper if not set.
- `channels.imessage.service`: `imessage | sms | auto`.
- `channels.imessage.region`: SMS region.
- `channels.imessage.dmPolicy`: `pairing | allowlist | open | disabled` (default: pairing).
-18
View File
@@ -53,24 +53,6 @@ Env var equivalents:
- `CLAWDBOT_LOAD_SHELL_ENV=1`
- `CLAWDBOT_SHELL_ENV_TIMEOUT_MS=15000`
## Env var substitution in config
You can reference env vars directly in config string values using `${VAR_NAME}` syntax:
```json5
{
models: {
providers: {
"vercel-gateway": {
apiKey: "${VERCEL_GATEWAY_API_KEY}"
}
}
}
}
```
See [Configuration: Env var substitution](/gateway/configuration#env-var-substitution-in-config) for full details.
## Related
- [Gateway configuration](/gateway/configuration)
+1 -51
View File
@@ -283,48 +283,6 @@ Env var equivalent:
- `CLAWDBOT_LOAD_SHELL_ENV=1`
- `CLAWDBOT_SHELL_ENV_TIMEOUT_MS=15000`
### Env var substitution in config
You can reference environment variables directly in any config string value using
`${VAR_NAME}` syntax. Variables are substituted at config load time, before validation.
```json5
{
models: {
providers: {
"vercel-gateway": {
apiKey: "${VERCEL_GATEWAY_API_KEY}"
}
}
},
gateway: {
auth: {
token: "${CLAWDBOT_GATEWAY_TOKEN}"
}
}
}
```
**Rules:**
- Only uppercase env var names are matched: `[A-Z_][A-Z0-9_]*`
- Missing or empty env vars throw an error at config load
- Escape with `$${VAR}` to output a literal `${VAR}`
- Works with `$include` (included files also get substitution)
**Inline substitution:**
```json5
{
models: {
providers: {
custom: {
baseUrl: "${CUSTOM_API_BASE}/v1" // → "https://api.example.com/v1"
}
}
}
}
```
### Auth storage (OAuth + API keys)
Clawdbot stores **per-agent** auth profiles (OAuth + API keys) in:
@@ -1206,7 +1164,6 @@ Clawdbot spawns `imsg rpc` (JSON-RPC over stdio). No daemon or port required.
enabled: true,
cliPath: "imsg",
dbPath: "~/Library/Messages/chat.db",
remoteHost: "user@gateway-host", // SCP for remote attachments when using SSH wrapper
dmPolicy: "pairing", // pairing | allowlist | open | disabled
allowFrom: ["+15555550123", "user@example.com", "chat_id:123"],
historyLimit: 50, // include last N group messages as context (0 disables)
@@ -1226,12 +1183,11 @@ Notes:
- The first send will prompt for Messages automation permission.
- Prefer `chat_id:<id>` targets. Use `imsg chats --limit 20` to list chats.
- `channels.imessage.cliPath` can point to a wrapper script (e.g. `ssh` to another Mac that runs `imsg rpc`); use SSH keys to avoid password prompts.
- For remote SSH wrappers, set `channels.imessage.remoteHost` to fetch attachments via SCP when `includeAttachments` is enabled.
Example wrapper:
```bash
#!/usr/bin/env bash
exec ssh -T gateway-host imsg "$@"
exec ssh -T mac-mini "imsg rpc"
```
### `agents.defaults.workspace`
@@ -1759,12 +1715,6 @@ Legacy: `tools.bash` is still accepted as an alias.
- `tools.web.fetch.cacheTtlMinutes` (default 15)
- `tools.web.fetch.userAgent` (optional override)
- `tools.web.fetch.readability` (default true; disable to use basic HTML cleanup only)
- `tools.web.fetch.firecrawl.enabled` (default true when an API key is set)
- `tools.web.fetch.firecrawl.apiKey` (optional; defaults to `FIRECRAWL_API_KEY`)
- `tools.web.fetch.firecrawl.baseUrl` (default https://api.firecrawl.dev)
- `tools.web.fetch.firecrawl.onlyMainContent` (default true)
- `tools.web.fetch.firecrawl.maxAgeMs` (optional)
- `tools.web.fetch.firecrawl.timeoutSeconds` (optional)
`agents.defaults.subagents` configures sub-agent defaults:
- `model`: default model for spawned sub-agents (string or `{ primary, fallbacks }`). If omitted, sub-agents inherit the caller’s model unless overridden per agent or per call.
-58
View File
@@ -1,58 +0,0 @@
---
summary: "Firecrawl fallback for web_fetch (anti-bot + cached extraction)"
read_when:
- You want Firecrawl-backed web extraction
- You need a Firecrawl API key
- You want anti-bot extraction for web_fetch
---
# Firecrawl
Clawdbot can use **Firecrawl** as a fallback extractor for `web_fetch`. It is a hosted
content extraction service that supports bot circumvention and caching, which helps
with JS-heavy sites or pages that block plain HTTP fetches.
## Get an API key
1) Create a Firecrawl account and generate an API key.
2) Store it in config or set `FIRECRAWL_API_KEY` in the gateway environment.
## Configure Firecrawl
```json5
{
tools: {
web: {
fetch: {
firecrawl: {
apiKey: "FIRECRAWL_API_KEY_HERE",
baseUrl: "https://api.firecrawl.dev",
onlyMainContent: true,
maxAgeMs: 172800000,
timeoutSeconds: 60
}
}
}
}
}
```
Notes:
- `firecrawl.enabled` defaults to true when an API key is present.
- `maxAgeMs` controls how old cached results can be (ms). Default is 2 days.
## Stealth / bot circumvention
Firecrawl exposes a **proxy mode** parameter for bot circumvention (`basic`, `stealth`, or `auto`).
Clawdbot always uses `proxy: "auto"` plus `storeInCache: true` for Firecrawl requests.
If proxy is omitted, Firecrawl defaults to `auto`. `auto` retries with stealth proxies if a basic attempt fails, which may use more credits
than basic-only scraping.
## How `web_fetch` uses Firecrawl
`web_fetch` extraction order:
1) Readability (local)
2) Firecrawl (if configured)
3) Basic HTML cleanup (last fallback)
See [Web tools](/tools/web) for the full web tool setup.
-1
View File
@@ -215,7 +215,6 @@ Notes:
- Responses are cached (default 15 min).
- For JS-heavy sites, prefer the browser tool.
- See [Web tools](/tools/web) for setup.
- See [Firecrawl](/tools/firecrawl) for the optional anti-bot fallback.
### `browser`
Control the dedicated clawd browser.
+1
View File
@@ -94,6 +94,7 @@ Notes:
- `/restart` is disabled by default; set `commands.restart: true` to enable it.
- `/verbose` is meant for debugging and extra visibility; keep it **off** in normal use.
- `/reasoning` (and `/verbose`) are risky in group settings: they may reveal internal reasoning or tool output you did not intend to expose. Prefer leaving them off, especially in group chats.
- **Unknown slash commands:** if a command-only message starts with `/foo` and no built-in or skill command matches, Clawdbot looks for `~/.codex/prompts/foo.*` (or `$CODEX_HOME/prompts/foo.*`) and uses that file as the request body (front matter is stripped). Arguments are available as `$1`, `$2`, and `$*`.
- **Fast path:** command-only messages from allowlisted senders are handled immediately (bypass queue + model).
- **Group mention gating:** command-only messages from allowlisted senders bypass mention requirements.
- **Inline shortcuts (allowlisted senders only):** certain commands also work when embedded in a normal message and are stripped before the model sees the remaining text.
+3 -15
View File
@@ -104,7 +104,6 @@ Fetch a URL and extract readable content.
### Requirements
- `tools.web.fetch.enabled` must not be `false` (default: enabled)
- Optional Firecrawl fallback: set `tools.web.fetch.firecrawl.apiKey` or `FIRECRAWL_API_KEY`.
### Config
@@ -117,16 +116,8 @@ Fetch a URL and extract readable content.
maxChars: 50000,
timeoutSeconds: 30,
cacheTtlMinutes: 15,
userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
readability: true,
firecrawl: {
enabled: true,
apiKey: "FIRECRAWL_API_KEY_HERE", // optional if FIRECRAWL_API_KEY is set
baseUrl: "https://api.firecrawl.dev",
onlyMainContent: true,
maxAgeMs: 86400000, // ms (1 day)
timeoutSeconds: 60
}
userAgent: "clawdbot/2026.1.15",
readability: true
}
}
}
@@ -140,11 +131,8 @@ Fetch a URL and extract readable content.
- `maxChars` (truncate long pages)
Notes:
- `web_fetch` uses Readability (main-content extraction) first, then Firecrawl (if configured). If both fail, the tool returns an error.
- Firecrawl requests use bot-circumvention mode and cache results by default.
- `web_fetch` sends a Chrome-like User-Agent and `Accept-Language` by default; override `userAgent` if needed.
- `web_fetch` uses Readability (main-content extraction) by default and falls back to basic HTML cleanup if it fails.
- `web_fetch` is best-effort extraction; some sites will need the browser tool.
- See [Firecrawl](/tools/firecrawl) for key setup and service details.
- Responses are cached (default 15 minutes) to reduce repeated fetches.
- If you use tool profiles/allowlists, add `web_search`/`web_fetch` or `group:web`.
- If the Brave key is missing, `web_search` returns a short setup hint with a docs link.
-131
View File
@@ -1,131 +0,0 @@
import { extractReadableContent, fetchFirecrawlContent } from "../src/agents/tools/web-tools.js";
const DEFAULT_URLS = [
"https://en.wikipedia.org/wiki/Web_scraping",
"https://news.ycombinator.com/",
"https://www.apple.com/iphone/",
"https://www.nytimes.com/",
"https://www.reddit.com/r/javascript/",
];
const urls = process.argv.slice(2);
const targets = urls.length > 0 ? urls : DEFAULT_URLS;
const apiKey = process.env.FIRECRAWL_API_KEY;
const baseUrl = process.env.FIRECRAWL_BASE_URL ?? "https://api.firecrawl.dev";
const userAgent =
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
const timeoutMs = 30_000;
function truncate(value: string, max = 180): string {
if (!value) return "";
return value.length > max ? `${value.slice(0, max)}…` : value;
}
async function fetchHtml(url: string): Promise<{
ok: boolean;
status: number;
contentType: string;
finalUrl: string;
body: string;
}> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
try {
const res = await fetch(url, {
method: "GET",
headers: { Accept: "*/*", "User-Agent": userAgent },
signal: controller.signal,
});
const contentType = res.headers.get("content-type") ?? "application/octet-stream";
const body = await res.text();
return {
ok: res.ok,
status: res.status,
contentType,
finalUrl: res.url || url,
body,
};
} finally {
clearTimeout(timer);
}
}
async function run() {
if (!apiKey) {
console.log("FIRECRAWL_API_KEY not set. Firecrawl comparisons will be skipped.");
}
for (const url of targets) {
console.log(`\n=== ${url}`);
let localStatus = "skipped";
let localTitle = "";
let localText = "";
let localError: string | undefined;
try {
const res = await fetchHtml(url);
if (!res.ok) {
localStatus = `http ${res.status}`;
} else if (!res.contentType.includes("text/html")) {
localStatus = `non-html (${res.contentType})`;
} else {
const readable = await extractReadableContent({
html: res.body,
url: res.finalUrl,
extractMode: "markdown",
});
if (readable?.text) {
localStatus = "readability";
localTitle = readable.title ?? "";
localText = readable.text;
} else {
localStatus = "readability-empty";
}
}
} catch (error) {
localStatus = "error";
localError = error instanceof Error ? error.message : String(error);
}
console.log(
`local: ${localStatus} len=${localText.length} title=${truncate(localTitle, 80)}`
);
if (localError) console.log(`local error: ${localError}`);
if (localText) console.log(`local sample: ${truncate(localText)}`);
if (apiKey) {
try {
const firecrawl = await fetchFirecrawlContent({
url,
extractMode: "markdown",
apiKey,
baseUrl,
onlyMainContent: true,
maxAgeMs: 172_800_000,
proxy: "auto",
storeInCache: true,
timeoutSeconds: 60,
});
console.log(
`firecrawl: ok len=${firecrawl.text.length} title=${truncate(
firecrawl.title ?? "",
80,
)} status=${firecrawl.status ?? "n/a"}`
);
if (firecrawl.warning) console.log(`firecrawl warning: ${firecrawl.warning}`);
if (firecrawl.text) console.log(`firecrawl sample: ${truncate(firecrawl.text)}`);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.log(`firecrawl: error ${message}`);
}
}
}
process.exit(0);
}
run().catch((error) => {
console.error(error);
process.exit(1);
});
-60
View File
@@ -1,60 +0,0 @@
import { createWebFetchTool } from "../src/agents/tools/web-tools.js";
const DEFAULT_URLS = [
"https://example.com/",
"https://news.ycombinator.com/",
"https://www.reddit.com/r/javascript/",
"https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent",
"https://httpbin.org/html",
];
const urls = process.argv.slice(2);
const targets = urls.length > 0 ? urls : DEFAULT_URLS;
async function runFetch(url: string, readability: boolean) {
if (!readability) {
throw new Error("Basic extraction removed. Set readability=true or enable Firecrawl.");
}
const tool = createWebFetchTool({
config: {
tools: {
web: { fetch: { readability, cacheTtlMinutes: 0, firecrawl: { enabled: false } } },
},
},
sandboxed: false,
});
if (!tool) throw new Error("web_fetch tool is disabled");
const result = await tool.execute("test", { url, extractMode: "markdown" });
return result.details as {
text?: string;
title?: string;
extractor?: string;
length?: number;
truncated?: boolean;
};
}
function truncate(value: string, max = 160): string {
if (!value) return "";
return value.length > max ? `${value.slice(0, max)}…` : value;
}
async function run() {
for (const url of targets) {
console.log(`\n=== ${url}`);
const readable = await runFetch(url, true);
console.log(
`readability: ${readable.extractor ?? "unknown"} len=${readable.length ?? 0} title=${truncate(
readable.title ?? "",
80,
)}`,
);
if (readable.text) console.log(`readability sample: ${truncate(readable.text)}`);
}
}
run().catch((error) => {
console.error(error);
process.exit(1);
});
+8 -12
View File
@@ -1,6 +1,5 @@
#!/usr/bin/env python3
import argparse
import base64
import datetime as dt
import json
import os
@@ -212,21 +211,18 @@ def main() -> int:
args.output_format,
args.style,
)
data = res.get("data", [{}])[0]
image_b64 = data.get("b64_json")
image_url = data.get("url")
if not image_b64 and not image_url:
# OpenAI Images API now returns URLs by default
image_url = res.get("data", [{}])[0].get("url")
if not image_url:
raise RuntimeError(f"Unexpected response: {json.dumps(res)[:400]}")
# Download image from URL
filename = f"{idx:03d}-{slugify(prompt)[:40]}.{file_ext}"
filepath = out_dir / filename
if image_b64:
filepath.write_bytes(base64.b64decode(image_b64))
else:
try:
urllib.request.urlretrieve(image_url, filepath)
except urllib.error.URLError as e:
raise RuntimeError(f"Failed to download image from {image_url}: {e}") from e
try:
urllib.request.urlretrieve(image_url, filepath)
except urllib.error.URLError as e:
raise RuntimeError(f"Failed to download image from {image_url}: {e}") from e
items.append({"prompt": prompt, "file": filename})
@@ -1,30 +0,0 @@
import { describe, expect, it } from "vitest";
import { extractMessagingToolSend } from "./pi-embedded-subscribe.tools.js";
describe("extractMessagingToolSend", () => {
it("uses channel as provider for message tool", () => {
const result = extractMessagingToolSend("message", {
action: "send",
channel: "telegram",
to: "123",
});
expect(result?.tool).toBe("message");
expect(result?.provider).toBe("telegram");
expect(result?.to).toBe("telegram:123");
});
it("prefers provider when both provider and channel are set", () => {
const result = extractMessagingToolSend("message", {
action: "send",
provider: "slack",
channel: "telegram",
to: "channel:C1",
});
expect(result?.tool).toBe("message");
expect(result?.provider).toBe("slack");
expect(result?.to).toBe("channel:c1");
});
});
+2 -4
View File
@@ -57,10 +57,8 @@ export function extractMessagingToolSend(
const toRaw = typeof args.to === "string" ? args.to : undefined;
if (!toRaw) return undefined;
const providerRaw = typeof args.provider === "string" ? args.provider.trim() : "";
const channelRaw = typeof args.channel === "string" ? args.channel.trim() : "";
const providerHint = providerRaw || channelRaw;
const providerId = providerHint ? normalizeChannelId(providerHint) : null;
const provider = providerId ?? (providerHint ? providerHint.toLowerCase() : "message");
const providerId = providerRaw ? normalizeChannelId(providerRaw) : null;
const provider = providerId ?? (providerRaw ? providerRaw.toLowerCase() : "message");
const to = normalizeTargetForProvider(provider, toRaw);
return to ? { tool: toolName, provider, accountId, to } : undefined;
}
-1
View File
@@ -216,7 +216,6 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).toContain("message: Send messages and channel actions");
expect(prompt).toContain("### message tool");
expect(prompt).toContain("respond with ONLY: NO_REPLY");
});
it("includes runtime provider capabilities when present", () => {
-1
View File
@@ -95,7 +95,6 @@ function buildMessagingSection(params: {
"- Use `message` for proactive sends + channel actions (polls, reactions, etc.).",
"- For `action=send`, include `to` and `message`.",
`- If multiple channels are configured, pass \`channel\` (${params.messageChannelOptions}).`,
`- If you use \`message\` (\`action=send\`) to deliver your user-visible reply, respond with ONLY: ${SILENT_REPLY_TOKEN} (avoid duplicate replies).`,
params.inlineButtonsEnabled
? "- Inline buttons supported. Use `action=send` with `buttons=[[{text,callback_data}]]` (callback_data routes back as a user message)."
: params.runtimeChannel
-185
View File
@@ -1,185 +0,0 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { createWebFetchTool } from "./web-tools.js";
type MockResponse = {
ok: boolean;
status: number;
url?: string;
headers?: { get: (key: string) => string | null };
text?: () => Promise<string>;
json?: () => Promise<unknown>;
};
function makeHeaders(map: Record<string, string>): { get: (key: string) => string | null } {
return {
get: (key) => map[key.toLowerCase()] ?? null,
};
}
function htmlResponse(html: string, url = "https://example.com/"): MockResponse {
return {
ok: true,
status: 200,
url,
headers: makeHeaders({ "content-type": "text/html; charset=utf-8" }),
text: async () => html,
};
}
function firecrawlResponse(markdown: string, url = "https://example.com/"): MockResponse {
return {
ok: true,
status: 200,
json: async () => ({
success: true,
data: {
markdown,
metadata: { title: "Firecrawl Title", sourceURL: url, statusCode: 200 },
},
}),
};
}
function firecrawlError(): MockResponse {
return {
ok: false,
status: 403,
json: async () => ({ success: false, error: "blocked" }),
};
}
function requestUrl(input: RequestInfo): string {
if (typeof input === "string") return input;
if (input instanceof URL) return input.toString();
if ("url" in input && typeof input.url === "string") return input.url;
return "";
}
describe("web_fetch extraction fallbacks", () => {
const priorFetch = global.fetch;
afterEach(() => {
// @ts-expect-error restore
global.fetch = priorFetch;
vi.restoreAllMocks();
});
it("falls back to firecrawl when readability returns no content", async () => {
const mockFetch = vi.fn((input: RequestInfo) => {
const url = requestUrl(input);
if (url.includes("api.firecrawl.dev")) {
return Promise.resolve(firecrawlResponse("firecrawl content")) as Promise<Response>;
}
return Promise.resolve(
htmlResponse("<!doctype html><html><head></head><body></body></html>", url),
) as Promise<Response>;
});
// @ts-expect-error mock fetch
global.fetch = mockFetch;
const tool = createWebFetchTool({
config: {
tools: {
web: {
fetch: {
cacheTtlMinutes: 0,
firecrawl: { apiKey: "firecrawl-test" },
},
},
},
},
sandboxed: false,
});
const result = await tool?.execute?.("call", { url: "https://example.com/empty" });
const details = result?.details as { extractor?: string; text?: string };
expect(details.extractor).toBe("firecrawl");
expect(details.text).toContain("firecrawl content");
});
it("throws when readability is disabled and firecrawl is unavailable", async () => {
const mockFetch = vi.fn((input: RequestInfo) =>
Promise.resolve(htmlResponse("<html><body>hi</body></html>", requestUrl(input))),
);
// @ts-expect-error mock fetch
global.fetch = mockFetch;
const tool = createWebFetchTool({
config: {
tools: {
web: {
fetch: { readability: false, cacheTtlMinutes: 0, firecrawl: { enabled: false } },
},
},
},
sandboxed: false,
});
await expect(
tool?.execute?.("call", { url: "https://example.com/readability-off" }),
).rejects.toThrow("Readability disabled");
});
it("throws when readability is empty and firecrawl fails", async () => {
const mockFetch = vi.fn((input: RequestInfo) => {
const url = requestUrl(input);
if (url.includes("api.firecrawl.dev")) {
return Promise.resolve(firecrawlError()) as Promise<Response>;
}
return Promise.resolve(
htmlResponse("<!doctype html><html><head></head><body></body></html>", url),
) as Promise<Response>;
});
// @ts-expect-error mock fetch
global.fetch = mockFetch;
const tool = createWebFetchTool({
config: {
tools: {
web: {
fetch: { cacheTtlMinutes: 0, firecrawl: { apiKey: "firecrawl-test" } },
},
},
},
sandboxed: false,
});
await expect(
tool?.execute?.("call", { url: "https://example.com/readability-empty" }),
).rejects.toThrow("Readability and Firecrawl returned no content");
});
it("uses firecrawl when direct fetch fails", async () => {
const mockFetch = vi.fn((input: RequestInfo) => {
const url = requestUrl(input);
if (url.includes("api.firecrawl.dev")) {
return Promise.resolve(firecrawlResponse("firecrawl fallback", url)) as Promise<Response>;
}
return Promise.resolve({
ok: false,
status: 403,
headers: makeHeaders({ "content-type": "text/html" }),
text: async () => "blocked",
} as Response);
});
// @ts-expect-error mock fetch
global.fetch = mockFetch;
const tool = createWebFetchTool({
config: {
tools: {
web: {
fetch: { cacheTtlMinutes: 0, firecrawl: { apiKey: "firecrawl-test" } },
},
},
},
sandboxed: false,
});
const result = await tool?.execute?.("call", { url: "https://example.com/blocked" });
const details = result?.details as { extractor?: string; text?: string };
expect(details.extractor).toBe("firecrawl");
expect(details.text).toContain("firecrawl fallback");
});
});
+20 -323
View File
@@ -1,6 +1,7 @@
import { Type } from "@sinclair/typebox";
import type { ClawdbotConfig } from "../../config/config.js";
import { VERSION } from "../../version.js";
import { stringEnum } from "../schema/typebox.js";
import type { AnyAgentTool } from "./common.js";
import { jsonResult, readNumberParam, readStringParam } from "./common.js";
@@ -14,10 +15,6 @@ const DEFAULT_FETCH_MAX_CHARS = 50_000;
const DEFAULT_TIMEOUT_SECONDS = 30;
const DEFAULT_CACHE_TTL_MINUTES = 15;
const DEFAULT_CACHE_MAX_ENTRIES = 100;
const DEFAULT_FIRECRAWL_BASE_URL = "https://api.firecrawl.dev";
const DEFAULT_FIRECRAWL_MAX_AGE_MS = 172_800_000;
const DEFAULT_FETCH_USER_AGENT =
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
const BRAVE_SEARCH_ENDPOINT = "https://api.search.brave.com/res/v1/web/search";
@@ -33,17 +30,6 @@ type WebFetchConfig = NonNullable<ClawdbotConfig["tools"]>["web"] extends infer
: undefined
: undefined;
type FirecrawlFetchConfig =
| {
enabled?: boolean;
apiKey?: string;
baseUrl?: string;
onlyMainContent?: boolean;
maxAgeMs?: number;
timeoutSeconds?: number;
}
| undefined;
type CacheEntry<T> = {
value: T;
expiresAt: number;
@@ -137,13 +123,6 @@ function resolveFetchReadabilityEnabled(fetch?: WebFetchConfig): boolean {
return true;
}
function resolveFirecrawlConfig(fetch?: WebFetchConfig): FirecrawlFetchConfig {
if (!fetch || typeof fetch !== "object") return undefined;
const firecrawl = "firecrawl" in fetch ? fetch.firecrawl : undefined;
if (!firecrawl || typeof firecrawl !== "object") return undefined;
return firecrawl as FirecrawlFetchConfig;
}
function resolveSearchApiKey(search?: WebSearchConfig): string | undefined {
const fromConfig =
search && "apiKey" in search && typeof search.apiKey === "string" ? search.apiKey.trim() : "";
@@ -151,52 +130,6 @@ function resolveSearchApiKey(search?: WebSearchConfig): string | undefined {
return fromConfig || fromEnv || undefined;
}
function resolveFirecrawlApiKey(firecrawl?: FirecrawlFetchConfig): string | undefined {
const fromConfig =
firecrawl && "apiKey" in firecrawl && typeof firecrawl.apiKey === "string"
? firecrawl.apiKey.trim()
: "";
const fromEnv = (process.env.FIRECRAWL_API_KEY ?? "").trim();
return fromConfig || fromEnv || undefined;
}
function resolveFirecrawlEnabled(params: {
firecrawl?: FirecrawlFetchConfig;
apiKey?: string;
}): boolean {
if (typeof params.firecrawl?.enabled === "boolean") return params.firecrawl.enabled;
return Boolean(params.apiKey);
}
function resolveFirecrawlBaseUrl(firecrawl?: FirecrawlFetchConfig): string {
const raw =
firecrawl && "baseUrl" in firecrawl && typeof firecrawl.baseUrl === "string"
? firecrawl.baseUrl.trim()
: "";
return raw || DEFAULT_FIRECRAWL_BASE_URL;
}
function resolveFirecrawlOnlyMainContent(firecrawl?: FirecrawlFetchConfig): boolean {
if (typeof firecrawl?.onlyMainContent === "boolean") return firecrawl.onlyMainContent;
return true;
}
function resolveFirecrawlMaxAgeMs(firecrawl?: FirecrawlFetchConfig): number | undefined {
const raw =
firecrawl && "maxAgeMs" in firecrawl && typeof firecrawl.maxAgeMs === "number"
? firecrawl.maxAgeMs
: undefined;
if (typeof raw !== "number" || !Number.isFinite(raw)) return undefined;
const parsed = Math.max(0, Math.floor(raw));
return parsed > 0 ? parsed : undefined;
}
function resolveFirecrawlMaxAgeMsOrDefault(firecrawl?: FirecrawlFetchConfig): number {
const resolved = resolveFirecrawlMaxAgeMs(firecrawl);
if (typeof resolved === "number") return resolved;
return DEFAULT_FIRECRAWL_MAX_AGE_MS;
}
function missingSearchKeyPayload() {
return {
error: "missing_brave_api_key",
@@ -345,18 +278,9 @@ function htmlToMarkdown(html: string): { text: string; title?: string } {
return { text, title };
}
function markdownToText(markdown: string): string {
let text = markdown;
text = text.replace(/!\[[^\]]*]\([^)]+\)/g, "");
text = text.replace(/\[([^\]]+)]\([^)]+\)/g, "$1");
text = text.replace(/```[\s\S]*?```/g, (block) =>
block.replace(/```[^\n]*\n?/g, "").replace(/```/g, ""),
);
text = text.replace(/`([^`]+)`/g, "$1");
text = text.replace(/^#{1,6}\s+/gm, "");
text = text.replace(/^\s*[-*+]\s+/gm, "");
text = text.replace(/^\s*\d+\.\s+/gm, "");
return normalizeWhitespace(text);
function htmlToText(html: string): { text: string; title?: string } {
const { text, title } = htmlToMarkdown(html);
return { text, title };
}
function truncateText(value: string, maxChars: number): { text: string; truncated: boolean } {
@@ -412,81 +336,6 @@ export async function extractReadableContent(params: {
}
}
export async function fetchFirecrawlContent(params: {
url: string;
extractMode: (typeof EXTRACT_MODES)[number];
apiKey: string;
baseUrl: string;
onlyMainContent: boolean;
maxAgeMs: number;
proxy: "auto" | "basic" | "stealth";
storeInCache: boolean;
timeoutSeconds: number;
}): Promise<{
text: string;
title?: string;
finalUrl?: string;
status?: number;
warning?: string;
}> {
const endpoint = resolveFirecrawlEndpoint(params.baseUrl);
const body: Record<string, unknown> = {
url: params.url,
formats: ["markdown"],
onlyMainContent: params.onlyMainContent,
timeout: params.timeoutSeconds * 1000,
maxAge: params.maxAgeMs,
proxy: params.proxy,
storeInCache: params.storeInCache,
};
const res = await fetch(endpoint, {
method: "POST",
headers: {
Authorization: `Bearer ${params.apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
});
const payload = (await res.json()) as {
success?: boolean;
data?: {
markdown?: string;
content?: string;
metadata?: {
title?: string;
sourceURL?: string;
statusCode?: number;
};
};
warning?: string;
error?: string;
};
if (!res.ok || payload?.success === false) {
const detail = payload?.error || res.statusText;
throw new Error(`Firecrawl fetch failed (${res.status}): ${detail}`.trim());
}
const data = payload?.data ?? {};
const rawText =
typeof data.markdown === "string"
? data.markdown
: typeof data.content === "string"
? data.content
: "";
const text = params.extractMode === "text" ? markdownToText(rawText) : rawText;
return {
text,
title: data.metadata?.title,
finalUrl: data.metadata?.sourceURL,
status: data.metadata?.statusCode,
warning: payload?.warning,
};
}
async function runWebSearch(params: {
query: string;
count: number;
@@ -499,7 +348,7 @@ async function runWebSearch(params: {
ui_lang?: string;
}): Promise<Record<string, unknown>> {
const cacheKey = normalizeCacheKey(
`${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`,
`${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`
);
const cached = readCache(SEARCH_CACHE, cacheKey);
if (cached) return { ...cached.value, cached: true };
@@ -565,14 +414,6 @@ async function runWebFetch(params: {
cacheTtlMs: number;
userAgent: string;
readabilityEnabled: boolean;
firecrawlEnabled: boolean;
firecrawlApiKey?: string;
firecrawlBaseUrl: string;
firecrawlOnlyMainContent: boolean;
firecrawlMaxAgeMs: number;
firecrawlProxy: "auto" | "basic" | "stealth";
firecrawlStoreInCache: boolean;
firecrawlTimeoutSeconds: number;
}): Promise<Record<string, unknown>> {
const cacheKey = normalizeCacheKey(
`fetch:${params.url}:${params.extractMode}:${params.maxChars}`,
@@ -591,84 +432,16 @@ async function runWebFetch(params: {
}
const start = Date.now();
let res: Response;
try {
res = await fetch(parsedUrl.toString(), {
method: "GET",
headers: {
Accept: "*/*",
"User-Agent": params.userAgent,
"Accept-Language": "en-US,en;q=0.9",
},
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
});
} catch (error) {
if (params.firecrawlEnabled && params.firecrawlApiKey) {
const firecrawl = await fetchFirecrawlContent({
url: params.url,
extractMode: params.extractMode,
apiKey: params.firecrawlApiKey,
baseUrl: params.firecrawlBaseUrl,
onlyMainContent: params.firecrawlOnlyMainContent,
maxAgeMs: params.firecrawlMaxAgeMs,
proxy: params.firecrawlProxy,
storeInCache: params.firecrawlStoreInCache,
timeoutSeconds: params.firecrawlTimeoutSeconds,
});
const truncated = truncateText(firecrawl.text, params.maxChars);
const payload = {
url: params.url,
finalUrl: firecrawl.finalUrl || params.url,
status: firecrawl.status ?? 200,
contentType: "text/markdown",
title: firecrawl.title,
extractMode: params.extractMode,
extractor: "firecrawl",
truncated: truncated.truncated,
length: truncated.text.length,
fetchedAt: new Date().toISOString(),
tookMs: Date.now() - start,
text: truncated.text,
warning: firecrawl.warning,
};
writeCache(FETCH_CACHE, cacheKey, payload, params.cacheTtlMs);
return payload;
}
throw error;
}
const res = await fetch(parsedUrl.toString(), {
method: "GET",
headers: {
Accept: "*/*",
"User-Agent": params.userAgent,
},
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
});
if (!res.ok) {
if (params.firecrawlEnabled && params.firecrawlApiKey) {
const firecrawl = await fetchFirecrawlContent({
url: params.url,
extractMode: params.extractMode,
apiKey: params.firecrawlApiKey,
baseUrl: params.firecrawlBaseUrl,
onlyMainContent: params.firecrawlOnlyMainContent,
maxAgeMs: params.firecrawlMaxAgeMs,
proxy: params.firecrawlProxy,
storeInCache: params.firecrawlStoreInCache,
timeoutSeconds: params.firecrawlTimeoutSeconds,
});
const truncated = truncateText(firecrawl.text, params.maxChars);
const payload = {
url: params.url,
finalUrl: firecrawl.finalUrl || params.url,
status: firecrawl.status ?? res.status,
contentType: "text/markdown",
title: firecrawl.title,
extractMode: params.extractMode,
extractor: "firecrawl",
truncated: truncated.truncated,
length: truncated.text.length,
fetchedAt: new Date().toISOString(),
tookMs: Date.now() - start,
text: truncated.text,
warning: firecrawl.warning,
};
writeCache(FETCH_CACHE, cacheKey, payload, params.cacheTtlMs);
return payload;
}
const detail = await readResponseText(res);
throw new Error(`Web fetch failed (${res.status}): ${detail || res.statusText}`);
}
@@ -677,7 +450,6 @@ async function runWebFetch(params: {
const body = await readResponseText(res);
let title: string | undefined;
let extractor = "raw";
let text = body;
if (contentType.includes("text/html")) {
if (params.readabilityEnabled) {
@@ -689,31 +461,21 @@ async function runWebFetch(params: {
if (readable?.text) {
text = readable.text;
title = readable.title;
extractor = "readability";
} else {
const firecrawl = await tryFirecrawlFallback(params);
if (firecrawl) {
text = firecrawl.text;
title = firecrawl.title;
extractor = "firecrawl";
} else {
throw new Error(
"Web fetch extraction failed: Readability and Firecrawl returned no content.",
);
}
const parsed = params.extractMode === "text" ? htmlToText(body) : htmlToMarkdown(body);
text = parsed.text;
title = parsed.title;
}
} else {
throw new Error(
"Web fetch extraction failed: Readability disabled and Firecrawl unavailable.",
);
const parsed = params.extractMode === "text" ? htmlToText(body) : htmlToMarkdown(body);
text = parsed.text;
title = parsed.title;
}
} else if (contentType.includes("application/json")) {
try {
text = JSON.stringify(JSON.parse(body), null, 2);
extractor = "json";
} catch {
text = body;
extractor = "raw";
}
}
@@ -725,7 +487,6 @@ async function runWebFetch(params: {
contentType,
title,
extractMode: params.extractMode,
extractor,
truncated: truncated.truncated,
length: truncated.text.length,
fetchedAt: new Date().toISOString(),
@@ -736,37 +497,6 @@ async function runWebFetch(params: {
return payload;
}
async function tryFirecrawlFallback(params: {
url: string;
extractMode: (typeof EXTRACT_MODES)[number];
firecrawlEnabled: boolean;
firecrawlApiKey?: string;
firecrawlBaseUrl: string;
firecrawlOnlyMainContent: boolean;
firecrawlMaxAgeMs: number;
firecrawlProxy: "auto" | "basic" | "stealth";
firecrawlStoreInCache: boolean;
firecrawlTimeoutSeconds: number;
}): Promise<{ text: string; title?: string } | null> {
if (!params.firecrawlEnabled || !params.firecrawlApiKey) return null;
try {
const firecrawl = await fetchFirecrawlContent({
url: params.url,
extractMode: params.extractMode,
apiKey: params.firecrawlApiKey,
baseUrl: params.firecrawlBaseUrl,
onlyMainContent: params.firecrawlOnlyMainContent,
maxAgeMs: params.firecrawlMaxAgeMs,
proxy: params.firecrawlProxy,
storeInCache: params.firecrawlStoreInCache,
timeoutSeconds: params.firecrawlTimeoutSeconds,
});
return { text: firecrawl.text, title: firecrawl.title };
} catch {
return null;
}
}
export function createWebSearchTool(options?: {
config?: ClawdbotConfig;
sandboxed?: boolean;
@@ -807,21 +537,6 @@ export function createWebSearchTool(options?: {
};
}
function resolveFirecrawlEndpoint(baseUrl: string): string {
const trimmed = baseUrl.trim();
if (!trimmed) return `${DEFAULT_FIRECRAWL_BASE_URL}/v2/scrape`;
try {
const url = new URL(trimmed);
if (url.pathname && url.pathname !== "/") {
return url.toString();
}
url.pathname = "/v2/scrape";
return url.toString();
} catch {
return `${DEFAULT_FIRECRAWL_BASE_URL}/v2/scrape`;
}
}
export function createWebFetchTool(options?: {
config?: ClawdbotConfig;
sandboxed?: boolean;
@@ -829,19 +544,9 @@ export function createWebFetchTool(options?: {
const fetch = resolveFetchConfig(options?.config);
if (!resolveFetchEnabled({ fetch, sandboxed: options?.sandboxed })) return null;
const readabilityEnabled = resolveFetchReadabilityEnabled(fetch);
const firecrawl = resolveFirecrawlConfig(fetch);
const firecrawlApiKey = resolveFirecrawlApiKey(firecrawl);
const firecrawlEnabled = resolveFirecrawlEnabled({ firecrawl, apiKey: firecrawlApiKey });
const firecrawlBaseUrl = resolveFirecrawlBaseUrl(firecrawl);
const firecrawlOnlyMainContent = resolveFirecrawlOnlyMainContent(firecrawl);
const firecrawlMaxAgeMs = resolveFirecrawlMaxAgeMsOrDefault(firecrawl);
const firecrawlTimeoutSeconds = resolveTimeoutSeconds(
firecrawl?.timeoutSeconds ?? fetch?.timeoutSeconds,
DEFAULT_TIMEOUT_SECONDS,
);
const userAgent =
(fetch && "userAgent" in fetch && typeof fetch.userAgent === "string" && fetch.userAgent) ||
DEFAULT_FETCH_USER_AGENT;
`clawdbot/${VERSION}`;
return {
label: "Web Fetch",
name: "web_fetch",
@@ -861,14 +566,6 @@ export function createWebFetchTool(options?: {
cacheTtlMs: resolveCacheTtlMs(fetch?.cacheTtlMinutes, DEFAULT_CACHE_TTL_MINUTES),
userAgent,
readabilityEnabled,
firecrawlEnabled,
firecrawlApiKey,
firecrawlBaseUrl,
firecrawlOnlyMainContent,
firecrawlMaxAgeMs,
firecrawlProxy: "auto",
firecrawlStoreInCache: true,
firecrawlTimeoutSeconds,
});
return jsonResult(result);
},
@@ -0,0 +1,50 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
parseSlashCommand,
renderCodexPrompt,
resolveCodexPrompt,
stripFrontMatter,
} from "./codex-prompts.js";
describe("codex prompts", () => {
it("parses slash command names and args", () => {
expect(parseSlashCommand("/landpr 123 abc")).toEqual({ name: "landpr", args: "123 abc" });
expect(parseSlashCommand("nope")).toBeNull();
});
it("resolves prompt files and substitutes args", async () => {
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-codex-prompts-"));
const promptDir = path.join(tmp, "prompts");
await fs.mkdir(promptDir, { recursive: true });
await fs.writeFile(
path.join(promptDir, "landpr.md"),
`---\nsummary: test\n---\nHello $1 [$*] $0\n`,
"utf-8",
);
const prev = process.env.CODEX_HOME;
process.env.CODEX_HOME = tmp;
try {
const resolved = await resolveCodexPrompt("landpr");
expect(resolved?.path).toContain("landpr.md");
const stripped = stripFrontMatter(`---\nsummary: test\n---\nHello`);
expect(stripped).toBe("Hello");
const rendered = renderCodexPrompt({
body: resolved?.body ?? "",
args: "123 abc",
commandName: "landpr",
});
expect(rendered).toContain("Hello 123 [123 abc] landpr");
} finally {
if (prev === undefined) {
delete process.env.CODEX_HOME;
} else {
process.env.CODEX_HOME = prev;
}
}
});
});
+78
View File
@@ -0,0 +1,78 @@
import type { Dirent } from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
import { resolveUserPath } from "../../utils.js";
const PROMPT_NAME_RE = /^[a-z0-9_-]+$/i;
export type CodexPrompt = {
path: string;
body: string;
};
export function parseSlashCommand(input: string): { name: string; args?: string } | null {
const trimmed = input.trim();
if (!trimmed.startsWith("/")) return null;
const match = trimmed.match(/^\/([^\s]+)(?:\s+([\s\S]+))?$/);
if (!match) return null;
const name = match[1]?.trim();
if (!name) return null;
const args = match[2]?.trim();
return { name, args: args || undefined };
}
export function stripFrontMatter(input: string): string {
if (!input.startsWith("---")) return input;
const match = input.match(/^---\s*\r?\n[\s\S]*?\r?\n---\s*\r?\n?/);
return match ? input.slice(match[0].length) : input;
}
export async function resolveCodexPrompt(
commandName: string,
env: NodeJS.ProcessEnv = process.env,
): Promise<CodexPrompt | null> {
const normalized = commandName.trim().toLowerCase();
if (!PROMPT_NAME_RE.test(normalized)) return null;
const home = env.CODEX_HOME?.trim()
? resolveUserPath(env.CODEX_HOME.trim())
: resolveUserPath("~/.codex");
const promptsDir = path.join(home, "prompts");
let entries: Array<Dirent> | null = null;
try {
entries = await fs.readdir(promptsDir, { withFileTypes: true });
} catch {
return null;
}
const match = entries.find((entry) => {
if (!entry.isFile()) return false;
const base = path.parse(entry.name).name.toLowerCase();
return base === normalized;
});
if (!match) return null;
const promptPath = path.join(promptsDir, match.name);
try {
const raw = await fs.readFile(promptPath, "utf-8");
const body = stripFrontMatter(raw).trim();
if (!body) return null;
return { path: promptPath, body };
} catch {
return null;
}
}
export function renderCodexPrompt(params: {
body: string;
args?: string;
commandName?: string;
}): string {
const args = params.args?.trim() ?? "";
const tokens = args ? args.split(/\s+/) : [];
return params.body.replace(/\$(\d+|\*|@|0)/g, (match, token) => {
if (token === "*" || token === "@") return args;
if (token === "0") return params.commandName ?? "";
const index = Number(token);
if (!Number.isFinite(index) || index < 1) return match;
return tokens[index - 1] ?? "";
});
}
@@ -5,8 +5,10 @@ import type { SessionEntry } from "../../config/sessions.js";
import type { MsgContext, TemplateContext } from "../templating.js";
import type { ElevatedLevel, ReasoningLevel, ThinkLevel, VerboseLevel } from "../thinking.js";
import type { GetReplyOptions, ReplyPayload } from "../types.js";
import { listChatCommands } from "../commands-registry.js";
import { getAbortMemory } from "./abort.js";
import { buildStatusReply, handleCommands } from "./commands.js";
import { parseSlashCommand, renderCodexPrompt, resolveCodexPrompt } from "./codex-prompts.js";
import type { InlineDirectives } from "./directive-handling.js";
import { isDirectiveOnly } from "./directive-handling.js";
import type { createModelSelectionState } from "./model-selection.js";
@@ -138,6 +140,46 @@ export async function handleInlineActions(params: {
cleanedBody = rewrittenBody;
}
const slashCommand =
allowTextCommands && command.isAuthorizedSender
? parseSlashCommand(command.commandBodyNormalized)
: null;
if (slashCommand) {
const reserved = new Set<string>();
for (const entry of listChatCommands()) {
if (entry.nativeName) reserved.add(entry.nativeName.toLowerCase());
for (const alias of entry.textAliases) {
if (!alias.startsWith("/")) continue;
reserved.add(alias.slice(1).toLowerCase());
}
}
for (const entry of skillCommands) {
reserved.add(entry.name.toLowerCase());
}
const hasInlineDirective =
directives.hasThinkDirective ||
directives.hasVerboseDirective ||
directives.hasReasoningDirective ||
directives.hasElevatedDirective ||
directives.hasModelDirective ||
directives.hasQueueDirective ||
directives.hasStatusDirective;
if (!hasInlineDirective && !reserved.has(slashCommand.name.toLowerCase())) {
const prompt = await resolveCodexPrompt(slashCommand.name);
if (prompt) {
const rendered = renderCodexPrompt({
body: prompt.body,
args: slashCommand.args,
commandName: slashCommand.name,
});
ctx.Body = rendered;
sessionCtx.Body = rendered;
sessionCtx.BodyStripped = rendered;
cleanedBody = rendered;
}
}
}
const sendInlineReply = async (reply?: ReplyPayload) => {
if (!reply) return;
if (!opts?.onBlockReply) return;
+8 -9
View File
@@ -278,13 +278,10 @@ export async function initSessionState(params: {
ctx.MessageThreadId,
);
}
if (isNewSession) {
sessionEntry.compactionCount = 0;
sessionEntry.memoryFlushCompactionCount = undefined;
sessionEntry.memoryFlushAt = undefined;
}
// Preserve per-session overrides while resetting compaction state on /new.
sessionStore[sessionKey] = { ...sessionStore[sessionKey], ...sessionEntry };
// Fresh start for new sessions - don't inherit old state like compactionCount
sessionStore[sessionKey] = isNewSession
? sessionEntry
: { ...sessionStore[sessionKey], ...sessionEntry };
await updateSessionStore(storePath, (store) => {
if (groupResolution?.legacyKey && groupResolution.legacyKey !== sessionKey) {
if (store[groupResolution.legacyKey] && !store[sessionKey]) {
@@ -292,8 +289,10 @@ export async function initSessionState(params: {
}
delete store[groupResolution.legacyKey];
}
// Preserve per-session overrides while resetting compaction state on /new.
store[sessionKey] = { ...store[sessionKey], ...sessionEntry };
// Fresh start for new sessions - don't inherit old state like compactionCount
store[sessionKey] = isNewSession
? sessionEntry
: { ...store[sessionKey], ...sessionEntry };
});
const sessionCtx: TemplateContext = {
+9 -29
View File
@@ -1,12 +1,9 @@
import crypto from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { ensureSandboxWorkspaceForSession } from "../../agents/sandbox.js";
import type { ClawdbotConfig } from "../../config/config.js";
import { logVerbose } from "../../globals.js";
import { copyRemoteFileViaScp } from "../../media/remote.js";
import { ensureMediaDir } from "../../media/store.js";
import type { MsgContext, TemplateContext } from "../templating.js";
export async function stageSandboxMedia(params: {
@@ -32,8 +29,7 @@ export async function stageSandboxMedia(params: {
sessionKey,
workspaceDir,
});
const usesRemoteMedia = Boolean(ctx.MediaRemoteHost);
if (!sandbox && !usesRemoteMedia) return;
if (!sandbox) return;
const resolveAbsolutePath = (value: string): string | null => {
let resolved = value.trim();
@@ -50,9 +46,7 @@ export async function stageSandboxMedia(params: {
};
try {
const destDir = sandbox
? path.join(sandbox.workspaceDir, "media", "inbound")
: path.join(await ensureMediaDir(), "inbound");
const destDir = path.join(sandbox.workspaceDir, "media", "inbound");
await fs.mkdir(destDir, { recursive: true });
const usedNames = new Set<string>();
@@ -67,31 +61,17 @@ export async function stageSandboxMedia(params: {
if (!baseName) continue;
const parsed = path.parse(baseName);
let fileName = baseName;
if (!sandbox && usesRemoteMedia) {
fileName = `${parsed.name}-${crypto.randomUUID()}${parsed.ext}`;
} else {
let suffix = 1;
while (usedNames.has(fileName)) {
fileName = `${parsed.name}-${suffix}${parsed.ext}`;
suffix += 1;
}
let suffix = 1;
while (usedNames.has(fileName)) {
fileName = `${parsed.name}-${suffix}${parsed.ext}`;
suffix += 1;
}
usedNames.add(fileName);
const dest = path.join(destDir, fileName);
if (ctx.MediaRemoteHost) {
// Always use SCP when remote host is configured - local paths refer to remote machine
await copyRemoteFileViaScp({
remoteHost: ctx.MediaRemoteHost,
remotePath: source,
localPath: dest,
});
} else {
await fs.copyFile(source, dest);
}
// For sandbox use relative path, otherwise use absolute path
const stagedPath = sandbox ? path.posix.join("media", "inbound", fileName) : dest;
staged.set(source, stagedPath);
await fs.copyFile(source, dest);
const relative = path.posix.join("media", "inbound", fileName);
staged.set(source, relative);
}
const rewriteIfStaged = (value: string | undefined): string | undefined => {
-2
View File
@@ -38,8 +38,6 @@ export type MsgContext = {
MediaPaths?: string[];
MediaUrls?: string[];
MediaTypes?: string[];
/** Internal: remote host for inbound media fetch (iMessage SSH wrappers). */
MediaRemoteHost?: string;
Transcript?: string;
ChatType?: string;
GroupSubject?: string;
-1
View File
@@ -109,7 +109,6 @@ export const whatsappPlugin: ChannelPlugin<ResolvedWhatsAppAccount> = {
name: account.name,
enabled: account.enabled,
configured: Boolean(account.authDir),
linked: Boolean(account.authDir),
dmPolicy: account.dmPolicy,
allowFrom: account.allowFrom,
}),
-10
View File
@@ -84,13 +84,3 @@ export function normalizeAlias(alias: string): string {
export { modelKey };
export { DEFAULT_MODEL, DEFAULT_PROVIDER };
/**
* Model key format: "provider/model"
*
* The model key is displayed in `/model status` and used to reference models.
* When using `/model <key>`, use the exact format shown (e.g., "openrouter/moonshotai/kimi-k2").
*
* For providers with hierarchical model IDs (e.g., OpenRouter), the model ID may include
* sub-providers (e.g., "moonshotai/kimi-k2"), resulting in a key like "openrouter/moonshotai/kimi-k2".
*/
-275
View File
@@ -1,275 +0,0 @@
import { describe, expect, it } from "vitest";
import { MissingEnvVarError, resolveConfigEnvVars } from "./env-substitution.js";
describe("resolveConfigEnvVars", () => {
describe("basic substitution", () => {
it("substitutes a single env var", () => {
const result = resolveConfigEnvVars({ key: "${FOO}" }, { FOO: "bar" });
expect(result).toEqual({ key: "bar" });
});
it("substitutes multiple different env vars in same string", () => {
const result = resolveConfigEnvVars({ key: "${A}/${B}" }, { A: "x", B: "y" });
expect(result).toEqual({ key: "x/y" });
});
it("substitutes inline with prefix and suffix", () => {
const result = resolveConfigEnvVars({ key: "prefix-${FOO}-suffix" }, { FOO: "bar" });
expect(result).toEqual({ key: "prefix-bar-suffix" });
});
it("substitutes same var multiple times", () => {
const result = resolveConfigEnvVars({ key: "${FOO}:${FOO}" }, { FOO: "bar" });
expect(result).toEqual({ key: "bar:bar" });
});
});
describe("nested structures", () => {
it("substitutes in nested objects", () => {
const result = resolveConfigEnvVars(
{
outer: {
inner: {
key: "${API_KEY}",
},
},
},
{ API_KEY: "secret123" },
);
expect(result).toEqual({
outer: {
inner: {
key: "secret123",
},
},
});
});
it("substitutes in arrays", () => {
const result = resolveConfigEnvVars(
{ items: ["${A}", "${B}", "${C}"] },
{ A: "1", B: "2", C: "3" },
);
expect(result).toEqual({ items: ["1", "2", "3"] });
});
it("substitutes in deeply nested arrays and objects", () => {
const result = resolveConfigEnvVars(
{
providers: [
{ name: "openai", apiKey: "${OPENAI_KEY}" },
{ name: "anthropic", apiKey: "${ANTHROPIC_KEY}" },
],
},
{ OPENAI_KEY: "sk-xxx", ANTHROPIC_KEY: "sk-yyy" },
);
expect(result).toEqual({
providers: [
{ name: "openai", apiKey: "sk-xxx" },
{ name: "anthropic", apiKey: "sk-yyy" },
],
});
});
});
describe("missing env var handling", () => {
it("throws MissingEnvVarError for missing env var", () => {
expect(() => resolveConfigEnvVars({ key: "${MISSING}" }, {})).toThrow(MissingEnvVarError);
});
it("includes var name in error", () => {
try {
resolveConfigEnvVars({ key: "${MISSING_VAR}" }, {});
throw new Error("Expected to throw");
} catch (err) {
expect(err).toBeInstanceOf(MissingEnvVarError);
const error = err as MissingEnvVarError;
expect(error.varName).toBe("MISSING_VAR");
}
});
it("includes config path in error", () => {
try {
resolveConfigEnvVars({ outer: { inner: { key: "${MISSING}" } } }, {});
throw new Error("Expected to throw");
} catch (err) {
expect(err).toBeInstanceOf(MissingEnvVarError);
const error = err as MissingEnvVarError;
expect(error.configPath).toBe("outer.inner.key");
}
});
it("includes array index in config path", () => {
try {
resolveConfigEnvVars({ items: ["ok", "${MISSING}"] }, { OK: "val" });
throw new Error("Expected to throw");
} catch (err) {
expect(err).toBeInstanceOf(MissingEnvVarError);
const error = err as MissingEnvVarError;
expect(error.configPath).toBe("items[1]");
}
});
it("treats empty string env var as missing", () => {
expect(() => resolveConfigEnvVars({ key: "${EMPTY}" }, { EMPTY: "" })).toThrow(
MissingEnvVarError,
);
});
});
describe("escape syntax", () => {
it("outputs literal ${VAR} when escaped with $$", () => {
const result = resolveConfigEnvVars({ key: "$${VAR}" }, { VAR: "value" });
expect(result).toEqual({ key: "${VAR}" });
});
it("handles mix of escaped and unescaped", () => {
const result = resolveConfigEnvVars({ key: "${REAL}/$${LITERAL}" }, { REAL: "resolved" });
expect(result).toEqual({ key: "resolved/${LITERAL}" });
});
it("handles multiple escaped vars", () => {
const result = resolveConfigEnvVars({ key: "$${A}:$${B}" }, {});
expect(result).toEqual({ key: "${A}:${B}" });
});
});
describe("non-matching patterns unchanged", () => {
it("leaves $VAR (no braces) unchanged", () => {
const result = resolveConfigEnvVars({ key: "$VAR" }, { VAR: "value" });
expect(result).toEqual({ key: "$VAR" });
});
it("leaves ${lowercase} unchanged (uppercase only)", () => {
const result = resolveConfigEnvVars({ key: "${lowercase}" }, { lowercase: "value" });
expect(result).toEqual({ key: "${lowercase}" });
});
it("leaves ${MixedCase} unchanged", () => {
const result = resolveConfigEnvVars({ key: "${MixedCase}" }, { MixedCase: "value" });
expect(result).toEqual({ key: "${MixedCase}" });
});
it("leaves ${123INVALID} unchanged (must start with letter or underscore)", () => {
const result = resolveConfigEnvVars({ key: "${123INVALID}" }, {});
expect(result).toEqual({ key: "${123INVALID}" });
});
it("substitutes ${_UNDERSCORE_START} (valid)", () => {
const result = resolveConfigEnvVars(
{ key: "${_UNDERSCORE_START}" },
{ _UNDERSCORE_START: "valid" },
);
expect(result).toEqual({ key: "valid" });
});
it("substitutes ${VAR_WITH_NUMBERS_123} (valid)", () => {
const result = resolveConfigEnvVars(
{ key: "${VAR_WITH_NUMBERS_123}" },
{ VAR_WITH_NUMBERS_123: "valid" },
);
expect(result).toEqual({ key: "valid" });
});
});
describe("passthrough behavior", () => {
it("passes through primitives unchanged", () => {
expect(resolveConfigEnvVars("hello", {})).toBe("hello");
expect(resolveConfigEnvVars(42, {})).toBe(42);
expect(resolveConfigEnvVars(true, {})).toBe(true);
expect(resolveConfigEnvVars(null, {})).toBe(null);
});
it("passes through empty object", () => {
expect(resolveConfigEnvVars({}, {})).toEqual({});
});
it("passes through empty array", () => {
expect(resolveConfigEnvVars([], {})).toEqual([]);
});
it("passes through non-string values in objects", () => {
const result = resolveConfigEnvVars({ num: 42, bool: true, nil: null, arr: [1, 2] }, {});
expect(result).toEqual({ num: 42, bool: true, nil: null, arr: [1, 2] });
});
});
describe("real-world config patterns", () => {
it("substitutes API keys in provider config", () => {
const config = {
models: {
providers: {
"vercel-gateway": {
apiKey: "${VERCEL_GATEWAY_API_KEY}",
},
openai: {
apiKey: "${OPENAI_API_KEY}",
},
},
},
};
const env = {
VERCEL_GATEWAY_API_KEY: "vg_key_123",
OPENAI_API_KEY: "sk-xxx",
};
const result = resolveConfigEnvVars(config, env);
expect(result).toEqual({
models: {
providers: {
"vercel-gateway": {
apiKey: "vg_key_123",
},
openai: {
apiKey: "sk-xxx",
},
},
},
});
});
it("substitutes gateway auth token", () => {
const config = {
gateway: {
auth: {
token: "${CLAWDBOT_GATEWAY_TOKEN}",
},
},
};
const result = resolveConfigEnvVars(config, {
CLAWDBOT_GATEWAY_TOKEN: "secret-token",
});
expect(result).toEqual({
gateway: {
auth: {
token: "secret-token",
},
},
});
});
it("substitutes base URL with env var", () => {
const config = {
models: {
providers: {
custom: {
baseUrl: "${CUSTOM_API_BASE}/v1",
},
},
},
};
const result = resolveConfigEnvVars(config, {
CUSTOM_API_BASE: "https://api.example.com",
});
expect(result).toEqual({
models: {
providers: {
custom: {
baseUrl: "https://api.example.com/v1",
},
},
},
});
});
});
});
-103
View File
@@ -1,103 +0,0 @@
/**
* Environment variable substitution for config values.
*
* Supports `${VAR_NAME}` syntax in string values, substituted at config load time.
* - Only uppercase env vars are matched: `[A-Z_][A-Z0-9_]*`
* - Escape with `$${}` to output literal `${}`
* - Missing env vars throw `MissingEnvVarError` with context
*
* @example
* ```json5
* {
* models: {
* providers: {
* "vercel-gateway": {
* apiKey: "${VERCEL_GATEWAY_API_KEY}"
* }
* }
* }
* }
* ```
*/
// Pattern for valid uppercase env var names: starts with letter or underscore,
// followed by letters, numbers, or underscores (all uppercase)
const ENV_VAR_PATTERN = /\$\{([A-Z_][A-Z0-9_]*)\}/g;
// Pattern for escaped env vars: $${...} -> ${...}
const ESCAPED_PATTERN = /\$\$\{([A-Z_][A-Z0-9_]*)\}/g;
export class MissingEnvVarError extends Error {
constructor(
public readonly varName: string,
public readonly configPath: string,
) {
super(`Missing env var "${varName}" referenced at config path: ${configPath}`);
this.name = "MissingEnvVarError";
}
}
function isPlainObject(value: unknown): value is Record<string, unknown> {
return (
typeof value === "object" &&
value !== null &&
!Array.isArray(value) &&
Object.prototype.toString.call(value) === "[object Object]"
);
}
function substituteString(value: string, env: NodeJS.ProcessEnv, configPath: string): string {
// First pass: substitute real env vars
const substituted = value.replace(ENV_VAR_PATTERN, (match, varName: string) => {
// Check if this is actually an escaped var (preceded by $)
// We handle this in the second pass, so check the original string
const idx = value.indexOf(match);
if (idx > 0 && value[idx - 1] === "$") {
// This will be handled by ESCAPED_PATTERN, return as-is for now
return match;
}
const envValue = env[varName];
if (envValue === undefined || envValue === "") {
throw new MissingEnvVarError(varName, configPath);
}
return envValue;
});
// Second pass: convert escaped $${VAR} to literal ${VAR}
return substituted.replace(ESCAPED_PATTERN, (_, varName: string) => `\${${varName}}`);
}
function substituteAny(value: unknown, env: NodeJS.ProcessEnv, path: string): unknown {
if (typeof value === "string") {
return substituteString(value, env, path);
}
if (Array.isArray(value)) {
return value.map((item, index) => substituteAny(item, env, `${path}[${index}]`));
}
if (isPlainObject(value)) {
const result: Record<string, unknown> = {};
for (const [key, val] of Object.entries(value)) {
const childPath = path ? `${path}.${key}` : key;
result[key] = substituteAny(val, env, childPath);
}
return result;
}
// Primitives (number, boolean, null) pass through unchanged
return value;
}
/**
* Resolves `${VAR_NAME}` environment variable references in config values.
*
* @param obj - The parsed config object (after JSON5 parse and $include resolution)
* @param env - Environment variables to use for substitution (defaults to process.env)
* @returns The config object with env vars substituted
* @throws {MissingEnvVarError} If a referenced env var is not set or empty
*/
export function resolveConfigEnvVars(obj: unknown, env: NodeJS.ProcessEnv = process.env): unknown {
return substituteAny(obj, env, "");
}
+4 -31
View File
@@ -19,7 +19,6 @@ import {
applySessionDefaults,
applyTalkApiKey,
} from "./defaults.js";
import { MissingEnvVarError, resolveConfigEnvVars } from "./env-substitution.js";
import { ConfigIncludeError, resolveConfigIncludes } from "./includes.js";
import { applyLegacyMigrations, findLegacyConfigIssues } from "./legacy.js";
import { normalizeConfigPaths } from "./normalize-paths.js";
@@ -31,7 +30,6 @@ import { ClawdbotSchema } from "./zod-schema.js";
// Re-export for backwards compatibility
export { CircularIncludeError, ConfigIncludeError } from "./includes.js";
export { MissingEnvVarError } from "./env-substitution.js";
const SHELL_ENV_EXPECTED_KEYS = [
"OPENAI_API_KEY",
@@ -221,11 +219,8 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) {
parseJson: (raw) => deps.json5.parse(raw),
});
// Substitute ${VAR} env var references
const substituted = resolveConfigEnvVars(resolved, deps.env);
const migrated = applyLegacyMigrations(substituted);
const resolvedConfig = migrated.next ?? substituted;
const migrated = applyLegacyMigrations(resolved);
const resolvedConfig = migrated.next ?? resolved;
warnOnConfigMiskeys(resolvedConfig, deps.logger);
if (typeof resolvedConfig !== "object" || resolvedConfig === null) return {};
const validated = ClawdbotSchema.safeParse(resolvedConfig);
@@ -351,30 +346,8 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) {
};
}
// Substitute ${VAR} env var references
let substituted: unknown;
try {
substituted = resolveConfigEnvVars(resolved, deps.env);
} catch (err) {
const message =
err instanceof MissingEnvVarError
? err.message
: `Env var substitution failed: ${String(err)}`;
return {
path: configPath,
exists: true,
raw,
parsed: parsedRes.parsed,
valid: false,
config: {},
hash,
issues: [{ path: "", message }],
legacyIssues: [],
};
}
const migrated = applyLegacyMigrations(substituted);
const resolvedConfigRaw = migrated.next ?? substituted;
const migrated = applyLegacyMigrations(resolved);
const resolvedConfigRaw = migrated.next ?? resolved;
const legacyIssues = findLegacyConfigIssues(resolvedConfigRaw);
const validated = validateConfigObject(resolvedConfigRaw);
-9
View File
@@ -264,15 +264,6 @@ const FIELD_HELP: Record<string, string> = {
"tools.web.fetch.userAgent": "Override User-Agent header for web_fetch requests.",
"tools.web.fetch.readability":
"Use Readability to extract main content from HTML (fallbacks to basic HTML cleanup).",
"tools.web.fetch.firecrawl.enabled": "Enable Firecrawl fallback for web_fetch (if configured).",
"tools.web.fetch.firecrawl.apiKey": "Firecrawl API key (fallback: FIRECRAWL_API_KEY env var).",
"tools.web.fetch.firecrawl.baseUrl":
"Firecrawl base URL (e.g. https://api.firecrawl.dev or custom endpoint).",
"tools.web.fetch.firecrawl.onlyMainContent":
"When true, Firecrawl returns only the main content (default: true).",
"tools.web.fetch.firecrawl.maxAgeMs":
"Firecrawl maxAge (ms) for cached results when supported by the API.",
"tools.web.fetch.firecrawl.timeoutSeconds": "Timeout in seconds for Firecrawl requests.",
"channels.slack.allowBots":
"Allow bot-authored messages to trigger Slack replies (default: false).",
"channels.slack.thread.historyScope":
+37 -1
View File
@@ -1,7 +1,8 @@
import fs from "node:fs/promises";
import fsSync from "node:fs";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import {
buildGroupDisplayName,
@@ -157,6 +158,41 @@ describe("sessions", () => {
expect(store["agent:main:two"]?.sessionId).toBe("sess-2");
});
it("updateSessionStore preserves sessions.json permissions", async () => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
const storePath = path.join(dir, "sessions.json");
await fs.writeFile(storePath, "{}", "utf-8");
await fs.chmod(storePath, 0o600);
await updateSessionStore(storePath, (store) => {
store["agent:main:main"] = { sessionId: "sess-1", updatedAt: 1 };
});
const mode = (await fs.stat(storePath)).mode & 0o777;
if (process.platform === "win32") {
expect([0o600, 0o666, 0o777]).toContain(mode);
} else {
expect(mode).toBe(0o600);
}
});
it("updateSessionStore ignores chmod failures", async () => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
const storePath = path.join(dir, "sessions.json");
await fs.writeFile(storePath, "{}", "utf-8");
const spy = vi.spyOn(fsSync.promises, "chmod").mockRejectedValueOnce(new Error("nope"));
try {
await expect(
updateSessionStore(storePath, (store) => {
store["agent:main:main"] = { sessionId: "sess-1", updatedAt: 1 };
}),
).resolves.toBeUndefined();
} finally {
spy.mockRestore();
}
});
it("updateSessionStore keeps deletions when concurrent writes happen", async () => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
const storePath = path.join(dir, "sessions.json");
+16 -6
View File
@@ -45,6 +45,16 @@ export function clearSessionStoreCacheForTest(): void {
SESSION_STORE_CACHE.clear();
}
const sessionStoreWriteOptions = { mode: 0o600, encoding: "utf-8" } as const;
async function ensurePrivateMode(storePath: string): Promise<void> {
try {
await fs.promises.chmod(storePath, 0o600);
} catch {
// Best-effort: ignore chmod failures (e.g., Windows or filesystem quirks).
}
}
type LoadSessionStoreOptions = {
skipCache?: boolean;
};
@@ -121,7 +131,8 @@ async function saveSessionStoreUnlocked(
// We serialize writers via the session-store lock instead.
if (process.platform === "win32") {
try {
await fs.promises.writeFile(storePath, json, "utf-8");
await fs.promises.writeFile(storePath, json, sessionStoreWriteOptions);
await ensurePrivateMode(storePath);
} catch (err) {
const code =
err && typeof err === "object" && "code" in err
@@ -135,10 +146,9 @@ async function saveSessionStoreUnlocked(
const tmp = `${storePath}.${process.pid}.${crypto.randomUUID()}.tmp`;
try {
await fs.promises.writeFile(tmp, json, { mode: 0o600, encoding: "utf-8" });
await fs.promises.writeFile(tmp, json, sessionStoreWriteOptions);
await fs.promises.rename(tmp, storePath);
// Ensure permissions are set even if rename loses them
await fs.promises.chmod(storePath, 0o600);
await ensurePrivateMode(storePath);
} catch (err) {
const code =
err && typeof err === "object" && "code" in err
@@ -150,8 +160,8 @@ async function saveSessionStoreUnlocked(
// Best-effort: try a direct write (recreating the parent dir), otherwise ignore.
try {
await fs.promises.mkdir(path.dirname(storePath), { recursive: true });
await fs.promises.writeFile(storePath, json, { mode: 0o600, encoding: "utf-8" });
await fs.promises.chmod(storePath, 0o600);
await fs.promises.writeFile(storePath, json, sessionStoreWriteOptions);
await ensurePrivateMode(storePath);
} catch (err2) {
const code2 =
err2 && typeof err2 === "object" && "code" in err2
-2
View File
@@ -14,8 +14,6 @@ export type IMessageAccountConfig = {
cliPath?: string;
/** Optional Messages db path override. */
dbPath?: string;
/** Remote host for SCP when attachments live on a different machine (e.g., user@gateway-host). */
remoteHost?: string;
/** Optional default send service (imessage|sms|auto). */
service?: "imessage" | "sms" | "auto";
/** Optional default region (used when sending SMS). */
-14
View File
@@ -111,20 +111,6 @@ export type ToolsConfig = {
userAgent?: string;
/** Use Readability to extract main content (default: true). */
readability?: boolean;
firecrawl?: {
/** Enable Firecrawl fallback (default: true when apiKey is set). */
enabled?: boolean;
/** Firecrawl API key (optional; defaults to FIRECRAWL_API_KEY env var). */
apiKey?: string;
/** Firecrawl base URL (default: https://api.firecrawl.dev). */
baseUrl?: string;
/** Whether to keep only main content (default: true). */
onlyMainContent?: boolean;
/** Max age (ms) for cached Firecrawl content. */
maxAgeMs?: number;
/** Timeout in seconds for Firecrawl requests. */
timeoutSeconds?: number;
};
};
};
audio?: {
-1
View File
@@ -367,7 +367,6 @@ export const IMessageAccountSchemaBase = z.object({
configWrites: z.boolean().optional(),
cliPath: ExecutableTokenSchema.optional(),
dbPath: z.string().optional(),
remoteHost: z.string().optional(),
service: z.union([z.literal("imessage"), z.literal("sms"), z.literal("auto")]).optional(),
region: z.string().optional(),
dmPolicy: DmPolicySchema.optional().default("pairing"),
+1 -73
View File
@@ -1,5 +1,3 @@
import fs from "node:fs/promises";
import {
resolveEffectiveMessagesConfig,
resolveHumanDelayConfig,
@@ -41,7 +39,7 @@ import {
upsertChannelPairingRequest,
} from "../../pairing/pairing-store.js";
import { resolveAgentRoute } from "../../routing/resolve-route.js";
import { resolveUserPath, truncateUtf16Safe } from "../../utils.js";
import { truncateUtf16Safe } from "../../utils.js";
import { resolveIMessageAccount } from "../accounts.js";
import { createIMessageRpcClient } from "../client.js";
import { probeIMessage } from "../probe.js";
@@ -55,66 +53,6 @@ import { deliverReplies } from "./deliver.js";
import { normalizeAllowList, resolveRuntime } from "./runtime.js";
import type { IMessagePayload, MonitorIMessageOpts } from "./types.js";
/**
* Try to detect remote host from an SSH wrapper script like:
* exec ssh -T clawdbot@192.168.64.3 /opt/homebrew/bin/imsg "$@"
* exec ssh -T mac-mini imsg "$@"
* Returns the user@host or host portion if found, undefined otherwise.
*/
async function detectRemoteHostFromCliPath(cliPath: string): Promise<string | undefined> {
try {
const expanded = resolveUserPath(cliPath);
const stat = await fs.stat(expanded).catch(() => null);
if (!stat?.isFile()) return undefined;
const content = await fs.readFile(expanded, "utf8");
if (!content.includes("ssh")) return undefined;
const tokensFromLine = (line: string): string[] =>
line.match(/(?:[^\s"']+|"[^"]*"|'[^']*')+/g) ?? [];
const stripQuotes = (token: string): string => token.replace(/^['"]|['"]$/g, "");
const optsWithValue = new Set([
"-b",
"-c",
"-D",
"-E",
"-F",
"-I",
"-i",
"-J",
"-L",
"-o",
"-p",
"-R",
"-S",
"-W",
"-w",
]);
for (const rawLine of content.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line || line.startsWith("#")) continue;
if (!line.includes("ssh") || !line.includes("imsg")) continue;
const tokens = tokensFromLine(line).map(stripQuotes).filter(Boolean);
const sshIndex = tokens.findIndex((token) => token === "ssh" || token.endsWith("/ssh"));
if (sshIndex < 0) continue;
for (let idx = sshIndex + 1; idx < tokens.length; idx += 1) {
const token = tokens[idx] ?? "";
if (!token) continue;
if (token.startsWith("-")) {
if (optsWithValue.has(token)) idx += 1;
continue;
}
if (token === "imsg" || token.endsWith("/imsg")) break;
return token;
}
}
return undefined;
} catch {
return undefined;
}
}
export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): Promise<void> {
const runtime = resolveRuntime(opts);
const cfg = opts.config ?? loadConfig();
@@ -144,15 +82,6 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
const cliPath = opts.cliPath ?? imessageCfg.cliPath ?? "imsg";
const dbPath = opts.dbPath ?? imessageCfg.dbPath;
// Resolve remoteHost: explicit config, or auto-detect from SSH wrapper script
let remoteHost = imessageCfg.remoteHost;
if (!remoteHost && cliPath && cliPath !== "imsg") {
remoteHost = await detectRemoteHostFromCliPath(cliPath);
if (remoteHost) {
logVerbose(`imessage: detected remoteHost=${remoteHost} from cliPath`);
}
}
const inboundDebounceMs = resolveInboundDebounceMs({ cfg, channel: "imessage" });
const inboundDebouncer = createInboundDebouncer<{ message: IMessagePayload }>({
debounceMs: inboundDebounceMs,
@@ -440,7 +369,6 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
MediaPath: mediaPath,
MediaType: mediaType,
MediaUrl: mediaPath,
MediaRemoteHost: remoteHost,
WasMentioned: effectiveWasMentioned,
CommandAuthorized: commandAuthorized,
// Originating channel for reply routing.
-25
View File
@@ -1,25 +0,0 @@
import { describe, expect, it } from "vitest";
import { formatScpSource, quoteScpPath } from "./remote.js";
describe("quoteScpPath", () => {
it("wraps paths in single quotes", () => {
expect(quoteScpPath("/Users/bot/Messages/Attachment One.jpg")).toBe(
"'/Users/bot/Messages/Attachment One.jpg'",
);
});
it("escapes single quotes for remote shell", () => {
expect(quoteScpPath("/Users/bot/It's/Photo.jpg")).toBe(
"'/Users/bot/It'\"'\"'s/Photo.jpg'",
);
});
});
describe("formatScpSource", () => {
it("formats the scp source with quoted path", () => {
expect(formatScpSource("user@gateway-host", "/Users/bot/Hello World.jpg")).toBe(
"user@gateway-host:'/Users/bot/Hello World.jpg'",
);
});
});
-44
View File
@@ -1,44 +0,0 @@
import { spawn } from "node:child_process";
const SINGLE_QUOTE_ESCAPE = "'\"'\"'";
export function quoteScpPath(value: string): string {
return `'${value.replace(/'/g, SINGLE_QUOTE_ESCAPE)}'`;
}
export function formatScpSource(remoteHost: string, remotePath: string): string {
return `${remoteHost}:${quoteScpPath(remotePath)}`;
}
export async function copyRemoteFileViaScp(params: {
remoteHost: string;
remotePath: string;
localPath: string;
}): Promise<void> {
return await new Promise((resolve, reject) => {
const child = spawn(
"/usr/bin/scp",
[
"-o",
"BatchMode=yes",
"-o",
"StrictHostKeyChecking=accept-new",
formatScpSource(params.remoteHost, params.remotePath),
params.localPath,
],
{ stdio: ["ignore", "ignore", "pipe"] },
);
let stderr = "";
child.stderr?.setEncoding("utf8");
child.stderr?.on("data", (chunk) => {
stderr += chunk;
});
child.once("error", reject);
child.once("exit", (code) => {
if (code === 0) resolve();
else reject(new Error(`scp failed (${code}): ${stderr.trim()}`));
});
});
}