Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Peter Steinberger
|
1f9a96df8e |
No files matched your search
+31
-42
@@ -2,48 +2,6 @@
|
||||
|
||||
## 2026.1.16 (unreleased)
|
||||
|
||||
### Highlights
|
||||
- Web search: add `country`/`language` parameters (schema + Brave API) and docs. (#1046) — thanks @YuriNachos.
|
||||
- Plugins: add Zalo Personal plugin (`@clawdbot/zalouser`) and unify channel directory for plugins. (#1032) — thanks @suminhthanh.
|
||||
- Models: add Vercel AI Gateway auth choice + onboarding updates. (#1016) — thanks @timolins.
|
||||
- Sessions: add `session.identityLinks` for cross-platform DM session linking. (#1033) — thanks @thewilloftheshadow.
|
||||
|
||||
### Breaking
|
||||
- **BREAKING:** Discord/Telegram channel tokens now prefer config over env (env is fallback only).
|
||||
- **BREAKING:** Matrix channel credentials now prefer config over env (env is fallback only).
|
||||
|
||||
### Changes
|
||||
- Tools: improve `web_fetch` extraction using Readability (with fallback).
|
||||
- Tools: add Firecrawl fallback for `web_fetch` when configured.
|
||||
- Tools: send Chrome-like headers by default for `web_fetch` to improve extraction on bot-sensitive sites.
|
||||
- Tools: Firecrawl fallback now uses bot-circumvention + cache by default; remove basic HTML fallback when extraction fails.
|
||||
- Directory: unify `clawdbot directory` across channels and plugin channels.
|
||||
- UI: allow deleting sessions from the Control UI.
|
||||
- Skills: add user-invocable skill commands and expanded skill command registration.
|
||||
- Telegram: default reaction level to minimal and enable reaction notifications by default.
|
||||
- Telegram: allow reply-chain messages to bypass mention gating in groups. (#1038) — thanks @adityashaw2.
|
||||
- Cron: isolated cron jobs now start a fresh session id on every run to prevent context buildup.
|
||||
- Docs: add `/help` hub, Node/npm PATH guide, and expand directory CLI docs.
|
||||
- Config: support env var substitution in config values. (#1044) — thanks @sebslight.
|
||||
|
||||
### Fixes
|
||||
- Messages: `/stop` now hard-aborts queued followups and sub-agent runs; suppress zero-count stop notes.
|
||||
- Sessions: reset `compactionCount` on `/new` and `/reset`, and preserve `sessions.json` file mode (0600).
|
||||
- Sessions: repair orphaned user turns before embedded prompts.
|
||||
- Channels: treat replies to the bot as implicit mentions across supported channels.
|
||||
- WhatsApp: scope self-chat response prefix; inject pending-only group history and clear after any processed message.
|
||||
- Agents: drop unsigned Gemini tool calls and avoid JSON Schema `format` keyword collisions.
|
||||
- Auth: inherit/merge sub-agent auth profiles from the main agent.
|
||||
- Gateway: resolve local auth for security probe and validate gateway token/password file modes. (#1011, #1022) — thanks @ivanrvpereira, @kkarimi.
|
||||
- Signal/iMessage: bound transport readiness waits to 30s with periodic logging. (#1014) — thanks @Szpadel.
|
||||
- OpenAI image-gen: remove deprecated `response_format` and use URL downloads.
|
||||
- CLI: auto-update global installs when installed via a package manager.
|
||||
- Discord: truncate skill command descriptions to 100 chars for slash command limits. (#1018) — thanks @evalexpr.
|
||||
- Security: bump `tar` to 7.5.3.
|
||||
- Models: align ZAI thinking toggles.
|
||||
|
||||
## 2026.1.15
|
||||
|
||||
### Highlights
|
||||
- Plugins: add provider auth registry + `clawdbot models auth login` for plugin-driven OAuth/API key flows.
|
||||
- Browser: improve remote CDP/Browserless support (auth passthrough, `wss` upgrade, timeouts, clearer errors).
|
||||
@@ -53,6 +11,8 @@
|
||||
### Breaking
|
||||
- **BREAKING:** iOS minimum version is now 18.0 to support Textual markdown rendering in native chat. (#702)
|
||||
- **BREAKING:** Microsoft Teams is now a plugin; install `@clawdbot/msteams` via `clawdbot plugins install @clawdbot/msteams`.
|
||||
- **BREAKING:** Discord/Telegram channel tokens now prefer config over env (env is fallback only).
|
||||
- **BREAKING:** Matrix channel credentials now prefer config over env (env is fallback only).
|
||||
|
||||
### Changes
|
||||
- CLI: set process titles to `clawdbot-<command>` for clearer process listings.
|
||||
@@ -60,25 +20,34 @@
|
||||
- Telegram: scope inline buttons with allowlist default + callback gating in DMs/groups.
|
||||
- Telegram: default reaction notifications to own.
|
||||
- Tools: improve `web_fetch` extraction using Readability (with fallback).
|
||||
- Channels: inject only pending (mention-gated) group history; clear history on any processed message.
|
||||
- Heartbeat: tighten prompt guidance + suppress duplicate alerts for 24h. (#980) — thanks @voidserf.
|
||||
- Repo: ignore local identity files to avoid accidental commits. (#1001) — thanks @gerardward2007.
|
||||
- Sessions/Security: add `session.dmScope` for multi-user DM isolation and audit warnings. (#948) — thanks @Alphonse-arianee.
|
||||
- Sessions: add `session.identityLinks` for cross-platform DM session linking. (#1033) — thanks @thewilloftheshadow.
|
||||
- Plugins: add provider auth registry + `clawdbot models auth login` for plugin-driven OAuth/API key flows.
|
||||
- Skills: add user-invocable skill commands (sanitized + unique), native skill registration gating, and an opt-out for model invocation.
|
||||
- Onboarding: switch channels setup to a single-select loop with per-channel actions and disabled hints in the picker.
|
||||
- TUI: show provider/model labels for the active session and default model.
|
||||
- Heartbeat: add per-agent heartbeat configuration and multi-agent docs example.
|
||||
- UI: show gateway auth guidance + doc link on unauthorized Control UI connections.
|
||||
- Security: warn on weak model tiers (Haiku, below GPT-5, below Claude 4.5) in `clawdbot security audit`.
|
||||
- Apps: store node auth tokens encrypted (Keychain/SecurePrefs).
|
||||
- Cron: isolated cron jobs now start a fresh session id on every run to prevent context buildup.
|
||||
- Daemon: share profile/state-dir resolution across service helpers and honor `CLAWDBOT_STATE_DIR` for Windows task scripts.
|
||||
- Docs: clarify multi-gateway rescue bot guidance. (#969) — thanks @bjesuiter.
|
||||
- Docs: add `/help` hub, Node/npm PATH sanity guide, and installer PATH warnings (for “installed but command not found” setups). (#861)
|
||||
- Docs: clarify web_search country defaults to Brave’s region choice. (#1046) — thanks @YuriNachos.
|
||||
- Agents: add Current Date & Time system prompt section with configurable time format (auto/12/24).
|
||||
- Agents: default to no narration for routine tool calls. (#1008) — thanks @cpojer.
|
||||
- Tools: normalize Slack/Discord message timestamps with `timestampMs`/`timestampUtc` while keeping raw provider fields.
|
||||
- macOS: add `system.which` for prompt-free remote skill discovery (with gateway fallback to `system.run`).
|
||||
- Docs: add Date & Time guide and update prompt/timezone configuration docs.
|
||||
- Tools: fall back to local Codex prompt files for unknown slash commands (supports `$1`, `$2`, `$*`).
|
||||
- Messages: debounce rapid inbound messages across channels with per-connector overrides. (#971) — thanks @juanpablodlc.
|
||||
- Messages: allow media-only sends (CLI/tool) and show Telegram voice recording status for voice notes. (#957) — thanks @rdev.
|
||||
- Auth/Status: keep auth profiles sticky per session (rotate on compaction/new), surface provider usage headers in `/status` and `clawdbot models status`, and update docs.
|
||||
- Models: add Vercel AI Gateway auth choice + onboarding docs. (#1016) — thanks @timolins.
|
||||
- CLI: add `--json` output for `clawdbot daemon` lifecycle/install commands.
|
||||
- Memory: make `node-llama-cpp` an optional dependency (avoid Node 25 install failures) and improve local-embeddings fallback/errors.
|
||||
- Browser: add `snapshot refs=aria` (Playwright aria-ref ids) for self-resolving refs across `snapshot` → `act`.
|
||||
@@ -87,17 +56,28 @@
|
||||
- Browser: increase remote CDP reachability timeouts + add `remoteCdpTimeoutMs`/`remoteCdpHandshakeTimeoutMs`.
|
||||
- Browser: preserve auth/query tokens for remote CDP endpoints and pass Basic auth for CDP HTTP/WS. (#895) — thanks @mukhtharcm.
|
||||
- Telegram: add bidirectional reaction support with configurable notifications and agent guidance. (#964) — thanks @bohdanpodvirnyi.
|
||||
- Telegram: default reaction level to minimal for more delightful baseline reactions.
|
||||
- Telegram: allow custom commands in the bot menu (merged with native; conflicts ignored). (#860) — thanks @nachoiacovino.
|
||||
- Discord: allow allowlisted guilds without channel lists to receive messages when `groupPolicy="allowlist"`. — thanks @thewilloftheshadow.
|
||||
- Discord: allow emoji/sticker uploads + channel actions in config defaults. (#870) — thanks @JDIVE.
|
||||
- Channels/Plugins: add Zalo Personal plugin (`@clawdbot/zalouser`) via zca-cli. (#1032) — thanks @suminhthanh.
|
||||
|
||||
### Fixes
|
||||
- Messages: make `/stop` clear queued followups and pending session lane work for a hard abort.
|
||||
- Messages: make `/stop` abort active sub-agent runs spawned from the requester session and report how many were stopped.
|
||||
- WhatsApp: default response prefix only for self-chat, using identity name when set.
|
||||
- Signal/iMessage: bound transport readiness waits to 30s with periodic logging. (#1014) — thanks @Szpadel.
|
||||
- Auth: merge main auth profiles into per-agent stores for sub-agents and document inheritance. (#1013) — thanks @marcmarg.
|
||||
- Agents: avoid JSON Schema `format` collisions in tool params by renaming snapshot format fields. (#1013) — thanks @marcmarg.
|
||||
- Fix: make `clawdbot update` auto-update global installs when installed via a package manager.
|
||||
- Fix: list model picker entries as provider/model pairs for explicit selection. (#970) — thanks @mcinteerj.
|
||||
- Fix: align OpenAI image-gen defaults with DALL-E 3 standard quality and document output formats. (#880) — thanks @mkbehr.
|
||||
- Fix: persist `gateway.mode=local` after selecting Local run mode in `clawdbot configure`, even if no other sections are chosen.
|
||||
- Sessions: keep session store writes private on disk even when `chmod` fails. (#1049) — thanks @YuriNachos.
|
||||
- Daemon: fix profile-aware service label resolution (env-driven) and add coverage for launchd/systemd/schtasks. (#969) — thanks @bjesuiter.
|
||||
- Agents: avoid false positives when logging unsupported Google tool schema keywords.
|
||||
- Agents: skip Gemini history downgrades for google-antigravity to preserve tool calls. (#894) — thanks @mukhtharcm.
|
||||
- Agents: drop unsigned Gemini tool calls (and matching results) from context to prevent tool-call markup leakage across models.
|
||||
- Status: restore usage summary line for current provider when no OAuth profiles exist.
|
||||
- Fix: guard model fallback against undefined provider/model values. (#954) — thanks @roshanasingh4.
|
||||
- Fix: refactor session store updates, add chat.inject, and harden subagent cleanup flow. (#944) — thanks @tyler6204.
|
||||
@@ -112,6 +92,15 @@
|
||||
- Fix: normalize pairing CLI aliases, allow extension channels, and harden Zalo webhook payload parsing. (#991) — thanks @longmaba.
|
||||
- Fix: allow local Tailscale Serve hostnames without treating tailnet clients as direct. (#885) — thanks @oswalpalash.
|
||||
- Fix: reset sessions after role-ordering conflicts to recover from consecutive user turns. (#998)
|
||||
- Fix: repair orphaned user turns before embedded prompts to avoid role-ordering lockouts. (#1026) — thanks @odrobnik.
|
||||
- Dev: auto-sync Peekaboo submodule on install when clean to avoid dirty checkouts.
|
||||
- Fix: keep background exec aborts from killing backgrounded sessions while honoring timeouts.
|
||||
- Fix: use local auth for gateway security probe unless remote mode has a URL. (#1011) — thanks @ivanrvpereira.
|
||||
- Discord: truncate skill command descriptions for slash command limits. (#1018) — thanks @evalexpr.
|
||||
- macOS: resolve gateway token/password using config mode/remote URL, and warn when `launchctl setenv` overrides config. (#1022, #1021) — thanks @kkarimi.
|
||||
- Telegram: allow reply-chain messages to bypass mention gating in groups. (#1038) — thanks @adityashaw2.
|
||||
- Groups: treat replies to the bot as implicit mentions across supported channels.
|
||||
- Security: bump dependency `tar` to 7.5.3 to fix GHSA-8qq5-rm4j-mr97.
|
||||
|
||||
## 2026.1.14-1
|
||||
|
||||
|
||||
@@ -53,24 +53,6 @@ Env var equivalents:
|
||||
- `CLAWDBOT_LOAD_SHELL_ENV=1`
|
||||
- `CLAWDBOT_SHELL_ENV_TIMEOUT_MS=15000`
|
||||
|
||||
## Env var substitution in config
|
||||
|
||||
You can reference env vars directly in config string values using `${VAR_NAME}` syntax:
|
||||
|
||||
```json5
|
||||
{
|
||||
models: {
|
||||
providers: {
|
||||
"vercel-gateway": {
|
||||
apiKey: "${VERCEL_GATEWAY_API_KEY}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See [Configuration: Env var substitution](/gateway/configuration#env-var-substitution-in-config) for full details.
|
||||
|
||||
## Related
|
||||
|
||||
- [Gateway configuration](/gateway/configuration)
|
||||
|
||||
@@ -283,48 +283,6 @@ Env var equivalent:
|
||||
- `CLAWDBOT_LOAD_SHELL_ENV=1`
|
||||
- `CLAWDBOT_SHELL_ENV_TIMEOUT_MS=15000`
|
||||
|
||||
### Env var substitution in config
|
||||
|
||||
You can reference environment variables directly in any config string value using
|
||||
`${VAR_NAME}` syntax. Variables are substituted at config load time, before validation.
|
||||
|
||||
```json5
|
||||
{
|
||||
models: {
|
||||
providers: {
|
||||
"vercel-gateway": {
|
||||
apiKey: "${VERCEL_GATEWAY_API_KEY}"
|
||||
}
|
||||
}
|
||||
},
|
||||
gateway: {
|
||||
auth: {
|
||||
token: "${CLAWDBOT_GATEWAY_TOKEN}"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
- Only uppercase env var names are matched: `[A-Z_][A-Z0-9_]*`
|
||||
- Missing or empty env vars throw an error at config load
|
||||
- Escape with `$${VAR}` to output a literal `${VAR}`
|
||||
- Works with `$include` (included files also get substitution)
|
||||
|
||||
**Inline substitution:**
|
||||
|
||||
```json5
|
||||
{
|
||||
models: {
|
||||
providers: {
|
||||
custom: {
|
||||
baseUrl: "${CUSTOM_API_BASE}/v1" // → "https://api.example.com/v1"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Auth storage (OAuth + API keys)
|
||||
|
||||
Clawdbot stores **per-agent** auth profiles (OAuth + API keys) in:
|
||||
@@ -1757,12 +1715,6 @@ Legacy: `tools.bash` is still accepted as an alias.
|
||||
- `tools.web.fetch.cacheTtlMinutes` (default 15)
|
||||
- `tools.web.fetch.userAgent` (optional override)
|
||||
- `tools.web.fetch.readability` (default true; disable to use basic HTML cleanup only)
|
||||
- `tools.web.fetch.firecrawl.enabled` (default true when an API key is set)
|
||||
- `tools.web.fetch.firecrawl.apiKey` (optional; defaults to `FIRECRAWL_API_KEY`)
|
||||
- `tools.web.fetch.firecrawl.baseUrl` (default https://api.firecrawl.dev)
|
||||
- `tools.web.fetch.firecrawl.onlyMainContent` (default true)
|
||||
- `tools.web.fetch.firecrawl.maxAgeMs` (optional)
|
||||
- `tools.web.fetch.firecrawl.timeoutSeconds` (optional)
|
||||
|
||||
`agents.defaults.subagents` configures sub-agent defaults:
|
||||
- `model`: default model for spawned sub-agents (string or `{ primary, fallbacks }`). If omitted, sub-agents inherit the caller’s model unless overridden per agent or per call.
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
---
|
||||
summary: "Firecrawl fallback for web_fetch (anti-bot + cached extraction)"
|
||||
read_when:
|
||||
- You want Firecrawl-backed web extraction
|
||||
- You need a Firecrawl API key
|
||||
- You want anti-bot extraction for web_fetch
|
||||
---
|
||||
|
||||
# Firecrawl
|
||||
|
||||
Clawdbot can use **Firecrawl** as a fallback extractor for `web_fetch`. It is a hosted
|
||||
content extraction service that supports bot circumvention and caching, which helps
|
||||
with JS-heavy sites or pages that block plain HTTP fetches.
|
||||
|
||||
## Get an API key
|
||||
|
||||
1) Create a Firecrawl account and generate an API key.
|
||||
2) Store it in config or set `FIRECRAWL_API_KEY` in the gateway environment.
|
||||
|
||||
## Configure Firecrawl
|
||||
|
||||
```json5
|
||||
{
|
||||
tools: {
|
||||
web: {
|
||||
fetch: {
|
||||
firecrawl: {
|
||||
apiKey: "FIRECRAWL_API_KEY_HERE",
|
||||
baseUrl: "https://api.firecrawl.dev",
|
||||
onlyMainContent: true,
|
||||
maxAgeMs: 172800000,
|
||||
timeoutSeconds: 60
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `firecrawl.enabled` defaults to true when an API key is present.
|
||||
- `maxAgeMs` controls how old cached results can be (ms). Default is 2 days.
|
||||
|
||||
## Stealth / bot circumvention
|
||||
|
||||
Firecrawl exposes a **proxy mode** parameter for bot circumvention (`basic`, `stealth`, or `auto`).
|
||||
Clawdbot always uses `proxy: "auto"` plus `storeInCache: true` for Firecrawl requests.
|
||||
If proxy is omitted, Firecrawl defaults to `auto`. `auto` retries with stealth proxies if a basic attempt fails, which may use more credits
|
||||
than basic-only scraping.
|
||||
|
||||
## How `web_fetch` uses Firecrawl
|
||||
|
||||
`web_fetch` extraction order:
|
||||
1) Readability (local)
|
||||
2) Firecrawl (if configured)
|
||||
3) Basic HTML cleanup (last fallback)
|
||||
|
||||
See [Web tools](/tools/web) for the full web tool setup.
|
||||
@@ -215,7 +215,6 @@ Notes:
|
||||
- Responses are cached (default 15 min).
|
||||
- For JS-heavy sites, prefer the browser tool.
|
||||
- See [Web tools](/tools/web) for setup.
|
||||
- See [Firecrawl](/tools/firecrawl) for the optional anti-bot fallback.
|
||||
|
||||
### `browser`
|
||||
Control the dedicated clawd browser.
|
||||
|
||||
@@ -94,6 +94,7 @@ Notes:
|
||||
- `/restart` is disabled by default; set `commands.restart: true` to enable it.
|
||||
- `/verbose` is meant for debugging and extra visibility; keep it **off** in normal use.
|
||||
- `/reasoning` (and `/verbose`) are risky in group settings: they may reveal internal reasoning or tool output you did not intend to expose. Prefer leaving them off, especially in group chats.
|
||||
- **Unknown slash commands:** if a command-only message starts with `/foo` and no built-in or skill command matches, Clawdbot looks for `~/.codex/prompts/foo.*` (or `$CODEX_HOME/prompts/foo.*`) and uses that file as the request body (front matter is stripped). Arguments are available as `$1`, `$2`, and `$*`.
|
||||
- **Fast path:** command-only messages from allowlisted senders are handled immediately (bypass queue + model).
|
||||
- **Group mention gating:** command-only messages from allowlisted senders bypass mention requirements.
|
||||
- **Inline shortcuts (allowlisted senders only):** certain commands also work when embedded in a normal message and are stripped before the model sees the remaining text.
|
||||
|
||||
+3
-15
@@ -104,7 +104,6 @@ Fetch a URL and extract readable content.
|
||||
### Requirements
|
||||
|
||||
- `tools.web.fetch.enabled` must not be `false` (default: enabled)
|
||||
- Optional Firecrawl fallback: set `tools.web.fetch.firecrawl.apiKey` or `FIRECRAWL_API_KEY`.
|
||||
|
||||
### Config
|
||||
|
||||
@@ -117,16 +116,8 @@ Fetch a URL and extract readable content.
|
||||
maxChars: 50000,
|
||||
timeoutSeconds: 30,
|
||||
cacheTtlMinutes: 15,
|
||||
userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
|
||||
readability: true,
|
||||
firecrawl: {
|
||||
enabled: true,
|
||||
apiKey: "FIRECRAWL_API_KEY_HERE", // optional if FIRECRAWL_API_KEY is set
|
||||
baseUrl: "https://api.firecrawl.dev",
|
||||
onlyMainContent: true,
|
||||
maxAgeMs: 86400000, // ms (1 day)
|
||||
timeoutSeconds: 60
|
||||
}
|
||||
userAgent: "clawdbot/2026.1.15",
|
||||
readability: true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -140,11 +131,8 @@ Fetch a URL and extract readable content.
|
||||
- `maxChars` (truncate long pages)
|
||||
|
||||
Notes:
|
||||
- `web_fetch` uses Readability (main-content extraction) first, then Firecrawl (if configured). If both fail, the tool returns an error.
|
||||
- Firecrawl requests use bot-circumvention mode and cache results by default.
|
||||
- `web_fetch` sends a Chrome-like User-Agent and `Accept-Language` by default; override `userAgent` if needed.
|
||||
- `web_fetch` uses Readability (main-content extraction) by default and falls back to basic HTML cleanup if it fails.
|
||||
- `web_fetch` is best-effort extraction; some sites will need the browser tool.
|
||||
- See [Firecrawl](/tools/firecrawl) for key setup and service details.
|
||||
- Responses are cached (default 15 minutes) to reduce repeated fetches.
|
||||
- If you use tool profiles/allowlists, add `web_search`/`web_fetch` or `group:web`.
|
||||
- If the Brave key is missing, `web_search` returns a short setup hint with a docs link.
|
||||
@@ -1,131 +0,0 @@
|
||||
import { extractReadableContent, fetchFirecrawlContent } from "../src/agents/tools/web-tools.js";
|
||||
|
||||
const DEFAULT_URLS = [
|
||||
"https://en.wikipedia.org/wiki/Web_scraping",
|
||||
"https://news.ycombinator.com/",
|
||||
"https://www.apple.com/iphone/",
|
||||
"https://www.nytimes.com/",
|
||||
"https://www.reddit.com/r/javascript/",
|
||||
];
|
||||
|
||||
const urls = process.argv.slice(2);
|
||||
const targets = urls.length > 0 ? urls : DEFAULT_URLS;
|
||||
const apiKey = process.env.FIRECRAWL_API_KEY;
|
||||
const baseUrl = process.env.FIRECRAWL_BASE_URL ?? "https://api.firecrawl.dev";
|
||||
|
||||
const userAgent =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
|
||||
const timeoutMs = 30_000;
|
||||
|
||||
function truncate(value: string, max = 180): string {
|
||||
if (!value) return "";
|
||||
return value.length > max ? `${value.slice(0, max)}…` : value;
|
||||
}
|
||||
|
||||
async function fetchHtml(url: string): Promise<{
|
||||
ok: boolean;
|
||||
status: number;
|
||||
contentType: string;
|
||||
finalUrl: string;
|
||||
body: string;
|
||||
}> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
try {
|
||||
const res = await fetch(url, {
|
||||
method: "GET",
|
||||
headers: { Accept: "*/*", "User-Agent": userAgent },
|
||||
signal: controller.signal,
|
||||
});
|
||||
const contentType = res.headers.get("content-type") ?? "application/octet-stream";
|
||||
const body = await res.text();
|
||||
return {
|
||||
ok: res.ok,
|
||||
status: res.status,
|
||||
contentType,
|
||||
finalUrl: res.url || url,
|
||||
body,
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
async function run() {
|
||||
if (!apiKey) {
|
||||
console.log("FIRECRAWL_API_KEY not set. Firecrawl comparisons will be skipped.");
|
||||
}
|
||||
|
||||
for (const url of targets) {
|
||||
console.log(`\n=== ${url}`);
|
||||
let localStatus = "skipped";
|
||||
let localTitle = "";
|
||||
let localText = "";
|
||||
let localError: string | undefined;
|
||||
|
||||
try {
|
||||
const res = await fetchHtml(url);
|
||||
if (!res.ok) {
|
||||
localStatus = `http ${res.status}`;
|
||||
} else if (!res.contentType.includes("text/html")) {
|
||||
localStatus = `non-html (${res.contentType})`;
|
||||
} else {
|
||||
const readable = await extractReadableContent({
|
||||
html: res.body,
|
||||
url: res.finalUrl,
|
||||
extractMode: "markdown",
|
||||
});
|
||||
if (readable?.text) {
|
||||
localStatus = "readability";
|
||||
localTitle = readable.title ?? "";
|
||||
localText = readable.text;
|
||||
} else {
|
||||
localStatus = "readability-empty";
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
localStatus = "error";
|
||||
localError = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`local: ${localStatus} len=${localText.length} title=${truncate(localTitle, 80)}`
|
||||
);
|
||||
if (localError) console.log(`local error: ${localError}`);
|
||||
if (localText) console.log(`local sample: ${truncate(localText)}`);
|
||||
|
||||
if (apiKey) {
|
||||
try {
|
||||
const firecrawl = await fetchFirecrawlContent({
|
||||
url,
|
||||
extractMode: "markdown",
|
||||
apiKey,
|
||||
baseUrl,
|
||||
onlyMainContent: true,
|
||||
maxAgeMs: 172_800_000,
|
||||
proxy: "auto",
|
||||
storeInCache: true,
|
||||
timeoutSeconds: 60,
|
||||
});
|
||||
console.log(
|
||||
`firecrawl: ok len=${firecrawl.text.length} title=${truncate(
|
||||
firecrawl.title ?? "",
|
||||
80,
|
||||
)} status=${firecrawl.status ?? "n/a"}`
|
||||
);
|
||||
if (firecrawl.warning) console.log(`firecrawl warning: ${firecrawl.warning}`);
|
||||
if (firecrawl.text) console.log(`firecrawl sample: ${truncate(firecrawl.text)}`);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.log(`firecrawl: error ${message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
run().catch((error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,60 +0,0 @@
|
||||
import { createWebFetchTool } from "../src/agents/tools/web-tools.js";
|
||||
|
||||
const DEFAULT_URLS = [
|
||||
"https://example.com/",
|
||||
"https://news.ycombinator.com/",
|
||||
"https://www.reddit.com/r/javascript/",
|
||||
"https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent",
|
||||
"https://httpbin.org/html",
|
||||
];
|
||||
|
||||
const urls = process.argv.slice(2);
|
||||
const targets = urls.length > 0 ? urls : DEFAULT_URLS;
|
||||
|
||||
async function runFetch(url: string, readability: boolean) {
|
||||
if (!readability) {
|
||||
throw new Error("Basic extraction removed. Set readability=true or enable Firecrawl.");
|
||||
}
|
||||
const tool = createWebFetchTool({
|
||||
config: {
|
||||
tools: {
|
||||
web: { fetch: { readability, cacheTtlMinutes: 0, firecrawl: { enabled: false } } },
|
||||
},
|
||||
},
|
||||
sandboxed: false,
|
||||
});
|
||||
if (!tool) throw new Error("web_fetch tool is disabled");
|
||||
const result = await tool.execute("test", { url, extractMode: "markdown" });
|
||||
return result.details as {
|
||||
text?: string;
|
||||
title?: string;
|
||||
extractor?: string;
|
||||
length?: number;
|
||||
truncated?: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
function truncate(value: string, max = 160): string {
|
||||
if (!value) return "";
|
||||
return value.length > max ? `${value.slice(0, max)}…` : value;
|
||||
}
|
||||
|
||||
async function run() {
|
||||
for (const url of targets) {
|
||||
console.log(`\n=== ${url}`);
|
||||
const readable = await runFetch(url, true);
|
||||
|
||||
console.log(
|
||||
`readability: ${readable.extractor ?? "unknown"} len=${readable.length ?? 0} title=${truncate(
|
||||
readable.title ?? "",
|
||||
80,
|
||||
)}`,
|
||||
);
|
||||
if (readable.text) console.log(`readability sample: ${truncate(readable.text)}`);
|
||||
}
|
||||
}
|
||||
|
||||
run().catch((error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,6 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import base64
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
@@ -212,21 +211,18 @@ def main() -> int:
|
||||
args.output_format,
|
||||
args.style,
|
||||
)
|
||||
data = res.get("data", [{}])[0]
|
||||
image_b64 = data.get("b64_json")
|
||||
image_url = data.get("url")
|
||||
if not image_b64 and not image_url:
|
||||
# OpenAI Images API now returns URLs by default
|
||||
image_url = res.get("data", [{}])[0].get("url")
|
||||
if not image_url:
|
||||
raise RuntimeError(f"Unexpected response: {json.dumps(res)[:400]}")
|
||||
|
||||
# Download image from URL
|
||||
filename = f"{idx:03d}-{slugify(prompt)[:40]}.{file_ext}"
|
||||
filepath = out_dir / filename
|
||||
if image_b64:
|
||||
filepath.write_bytes(base64.b64decode(image_b64))
|
||||
else:
|
||||
try:
|
||||
urllib.request.urlretrieve(image_url, filepath)
|
||||
except urllib.error.URLError as e:
|
||||
raise RuntimeError(f"Failed to download image from {image_url}: {e}") from e
|
||||
try:
|
||||
urllib.request.urlretrieve(image_url, filepath)
|
||||
except urllib.error.URLError as e:
|
||||
raise RuntimeError(f"Failed to download image from {image_url}: {e}") from e
|
||||
|
||||
items.append({"prompt": prompt, "file": filename})
|
||||
|
||||
|
||||
@@ -1,185 +0,0 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createWebFetchTool } from "./web-tools.js";
|
||||
|
||||
type MockResponse = {
|
||||
ok: boolean;
|
||||
status: number;
|
||||
url?: string;
|
||||
headers?: { get: (key: string) => string | null };
|
||||
text?: () => Promise<string>;
|
||||
json?: () => Promise<unknown>;
|
||||
};
|
||||
|
||||
function makeHeaders(map: Record<string, string>): { get: (key: string) => string | null } {
|
||||
return {
|
||||
get: (key) => map[key.toLowerCase()] ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function htmlResponse(html: string, url = "https://example.com/"): MockResponse {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
url,
|
||||
headers: makeHeaders({ "content-type": "text/html; charset=utf-8" }),
|
||||
text: async () => html,
|
||||
};
|
||||
}
|
||||
|
||||
function firecrawlResponse(markdown: string, url = "https://example.com/"): MockResponse {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
json: async () => ({
|
||||
success: true,
|
||||
data: {
|
||||
markdown,
|
||||
metadata: { title: "Firecrawl Title", sourceURL: url, statusCode: 200 },
|
||||
},
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function firecrawlError(): MockResponse {
|
||||
return {
|
||||
ok: false,
|
||||
status: 403,
|
||||
json: async () => ({ success: false, error: "blocked" }),
|
||||
};
|
||||
}
|
||||
|
||||
function requestUrl(input: RequestInfo): string {
|
||||
if (typeof input === "string") return input;
|
||||
if (input instanceof URL) return input.toString();
|
||||
if ("url" in input && typeof input.url === "string") return input.url;
|
||||
return "";
|
||||
}
|
||||
|
||||
describe("web_fetch extraction fallbacks", () => {
|
||||
const priorFetch = global.fetch;
|
||||
|
||||
afterEach(() => {
|
||||
// @ts-expect-error restore
|
||||
global.fetch = priorFetch;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("falls back to firecrawl when readability returns no content", async () => {
|
||||
const mockFetch = vi.fn((input: RequestInfo) => {
|
||||
const url = requestUrl(input);
|
||||
if (url.includes("api.firecrawl.dev")) {
|
||||
return Promise.resolve(firecrawlResponse("firecrawl content")) as Promise<Response>;
|
||||
}
|
||||
return Promise.resolve(
|
||||
htmlResponse("<!doctype html><html><head></head><body></body></html>", url),
|
||||
) as Promise<Response>;
|
||||
});
|
||||
// @ts-expect-error mock fetch
|
||||
global.fetch = mockFetch;
|
||||
|
||||
const tool = createWebFetchTool({
|
||||
config: {
|
||||
tools: {
|
||||
web: {
|
||||
fetch: {
|
||||
cacheTtlMinutes: 0,
|
||||
firecrawl: { apiKey: "firecrawl-test" },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
sandboxed: false,
|
||||
});
|
||||
|
||||
const result = await tool?.execute?.("call", { url: "https://example.com/empty" });
|
||||
const details = result?.details as { extractor?: string; text?: string };
|
||||
expect(details.extractor).toBe("firecrawl");
|
||||
expect(details.text).toContain("firecrawl content");
|
||||
});
|
||||
|
||||
it("throws when readability is disabled and firecrawl is unavailable", async () => {
|
||||
const mockFetch = vi.fn((input: RequestInfo) =>
|
||||
Promise.resolve(htmlResponse("<html><body>hi</body></html>", requestUrl(input))),
|
||||
);
|
||||
// @ts-expect-error mock fetch
|
||||
global.fetch = mockFetch;
|
||||
|
||||
const tool = createWebFetchTool({
|
||||
config: {
|
||||
tools: {
|
||||
web: {
|
||||
fetch: { readability: false, cacheTtlMinutes: 0, firecrawl: { enabled: false } },
|
||||
},
|
||||
},
|
||||
},
|
||||
sandboxed: false,
|
||||
});
|
||||
|
||||
await expect(
|
||||
tool?.execute?.("call", { url: "https://example.com/readability-off" }),
|
||||
).rejects.toThrow("Readability disabled");
|
||||
});
|
||||
|
||||
it("throws when readability is empty and firecrawl fails", async () => {
|
||||
const mockFetch = vi.fn((input: RequestInfo) => {
|
||||
const url = requestUrl(input);
|
||||
if (url.includes("api.firecrawl.dev")) {
|
||||
return Promise.resolve(firecrawlError()) as Promise<Response>;
|
||||
}
|
||||
return Promise.resolve(
|
||||
htmlResponse("<!doctype html><html><head></head><body></body></html>", url),
|
||||
) as Promise<Response>;
|
||||
});
|
||||
// @ts-expect-error mock fetch
|
||||
global.fetch = mockFetch;
|
||||
|
||||
const tool = createWebFetchTool({
|
||||
config: {
|
||||
tools: {
|
||||
web: {
|
||||
fetch: { cacheTtlMinutes: 0, firecrawl: { apiKey: "firecrawl-test" } },
|
||||
},
|
||||
},
|
||||
},
|
||||
sandboxed: false,
|
||||
});
|
||||
|
||||
await expect(
|
||||
tool?.execute?.("call", { url: "https://example.com/readability-empty" }),
|
||||
).rejects.toThrow("Readability and Firecrawl returned no content");
|
||||
});
|
||||
|
||||
it("uses firecrawl when direct fetch fails", async () => {
|
||||
const mockFetch = vi.fn((input: RequestInfo) => {
|
||||
const url = requestUrl(input);
|
||||
if (url.includes("api.firecrawl.dev")) {
|
||||
return Promise.resolve(firecrawlResponse("firecrawl fallback", url)) as Promise<Response>;
|
||||
}
|
||||
return Promise.resolve({
|
||||
ok: false,
|
||||
status: 403,
|
||||
headers: makeHeaders({ "content-type": "text/html" }),
|
||||
text: async () => "blocked",
|
||||
} as Response);
|
||||
});
|
||||
// @ts-expect-error mock fetch
|
||||
global.fetch = mockFetch;
|
||||
|
||||
const tool = createWebFetchTool({
|
||||
config: {
|
||||
tools: {
|
||||
web: {
|
||||
fetch: { cacheTtlMinutes: 0, firecrawl: { apiKey: "firecrawl-test" } },
|
||||
},
|
||||
},
|
||||
},
|
||||
sandboxed: false,
|
||||
});
|
||||
|
||||
const result = await tool?.execute?.("call", { url: "https://example.com/blocked" });
|
||||
const details = result?.details as { extractor?: string; text?: string };
|
||||
expect(details.extractor).toBe("firecrawl");
|
||||
expect(details.text).toContain("firecrawl fallback");
|
||||
});
|
||||
});
|
||||
+20
-323
@@ -1,6 +1,7 @@
|
||||
import { Type } from "@sinclair/typebox";
|
||||
|
||||
import type { ClawdbotConfig } from "../../config/config.js";
|
||||
import { VERSION } from "../../version.js";
|
||||
import { stringEnum } from "../schema/typebox.js";
|
||||
import type { AnyAgentTool } from "./common.js";
|
||||
import { jsonResult, readNumberParam, readStringParam } from "./common.js";
|
||||
@@ -14,10 +15,6 @@ const DEFAULT_FETCH_MAX_CHARS = 50_000;
|
||||
const DEFAULT_TIMEOUT_SECONDS = 30;
|
||||
const DEFAULT_CACHE_TTL_MINUTES = 15;
|
||||
const DEFAULT_CACHE_MAX_ENTRIES = 100;
|
||||
const DEFAULT_FIRECRAWL_BASE_URL = "https://api.firecrawl.dev";
|
||||
const DEFAULT_FIRECRAWL_MAX_AGE_MS = 172_800_000;
|
||||
const DEFAULT_FETCH_USER_AGENT =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
|
||||
|
||||
const BRAVE_SEARCH_ENDPOINT = "https://api.search.brave.com/res/v1/web/search";
|
||||
|
||||
@@ -33,17 +30,6 @@ type WebFetchConfig = NonNullable<ClawdbotConfig["tools"]>["web"] extends infer
|
||||
: undefined
|
||||
: undefined;
|
||||
|
||||
type FirecrawlFetchConfig =
|
||||
| {
|
||||
enabled?: boolean;
|
||||
apiKey?: string;
|
||||
baseUrl?: string;
|
||||
onlyMainContent?: boolean;
|
||||
maxAgeMs?: number;
|
||||
timeoutSeconds?: number;
|
||||
}
|
||||
| undefined;
|
||||
|
||||
type CacheEntry<T> = {
|
||||
value: T;
|
||||
expiresAt: number;
|
||||
@@ -137,13 +123,6 @@ function resolveFetchReadabilityEnabled(fetch?: WebFetchConfig): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
function resolveFirecrawlConfig(fetch?: WebFetchConfig): FirecrawlFetchConfig {
|
||||
if (!fetch || typeof fetch !== "object") return undefined;
|
||||
const firecrawl = "firecrawl" in fetch ? fetch.firecrawl : undefined;
|
||||
if (!firecrawl || typeof firecrawl !== "object") return undefined;
|
||||
return firecrawl as FirecrawlFetchConfig;
|
||||
}
|
||||
|
||||
function resolveSearchApiKey(search?: WebSearchConfig): string | undefined {
|
||||
const fromConfig =
|
||||
search && "apiKey" in search && typeof search.apiKey === "string" ? search.apiKey.trim() : "";
|
||||
@@ -151,52 +130,6 @@ function resolveSearchApiKey(search?: WebSearchConfig): string | undefined {
|
||||
return fromConfig || fromEnv || undefined;
|
||||
}
|
||||
|
||||
function resolveFirecrawlApiKey(firecrawl?: FirecrawlFetchConfig): string | undefined {
|
||||
const fromConfig =
|
||||
firecrawl && "apiKey" in firecrawl && typeof firecrawl.apiKey === "string"
|
||||
? firecrawl.apiKey.trim()
|
||||
: "";
|
||||
const fromEnv = (process.env.FIRECRAWL_API_KEY ?? "").trim();
|
||||
return fromConfig || fromEnv || undefined;
|
||||
}
|
||||
|
||||
function resolveFirecrawlEnabled(params: {
|
||||
firecrawl?: FirecrawlFetchConfig;
|
||||
apiKey?: string;
|
||||
}): boolean {
|
||||
if (typeof params.firecrawl?.enabled === "boolean") return params.firecrawl.enabled;
|
||||
return Boolean(params.apiKey);
|
||||
}
|
||||
|
||||
function resolveFirecrawlBaseUrl(firecrawl?: FirecrawlFetchConfig): string {
|
||||
const raw =
|
||||
firecrawl && "baseUrl" in firecrawl && typeof firecrawl.baseUrl === "string"
|
||||
? firecrawl.baseUrl.trim()
|
||||
: "";
|
||||
return raw || DEFAULT_FIRECRAWL_BASE_URL;
|
||||
}
|
||||
|
||||
function resolveFirecrawlOnlyMainContent(firecrawl?: FirecrawlFetchConfig): boolean {
|
||||
if (typeof firecrawl?.onlyMainContent === "boolean") return firecrawl.onlyMainContent;
|
||||
return true;
|
||||
}
|
||||
|
||||
function resolveFirecrawlMaxAgeMs(firecrawl?: FirecrawlFetchConfig): number | undefined {
|
||||
const raw =
|
||||
firecrawl && "maxAgeMs" in firecrawl && typeof firecrawl.maxAgeMs === "number"
|
||||
? firecrawl.maxAgeMs
|
||||
: undefined;
|
||||
if (typeof raw !== "number" || !Number.isFinite(raw)) return undefined;
|
||||
const parsed = Math.max(0, Math.floor(raw));
|
||||
return parsed > 0 ? parsed : undefined;
|
||||
}
|
||||
|
||||
function resolveFirecrawlMaxAgeMsOrDefault(firecrawl?: FirecrawlFetchConfig): number {
|
||||
const resolved = resolveFirecrawlMaxAgeMs(firecrawl);
|
||||
if (typeof resolved === "number") return resolved;
|
||||
return DEFAULT_FIRECRAWL_MAX_AGE_MS;
|
||||
}
|
||||
|
||||
function missingSearchKeyPayload() {
|
||||
return {
|
||||
error: "missing_brave_api_key",
|
||||
@@ -345,18 +278,9 @@ function htmlToMarkdown(html: string): { text: string; title?: string } {
|
||||
return { text, title };
|
||||
}
|
||||
|
||||
function markdownToText(markdown: string): string {
|
||||
let text = markdown;
|
||||
text = text.replace(/!\[[^\]]*]\([^)]+\)/g, "");
|
||||
text = text.replace(/\[([^\]]+)]\([^)]+\)/g, "$1");
|
||||
text = text.replace(/```[\s\S]*?```/g, (block) =>
|
||||
block.replace(/```[^\n]*\n?/g, "").replace(/```/g, ""),
|
||||
);
|
||||
text = text.replace(/`([^`]+)`/g, "$1");
|
||||
text = text.replace(/^#{1,6}\s+/gm, "");
|
||||
text = text.replace(/^\s*[-*+]\s+/gm, "");
|
||||
text = text.replace(/^\s*\d+\.\s+/gm, "");
|
||||
return normalizeWhitespace(text);
|
||||
function htmlToText(html: string): { text: string; title?: string } {
|
||||
const { text, title } = htmlToMarkdown(html);
|
||||
return { text, title };
|
||||
}
|
||||
|
||||
function truncateText(value: string, maxChars: number): { text: string; truncated: boolean } {
|
||||
@@ -412,81 +336,6 @@ export async function extractReadableContent(params: {
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchFirecrawlContent(params: {
|
||||
url: string;
|
||||
extractMode: (typeof EXTRACT_MODES)[number];
|
||||
apiKey: string;
|
||||
baseUrl: string;
|
||||
onlyMainContent: boolean;
|
||||
maxAgeMs: number;
|
||||
proxy: "auto" | "basic" | "stealth";
|
||||
storeInCache: boolean;
|
||||
timeoutSeconds: number;
|
||||
}): Promise<{
|
||||
text: string;
|
||||
title?: string;
|
||||
finalUrl?: string;
|
||||
status?: number;
|
||||
warning?: string;
|
||||
}> {
|
||||
const endpoint = resolveFirecrawlEndpoint(params.baseUrl);
|
||||
const body: Record<string, unknown> = {
|
||||
url: params.url,
|
||||
formats: ["markdown"],
|
||||
onlyMainContent: params.onlyMainContent,
|
||||
timeout: params.timeoutSeconds * 1000,
|
||||
maxAge: params.maxAgeMs,
|
||||
proxy: params.proxy,
|
||||
storeInCache: params.storeInCache,
|
||||
};
|
||||
|
||||
const res = await fetch(endpoint, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${params.apiKey}`,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
|
||||
});
|
||||
|
||||
const payload = (await res.json()) as {
|
||||
success?: boolean;
|
||||
data?: {
|
||||
markdown?: string;
|
||||
content?: string;
|
||||
metadata?: {
|
||||
title?: string;
|
||||
sourceURL?: string;
|
||||
statusCode?: number;
|
||||
};
|
||||
};
|
||||
warning?: string;
|
||||
error?: string;
|
||||
};
|
||||
|
||||
if (!res.ok || payload?.success === false) {
|
||||
const detail = payload?.error || res.statusText;
|
||||
throw new Error(`Firecrawl fetch failed (${res.status}): ${detail}`.trim());
|
||||
}
|
||||
|
||||
const data = payload?.data ?? {};
|
||||
const rawText =
|
||||
typeof data.markdown === "string"
|
||||
? data.markdown
|
||||
: typeof data.content === "string"
|
||||
? data.content
|
||||
: "";
|
||||
const text = params.extractMode === "text" ? markdownToText(rawText) : rawText;
|
||||
return {
|
||||
text,
|
||||
title: data.metadata?.title,
|
||||
finalUrl: data.metadata?.sourceURL,
|
||||
status: data.metadata?.statusCode,
|
||||
warning: payload?.warning,
|
||||
};
|
||||
}
|
||||
|
||||
async function runWebSearch(params: {
|
||||
query: string;
|
||||
count: number;
|
||||
@@ -499,7 +348,7 @@ async function runWebSearch(params: {
|
||||
ui_lang?: string;
|
||||
}): Promise<Record<string, unknown>> {
|
||||
const cacheKey = normalizeCacheKey(
|
||||
`${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`,
|
||||
`${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`
|
||||
);
|
||||
const cached = readCache(SEARCH_CACHE, cacheKey);
|
||||
if (cached) return { ...cached.value, cached: true };
|
||||
@@ -565,14 +414,6 @@ async function runWebFetch(params: {
|
||||
cacheTtlMs: number;
|
||||
userAgent: string;
|
||||
readabilityEnabled: boolean;
|
||||
firecrawlEnabled: boolean;
|
||||
firecrawlApiKey?: string;
|
||||
firecrawlBaseUrl: string;
|
||||
firecrawlOnlyMainContent: boolean;
|
||||
firecrawlMaxAgeMs: number;
|
||||
firecrawlProxy: "auto" | "basic" | "stealth";
|
||||
firecrawlStoreInCache: boolean;
|
||||
firecrawlTimeoutSeconds: number;
|
||||
}): Promise<Record<string, unknown>> {
|
||||
const cacheKey = normalizeCacheKey(
|
||||
`fetch:${params.url}:${params.extractMode}:${params.maxChars}`,
|
||||
@@ -591,84 +432,16 @@ async function runWebFetch(params: {
|
||||
}
|
||||
|
||||
const start = Date.now();
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(parsedUrl.toString(), {
|
||||
method: "GET",
|
||||
headers: {
|
||||
Accept: "*/*",
|
||||
"User-Agent": params.userAgent,
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
|
||||
});
|
||||
} catch (error) {
|
||||
if (params.firecrawlEnabled && params.firecrawlApiKey) {
|
||||
const firecrawl = await fetchFirecrawlContent({
|
||||
url: params.url,
|
||||
extractMode: params.extractMode,
|
||||
apiKey: params.firecrawlApiKey,
|
||||
baseUrl: params.firecrawlBaseUrl,
|
||||
onlyMainContent: params.firecrawlOnlyMainContent,
|
||||
maxAgeMs: params.firecrawlMaxAgeMs,
|
||||
proxy: params.firecrawlProxy,
|
||||
storeInCache: params.firecrawlStoreInCache,
|
||||
timeoutSeconds: params.firecrawlTimeoutSeconds,
|
||||
});
|
||||
const truncated = truncateText(firecrawl.text, params.maxChars);
|
||||
const payload = {
|
||||
url: params.url,
|
||||
finalUrl: firecrawl.finalUrl || params.url,
|
||||
status: firecrawl.status ?? 200,
|
||||
contentType: "text/markdown",
|
||||
title: firecrawl.title,
|
||||
extractMode: params.extractMode,
|
||||
extractor: "firecrawl",
|
||||
truncated: truncated.truncated,
|
||||
length: truncated.text.length,
|
||||
fetchedAt: new Date().toISOString(),
|
||||
tookMs: Date.now() - start,
|
||||
text: truncated.text,
|
||||
warning: firecrawl.warning,
|
||||
};
|
||||
writeCache(FETCH_CACHE, cacheKey, payload, params.cacheTtlMs);
|
||||
return payload;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const res = await fetch(parsedUrl.toString(), {
|
||||
method: "GET",
|
||||
headers: {
|
||||
Accept: "*/*",
|
||||
"User-Agent": params.userAgent,
|
||||
},
|
||||
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
if (params.firecrawlEnabled && params.firecrawlApiKey) {
|
||||
const firecrawl = await fetchFirecrawlContent({
|
||||
url: params.url,
|
||||
extractMode: params.extractMode,
|
||||
apiKey: params.firecrawlApiKey,
|
||||
baseUrl: params.firecrawlBaseUrl,
|
||||
onlyMainContent: params.firecrawlOnlyMainContent,
|
||||
maxAgeMs: params.firecrawlMaxAgeMs,
|
||||
proxy: params.firecrawlProxy,
|
||||
storeInCache: params.firecrawlStoreInCache,
|
||||
timeoutSeconds: params.firecrawlTimeoutSeconds,
|
||||
});
|
||||
const truncated = truncateText(firecrawl.text, params.maxChars);
|
||||
const payload = {
|
||||
url: params.url,
|
||||
finalUrl: firecrawl.finalUrl || params.url,
|
||||
status: firecrawl.status ?? res.status,
|
||||
contentType: "text/markdown",
|
||||
title: firecrawl.title,
|
||||
extractMode: params.extractMode,
|
||||
extractor: "firecrawl",
|
||||
truncated: truncated.truncated,
|
||||
length: truncated.text.length,
|
||||
fetchedAt: new Date().toISOString(),
|
||||
tookMs: Date.now() - start,
|
||||
text: truncated.text,
|
||||
warning: firecrawl.warning,
|
||||
};
|
||||
writeCache(FETCH_CACHE, cacheKey, payload, params.cacheTtlMs);
|
||||
return payload;
|
||||
}
|
||||
const detail = await readResponseText(res);
|
||||
throw new Error(`Web fetch failed (${res.status}): ${detail || res.statusText}`);
|
||||
}
|
||||
@@ -677,7 +450,6 @@ async function runWebFetch(params: {
|
||||
const body = await readResponseText(res);
|
||||
|
||||
let title: string | undefined;
|
||||
let extractor = "raw";
|
||||
let text = body;
|
||||
if (contentType.includes("text/html")) {
|
||||
if (params.readabilityEnabled) {
|
||||
@@ -689,31 +461,21 @@ async function runWebFetch(params: {
|
||||
if (readable?.text) {
|
||||
text = readable.text;
|
||||
title = readable.title;
|
||||
extractor = "readability";
|
||||
} else {
|
||||
const firecrawl = await tryFirecrawlFallback(params);
|
||||
if (firecrawl) {
|
||||
text = firecrawl.text;
|
||||
title = firecrawl.title;
|
||||
extractor = "firecrawl";
|
||||
} else {
|
||||
throw new Error(
|
||||
"Web fetch extraction failed: Readability and Firecrawl returned no content.",
|
||||
);
|
||||
}
|
||||
const parsed = params.extractMode === "text" ? htmlToText(body) : htmlToMarkdown(body);
|
||||
text = parsed.text;
|
||||
title = parsed.title;
|
||||
}
|
||||
} else {
|
||||
throw new Error(
|
||||
"Web fetch extraction failed: Readability disabled and Firecrawl unavailable.",
|
||||
);
|
||||
const parsed = params.extractMode === "text" ? htmlToText(body) : htmlToMarkdown(body);
|
||||
text = parsed.text;
|
||||
title = parsed.title;
|
||||
}
|
||||
} else if (contentType.includes("application/json")) {
|
||||
try {
|
||||
text = JSON.stringify(JSON.parse(body), null, 2);
|
||||
extractor = "json";
|
||||
} catch {
|
||||
text = body;
|
||||
extractor = "raw";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -725,7 +487,6 @@ async function runWebFetch(params: {
|
||||
contentType,
|
||||
title,
|
||||
extractMode: params.extractMode,
|
||||
extractor,
|
||||
truncated: truncated.truncated,
|
||||
length: truncated.text.length,
|
||||
fetchedAt: new Date().toISOString(),
|
||||
@@ -736,37 +497,6 @@ async function runWebFetch(params: {
|
||||
return payload;
|
||||
}
|
||||
|
||||
async function tryFirecrawlFallback(params: {
|
||||
url: string;
|
||||
extractMode: (typeof EXTRACT_MODES)[number];
|
||||
firecrawlEnabled: boolean;
|
||||
firecrawlApiKey?: string;
|
||||
firecrawlBaseUrl: string;
|
||||
firecrawlOnlyMainContent: boolean;
|
||||
firecrawlMaxAgeMs: number;
|
||||
firecrawlProxy: "auto" | "basic" | "stealth";
|
||||
firecrawlStoreInCache: boolean;
|
||||
firecrawlTimeoutSeconds: number;
|
||||
}): Promise<{ text: string; title?: string } | null> {
|
||||
if (!params.firecrawlEnabled || !params.firecrawlApiKey) return null;
|
||||
try {
|
||||
const firecrawl = await fetchFirecrawlContent({
|
||||
url: params.url,
|
||||
extractMode: params.extractMode,
|
||||
apiKey: params.firecrawlApiKey,
|
||||
baseUrl: params.firecrawlBaseUrl,
|
||||
onlyMainContent: params.firecrawlOnlyMainContent,
|
||||
maxAgeMs: params.firecrawlMaxAgeMs,
|
||||
proxy: params.firecrawlProxy,
|
||||
storeInCache: params.firecrawlStoreInCache,
|
||||
timeoutSeconds: params.firecrawlTimeoutSeconds,
|
||||
});
|
||||
return { text: firecrawl.text, title: firecrawl.title };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function createWebSearchTool(options?: {
|
||||
config?: ClawdbotConfig;
|
||||
sandboxed?: boolean;
|
||||
@@ -807,21 +537,6 @@ export function createWebSearchTool(options?: {
|
||||
};
|
||||
}
|
||||
|
||||
function resolveFirecrawlEndpoint(baseUrl: string): string {
|
||||
const trimmed = baseUrl.trim();
|
||||
if (!trimmed) return `${DEFAULT_FIRECRAWL_BASE_URL}/v2/scrape`;
|
||||
try {
|
||||
const url = new URL(trimmed);
|
||||
if (url.pathname && url.pathname !== "/") {
|
||||
return url.toString();
|
||||
}
|
||||
url.pathname = "/v2/scrape";
|
||||
return url.toString();
|
||||
} catch {
|
||||
return `${DEFAULT_FIRECRAWL_BASE_URL}/v2/scrape`;
|
||||
}
|
||||
}
|
||||
|
||||
export function createWebFetchTool(options?: {
|
||||
config?: ClawdbotConfig;
|
||||
sandboxed?: boolean;
|
||||
@@ -829,19 +544,9 @@ export function createWebFetchTool(options?: {
|
||||
const fetch = resolveFetchConfig(options?.config);
|
||||
if (!resolveFetchEnabled({ fetch, sandboxed: options?.sandboxed })) return null;
|
||||
const readabilityEnabled = resolveFetchReadabilityEnabled(fetch);
|
||||
const firecrawl = resolveFirecrawlConfig(fetch);
|
||||
const firecrawlApiKey = resolveFirecrawlApiKey(firecrawl);
|
||||
const firecrawlEnabled = resolveFirecrawlEnabled({ firecrawl, apiKey: firecrawlApiKey });
|
||||
const firecrawlBaseUrl = resolveFirecrawlBaseUrl(firecrawl);
|
||||
const firecrawlOnlyMainContent = resolveFirecrawlOnlyMainContent(firecrawl);
|
||||
const firecrawlMaxAgeMs = resolveFirecrawlMaxAgeMsOrDefault(firecrawl);
|
||||
const firecrawlTimeoutSeconds = resolveTimeoutSeconds(
|
||||
firecrawl?.timeoutSeconds ?? fetch?.timeoutSeconds,
|
||||
DEFAULT_TIMEOUT_SECONDS,
|
||||
);
|
||||
const userAgent =
|
||||
(fetch && "userAgent" in fetch && typeof fetch.userAgent === "string" && fetch.userAgent) ||
|
||||
DEFAULT_FETCH_USER_AGENT;
|
||||
`clawdbot/${VERSION}`;
|
||||
return {
|
||||
label: "Web Fetch",
|
||||
name: "web_fetch",
|
||||
@@ -861,14 +566,6 @@ export function createWebFetchTool(options?: {
|
||||
cacheTtlMs: resolveCacheTtlMs(fetch?.cacheTtlMinutes, DEFAULT_CACHE_TTL_MINUTES),
|
||||
userAgent,
|
||||
readabilityEnabled,
|
||||
firecrawlEnabled,
|
||||
firecrawlApiKey,
|
||||
firecrawlBaseUrl,
|
||||
firecrawlOnlyMainContent,
|
||||
firecrawlMaxAgeMs,
|
||||
firecrawlProxy: "auto",
|
||||
firecrawlStoreInCache: true,
|
||||
firecrawlTimeoutSeconds,
|
||||
});
|
||||
return jsonResult(result);
|
||||
},
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
parseSlashCommand,
|
||||
renderCodexPrompt,
|
||||
resolveCodexPrompt,
|
||||
stripFrontMatter,
|
||||
} from "./codex-prompts.js";
|
||||
|
||||
describe("codex prompts", () => {
|
||||
it("parses slash command names and args", () => {
|
||||
expect(parseSlashCommand("/landpr 123 abc")).toEqual({ name: "landpr", args: "123 abc" });
|
||||
expect(parseSlashCommand("nope")).toBeNull();
|
||||
});
|
||||
|
||||
it("resolves prompt files and substitutes args", async () => {
|
||||
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-codex-prompts-"));
|
||||
const promptDir = path.join(tmp, "prompts");
|
||||
await fs.mkdir(promptDir, { recursive: true });
|
||||
await fs.writeFile(
|
||||
path.join(promptDir, "landpr.md"),
|
||||
`---\nsummary: test\n---\nHello $1 [$*] $0\n`,
|
||||
"utf-8",
|
||||
);
|
||||
|
||||
const prev = process.env.CODEX_HOME;
|
||||
process.env.CODEX_HOME = tmp;
|
||||
try {
|
||||
const resolved = await resolveCodexPrompt("landpr");
|
||||
expect(resolved?.path).toContain("landpr.md");
|
||||
const stripped = stripFrontMatter(`---\nsummary: test\n---\nHello`);
|
||||
expect(stripped).toBe("Hello");
|
||||
const rendered = renderCodexPrompt({
|
||||
body: resolved?.body ?? "",
|
||||
args: "123 abc",
|
||||
commandName: "landpr",
|
||||
});
|
||||
expect(rendered).toContain("Hello 123 [123 abc] landpr");
|
||||
} finally {
|
||||
if (prev === undefined) {
|
||||
delete process.env.CODEX_HOME;
|
||||
} else {
|
||||
process.env.CODEX_HOME = prev;
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,78 @@
|
||||
import type { Dirent } from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
import { resolveUserPath } from "../../utils.js";
|
||||
|
||||
const PROMPT_NAME_RE = /^[a-z0-9_-]+$/i;
|
||||
|
||||
export type CodexPrompt = {
|
||||
path: string;
|
||||
body: string;
|
||||
};
|
||||
|
||||
export function parseSlashCommand(input: string): { name: string; args?: string } | null {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed.startsWith("/")) return null;
|
||||
const match = trimmed.match(/^\/([^\s]+)(?:\s+([\s\S]+))?$/);
|
||||
if (!match) return null;
|
||||
const name = match[1]?.trim();
|
||||
if (!name) return null;
|
||||
const args = match[2]?.trim();
|
||||
return { name, args: args || undefined };
|
||||
}
|
||||
|
||||
export function stripFrontMatter(input: string): string {
|
||||
if (!input.startsWith("---")) return input;
|
||||
const match = input.match(/^---\s*\r?\n[\s\S]*?\r?\n---\s*\r?\n?/);
|
||||
return match ? input.slice(match[0].length) : input;
|
||||
}
|
||||
|
||||
export async function resolveCodexPrompt(
|
||||
commandName: string,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): Promise<CodexPrompt | null> {
|
||||
const normalized = commandName.trim().toLowerCase();
|
||||
if (!PROMPT_NAME_RE.test(normalized)) return null;
|
||||
const home = env.CODEX_HOME?.trim()
|
||||
? resolveUserPath(env.CODEX_HOME.trim())
|
||||
: resolveUserPath("~/.codex");
|
||||
const promptsDir = path.join(home, "prompts");
|
||||
let entries: Array<Dirent> | null = null;
|
||||
try {
|
||||
entries = await fs.readdir(promptsDir, { withFileTypes: true });
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const match = entries.find((entry) => {
|
||||
if (!entry.isFile()) return false;
|
||||
const base = path.parse(entry.name).name.toLowerCase();
|
||||
return base === normalized;
|
||||
});
|
||||
if (!match) return null;
|
||||
const promptPath = path.join(promptsDir, match.name);
|
||||
try {
|
||||
const raw = await fs.readFile(promptPath, "utf-8");
|
||||
const body = stripFrontMatter(raw).trim();
|
||||
if (!body) return null;
|
||||
return { path: promptPath, body };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function renderCodexPrompt(params: {
|
||||
body: string;
|
||||
args?: string;
|
||||
commandName?: string;
|
||||
}): string {
|
||||
const args = params.args?.trim() ?? "";
|
||||
const tokens = args ? args.split(/\s+/) : [];
|
||||
return params.body.replace(/\$(\d+|\*|@|0)/g, (match, token) => {
|
||||
if (token === "*" || token === "@") return args;
|
||||
if (token === "0") return params.commandName ?? "";
|
||||
const index = Number(token);
|
||||
if (!Number.isFinite(index) || index < 1) return match;
|
||||
return tokens[index - 1] ?? "";
|
||||
});
|
||||
}
|
||||
@@ -5,8 +5,10 @@ import type { SessionEntry } from "../../config/sessions.js";
|
||||
import type { MsgContext, TemplateContext } from "../templating.js";
|
||||
import type { ElevatedLevel, ReasoningLevel, ThinkLevel, VerboseLevel } from "../thinking.js";
|
||||
import type { GetReplyOptions, ReplyPayload } from "../types.js";
|
||||
import { listChatCommands } from "../commands-registry.js";
|
||||
import { getAbortMemory } from "./abort.js";
|
||||
import { buildStatusReply, handleCommands } from "./commands.js";
|
||||
import { parseSlashCommand, renderCodexPrompt, resolveCodexPrompt } from "./codex-prompts.js";
|
||||
import type { InlineDirectives } from "./directive-handling.js";
|
||||
import { isDirectiveOnly } from "./directive-handling.js";
|
||||
import type { createModelSelectionState } from "./model-selection.js";
|
||||
@@ -138,6 +140,46 @@ export async function handleInlineActions(params: {
|
||||
cleanedBody = rewrittenBody;
|
||||
}
|
||||
|
||||
const slashCommand =
|
||||
allowTextCommands && command.isAuthorizedSender
|
||||
? parseSlashCommand(command.commandBodyNormalized)
|
||||
: null;
|
||||
if (slashCommand) {
|
||||
const reserved = new Set<string>();
|
||||
for (const entry of listChatCommands()) {
|
||||
if (entry.nativeName) reserved.add(entry.nativeName.toLowerCase());
|
||||
for (const alias of entry.textAliases) {
|
||||
if (!alias.startsWith("/")) continue;
|
||||
reserved.add(alias.slice(1).toLowerCase());
|
||||
}
|
||||
}
|
||||
for (const entry of skillCommands) {
|
||||
reserved.add(entry.name.toLowerCase());
|
||||
}
|
||||
const hasInlineDirective =
|
||||
directives.hasThinkDirective ||
|
||||
directives.hasVerboseDirective ||
|
||||
directives.hasReasoningDirective ||
|
||||
directives.hasElevatedDirective ||
|
||||
directives.hasModelDirective ||
|
||||
directives.hasQueueDirective ||
|
||||
directives.hasStatusDirective;
|
||||
if (!hasInlineDirective && !reserved.has(slashCommand.name.toLowerCase())) {
|
||||
const prompt = await resolveCodexPrompt(slashCommand.name);
|
||||
if (prompt) {
|
||||
const rendered = renderCodexPrompt({
|
||||
body: prompt.body,
|
||||
args: slashCommand.args,
|
||||
commandName: slashCommand.name,
|
||||
});
|
||||
ctx.Body = rendered;
|
||||
sessionCtx.Body = rendered;
|
||||
sessionCtx.BodyStripped = rendered;
|
||||
cleanedBody = rendered;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const sendInlineReply = async (reply?: ReplyPayload) => {
|
||||
if (!reply) return;
|
||||
if (!opts?.onBlockReply) return;
|
||||
|
||||
@@ -278,13 +278,10 @@ export async function initSessionState(params: {
|
||||
ctx.MessageThreadId,
|
||||
);
|
||||
}
|
||||
if (isNewSession) {
|
||||
sessionEntry.compactionCount = 0;
|
||||
sessionEntry.memoryFlushCompactionCount = undefined;
|
||||
sessionEntry.memoryFlushAt = undefined;
|
||||
}
|
||||
// Preserve per-session overrides while resetting compaction state on /new.
|
||||
sessionStore[sessionKey] = { ...sessionStore[sessionKey], ...sessionEntry };
|
||||
// Fresh start for new sessions - don't inherit old state like compactionCount
|
||||
sessionStore[sessionKey] = isNewSession
|
||||
? sessionEntry
|
||||
: { ...sessionStore[sessionKey], ...sessionEntry };
|
||||
await updateSessionStore(storePath, (store) => {
|
||||
if (groupResolution?.legacyKey && groupResolution.legacyKey !== sessionKey) {
|
||||
if (store[groupResolution.legacyKey] && !store[sessionKey]) {
|
||||
@@ -292,8 +289,10 @@ export async function initSessionState(params: {
|
||||
}
|
||||
delete store[groupResolution.legacyKey];
|
||||
}
|
||||
// Preserve per-session overrides while resetting compaction state on /new.
|
||||
store[sessionKey] = { ...store[sessionKey], ...sessionEntry };
|
||||
// Fresh start for new sessions - don't inherit old state like compactionCount
|
||||
store[sessionKey] = isNewSession
|
||||
? sessionEntry
|
||||
: { ...store[sessionKey], ...sessionEntry };
|
||||
});
|
||||
|
||||
const sessionCtx: TemplateContext = {
|
||||
|
||||
@@ -109,7 +109,6 @@ export const whatsappPlugin: ChannelPlugin<ResolvedWhatsAppAccount> = {
|
||||
name: account.name,
|
||||
enabled: account.enabled,
|
||||
configured: Boolean(account.authDir),
|
||||
linked: Boolean(account.authDir),
|
||||
dmPolicy: account.dmPolicy,
|
||||
allowFrom: account.allowFrom,
|
||||
}),
|
||||
|
||||
@@ -84,13 +84,3 @@ export function normalizeAlias(alias: string): string {
|
||||
|
||||
export { modelKey };
|
||||
export { DEFAULT_MODEL, DEFAULT_PROVIDER };
|
||||
|
||||
/**
|
||||
* Model key format: "provider/model"
|
||||
*
|
||||
* The model key is displayed in `/model status` and used to reference models.
|
||||
* When using `/model <key>`, use the exact format shown (e.g., "openrouter/moonshotai/kimi-k2").
|
||||
*
|
||||
* For providers with hierarchical model IDs (e.g., OpenRouter), the model ID may include
|
||||
* sub-providers (e.g., "moonshotai/kimi-k2"), resulting in a key like "openrouter/moonshotai/kimi-k2".
|
||||
*/
|
||||
@@ -1,290 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { MissingEnvVarError, resolveConfigEnvVars } from "./env-substitution.js";
|
||||
|
||||
describe("resolveConfigEnvVars", () => {
|
||||
describe("basic substitution", () => {
|
||||
it("substitutes a single env var", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${FOO}" }, { FOO: "bar" });
|
||||
expect(result).toEqual({ key: "bar" });
|
||||
});
|
||||
|
||||
it("substitutes multiple different env vars in same string", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${A}/${B}" }, { A: "x", B: "y" });
|
||||
expect(result).toEqual({ key: "x/y" });
|
||||
});
|
||||
|
||||
it("substitutes inline with prefix and suffix", () => {
|
||||
const result = resolveConfigEnvVars({ key: "prefix-${FOO}-suffix" }, { FOO: "bar" });
|
||||
expect(result).toEqual({ key: "prefix-bar-suffix" });
|
||||
});
|
||||
|
||||
it("substitutes same var multiple times", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${FOO}:${FOO}" }, { FOO: "bar" });
|
||||
expect(result).toEqual({ key: "bar:bar" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("nested structures", () => {
|
||||
it("substitutes in nested objects", () => {
|
||||
const result = resolveConfigEnvVars(
|
||||
{
|
||||
outer: {
|
||||
inner: {
|
||||
key: "${API_KEY}",
|
||||
},
|
||||
},
|
||||
},
|
||||
{ API_KEY: "secret123" },
|
||||
);
|
||||
expect(result).toEqual({
|
||||
outer: {
|
||||
inner: {
|
||||
key: "secret123",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("substitutes in arrays", () => {
|
||||
const result = resolveConfigEnvVars(
|
||||
{ items: ["${A}", "${B}", "${C}"] },
|
||||
{ A: "1", B: "2", C: "3" },
|
||||
);
|
||||
expect(result).toEqual({ items: ["1", "2", "3"] });
|
||||
});
|
||||
|
||||
it("substitutes in deeply nested arrays and objects", () => {
|
||||
const result = resolveConfigEnvVars(
|
||||
{
|
||||
providers: [
|
||||
{ name: "openai", apiKey: "${OPENAI_KEY}" },
|
||||
{ name: "anthropic", apiKey: "${ANTHROPIC_KEY}" },
|
||||
],
|
||||
},
|
||||
{ OPENAI_KEY: "sk-xxx", ANTHROPIC_KEY: "sk-yyy" },
|
||||
);
|
||||
expect(result).toEqual({
|
||||
providers: [
|
||||
{ name: "openai", apiKey: "sk-xxx" },
|
||||
{ name: "anthropic", apiKey: "sk-yyy" },
|
||||
],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("missing env var handling", () => {
|
||||
it("throws MissingEnvVarError for missing env var", () => {
|
||||
expect(() => resolveConfigEnvVars({ key: "${MISSING}" }, {})).toThrow(MissingEnvVarError);
|
||||
});
|
||||
|
||||
it("includes var name in error", () => {
|
||||
try {
|
||||
resolveConfigEnvVars({ key: "${MISSING_VAR}" }, {});
|
||||
throw new Error("Expected to throw");
|
||||
} catch (err) {
|
||||
expect(err).toBeInstanceOf(MissingEnvVarError);
|
||||
const error = err as MissingEnvVarError;
|
||||
expect(error.varName).toBe("MISSING_VAR");
|
||||
}
|
||||
});
|
||||
|
||||
it("includes config path in error", () => {
|
||||
try {
|
||||
resolveConfigEnvVars({ outer: { inner: { key: "${MISSING}" } } }, {});
|
||||
throw new Error("Expected to throw");
|
||||
} catch (err) {
|
||||
expect(err).toBeInstanceOf(MissingEnvVarError);
|
||||
const error = err as MissingEnvVarError;
|
||||
expect(error.configPath).toBe("outer.inner.key");
|
||||
}
|
||||
});
|
||||
|
||||
it("includes array index in config path", () => {
|
||||
try {
|
||||
resolveConfigEnvVars({ items: ["ok", "${MISSING}"] }, { OK: "val" });
|
||||
throw new Error("Expected to throw");
|
||||
} catch (err) {
|
||||
expect(err).toBeInstanceOf(MissingEnvVarError);
|
||||
const error = err as MissingEnvVarError;
|
||||
expect(error.configPath).toBe("items[1]");
|
||||
}
|
||||
});
|
||||
|
||||
it("treats empty string env var as missing", () => {
|
||||
expect(() => resolveConfigEnvVars({ key: "${EMPTY}" }, { EMPTY: "" })).toThrow(
|
||||
MissingEnvVarError,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("escape syntax", () => {
|
||||
it("outputs literal ${VAR} when escaped with $$", () => {
|
||||
const result = resolveConfigEnvVars({ key: "$${VAR}" }, { VAR: "value" });
|
||||
expect(result).toEqual({ key: "${VAR}" });
|
||||
});
|
||||
|
||||
it("handles mix of escaped and unescaped", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${REAL}/$${LITERAL}" }, { REAL: "resolved" });
|
||||
expect(result).toEqual({ key: "resolved/${LITERAL}" });
|
||||
});
|
||||
|
||||
it("handles escaped and unescaped of the same var (escaped first)", () => {
|
||||
const result = resolveConfigEnvVars({ key: "$${FOO} ${FOO}" }, { FOO: "bar" });
|
||||
expect(result).toEqual({ key: "${FOO} bar" });
|
||||
});
|
||||
|
||||
it("handles escaped and unescaped of the same var (unescaped first)", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${FOO} $${FOO}" }, { FOO: "bar" });
|
||||
expect(result).toEqual({ key: "bar ${FOO}" });
|
||||
});
|
||||
|
||||
it("handles multiple escaped vars", () => {
|
||||
const result = resolveConfigEnvVars({ key: "$${A}:$${B}" }, {});
|
||||
expect(result).toEqual({ key: "${A}:${B}" });
|
||||
});
|
||||
|
||||
it("does not unescape $${VAR} sequences from env values", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${FOO}" }, { FOO: "$${BAR}" });
|
||||
expect(result).toEqual({ key: "$${BAR}" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("non-matching patterns unchanged", () => {
|
||||
it("leaves $VAR (no braces) unchanged", () => {
|
||||
const result = resolveConfigEnvVars({ key: "$VAR" }, { VAR: "value" });
|
||||
expect(result).toEqual({ key: "$VAR" });
|
||||
});
|
||||
|
||||
it("leaves ${lowercase} unchanged (uppercase only)", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${lowercase}" }, { lowercase: "value" });
|
||||
expect(result).toEqual({ key: "${lowercase}" });
|
||||
});
|
||||
|
||||
it("leaves ${MixedCase} unchanged", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${MixedCase}" }, { MixedCase: "value" });
|
||||
expect(result).toEqual({ key: "${MixedCase}" });
|
||||
});
|
||||
|
||||
it("leaves ${123INVALID} unchanged (must start with letter or underscore)", () => {
|
||||
const result = resolveConfigEnvVars({ key: "${123INVALID}" }, {});
|
||||
expect(result).toEqual({ key: "${123INVALID}" });
|
||||
});
|
||||
|
||||
it("substitutes ${_UNDERSCORE_START} (valid)", () => {
|
||||
const result = resolveConfigEnvVars(
|
||||
{ key: "${_UNDERSCORE_START}" },
|
||||
{ _UNDERSCORE_START: "valid" },
|
||||
);
|
||||
expect(result).toEqual({ key: "valid" });
|
||||
});
|
||||
|
||||
it("substitutes ${VAR_WITH_NUMBERS_123} (valid)", () => {
|
||||
const result = resolveConfigEnvVars(
|
||||
{ key: "${VAR_WITH_NUMBERS_123}" },
|
||||
{ VAR_WITH_NUMBERS_123: "valid" },
|
||||
);
|
||||
expect(result).toEqual({ key: "valid" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("passthrough behavior", () => {
|
||||
it("passes through primitives unchanged", () => {
|
||||
expect(resolveConfigEnvVars("hello", {})).toBe("hello");
|
||||
expect(resolveConfigEnvVars(42, {})).toBe(42);
|
||||
expect(resolveConfigEnvVars(true, {})).toBe(true);
|
||||
expect(resolveConfigEnvVars(null, {})).toBe(null);
|
||||
});
|
||||
|
||||
it("passes through empty object", () => {
|
||||
expect(resolveConfigEnvVars({}, {})).toEqual({});
|
||||
});
|
||||
|
||||
it("passes through empty array", () => {
|
||||
expect(resolveConfigEnvVars([], {})).toEqual([]);
|
||||
});
|
||||
|
||||
it("passes through non-string values in objects", () => {
|
||||
const result = resolveConfigEnvVars({ num: 42, bool: true, nil: null, arr: [1, 2] }, {});
|
||||
expect(result).toEqual({ num: 42, bool: true, nil: null, arr: [1, 2] });
|
||||
});
|
||||
});
|
||||
|
||||
describe("real-world config patterns", () => {
|
||||
it("substitutes API keys in provider config", () => {
|
||||
const config = {
|
||||
models: {
|
||||
providers: {
|
||||
"vercel-gateway": {
|
||||
apiKey: "${VERCEL_GATEWAY_API_KEY}",
|
||||
},
|
||||
openai: {
|
||||
apiKey: "${OPENAI_API_KEY}",
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const env = {
|
||||
VERCEL_GATEWAY_API_KEY: "vg_key_123",
|
||||
OPENAI_API_KEY: "sk-xxx",
|
||||
};
|
||||
const result = resolveConfigEnvVars(config, env);
|
||||
expect(result).toEqual({
|
||||
models: {
|
||||
providers: {
|
||||
"vercel-gateway": {
|
||||
apiKey: "vg_key_123",
|
||||
},
|
||||
openai: {
|
||||
apiKey: "sk-xxx",
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("substitutes gateway auth token", () => {
|
||||
const config = {
|
||||
gateway: {
|
||||
auth: {
|
||||
token: "${CLAWDBOT_GATEWAY_TOKEN}",
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = resolveConfigEnvVars(config, {
|
||||
CLAWDBOT_GATEWAY_TOKEN: "secret-token",
|
||||
});
|
||||
expect(result).toEqual({
|
||||
gateway: {
|
||||
auth: {
|
||||
token: "secret-token",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("substitutes base URL with env var", () => {
|
||||
const config = {
|
||||
models: {
|
||||
providers: {
|
||||
custom: {
|
||||
baseUrl: "${CUSTOM_API_BASE}/v1",
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = resolveConfigEnvVars(config, {
|
||||
CUSTOM_API_BASE: "https://api.example.com",
|
||||
});
|
||||
expect(result).toEqual({
|
||||
models: {
|
||||
providers: {
|
||||
custom: {
|
||||
baseUrl: "https://api.example.com/v1",
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,139 +0,0 @@
|
||||
/**
|
||||
* Environment variable substitution for config values.
|
||||
*
|
||||
* Supports `${VAR_NAME}` syntax in string values, substituted at config load time.
|
||||
* - Only uppercase env vars are matched: `[A-Z_][A-Z0-9_]*`
|
||||
* - Escape with `$${}` to output literal `${}`
|
||||
* - Missing env vars throw `MissingEnvVarError` with context
|
||||
*
|
||||
* @example
|
||||
* ```json5
|
||||
* {
|
||||
* models: {
|
||||
* providers: {
|
||||
* "vercel-gateway": {
|
||||
* apiKey: "${VERCEL_GATEWAY_API_KEY}"
|
||||
* }
|
||||
* }
|
||||
* }
|
||||
* }
|
||||
* ```
|
||||
*/
|
||||
|
||||
// Pattern for valid uppercase env var names: starts with letter or underscore,
|
||||
// followed by letters, numbers, or underscores (all uppercase)
|
||||
const ENV_VAR_NAME_PATTERN = /^[A-Z_][A-Z0-9_]*$/;
|
||||
|
||||
export class MissingEnvVarError extends Error {
|
||||
constructor(
|
||||
public readonly varName: string,
|
||||
public readonly configPath: string,
|
||||
) {
|
||||
super(`Missing env var "${varName}" referenced at config path: ${configPath}`);
|
||||
this.name = "MissingEnvVarError";
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
||||
return (
|
||||
typeof value === "object" &&
|
||||
value !== null &&
|
||||
!Array.isArray(value) &&
|
||||
Object.prototype.toString.call(value) === "[object Object]"
|
||||
);
|
||||
}
|
||||
|
||||
function readEnvVarName(
|
||||
value: string,
|
||||
start: number,
|
||||
): { name: string | null; end: number } {
|
||||
const end = value.indexOf("}", start);
|
||||
if (end === -1) {
|
||||
return { name: null, end: -1 };
|
||||
}
|
||||
|
||||
const name = value.slice(start, end);
|
||||
if (!ENV_VAR_NAME_PATTERN.test(name)) {
|
||||
return { name: null, end: -1 };
|
||||
}
|
||||
|
||||
return { name, end };
|
||||
}
|
||||
|
||||
function substituteString(value: string, env: NodeJS.ProcessEnv, configPath: string): string {
|
||||
let result = "";
|
||||
|
||||
for (let i = 0; i < value.length; i += 1) {
|
||||
const char = value[i];
|
||||
if (char !== "$") {
|
||||
result += char;
|
||||
continue;
|
||||
}
|
||||
|
||||
const next = value[i + 1];
|
||||
const afterNext = value[i + 2];
|
||||
|
||||
// Escaped: $${VAR} -> ${VAR}
|
||||
if (next === "$" && afterNext === "{") {
|
||||
const { name, end } = readEnvVarName(value, i + 3);
|
||||
if (name !== null) {
|
||||
result += `\${${name}}`;
|
||||
i = end;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Substitution: ${VAR} -> value
|
||||
if (next === "{") {
|
||||
const { name, end } = readEnvVarName(value, i + 2);
|
||||
if (name !== null) {
|
||||
const envValue = env[name];
|
||||
if (envValue === undefined || envValue === "") {
|
||||
throw new MissingEnvVarError(name, configPath);
|
||||
}
|
||||
result += envValue;
|
||||
i = end;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Leave untouched if not a recognized pattern
|
||||
result += char;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function substituteAny(value: unknown, env: NodeJS.ProcessEnv, path: string): unknown {
|
||||
if (typeof value === "string") {
|
||||
return substituteString(value, env, path);
|
||||
}
|
||||
|
||||
if (Array.isArray(value)) {
|
||||
return value.map((item, index) => substituteAny(item, env, `${path}[${index}]`));
|
||||
}
|
||||
|
||||
if (isPlainObject(value)) {
|
||||
const result: Record<string, unknown> = {};
|
||||
for (const [key, val] of Object.entries(value)) {
|
||||
const childPath = path ? `${path}.${key}` : key;
|
||||
result[key] = substituteAny(val, env, childPath);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Primitives (number, boolean, null) pass through unchanged
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves `${VAR_NAME}` environment variable references in config values.
|
||||
*
|
||||
* @param obj - The parsed config object (after JSON5 parse and $include resolution)
|
||||
* @param env - Environment variables to use for substitution (defaults to process.env)
|
||||
* @returns The config object with env vars substituted
|
||||
* @throws {MissingEnvVarError} If a referenced env var is not set or empty
|
||||
*/
|
||||
export function resolveConfigEnvVars(obj: unknown, env: NodeJS.ProcessEnv = process.env): unknown {
|
||||
return substituteAny(obj, env, "");
|
||||
}
|
||||
+4
-31
@@ -19,7 +19,6 @@ import {
|
||||
applySessionDefaults,
|
||||
applyTalkApiKey,
|
||||
} from "./defaults.js";
|
||||
import { MissingEnvVarError, resolveConfigEnvVars } from "./env-substitution.js";
|
||||
import { ConfigIncludeError, resolveConfigIncludes } from "./includes.js";
|
||||
import { applyLegacyMigrations, findLegacyConfigIssues } from "./legacy.js";
|
||||
import { normalizeConfigPaths } from "./normalize-paths.js";
|
||||
@@ -31,7 +30,6 @@ import { ClawdbotSchema } from "./zod-schema.js";
|
||||
|
||||
// Re-export for backwards compatibility
|
||||
export { CircularIncludeError, ConfigIncludeError } from "./includes.js";
|
||||
export { MissingEnvVarError } from "./env-substitution.js";
|
||||
|
||||
const SHELL_ENV_EXPECTED_KEYS = [
|
||||
"OPENAI_API_KEY",
|
||||
@@ -221,11 +219,8 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) {
|
||||
parseJson: (raw) => deps.json5.parse(raw),
|
||||
});
|
||||
|
||||
// Substitute ${VAR} env var references
|
||||
const substituted = resolveConfigEnvVars(resolved, deps.env);
|
||||
|
||||
const migrated = applyLegacyMigrations(substituted);
|
||||
const resolvedConfig = migrated.next ?? substituted;
|
||||
const migrated = applyLegacyMigrations(resolved);
|
||||
const resolvedConfig = migrated.next ?? resolved;
|
||||
warnOnConfigMiskeys(resolvedConfig, deps.logger);
|
||||
if (typeof resolvedConfig !== "object" || resolvedConfig === null) return {};
|
||||
const validated = ClawdbotSchema.safeParse(resolvedConfig);
|
||||
@@ -351,30 +346,8 @@ export function createConfigIO(overrides: ConfigIoDeps = {}) {
|
||||
};
|
||||
}
|
||||
|
||||
// Substitute ${VAR} env var references
|
||||
let substituted: unknown;
|
||||
try {
|
||||
substituted = resolveConfigEnvVars(resolved, deps.env);
|
||||
} catch (err) {
|
||||
const message =
|
||||
err instanceof MissingEnvVarError
|
||||
? err.message
|
||||
: `Env var substitution failed: ${String(err)}`;
|
||||
return {
|
||||
path: configPath,
|
||||
exists: true,
|
||||
raw,
|
||||
parsed: parsedRes.parsed,
|
||||
valid: false,
|
||||
config: {},
|
||||
hash,
|
||||
issues: [{ path: "", message }],
|
||||
legacyIssues: [],
|
||||
};
|
||||
}
|
||||
|
||||
const migrated = applyLegacyMigrations(substituted);
|
||||
const resolvedConfigRaw = migrated.next ?? substituted;
|
||||
const migrated = applyLegacyMigrations(resolved);
|
||||
const resolvedConfigRaw = migrated.next ?? resolved;
|
||||
const legacyIssues = findLegacyConfigIssues(resolvedConfigRaw);
|
||||
|
||||
const validated = validateConfigObject(resolvedConfigRaw);
|
||||
|
||||
@@ -264,15 +264,6 @@ const FIELD_HELP: Record<string, string> = {
|
||||
"tools.web.fetch.userAgent": "Override User-Agent header for web_fetch requests.",
|
||||
"tools.web.fetch.readability":
|
||||
"Use Readability to extract main content from HTML (fallbacks to basic HTML cleanup).",
|
||||
"tools.web.fetch.firecrawl.enabled": "Enable Firecrawl fallback for web_fetch (if configured).",
|
||||
"tools.web.fetch.firecrawl.apiKey": "Firecrawl API key (fallback: FIRECRAWL_API_KEY env var).",
|
||||
"tools.web.fetch.firecrawl.baseUrl":
|
||||
"Firecrawl base URL (e.g. https://api.firecrawl.dev or custom endpoint).",
|
||||
"tools.web.fetch.firecrawl.onlyMainContent":
|
||||
"When true, Firecrawl returns only the main content (default: true).",
|
||||
"tools.web.fetch.firecrawl.maxAgeMs":
|
||||
"Firecrawl maxAge (ms) for cached results when supported by the API.",
|
||||
"tools.web.fetch.firecrawl.timeoutSeconds": "Timeout in seconds for Firecrawl requests.",
|
||||
"channels.slack.allowBots":
|
||||
"Allow bot-authored messages to trigger Slack replies (default: false).",
|
||||
"channels.slack.thread.historyScope":
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import fs from "node:fs/promises";
|
||||
import fsSync from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import {
|
||||
buildGroupDisplayName,
|
||||
@@ -157,6 +158,41 @@ describe("sessions", () => {
|
||||
expect(store["agent:main:two"]?.sessionId).toBe("sess-2");
|
||||
});
|
||||
|
||||
it("updateSessionStore preserves sessions.json permissions", async () => {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
|
||||
const storePath = path.join(dir, "sessions.json");
|
||||
await fs.writeFile(storePath, "{}", "utf-8");
|
||||
await fs.chmod(storePath, 0o600);
|
||||
|
||||
await updateSessionStore(storePath, (store) => {
|
||||
store["agent:main:main"] = { sessionId: "sess-1", updatedAt: 1 };
|
||||
});
|
||||
|
||||
const mode = (await fs.stat(storePath)).mode & 0o777;
|
||||
if (process.platform === "win32") {
|
||||
expect([0o600, 0o666, 0o777]).toContain(mode);
|
||||
} else {
|
||||
expect(mode).toBe(0o600);
|
||||
}
|
||||
});
|
||||
|
||||
it("updateSessionStore ignores chmod failures", async () => {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
|
||||
const storePath = path.join(dir, "sessions.json");
|
||||
await fs.writeFile(storePath, "{}", "utf-8");
|
||||
|
||||
const spy = vi.spyOn(fsSync.promises, "chmod").mockRejectedValueOnce(new Error("nope"));
|
||||
try {
|
||||
await expect(
|
||||
updateSessionStore(storePath, (store) => {
|
||||
store["agent:main:main"] = { sessionId: "sess-1", updatedAt: 1 };
|
||||
}),
|
||||
).resolves.toBeUndefined();
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("updateSessionStore keeps deletions when concurrent writes happen", async () => {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "clawdbot-sessions-"));
|
||||
const storePath = path.join(dir, "sessions.json");
|
||||
|
||||
@@ -45,6 +45,16 @@ export function clearSessionStoreCacheForTest(): void {
|
||||
SESSION_STORE_CACHE.clear();
|
||||
}
|
||||
|
||||
const sessionStoreWriteOptions = { mode: 0o600, encoding: "utf-8" } as const;
|
||||
|
||||
async function ensurePrivateMode(storePath: string): Promise<void> {
|
||||
try {
|
||||
await fs.promises.chmod(storePath, 0o600);
|
||||
} catch {
|
||||
// Best-effort: ignore chmod failures (e.g., Windows or filesystem quirks).
|
||||
}
|
||||
}
|
||||
|
||||
type LoadSessionStoreOptions = {
|
||||
skipCache?: boolean;
|
||||
};
|
||||
@@ -121,7 +131,8 @@ async function saveSessionStoreUnlocked(
|
||||
// We serialize writers via the session-store lock instead.
|
||||
if (process.platform === "win32") {
|
||||
try {
|
||||
await fs.promises.writeFile(storePath, json, "utf-8");
|
||||
await fs.promises.writeFile(storePath, json, sessionStoreWriteOptions);
|
||||
await ensurePrivateMode(storePath);
|
||||
} catch (err) {
|
||||
const code =
|
||||
err && typeof err === "object" && "code" in err
|
||||
@@ -135,10 +146,9 @@ async function saveSessionStoreUnlocked(
|
||||
|
||||
const tmp = `${storePath}.${process.pid}.${crypto.randomUUID()}.tmp`;
|
||||
try {
|
||||
await fs.promises.writeFile(tmp, json, { mode: 0o600, encoding: "utf-8" });
|
||||
await fs.promises.writeFile(tmp, json, sessionStoreWriteOptions);
|
||||
await fs.promises.rename(tmp, storePath);
|
||||
// Ensure permissions are set even if rename loses them
|
||||
await fs.promises.chmod(storePath, 0o600);
|
||||
await ensurePrivateMode(storePath);
|
||||
} catch (err) {
|
||||
const code =
|
||||
err && typeof err === "object" && "code" in err
|
||||
@@ -150,8 +160,8 @@ async function saveSessionStoreUnlocked(
|
||||
// Best-effort: try a direct write (recreating the parent dir), otherwise ignore.
|
||||
try {
|
||||
await fs.promises.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.promises.writeFile(storePath, json, { mode: 0o600, encoding: "utf-8" });
|
||||
await fs.promises.chmod(storePath, 0o600);
|
||||
await fs.promises.writeFile(storePath, json, sessionStoreWriteOptions);
|
||||
await ensurePrivateMode(storePath);
|
||||
} catch (err2) {
|
||||
const code2 =
|
||||
err2 && typeof err2 === "object" && "code" in err2
|
||||
|
||||
@@ -111,20 +111,6 @@ export type ToolsConfig = {
|
||||
userAgent?: string;
|
||||
/** Use Readability to extract main content (default: true). */
|
||||
readability?: boolean;
|
||||
firecrawl?: {
|
||||
/** Enable Firecrawl fallback (default: true when apiKey is set). */
|
||||
enabled?: boolean;
|
||||
/** Firecrawl API key (optional; defaults to FIRECRAWL_API_KEY env var). */
|
||||
apiKey?: string;
|
||||
/** Firecrawl base URL (default: https://api.firecrawl.dev). */
|
||||
baseUrl?: string;
|
||||
/** Whether to keep only main content (default: true). */
|
||||
onlyMainContent?: boolean;
|
||||
/** Max age (ms) for cached Firecrawl content. */
|
||||
maxAgeMs?: number;
|
||||
/** Timeout in seconds for Firecrawl requests. */
|
||||
timeoutSeconds?: number;
|
||||
};
|
||||
};
|
||||
};
|
||||
audio?: {
|
||||
|
||||
Reference in new issue
Block a user