Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d16d721898 | ||
|
|
ebf6553c9e | ||
|
|
3211b6cb10 | ||
|
|
06d675d4ae | ||
|
|
eff75c395b | ||
|
|
62e80d030c | ||
|
|
f3f1640c7b | ||
|
|
4e8000e745 | ||
|
|
1cb4af074f |
No files matched your search
@@ -96,6 +96,119 @@ embeddings for memory search. For Gemini, use `GEMINI_API_KEY` or
|
||||
`models.providers.google.apiKey`. When using a custom OpenAI-compatible endpoint,
|
||||
set `memorySearch.remote.apiKey` (and optional `memorySearch.remote.headers`).
|
||||
|
||||
### QMD backend (experimental)
|
||||
|
||||
Set `memory.backend = "qmd"` to swap the built-in SQLite indexer for
|
||||
[QMD](https://github.com/tobi/qmd): a local-first search sidecar that combines
|
||||
BM25 + vectors + reranking. Markdown stays the source of truth; Moltbot shells
|
||||
out to QMD for retrieval. Key points:
|
||||
|
||||
**Prereqs**
|
||||
- Disabled by default. Opt in per-config (`memory.backend = "qmd"`).
|
||||
- Install the QMD CLI separately (`bun install -g github.com/tobi/qmd` or grab
|
||||
a release) and make sure the `qmd` binary is on the gateway’s `PATH`.
|
||||
- QMD needs an SQLite build that allows extensions (`brew install sqlite` on
|
||||
macOS).
|
||||
- QMD runs fully locally via Bun + `node-llama-cpp` and auto-downloads GGUF
|
||||
models from HuggingFace on first use (no separate Ollama daemon required).
|
||||
- The gateway runs QMD in a self-contained XDG home under
|
||||
`~/.clawdbot/agents/<agentId>/qmd/` by setting `XDG_CONFIG_HOME` and
|
||||
`XDG_CACHE_HOME`.
|
||||
- OS support: macOS and Linux work out of the box once Bun + SQLite are
|
||||
installed. Windows is best supported via WSL2.
|
||||
|
||||
**How the sidecar runs**
|
||||
- The gateway writes a self-contained QMD home under
|
||||
`~/.clawdbot/agents/<agentId>/qmd/` (config + cache + sqlite DB).
|
||||
- Collections are rewritten from `memory.qmd.paths` (plus default workspace
|
||||
memory files) into `index.yml`, then `qmd update` + `qmd embed` run on boot and
|
||||
on a configurable interval (`memory.qmd.update.interval`, default 5 m).
|
||||
- Searches run via `qmd query --json`. If QMD fails or the binary is missing,
|
||||
Moltbot automatically falls back to the builtin SQLite manager so memory tools
|
||||
keep working.
|
||||
- **First search may be slow**: QMD may download local GGUF models (reranker/query
|
||||
expansion) on the first `qmd query` run.
|
||||
- Moltbot sets `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` automatically when it runs QMD.
|
||||
- If you want to pre-download models manually (and warm the same index Moltbot
|
||||
uses), run a one-off query with the agent’s XDG dirs.
|
||||
|
||||
Moltbot’s QMD state lives under your **state dir** (usually `~/.clawdbot`, or
|
||||
`~/.moltbot` on newer installs). You can point `qmd` at the exact same index
|
||||
by exporting the same XDG vars Moltbot uses:
|
||||
|
||||
```bash
|
||||
# Pick the same state dir Moltbot uses
|
||||
STATE_DIR="${MOLTBOT_STATE_DIR:-${CLAWDBOT_STATE_DIR:-$HOME/.clawdbot}}"
|
||||
if [ -d "$HOME/.moltbot" ] && [ ! -d "$HOME/.clawdbot" ] \
|
||||
&& [ -z "${MOLTBOT_STATE_DIR:-}" ] && [ -z "${CLAWDBOT_STATE_DIR:-}" ]; then
|
||||
STATE_DIR="$HOME/.moltbot"
|
||||
fi
|
||||
|
||||
export XDG_CONFIG_HOME="$STATE_DIR/agents/main/qmd/xdg-config"
|
||||
export XDG_CACHE_HOME="$STATE_DIR/agents/main/qmd/xdg-cache"
|
||||
|
||||
# (Optional) force an index refresh + embeddings
|
||||
qmd update
|
||||
qmd embed
|
||||
|
||||
# Warm up / trigger first-time model downloads
|
||||
qmd query "test" -c memory-root --json >/dev/null 2>&1
|
||||
```
|
||||
|
||||
**Config surface (`memory.qmd.*`)**
|
||||
- `command` (default `qmd`): override the executable path.
|
||||
- `includeDefaultMemory` (default `true`): auto-index `MEMORY.md` + `memory/**/*.md`.
|
||||
- `paths[]`: add extra directories/files (`path`, optional `pattern`, optional
|
||||
stable `name`).
|
||||
- `sessions`: opt into session JSONL indexing (`enabled`, `retentionDays`,
|
||||
`exportDir`).
|
||||
- `update`: controls refresh cadence (`interval`, `debounceMs`, `onBoot`, `embedInterval`).
|
||||
- `limits`: clamp recall payload (`maxResults`, `maxSnippetChars`,
|
||||
`maxInjectedChars`, `timeoutMs`).
|
||||
- `scope`: same schema as [`session.sendPolicy`](/reference/configuration#session-sendpolicy).
|
||||
Default is DM-only (`deny` all, `allow` direct chats); loosen it to surface QMD
|
||||
hits in groups/channels.
|
||||
- Snippets sourced outside the workspace show up as
|
||||
`qmd/<collection>/<relative-path>` in `memory_search` results; `memory_get`
|
||||
understands that prefix and reads from the configured QMD collection root.
|
||||
- When `memory.qmd.sessions.enabled = true`, Moltbot exports sanitized session
|
||||
transcripts (User/Assistant turns) into a dedicated QMD collection under
|
||||
`~/.clawdbot/agents/<id>/qmd/sessions/`, so `memory_search` can recall recent
|
||||
conversations without touching the builtin SQLite index.
|
||||
- `memory_search` snippets now include a `Source: <path#line>` footer when
|
||||
`memory.citations` is `auto`/`on`; set `memory.citations = "off"` to keep
|
||||
the path metadata internal (the agent still receives the path for
|
||||
`memory_get`, but the snippet text omits the footer and the system prompt
|
||||
warns the agent not to cite it).
|
||||
|
||||
**Example**
|
||||
|
||||
```json5
|
||||
memory: {
|
||||
backend: "qmd",
|
||||
citations: "auto",
|
||||
qmd: {
|
||||
includeDefaultMemory: true,
|
||||
update: { interval: "5m", debounceMs: 15000 },
|
||||
limits: { maxResults: 6, timeoutMs: 4000 },
|
||||
scope: {
|
||||
default: "deny",
|
||||
rules: [{ action: "allow", match: { chatType: "direct" } }]
|
||||
},
|
||||
paths: [
|
||||
{ name: "docs", path: "~/notes", pattern: "**/*.md" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Citations & fallback**
|
||||
- `memory.citations` applies regardless of backend (`auto`/`on`/`off`).
|
||||
- When `qmd` runs, we tag `status().backend = "qmd"` so diagnostics show which
|
||||
engine served the results. If the QMD subprocess exits or JSON output can’t be
|
||||
parsed, the search manager logs a warning and returns the builtin provider
|
||||
(existing Markdown embeddings) until QMD recovers.
|
||||
|
||||
### Gemini embeddings (native)
|
||||
|
||||
Set the provider to `gemini` to use the Gemini embeddings API directly:
|
||||
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
name: bitwarden
|
||||
description: Manage passwords and credentials via Bitwarden CLI (bw). Use for storing, retrieving, creating, or updating logins, credit cards, secure notes, and identities. Trigger when automating authentication, filling payment forms, or managing secrets programmatically.
|
||||
---
|
||||
|
||||
# Bitwarden CLI
|
||||
|
||||
Full read/write vault access via `bw` command.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
brew install bitwarden-cli
|
||||
bw login <email> # one-time, prompts for master password
|
||||
```
|
||||
|
||||
## Session Management
|
||||
|
||||
Bitwarden requires an unlocked session. Use the helper script:
|
||||
|
||||
```bash
|
||||
source scripts/bw-session.sh <master_password>
|
||||
# Sets BW_SESSION env var
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
export BW_SESSION=$(echo '<password>' | bw unlock --raw)
|
||||
bw sync # always sync after unlock
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Retrieve credentials
|
||||
```bash
|
||||
bw get password "Site Name"
|
||||
bw get username "Site Name"
|
||||
bw get item "Site Name" --pretty | jq '.login'
|
||||
```
|
||||
|
||||
### Create login
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 1 |
|
||||
.name = "Site Name" |
|
||||
.login.username = "user@email.com" |
|
||||
.login.password = "secret123" |
|
||||
.login.uris = [{uri: "https://example.com"}]
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
### Create credit card
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 3 |
|
||||
.name = "Card Name" |
|
||||
.card.cardholderName = "John Doe" |
|
||||
.card.brand = "Visa" |
|
||||
.card.number = "4111111111111111" |
|
||||
.card.expMonth = "12" |
|
||||
.card.expYear = "2030" |
|
||||
.card.code = "123"
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
### Get card for payment automation
|
||||
```bash
|
||||
bw get item "Card Name" | jq -r '.card | "\(.number) \(.expMonth)/\(.expYear) \(.code)"'
|
||||
```
|
||||
|
||||
### List items
|
||||
```bash
|
||||
bw list items | jq -r '.[] | "\(.type)|\(.name)"'
|
||||
# Types: 1=login, 2=note, 3=card, 4=identity
|
||||
```
|
||||
|
||||
### Search
|
||||
```bash
|
||||
bw list items --search "vilaviniteca" | jq '.[0]'
|
||||
```
|
||||
|
||||
## Item Types
|
||||
|
||||
| Type | Value | Use |
|
||||
|------|-------|-----|
|
||||
| Login | 1 | Website credentials |
|
||||
| Secure Note | 2 | Freeform text |
|
||||
| Card | 3 | Credit/debit cards |
|
||||
| Identity | 4 | Personal info |
|
||||
|
||||
## References
|
||||
|
||||
- [templates.md](references/templates.md) — Full jq templates for all item types
|
||||
- [Bitwarden CLI docs](https://bitwarden.com/help/cli/)
|
||||
|
||||
## Tips
|
||||
|
||||
1. **Always sync** after creating/editing items: `bw sync`
|
||||
2. **Session expires** — re-unlock if you get auth errors
|
||||
3. **Delete sensitive messages** after receiving credentials
|
||||
4. **Card numbers** may not import from other managers (security restriction)
|
||||
@@ -1,116 +0,0 @@
|
||||
# Bitwarden Item Templates
|
||||
|
||||
jq patterns for creating vault items via CLI.
|
||||
|
||||
## Login (type=1)
|
||||
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 1 |
|
||||
.name = "Example Site" |
|
||||
.notes = "Optional notes" |
|
||||
.favorite = false |
|
||||
.login.username = "user@example.com" |
|
||||
.login.password = "secretPassword123" |
|
||||
.login.totp = "otpauth://totp/..." |
|
||||
.login.uris = [
|
||||
{uri: "https://example.com", match: null},
|
||||
{uri: "https://app.example.com", match: null}
|
||||
]
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
## Credit Card (type=3)
|
||||
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 3 |
|
||||
.name = "Visa ending 1234" |
|
||||
.notes = "Primary card" |
|
||||
.card.cardholderName = "JOHN DOE" |
|
||||
.card.brand = "Visa" |
|
||||
.card.number = "4111111111111111" |
|
||||
.card.expMonth = "12" |
|
||||
.card.expYear = "2030" |
|
||||
.card.code = "123"
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
**Brands:** Visa, Mastercard, Amex, Discover, Diners Club, JCB, Maestro, UnionPay, Other
|
||||
|
||||
## Secure Note (type=2)
|
||||
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 2 |
|
||||
.name = "API Keys" |
|
||||
.notes = "OPENAI_KEY=sk-xxx\nANTHROPIC_KEY=sk-ant-xxx" |
|
||||
.secureNote.type = 0
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
## Identity (type=4)
|
||||
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 4 |
|
||||
.name = "Personal Info" |
|
||||
.identity.title = "Mr" |
|
||||
.identity.firstName = "John" |
|
||||
.identity.lastName = "Doe" |
|
||||
.identity.email = "john@example.com" |
|
||||
.identity.phone = "+34612345678" |
|
||||
.identity.address1 = "123 Main St" |
|
||||
.identity.city = "Barcelona" |
|
||||
.identity.state = "Catalunya" |
|
||||
.identity.postalCode = "08001" |
|
||||
.identity.country = "ES"
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
## Edit Existing Item
|
||||
|
||||
```bash
|
||||
# Get item, modify, update
|
||||
bw get item <id> | jq '.login.password = "newPassword"' | bw encode | bw edit item <id>
|
||||
```
|
||||
|
||||
## Custom Fields
|
||||
|
||||
```bash
|
||||
bw get template item | jq '
|
||||
.type = 1 |
|
||||
.name = "With Custom Fields" |
|
||||
.fields = [
|
||||
{name: "Security Question", value: "Pet name", type: 0},
|
||||
{name: "PIN", value: "1234", type: 1}
|
||||
]
|
||||
' | bw encode | bw create item
|
||||
```
|
||||
|
||||
**Field types:** 0=text, 1=hidden, 2=boolean
|
||||
|
||||
## Retrieve Patterns
|
||||
|
||||
```bash
|
||||
# Password only
|
||||
bw get password "Site Name"
|
||||
|
||||
# Username only
|
||||
bw get username "Site Name"
|
||||
|
||||
# Full login object
|
||||
bw get item "Site Name" | jq '.login'
|
||||
|
||||
# Card number
|
||||
bw get item "Card Name" | jq -r '.card.number'
|
||||
|
||||
# All card fields for form filling
|
||||
bw get item "Card Name" | jq -r '.card | [.number, .expMonth, .expYear, .code] | @tsv'
|
||||
|
||||
# Search by URL
|
||||
bw list items --url "example.com" | jq '.[0].login'
|
||||
|
||||
# List all cards
|
||||
bw list items | jq '.[] | select(.type == 3) | .name'
|
||||
```
|
||||
@@ -1,33 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Unlock Bitwarden vault and export session key
|
||||
# Usage: source bw-session.sh <master_password>
|
||||
# Or: source bw-session.sh (prompts for password)
|
||||
|
||||
set -e
|
||||
|
||||
if [ -n "$1" ]; then
|
||||
MASTER_PW="$1"
|
||||
else
|
||||
read -sp "Bitwarden master password: " MASTER_PW
|
||||
echo
|
||||
fi
|
||||
|
||||
# Check if already logged in
|
||||
if ! bw login --check &>/dev/null; then
|
||||
echo "Not logged in. Run: bw login <email>"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Unlock and get session
|
||||
export BW_SESSION=$(echo "$MASTER_PW" | bw unlock --raw 2>/dev/null)
|
||||
|
||||
if [ -z "$BW_SESSION" ]; then
|
||||
echo "Failed to unlock vault"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Sync to get latest
|
||||
bw sync &>/dev/null
|
||||
|
||||
echo "✓ Vault unlocked and synced"
|
||||
echo "Session valid for this shell"
|
||||
@@ -212,6 +212,7 @@ export function buildSystemPrompt(params: {
|
||||
userTimeFormat,
|
||||
contextFiles: params.contextFiles,
|
||||
ttsHint,
|
||||
memoryCitationsMode: params.config?.memory?.citations,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -346,6 +346,7 @@ export async function compactEmbeddedPiSessionDirect(
|
||||
userTime,
|
||||
userTimeFormat,
|
||||
contextFiles,
|
||||
memoryCitationsMode: params.config?.memory?.citations,
|
||||
});
|
||||
const systemPrompt = createSystemPromptOverride(appendPrompt);
|
||||
|
||||
|
||||
@@ -359,6 +359,7 @@ export async function runEmbeddedAttempt(
|
||||
userTime,
|
||||
userTimeFormat,
|
||||
contextFiles,
|
||||
memoryCitationsMode: params.config?.memory?.citations,
|
||||
});
|
||||
const systemPromptReport = buildSystemPromptReport({
|
||||
source: "run",
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { AgentTool } from "@mariozechner/pi-agent-core";
|
||||
import type { MemoryCitationsMode } from "../../config/types.memory.js";
|
||||
import type { ResolvedTimeFormat } from "../date-time.js";
|
||||
import type { EmbeddedContextFile } from "../pi-embedded-helpers.js";
|
||||
import { buildAgentSystemPrompt, type PromptMode } from "../system-prompt.js";
|
||||
@@ -45,6 +46,7 @@ export function buildEmbeddedSystemPrompt(params: {
|
||||
userTime?: string;
|
||||
userTimeFormat?: ResolvedTimeFormat;
|
||||
contextFiles?: EmbeddedContextFile[];
|
||||
memoryCitationsMode?: MemoryCitationsMode;
|
||||
}): string {
|
||||
return buildAgentSystemPrompt({
|
||||
workspaceDir: params.workspaceDir,
|
||||
@@ -70,6 +72,7 @@ export function buildEmbeddedSystemPrompt(params: {
|
||||
userTime: params.userTime,
|
||||
userTimeFormat: params.userTimeFormat,
|
||||
contextFiles: params.contextFiles,
|
||||
memoryCitationsMode: params.memoryCitationsMode,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -52,28 +52,6 @@ describe("installSessionToolResultGuard", () => {
|
||||
expect(messages.map((m) => m.role)).toEqual(["assistant", "toolResult"]);
|
||||
});
|
||||
|
||||
it("ignores tool calls from aborted assistant messages", () => {
|
||||
const sm = SessionManager.inMemory();
|
||||
installSessionToolResultGuard(sm);
|
||||
|
||||
sm.appendMessage({
|
||||
role: "assistant",
|
||||
stopReason: "aborted",
|
||||
content: [{ type: "toolCall", id: "call_abort", name: "read", arguments: {} }],
|
||||
} as AgentMessage);
|
||||
sm.appendMessage({
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: "next" }],
|
||||
} as AgentMessage);
|
||||
|
||||
const messages = sm
|
||||
.getEntries()
|
||||
.filter((e) => e.type === "message")
|
||||
.map((e) => (e as { message: AgentMessage }).message);
|
||||
|
||||
expect(messages.map((m) => m.role)).toEqual(["assistant", "assistant"]);
|
||||
});
|
||||
|
||||
it("does not add synthetic toolResult when a matching one exists", () => {
|
||||
const sm = SessionManager.inMemory();
|
||||
installSessionToolResultGuard(sm);
|
||||
|
||||
@@ -7,7 +7,6 @@ import { emitSessionTranscriptUpdate } from "../sessions/transcript-events.js";
|
||||
type ToolCall = { id: string; name?: string };
|
||||
|
||||
function extractAssistantToolCalls(msg: Extract<AgentMessage, { role: "assistant" }>): ToolCall[] {
|
||||
if (msg.stopReason === "aborted") return [];
|
||||
const content = msg.content;
|
||||
if (!Array.isArray(content)) return [];
|
||||
|
||||
|
||||
@@ -8,7 +8,6 @@ type ToolCallLike = {
|
||||
function extractToolCallsFromAssistant(
|
||||
msg: Extract<AgentMessage, { role: "assistant" }>,
|
||||
): ToolCallLike[] {
|
||||
if (msg.stopReason === "aborted") return [];
|
||||
const content = msg.content;
|
||||
if (!Array.isArray(content)) return [];
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { ReasoningLevel, ThinkLevel } from "../auto-reply/thinking.js";
|
||||
import { SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
|
||||
import { listDeliverableMessageChannels } from "../utils/message-channel.js";
|
||||
import type { MemoryCitationsMode } from "../config/types.memory.js";
|
||||
import type { ResolvedTimeFormat } from "./date-time.js";
|
||||
import type { EmbeddedContextFile } from "./pi-embedded-helpers.js";
|
||||
|
||||
@@ -32,16 +33,30 @@ function buildSkillsSection(params: {
|
||||
];
|
||||
}
|
||||
|
||||
function buildMemorySection(params: { isMinimal: boolean; availableTools: Set<string> }) {
|
||||
function buildMemorySection(params: {
|
||||
isMinimal: boolean;
|
||||
availableTools: Set<string>;
|
||||
citationsMode?: MemoryCitationsMode;
|
||||
}) {
|
||||
if (params.isMinimal) return [];
|
||||
if (!params.availableTools.has("memory_search") && !params.availableTools.has("memory_get")) {
|
||||
return [];
|
||||
}
|
||||
return [
|
||||
const lines = [
|
||||
"## Memory Recall",
|
||||
"Before answering anything about prior work, decisions, dates, people, preferences, or todos: run memory_search on MEMORY.md + memory/*.md; then use memory_get to pull only the needed lines. If low confidence after search, say you checked.",
|
||||
"",
|
||||
];
|
||||
if (params.citationsMode === "off") {
|
||||
lines.push(
|
||||
"Citations are disabled: do not mention file paths or line numbers in replies unless the user explicitly asks.",
|
||||
);
|
||||
} else {
|
||||
lines.push(
|
||||
"Citations: include Source: <path#line> when it helps the user verify memory snippets.",
|
||||
);
|
||||
}
|
||||
lines.push("");
|
||||
return lines;
|
||||
}
|
||||
|
||||
function buildUserIdentitySection(ownerLine: string | undefined, isMinimal: boolean) {
|
||||
@@ -178,6 +193,7 @@ export function buildAgentSystemPrompt(params: {
|
||||
level: "minimal" | "extensive";
|
||||
channel: string;
|
||||
};
|
||||
memoryCitationsMode?: MemoryCitationsMode;
|
||||
}) {
|
||||
const coreToolSummaries: Record<string, string> = {
|
||||
read: "Read file contents",
|
||||
@@ -314,7 +330,11 @@ export function buildAgentSystemPrompt(params: {
|
||||
isMinimal,
|
||||
readToolName,
|
||||
});
|
||||
const memorySection = buildMemorySection({ isMinimal, availableTools });
|
||||
const memorySection = buildMemorySection({
|
||||
isMinimal,
|
||||
availableTools,
|
||||
citationsMode: params.memoryCitationsMode,
|
||||
});
|
||||
const docsSection = buildDocsSection({
|
||||
docsPath: params.docsPath,
|
||||
isMinimal,
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const stubManager = {
|
||||
search: vi.fn(async () => [
|
||||
{
|
||||
path: "MEMORY.md",
|
||||
startLine: 5,
|
||||
endLine: 7,
|
||||
score: 0.9,
|
||||
snippet: "@@ -5,3 @@\nAssistant: noted",
|
||||
source: "memory" as const,
|
||||
},
|
||||
]),
|
||||
readFile: vi.fn(),
|
||||
status: () => ({
|
||||
backend: "builtin" as const,
|
||||
files: 1,
|
||||
chunks: 1,
|
||||
dirty: false,
|
||||
workspaceDir: "/workspace",
|
||||
dbPath: "/workspace/.memory/index.sqlite",
|
||||
provider: "builtin",
|
||||
model: "builtin",
|
||||
requestedProvider: "builtin",
|
||||
sources: ["memory" as const],
|
||||
sourceCounts: [{ source: "memory" as const, files: 1, chunks: 1 }],
|
||||
}),
|
||||
sync: vi.fn(),
|
||||
probeVectorAvailability: vi.fn(async () => true),
|
||||
close: vi.fn(),
|
||||
};
|
||||
|
||||
vi.mock("../../memory/index.js", () => {
|
||||
return {
|
||||
getMemorySearchManager: async () => ({ manager: stubManager }),
|
||||
};
|
||||
});
|
||||
|
||||
import { createMemorySearchTool } from "./memory-tool.js";
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
describe("memory search citations", () => {
|
||||
it("appends source information when citations are enabled", async () => {
|
||||
const cfg = { memory: { citations: "on" }, agents: { list: [{ id: "main", default: true }] } };
|
||||
const tool = createMemorySearchTool({ config: cfg });
|
||||
if (!tool) throw new Error("tool missing");
|
||||
const result = await tool.execute("call_citations_on", { query: "notes" });
|
||||
const details = result.details as { results: Array<{ snippet: string; citation?: string }> };
|
||||
expect(details.results[0]?.snippet).toMatch(/Source: MEMORY.md#L5-L7/);
|
||||
expect(details.results[0]?.citation).toBe("MEMORY.md#L5-L7");
|
||||
});
|
||||
|
||||
it("leaves snippet untouched when citations are off", async () => {
|
||||
const cfg = { memory: { citations: "off" }, agents: { list: [{ id: "main", default: true }] } };
|
||||
const tool = createMemorySearchTool({ config: cfg });
|
||||
if (!tool) throw new Error("tool missing");
|
||||
const result = await tool.execute("call_citations_off", { query: "notes" });
|
||||
const details = result.details as { results: Array<{ snippet: string; citation?: string }> };
|
||||
expect(details.results[0]?.snippet).not.toMatch(/Source:/);
|
||||
expect(details.results[0]?.citation).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,9 @@
|
||||
import { Type } from "@sinclair/typebox";
|
||||
|
||||
import type { MoltbotConfig } from "../../config/config.js";
|
||||
import type { MemoryCitationsMode } from "../../config/types.memory.js";
|
||||
import { getMemorySearchManager } from "../../memory/index.js";
|
||||
import type { MemorySearchResult } from "../../memory/types.js";
|
||||
import { resolveSessionAgentId } from "../agent-scope.js";
|
||||
import { resolveMemorySearchConfig } from "../memory-search.js";
|
||||
import type { AnyAgentTool } from "./common.js";
|
||||
@@ -48,17 +50,24 @@ export function createMemorySearchTool(options: {
|
||||
return jsonResult({ results: [], disabled: true, error });
|
||||
}
|
||||
try {
|
||||
const results = await manager.search(query, {
|
||||
const citationsMode = resolveMemoryCitationsMode(cfg);
|
||||
const includeCitations = shouldIncludeCitations({
|
||||
mode: citationsMode,
|
||||
sessionKey: options.agentSessionKey,
|
||||
});
|
||||
const rawResults = await manager.search(query, {
|
||||
maxResults,
|
||||
minScore,
|
||||
sessionKey: options.agentSessionKey,
|
||||
});
|
||||
const status = manager.status();
|
||||
const results = decorateCitations(rawResults, includeCitations);
|
||||
return jsonResult({
|
||||
results,
|
||||
provider: status.provider,
|
||||
model: status.model,
|
||||
fallback: status.fallback,
|
||||
citations: citationsMode,
|
||||
});
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
@@ -110,3 +119,46 @@ export function createMemoryGetTool(options: {
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function resolveMemoryCitationsMode(cfg: MoltbotConfig): MemoryCitationsMode {
|
||||
const mode = cfg.memory?.citations;
|
||||
if (mode === "on" || mode === "off" || mode === "auto") return mode;
|
||||
return "auto";
|
||||
}
|
||||
|
||||
function decorateCitations(results: MemorySearchResult[], include: boolean): MemorySearchResult[] {
|
||||
if (!include) {
|
||||
return results.map((entry) => ({ ...entry, citation: undefined }));
|
||||
}
|
||||
return results.map((entry) => {
|
||||
const citation = formatCitation(entry);
|
||||
const snippet = `${entry.snippet.trim()}\n\nSource: ${citation}`;
|
||||
return { ...entry, citation, snippet };
|
||||
});
|
||||
}
|
||||
|
||||
function formatCitation(entry: MemorySearchResult): string {
|
||||
const lineRange =
|
||||
entry.startLine === entry.endLine
|
||||
? `#L${entry.startLine}`
|
||||
: `#L${entry.startLine}-L${entry.endLine}`;
|
||||
return `${entry.path}${lineRange}`;
|
||||
}
|
||||
|
||||
function shouldIncludeCitations(params: {
|
||||
mode: MemoryCitationsMode;
|
||||
sessionKey?: string;
|
||||
}): boolean {
|
||||
if (params.mode === "on") return true;
|
||||
if (params.mode === "off") return false;
|
||||
// auto: show citations in direct chats; suppress in groups/channels by default.
|
||||
const chatType = deriveChatTypeFromSessionKey(params.sessionKey);
|
||||
return chatType === "direct";
|
||||
}
|
||||
|
||||
function deriveChatTypeFromSessionKey(sessionKey?: string): "direct" | "group" | "channel" {
|
||||
if (!sessionKey) return "direct";
|
||||
if (sessionKey.includes(":group:")) return "group";
|
||||
if (sessionKey.includes(":channel:")) return "channel";
|
||||
return "direct";
|
||||
}
|
||||
@@ -150,6 +150,7 @@ async function resolveContextReport(
|
||||
ttsHint,
|
||||
runtimeInfo,
|
||||
sandboxInfo,
|
||||
memoryCitationsMode: params.cfg?.memory?.citations,
|
||||
});
|
||||
|
||||
return buildSystemPromptReport({
|
||||
|
||||
@@ -242,7 +242,7 @@ describe("memory cli", () => {
|
||||
await program.parseAsync(["memory", "status", "--index"], { from: "user" });
|
||||
|
||||
expect(sync).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ reason: "cli", progress: expect.any(Function) }),
|
||||
expect.objectContaining({ reason: "cli", force: false, progress: expect.any(Function) }),
|
||||
);
|
||||
expect(probeEmbeddingAvailability).toHaveBeenCalled();
|
||||
expect(close).toHaveBeenCalled();
|
||||
|
||||
+50
-28
@@ -24,6 +24,7 @@ type MemoryCommandOptions = {
|
||||
json?: boolean;
|
||||
deep?: boolean;
|
||||
index?: boolean;
|
||||
force?: boolean;
|
||||
verbose?: boolean;
|
||||
};
|
||||
|
||||
@@ -213,13 +214,16 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
|
||||
onCloseError: (err) =>
|
||||
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
|
||||
close: (manager) => manager.close(),
|
||||
close: async (manager) => {
|
||||
await manager.close?.();
|
||||
},
|
||||
run: async (manager) => {
|
||||
const deep = Boolean(opts.deep || opts.index);
|
||||
let embeddingProbe:
|
||||
| Awaited<ReturnType<typeof manager.probeEmbeddingAvailability>>
|
||||
| undefined;
|
||||
let indexError: string | undefined;
|
||||
const syncFn = manager.sync ? manager.sync.bind(manager) : undefined;
|
||||
if (deep) {
|
||||
await withProgress({ label: "Checking memory…", total: 2 }, async (progress) => {
|
||||
progress.setLabel("Probing vector…");
|
||||
@@ -229,7 +233,7 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
embeddingProbe = await manager.probeEmbeddingAvailability();
|
||||
progress.tick();
|
||||
});
|
||||
if (opts.index) {
|
||||
if (opts.index && syncFn) {
|
||||
await withProgressTotals(
|
||||
{
|
||||
label: "Indexing memory…",
|
||||
@@ -238,8 +242,9 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
},
|
||||
async (update, progress) => {
|
||||
try {
|
||||
await manager.sync({
|
||||
await syncFn({
|
||||
reason: "cli",
|
||||
force: Boolean(opts.force),
|
||||
progress: (syncUpdate) => {
|
||||
update({
|
||||
completed: syncUpdate.completed,
|
||||
@@ -256,6 +261,8 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
}
|
||||
},
|
||||
);
|
||||
} else if (opts.index && !syncFn) {
|
||||
defaultRuntime.log("Memory backend does not support manual reindex.");
|
||||
}
|
||||
} else {
|
||||
await manager.probeVectorAvailability();
|
||||
@@ -264,11 +271,10 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
const sources = (
|
||||
status.sources?.length ? status.sources : ["memory"]
|
||||
) as MemorySourceName[];
|
||||
const scan = await scanMemorySources({
|
||||
workspaceDir: status.workspaceDir,
|
||||
agentId,
|
||||
sources,
|
||||
});
|
||||
const workspaceDir = status.workspaceDir;
|
||||
const scan = workspaceDir
|
||||
? await scanMemorySources({ workspaceDir, agentId, sources })
|
||||
: undefined;
|
||||
allResults.push({ agentId, status, embeddingProbe, indexError, scan });
|
||||
},
|
||||
});
|
||||
@@ -290,26 +296,31 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
|
||||
for (const result of allResults) {
|
||||
const { agentId, status, embeddingProbe, indexError, scan } = result;
|
||||
const filesIndexed = status.files ?? 0;
|
||||
const chunksIndexed = status.chunks ?? 0;
|
||||
const totalFiles = scan?.totalFiles ?? null;
|
||||
const indexedLabel =
|
||||
totalFiles === null
|
||||
? `${status.files}/? files · ${status.chunks} chunks`
|
||||
: `${status.files}/${totalFiles} files · ${status.chunks} chunks`;
|
||||
? `${filesIndexed}/? files · ${chunksIndexed} chunks`
|
||||
: `${filesIndexed}/${totalFiles} files · ${chunksIndexed} chunks`;
|
||||
if (opts.index) {
|
||||
const line = indexError ? `Memory index failed: ${indexError}` : "Memory index complete.";
|
||||
defaultRuntime.log(line);
|
||||
}
|
||||
const requestedProvider = status.requestedProvider ?? status.provider;
|
||||
const modelLabel = status.model ?? status.provider;
|
||||
const storePath = status.dbPath ? shortenHomePath(status.dbPath) : "<unknown>";
|
||||
const workspacePath = status.workspaceDir ? shortenHomePath(status.workspaceDir) : "<unknown>";
|
||||
const sourceList = status.sources?.length ? status.sources.join(", ") : null;
|
||||
const lines = [
|
||||
`${heading("Memory Search")} ${muted(`(${agentId})`)}`,
|
||||
`${label("Provider")} ${info(status.provider)} ${muted(
|
||||
`(requested: ${status.requestedProvider})`,
|
||||
)}`,
|
||||
`${label("Model")} ${info(status.model)}`,
|
||||
status.sources?.length ? `${label("Sources")} ${info(status.sources.join(", "))}` : null,
|
||||
`${label("Provider")} ${info(status.provider)} ${muted(`(requested: ${requestedProvider})`)}`,
|
||||
`${label("Model")} ${info(modelLabel)}`,
|
||||
sourceList ? `${label("Sources")} ${info(sourceList)}` : null,
|
||||
`${label("Indexed")} ${success(indexedLabel)}`,
|
||||
`${label("Dirty")} ${status.dirty ? warn("yes") : muted("no")}`,
|
||||
`${label("Store")} ${info(shortenHomePath(status.dbPath))}`,
|
||||
`${label("Workspace")} ${info(shortenHomePath(status.workspaceDir))}`,
|
||||
`${label("Store")} ${info(storePath)}`,
|
||||
`${label("Workspace")} ${info(workspacePath)}`,
|
||||
].filter(Boolean) as string[];
|
||||
if (embeddingProbe) {
|
||||
const state = embeddingProbe.ok ? "ready" : "unavailable";
|
||||
@@ -322,7 +333,7 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
|
||||
if (status.sourceCounts?.length) {
|
||||
lines.push(label("By source"));
|
||||
for (const entry of status.sourceCounts) {
|
||||
const total = scan?.sources.find(
|
||||
const total = scan?.sources?.find(
|
||||
(scanEntry) => scanEntry.source === entry.source,
|
||||
)?.totalFiles;
|
||||
const counts =
|
||||
@@ -435,7 +446,7 @@ export function registerMemoryCli(program: Command) {
|
||||
.option("--deep", "Probe embedding provider availability")
|
||||
.option("--index", "Reindex if dirty (implies --deep)")
|
||||
.option("--verbose", "Verbose logging", false)
|
||||
.action(async (opts: MemoryCommandOptions) => {
|
||||
.action(async (opts: MemoryCommandOptions & { force?: boolean }) => {
|
||||
await runMemoryStatus(opts);
|
||||
});
|
||||
|
||||
@@ -445,7 +456,7 @@ export function registerMemoryCli(program: Command) {
|
||||
.option("--agent <id>", "Agent id (default: default agent)")
|
||||
.option("--force", "Force full reindex", false)
|
||||
.option("--verbose", "Verbose logging", false)
|
||||
.action(async (opts: MemoryCommandOptions & { force?: boolean }) => {
|
||||
.action(async (opts: MemoryCommandOptions) => {
|
||||
setVerbose(Boolean(opts.verbose));
|
||||
const cfg = loadConfig();
|
||||
const agentIds = resolveAgentIds(cfg, opts.agent);
|
||||
@@ -455,9 +466,12 @@ export function registerMemoryCli(program: Command) {
|
||||
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
|
||||
onCloseError: (err) =>
|
||||
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
|
||||
close: (manager) => manager.close(),
|
||||
close: async (manager) => {
|
||||
await manager.close?.();
|
||||
},
|
||||
run: async (manager) => {
|
||||
try {
|
||||
const syncFn = manager.sync ? manager.sync.bind(manager) : undefined;
|
||||
if (opts.verbose) {
|
||||
const status = manager.status();
|
||||
const rich = isRich();
|
||||
@@ -466,15 +480,17 @@ export function registerMemoryCli(program: Command) {
|
||||
const info = (text: string) => colorize(rich, theme.info, text);
|
||||
const warn = (text: string) => colorize(rich, theme.warn, text);
|
||||
const label = (text: string) => muted(`${text}:`);
|
||||
const sourceLabels = status.sources.map((source) =>
|
||||
formatSourceLabel(source, status.workspaceDir, agentId),
|
||||
const sourceLabels = (status.sources ?? []).map((source) =>
|
||||
formatSourceLabel(source, status.workspaceDir ?? "", agentId),
|
||||
);
|
||||
const requestedProvider = status.requestedProvider ?? status.provider;
|
||||
const modelLabel = status.model ?? status.provider;
|
||||
const lines = [
|
||||
`${heading("Memory Index")} ${muted(`(${agentId})`)}`,
|
||||
`${label("Provider")} ${info(status.provider)} ${muted(
|
||||
`(requested: ${status.requestedProvider})`,
|
||||
`(requested: ${requestedProvider})`,
|
||||
)}`,
|
||||
`${label("Model")} ${info(status.model)}`,
|
||||
`${label("Model")} ${info(modelLabel)}`,
|
||||
sourceLabels.length
|
||||
? `${label("Sources")} ${info(sourceLabels.join(", "))}`
|
||||
: null,
|
||||
@@ -514,6 +530,10 @@ export function registerMemoryCli(program: Command) {
|
||||
? `${lastLabel} · elapsed ${elapsed} · eta ${eta}`
|
||||
: `${lastLabel} · elapsed ${elapsed}`;
|
||||
};
|
||||
if (!syncFn) {
|
||||
defaultRuntime.log("Memory backend does not support manual reindex.");
|
||||
return;
|
||||
}
|
||||
await withProgressTotals(
|
||||
{
|
||||
label: "Indexing memory…",
|
||||
@@ -525,9 +545,9 @@ export function registerMemoryCli(program: Command) {
|
||||
progress.setLabel(buildLabel());
|
||||
}, 1000);
|
||||
try {
|
||||
await manager.sync({
|
||||
await syncFn({
|
||||
reason: "cli",
|
||||
force: opts.force,
|
||||
force: Boolean(opts.force),
|
||||
progress: (syncUpdate) => {
|
||||
if (syncUpdate.label) lastLabel = syncUpdate.label;
|
||||
lastCompleted = syncUpdate.completed;
|
||||
@@ -579,7 +599,9 @@ export function registerMemoryCli(program: Command) {
|
||||
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
|
||||
onCloseError: (err) =>
|
||||
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
|
||||
close: (manager) => manager.close(),
|
||||
close: async (manager) => {
|
||||
await manager.close?.();
|
||||
},
|
||||
run: async (manager) => {
|
||||
let results: Awaited<ReturnType<typeof manager.search>>;
|
||||
try {
|
||||
|
||||
@@ -6,7 +6,8 @@ import { probeGateway } from "../gateway/probe.js";
|
||||
import { collectChannelStatusIssues } from "../infra/channels-status-issues.js";
|
||||
import { resolveOsSummary } from "../infra/os-summary.js";
|
||||
import { getTailnetHostname } from "../infra/tailscale.js";
|
||||
import type { MemoryIndexManager } from "../memory/manager.js";
|
||||
import { getMemorySearchManager } from "../memory/index.js";
|
||||
import type { MemoryProviderStatus } from "../memory/types.js";
|
||||
import { runExec } from "../process/exec.js";
|
||||
import type { RuntimeEnv } from "../runtime.js";
|
||||
import { getAgentLocalStatuses } from "./status.agent-local.js";
|
||||
@@ -15,7 +16,7 @@ import { getStatusSummary } from "./status.summary.js";
|
||||
import { getUpdateCheckResult } from "./status.update.js";
|
||||
import { buildChannelsTable } from "./status-all/channels.js";
|
||||
|
||||
type MemoryStatusSnapshot = ReturnType<MemoryIndexManager["status"]> & {
|
||||
type MemoryStatusSnapshot = MemoryProviderStatus & {
|
||||
agentId: string;
|
||||
};
|
||||
|
||||
@@ -151,14 +152,13 @@ export async function scanStatus(
|
||||
if (!memoryPlugin.enabled) return null;
|
||||
if (memoryPlugin.slot !== "memory-core") return null;
|
||||
const agentId = agentStatus.defaultId ?? "main";
|
||||
const { MemoryIndexManager } = await import("../memory/manager.js");
|
||||
const manager = await MemoryIndexManager.get({ cfg, agentId }).catch(() => null);
|
||||
const { manager } = await getMemorySearchManager({ cfg, agentId });
|
||||
if (!manager) return null;
|
||||
try {
|
||||
await manager.probeVectorAvailability();
|
||||
} catch {}
|
||||
const status = manager.status();
|
||||
await manager.close().catch(() => {});
|
||||
await manager.close?.().catch(() => {});
|
||||
return { agentId, ...status };
|
||||
})();
|
||||
progress.tick();
|
||||
|
||||
@@ -252,6 +252,27 @@ const FIELD_LABELS: Record<string, string> = {
|
||||
"Memory Search Hybrid Candidate Multiplier",
|
||||
"agents.defaults.memorySearch.cache.enabled": "Memory Search Embedding Cache",
|
||||
"agents.defaults.memorySearch.cache.maxEntries": "Memory Search Embedding Cache Max Entries",
|
||||
memory: "Memory",
|
||||
"memory.backend": "Memory Backend",
|
||||
"memory.citations": "Memory Citations Mode",
|
||||
"memory.qmd.command": "QMD Binary",
|
||||
"memory.qmd.includeDefaultMemory": "QMD Include Default Memory",
|
||||
"memory.qmd.paths": "QMD Extra Paths",
|
||||
"memory.qmd.paths.path": "QMD Path",
|
||||
"memory.qmd.paths.pattern": "QMD Path Pattern",
|
||||
"memory.qmd.paths.name": "QMD Path Name",
|
||||
"memory.qmd.sessions.enabled": "QMD Session Indexing",
|
||||
"memory.qmd.sessions.exportDir": "QMD Session Export Directory",
|
||||
"memory.qmd.sessions.retentionDays": "QMD Session Retention (days)",
|
||||
"memory.qmd.update.interval": "QMD Update Interval",
|
||||
"memory.qmd.update.debounceMs": "QMD Update Debounce (ms)",
|
||||
"memory.qmd.update.onBoot": "QMD Update on Startup",
|
||||
"memory.qmd.update.embedInterval": "QMD Embed Interval",
|
||||
"memory.qmd.limits.maxResults": "QMD Max Results",
|
||||
"memory.qmd.limits.maxSnippetChars": "QMD Max Snippet Chars",
|
||||
"memory.qmd.limits.maxInjectedChars": "QMD Max Injected Chars",
|
||||
"memory.qmd.limits.timeoutMs": "QMD Search Timeout (ms)",
|
||||
"memory.qmd.scope": "QMD Surface Scope",
|
||||
"auth.profiles": "Auth Profiles",
|
||||
"auth.order": "Auth Profile Order",
|
||||
"auth.cooldowns.billingBackoffHours": "Billing Backoff (hours)",
|
||||
@@ -537,6 +558,37 @@ const FIELD_HELP: Record<string, string> = {
|
||||
"Multiplier for candidate pool size (default: 4).",
|
||||
"agents.defaults.memorySearch.cache.enabled":
|
||||
"Cache chunk embeddings in SQLite to speed up reindexing and frequent updates (default: true).",
|
||||
memory: "Memory backend configuration (global).",
|
||||
"memory.backend": 'Memory backend ("builtin" for Moltbot embeddings, "qmd" for QMD sidecar).',
|
||||
"memory.citations": 'Default citation behavior ("auto", "on", or "off").',
|
||||
"memory.qmd.command": "Path to the qmd binary (default: resolves from PATH).",
|
||||
"memory.qmd.includeDefaultMemory":
|
||||
"Whether to automatically index MEMORY.md + memory/**/*.md (default: true).",
|
||||
"memory.qmd.paths":
|
||||
"Additional directories/files to index with QMD (path + optional glob pattern).",
|
||||
"memory.qmd.paths.path": "Absolute or ~-relative path to index via QMD.",
|
||||
"memory.qmd.paths.pattern": "Glob pattern relative to the path root (default: **/*.md).",
|
||||
"memory.qmd.paths.name":
|
||||
"Optional stable name for the QMD collection (default derived from path).",
|
||||
"memory.qmd.sessions.enabled":
|
||||
"Enable QMD session transcript indexing (experimental, default: false).",
|
||||
"memory.qmd.sessions.exportDir":
|
||||
"Override directory for sanitized session exports before indexing.",
|
||||
"memory.qmd.sessions.retentionDays":
|
||||
"Retention window for exported sessions before pruning (default: unlimited).",
|
||||
"memory.qmd.update.interval":
|
||||
"How often the QMD sidecar refreshes indexes (duration string, default: 5m).",
|
||||
"memory.qmd.update.debounceMs":
|
||||
"Minimum delay between successive QMD refresh runs (default: 15000).",
|
||||
"memory.qmd.update.onBoot": "Run QMD update once on gateway startup (default: true).",
|
||||
"memory.qmd.update.embedInterval":
|
||||
"How often QMD embeddings are refreshed (duration string, default: 60m). Set to 0 to disable periodic embed.",
|
||||
"memory.qmd.limits.maxResults": "Max QMD results returned to the agent loop (default: 6).",
|
||||
"memory.qmd.limits.maxSnippetChars": "Max characters per snippet pulled from QMD (default: 700).",
|
||||
"memory.qmd.limits.maxInjectedChars": "Max total characters injected from QMD hits per turn.",
|
||||
"memory.qmd.limits.timeoutMs": "Per-query timeout for QMD searches (default: 4000).",
|
||||
"memory.qmd.scope":
|
||||
"Session/channel scope for QMD recall (same syntax as session.sendPolicy; default: direct-only).",
|
||||
"agents.defaults.memorySearch.cache.maxEntries":
|
||||
"Optional cap on cached embeddings (best-effort).",
|
||||
"agents.defaults.memorySearch.sync.onSearch":
|
||||
|
||||
@@ -23,6 +23,7 @@ import type { NodeHostConfig } from "./types.node-host.js";
|
||||
import type { PluginsConfig } from "./types.plugins.js";
|
||||
import type { SkillsConfig } from "./types.skills.js";
|
||||
import type { ToolsConfig } from "./types.tools.js";
|
||||
import type { MemoryConfig } from "./types.memory.js";
|
||||
|
||||
export type MoltbotConfig = {
|
||||
meta?: {
|
||||
@@ -95,6 +96,7 @@ export type MoltbotConfig = {
|
||||
canvasHost?: CanvasHostConfig;
|
||||
talk?: TalkConfig;
|
||||
gateway?: GatewayConfig;
|
||||
memory?: MemoryConfig;
|
||||
};
|
||||
|
||||
export type ConfigValidationIssue = {
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import type { SessionSendPolicyConfig } from "./types.base.js";
|
||||
|
||||
export type MemoryBackend = "builtin" | "qmd";
|
||||
export type MemoryCitationsMode = "auto" | "on" | "off";
|
||||
|
||||
export type MemoryConfig = {
|
||||
backend?: MemoryBackend;
|
||||
citations?: MemoryCitationsMode;
|
||||
qmd?: MemoryQmdConfig;
|
||||
};
|
||||
|
||||
export type MemoryQmdConfig = {
|
||||
command?: string;
|
||||
includeDefaultMemory?: boolean;
|
||||
paths?: MemoryQmdIndexPath[];
|
||||
sessions?: MemoryQmdSessionConfig;
|
||||
update?: MemoryQmdUpdateConfig;
|
||||
limits?: MemoryQmdLimitsConfig;
|
||||
scope?: SessionSendPolicyConfig;
|
||||
};
|
||||
|
||||
export type MemoryQmdIndexPath = {
|
||||
path: string;
|
||||
name?: string;
|
||||
pattern?: string;
|
||||
};
|
||||
|
||||
export type MemoryQmdSessionConfig = {
|
||||
enabled?: boolean;
|
||||
exportDir?: string;
|
||||
retentionDays?: number;
|
||||
};
|
||||
|
||||
export type MemoryQmdUpdateConfig = {
|
||||
interval?: string;
|
||||
debounceMs?: number;
|
||||
onBoot?: boolean;
|
||||
embedInterval?: string;
|
||||
};
|
||||
|
||||
export type MemoryQmdLimitsConfig = {
|
||||
maxResults?: number;
|
||||
maxSnippetChars?: number;
|
||||
maxInjectedChars?: number;
|
||||
timeoutMs?: number;
|
||||
};
|
||||
@@ -28,3 +28,4 @@ export * from "./types.telegram.js";
|
||||
export * from "./types.tts.js";
|
||||
export * from "./types.tools.js";
|
||||
export * from "./types.whatsapp.js";
|
||||
export * from "./types.memory.js";
|
||||
@@ -16,6 +16,31 @@ const SessionResetConfigSchema = z
|
||||
})
|
||||
.strict();
|
||||
|
||||
export const SessionSendPolicySchema = z
|
||||
.object({
|
||||
default: z.union([z.literal("allow"), z.literal("deny")]).optional(),
|
||||
rules: z
|
||||
.array(
|
||||
z
|
||||
.object({
|
||||
action: z.union([z.literal("allow"), z.literal("deny")]),
|
||||
match: z
|
||||
.object({
|
||||
channel: z.string().optional(),
|
||||
chatType: z
|
||||
.union([z.literal("direct"), z.literal("group"), z.literal("channel")])
|
||||
.optional(),
|
||||
keyPrefix: z.string().optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
})
|
||||
.strict(),
|
||||
)
|
||||
.optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
export const SessionSchema = z
|
||||
.object({
|
||||
scope: z.union([z.literal("per-sender"), z.literal("global")]).optional(),
|
||||
@@ -51,31 +76,7 @@ export const SessionSchema = z
|
||||
])
|
||||
.optional(),
|
||||
mainKey: z.string().optional(),
|
||||
sendPolicy: z
|
||||
.object({
|
||||
default: z.union([z.literal("allow"), z.literal("deny")]).optional(),
|
||||
rules: z
|
||||
.array(
|
||||
z
|
||||
.object({
|
||||
action: z.union([z.literal("allow"), z.literal("deny")]),
|
||||
match: z
|
||||
.object({
|
||||
channel: z.string().optional(),
|
||||
chatType: z
|
||||
.union([z.literal("direct"), z.literal("group"), z.literal("channel")])
|
||||
.optional(),
|
||||
keyPrefix: z.string().optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
})
|
||||
.strict(),
|
||||
)
|
||||
.optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
sendPolicy: SessionSendPolicySchema.optional(),
|
||||
agentToAgent: z
|
||||
.object({
|
||||
maxPingPongTurns: z.number().int().min(0).max(5).optional(),
|
||||
|
||||
@@ -5,7 +5,12 @@ import { AgentsSchema, AudioSchema, BindingsSchema, BroadcastSchema } from "./zo
|
||||
import { HexColorSchema, ModelsConfigSchema } from "./zod-schema.core.js";
|
||||
import { HookMappingSchema, HooksGmailSchema, InternalHooksSchema } from "./zod-schema.hooks.js";
|
||||
import { ChannelsSchema } from "./zod-schema.providers.js";
|
||||
import { CommandsSchema, MessagesSchema, SessionSchema } from "./zod-schema.session.js";
|
||||
import {
|
||||
CommandsSchema,
|
||||
MessagesSchema,
|
||||
SessionSchema,
|
||||
SessionSendPolicySchema,
|
||||
} from "./zod-schema.session.js";
|
||||
|
||||
const BrowserSnapshotDefaultsSchema = z
|
||||
.object({
|
||||
@@ -27,6 +32,61 @@ const NodeHostSchema = z
|
||||
.strict()
|
||||
.optional();
|
||||
|
||||
const MemoryQmdPathSchema = z
|
||||
.object({
|
||||
path: z.string(),
|
||||
name: z.string().optional(),
|
||||
pattern: z.string().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const MemoryQmdSessionSchema = z
|
||||
.object({
|
||||
enabled: z.boolean().optional(),
|
||||
exportDir: z.string().optional(),
|
||||
retentionDays: z.number().int().nonnegative().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const MemoryQmdUpdateSchema = z
|
||||
.object({
|
||||
interval: z.string().optional(),
|
||||
debounceMs: z.number().int().nonnegative().optional(),
|
||||
onBoot: z.boolean().optional(),
|
||||
embedInterval: z.string().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const MemoryQmdLimitsSchema = z
|
||||
.object({
|
||||
maxResults: z.number().int().positive().optional(),
|
||||
maxSnippetChars: z.number().int().positive().optional(),
|
||||
maxInjectedChars: z.number().int().positive().optional(),
|
||||
timeoutMs: z.number().int().nonnegative().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const MemoryQmdSchema = z
|
||||
.object({
|
||||
command: z.string().optional(),
|
||||
includeDefaultMemory: z.boolean().optional(),
|
||||
paths: z.array(MemoryQmdPathSchema).optional(),
|
||||
sessions: MemoryQmdSessionSchema.optional(),
|
||||
update: MemoryQmdUpdateSchema.optional(),
|
||||
limits: MemoryQmdLimitsSchema.optional(),
|
||||
scope: SessionSendPolicySchema.optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const MemorySchema = z
|
||||
.object({
|
||||
backend: z.union([z.literal("builtin"), z.literal("qmd")]).optional(),
|
||||
citations: z.union([z.literal("auto"), z.literal("on"), z.literal("off")]).optional(),
|
||||
qmd: MemoryQmdSchema.optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional();
|
||||
|
||||
export const MoltbotSchema = z
|
||||
.object({
|
||||
meta: z
|
||||
@@ -445,6 +505,7 @@ export const MoltbotSchema = z
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
memory: MemorySchema,
|
||||
skills: z
|
||||
.object({
|
||||
allowBundled: z.array(z.string()).optional(),
|
||||
|
||||
@@ -84,10 +84,14 @@ export async function resolveDiscordTarget(
|
||||
|
||||
const likelyUsername = isLikelyUsername(trimmed);
|
||||
const shouldLookup = isExplicitUserLookup(trimmed, parseOptions) || likelyUsername;
|
||||
|
||||
// Parse directly if it's already a known format. Use a safe parse so ambiguous
|
||||
// numeric targets don't throw when we still want to attempt username lookup.
|
||||
const directParse = safeParseDiscordTarget(trimmed, parseOptions);
|
||||
if (directParse && directParse.kind !== "channel" && !likelyUsername) {
|
||||
return directParse;
|
||||
}
|
||||
|
||||
if (!shouldLookup) {
|
||||
return directParse ?? parseDiscordTarget(trimmed, parseOptions);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import type { MoltbotConfig } from "../config/config.js";
|
||||
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
|
||||
import { resolveMemoryBackendConfig } from "./backend-config.js";
|
||||
|
||||
describe("resolveMemoryBackendConfig", () => {
|
||||
it("defaults to builtin backend when config missing", () => {
|
||||
const cfg = { agents: { defaults: { workspace: "/tmp/memory-test" } } } as MoltbotConfig;
|
||||
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
|
||||
expect(resolved.backend).toBe("builtin");
|
||||
expect(resolved.citations).toBe("auto");
|
||||
expect(resolved.qmd).toBeUndefined();
|
||||
});
|
||||
|
||||
it("resolves qmd backend with default collections", () => {
|
||||
const cfg = {
|
||||
agents: { defaults: { workspace: "/tmp/memory-test" } },
|
||||
memory: {
|
||||
backend: "qmd",
|
||||
qmd: {},
|
||||
},
|
||||
} as MoltbotConfig;
|
||||
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
|
||||
expect(resolved.backend).toBe("qmd");
|
||||
expect(resolved.qmd?.collections.length).toBeGreaterThanOrEqual(3);
|
||||
expect(resolved.qmd?.command).toBe("qmd");
|
||||
expect(resolved.qmd?.update.intervalMs).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("resolves custom paths relative to workspace", () => {
|
||||
const cfg = {
|
||||
agents: {
|
||||
defaults: { workspace: "/workspace/root" },
|
||||
list: [{ id: "main", workspace: "/workspace/root" }],
|
||||
},
|
||||
memory: {
|
||||
backend: "qmd",
|
||||
qmd: {
|
||||
paths: [
|
||||
{
|
||||
path: "notes",
|
||||
name: "custom-notes",
|
||||
pattern: "**/*.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
} as MoltbotConfig;
|
||||
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
|
||||
const custom = resolved.qmd?.collections.find((c) => c.name.startsWith("custom-notes"));
|
||||
expect(custom).toBeDefined();
|
||||
const workspaceRoot = resolveAgentWorkspaceDir(cfg, "main");
|
||||
expect(custom?.path).toBe(path.resolve(workspaceRoot, "notes"));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,255 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { parseDurationMs } from "../cli/parse-duration.js";
|
||||
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
|
||||
import type { MoltbotConfig } from "../config/config.js";
|
||||
import type {
|
||||
MemoryBackend,
|
||||
MemoryCitationsMode,
|
||||
MemoryQmdConfig,
|
||||
MemoryQmdIndexPath,
|
||||
} from "../config/types.memory.js";
|
||||
import type { SessionSendPolicyConfig } from "../config/types.base.js";
|
||||
import { resolveUserPath } from "../utils.js";
|
||||
|
||||
export type ResolvedMemoryBackendConfig = {
|
||||
backend: MemoryBackend;
|
||||
citations: MemoryCitationsMode;
|
||||
qmd?: ResolvedQmdConfig;
|
||||
};
|
||||
|
||||
export type ResolvedQmdCollection = {
|
||||
name: string;
|
||||
path: string;
|
||||
pattern: string;
|
||||
kind: "memory" | "custom" | "sessions";
|
||||
};
|
||||
|
||||
export type ResolvedQmdUpdateConfig = {
|
||||
intervalMs: number;
|
||||
debounceMs: number;
|
||||
onBoot: boolean;
|
||||
embedIntervalMs: number;
|
||||
};
|
||||
|
||||
export type ResolvedQmdLimitsConfig = {
|
||||
maxResults: number;
|
||||
maxSnippetChars: number;
|
||||
maxInjectedChars: number;
|
||||
timeoutMs: number;
|
||||
};
|
||||
|
||||
export type ResolvedQmdSessionConfig = {
|
||||
enabled: boolean;
|
||||
exportDir?: string;
|
||||
retentionDays?: number;
|
||||
};
|
||||
|
||||
export type ResolvedQmdConfig = {
|
||||
command: string;
|
||||
collections: ResolvedQmdCollection[];
|
||||
sessions: ResolvedQmdSessionConfig;
|
||||
update: ResolvedQmdUpdateConfig;
|
||||
limits: ResolvedQmdLimitsConfig;
|
||||
includeDefaultMemory: boolean;
|
||||
scope?: SessionSendPolicyConfig;
|
||||
};
|
||||
|
||||
const DEFAULT_BACKEND: MemoryBackend = "builtin";
|
||||
const DEFAULT_CITATIONS: MemoryCitationsMode = "auto";
|
||||
const DEFAULT_QMD_INTERVAL = "5m";
|
||||
const DEFAULT_QMD_DEBOUNCE_MS = 15_000;
|
||||
const DEFAULT_QMD_TIMEOUT_MS = 4_000;
|
||||
const DEFAULT_QMD_EMBED_INTERVAL = "60m";
|
||||
const DEFAULT_QMD_LIMITS: ResolvedQmdLimitsConfig = {
|
||||
maxResults: 6,
|
||||
maxSnippetChars: 700,
|
||||
maxInjectedChars: 4_000,
|
||||
timeoutMs: DEFAULT_QMD_TIMEOUT_MS,
|
||||
};
|
||||
const DEFAULT_QMD_SCOPE: SessionSendPolicyConfig = {
|
||||
default: "deny",
|
||||
rules: [
|
||||
{
|
||||
action: "allow",
|
||||
match: { chatType: "direct" },
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
function sanitizeName(input: string): string {
|
||||
const lower = input.toLowerCase().replace(/[^a-z0-9-]+/g, "-");
|
||||
const trimmed = lower.replace(/^-+|-+$/g, "");
|
||||
return trimmed || "collection";
|
||||
}
|
||||
|
||||
function ensureUniqueName(base: string, existing: Set<string>): string {
|
||||
let name = sanitizeName(base);
|
||||
if (!existing.has(name)) {
|
||||
existing.add(name);
|
||||
return name;
|
||||
}
|
||||
let suffix = 2;
|
||||
while (existing.has(`${name}-${suffix}`)) {
|
||||
suffix += 1;
|
||||
}
|
||||
const unique = `${name}-${suffix}`;
|
||||
existing.add(unique);
|
||||
return unique;
|
||||
}
|
||||
|
||||
function resolvePath(raw: string, workspaceDir: string): string {
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) throw new Error("path required");
|
||||
if (trimmed.startsWith("~") || path.isAbsolute(trimmed)) {
|
||||
return path.normalize(resolveUserPath(trimmed));
|
||||
}
|
||||
return path.normalize(path.resolve(workspaceDir, trimmed));
|
||||
}
|
||||
|
||||
function resolveIntervalMs(raw: string | undefined): number {
|
||||
const value = raw?.trim();
|
||||
if (!value) return parseDurationMs(DEFAULT_QMD_INTERVAL, { defaultUnit: "m" });
|
||||
try {
|
||||
return parseDurationMs(value, { defaultUnit: "m" });
|
||||
} catch {
|
||||
return parseDurationMs(DEFAULT_QMD_INTERVAL, { defaultUnit: "m" });
|
||||
}
|
||||
}
|
||||
|
||||
function resolveEmbedIntervalMs(raw: string | undefined): number {
|
||||
const value = raw?.trim();
|
||||
if (!value) return parseDurationMs(DEFAULT_QMD_EMBED_INTERVAL, { defaultUnit: "m" });
|
||||
try {
|
||||
return parseDurationMs(value, { defaultUnit: "m" });
|
||||
} catch {
|
||||
return parseDurationMs(DEFAULT_QMD_EMBED_INTERVAL, { defaultUnit: "m" });
|
||||
}
|
||||
}
|
||||
|
||||
function resolveDebounceMs(raw: number | undefined): number {
|
||||
if (typeof raw === "number" && Number.isFinite(raw) && raw >= 0) {
|
||||
return Math.floor(raw);
|
||||
}
|
||||
return DEFAULT_QMD_DEBOUNCE_MS;
|
||||
}
|
||||
|
||||
function resolveLimits(raw?: MemoryQmdConfig["limits"]): ResolvedQmdLimitsConfig {
|
||||
const parsed: ResolvedQmdLimitsConfig = { ...DEFAULT_QMD_LIMITS };
|
||||
if (raw?.maxResults && raw.maxResults > 0) parsed.maxResults = Math.floor(raw.maxResults);
|
||||
if (raw?.maxSnippetChars && raw.maxSnippetChars > 0) {
|
||||
parsed.maxSnippetChars = Math.floor(raw.maxSnippetChars);
|
||||
}
|
||||
if (raw?.maxInjectedChars && raw.maxInjectedChars > 0) {
|
||||
parsed.maxInjectedChars = Math.floor(raw.maxInjectedChars);
|
||||
}
|
||||
if (raw?.timeoutMs && raw.timeoutMs > 0) {
|
||||
parsed.timeoutMs = Math.floor(raw.timeoutMs);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function resolveSessionConfig(
|
||||
cfg: MemoryQmdConfig["sessions"],
|
||||
workspaceDir: string,
|
||||
): ResolvedQmdSessionConfig {
|
||||
const enabled = Boolean(cfg?.enabled);
|
||||
const exportDirRaw = cfg?.exportDir?.trim();
|
||||
const exportDir = exportDirRaw ? resolvePath(exportDirRaw, workspaceDir) : undefined;
|
||||
const retentionDays =
|
||||
cfg?.retentionDays && cfg.retentionDays > 0 ? Math.floor(cfg.retentionDays) : undefined;
|
||||
return {
|
||||
enabled,
|
||||
exportDir,
|
||||
retentionDays,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveCustomPaths(
|
||||
rawPaths: MemoryQmdIndexPath[] | undefined,
|
||||
workspaceDir: string,
|
||||
existing: Set<string>,
|
||||
): ResolvedQmdCollection[] {
|
||||
if (!rawPaths?.length) return [];
|
||||
const collections: ResolvedQmdCollection[] = [];
|
||||
rawPaths.forEach((entry, index) => {
|
||||
const trimmedPath = entry?.path?.trim();
|
||||
if (!trimmedPath) return;
|
||||
let resolved: string;
|
||||
try {
|
||||
resolved = resolvePath(trimmedPath, workspaceDir);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
const pattern = entry.pattern?.trim() || "**/*.md";
|
||||
const baseName = entry.name?.trim() || `custom-${index + 1}`;
|
||||
const name = ensureUniqueName(baseName, existing);
|
||||
collections.push({
|
||||
name,
|
||||
path: resolved,
|
||||
pattern,
|
||||
kind: "custom",
|
||||
});
|
||||
});
|
||||
return collections;
|
||||
}
|
||||
|
||||
function resolveDefaultCollections(
|
||||
include: boolean,
|
||||
workspaceDir: string,
|
||||
existing: Set<string>,
|
||||
): ResolvedQmdCollection[] {
|
||||
if (!include) return [];
|
||||
const entries: Array<{ path: string; pattern: string; base: string }> = [
|
||||
{ path: workspaceDir, pattern: "MEMORY.md", base: "memory-root" },
|
||||
{ path: workspaceDir, pattern: "memory.md", base: "memory-alt" },
|
||||
{ path: path.join(workspaceDir, "memory"), pattern: "**/*.md", base: "memory-dir" },
|
||||
];
|
||||
return entries.map((entry) => ({
|
||||
name: ensureUniqueName(entry.base, existing),
|
||||
path: entry.path,
|
||||
pattern: entry.pattern,
|
||||
kind: "memory",
|
||||
}));
|
||||
}
|
||||
|
||||
export function resolveMemoryBackendConfig(params: {
|
||||
cfg: MoltbotConfig;
|
||||
agentId: string;
|
||||
}): ResolvedMemoryBackendConfig {
|
||||
const backend = params.cfg.memory?.backend ?? DEFAULT_BACKEND;
|
||||
const citations = params.cfg.memory?.citations ?? DEFAULT_CITATIONS;
|
||||
if (backend !== "qmd") {
|
||||
return { backend: "builtin", citations };
|
||||
}
|
||||
|
||||
const workspaceDir = resolveAgentWorkspaceDir(params.cfg, params.agentId);
|
||||
const qmdCfg = params.cfg.memory?.qmd;
|
||||
const includeDefaultMemory = qmdCfg?.includeDefaultMemory !== false;
|
||||
const nameSet = new Set<string>();
|
||||
const collections = [
|
||||
...resolveDefaultCollections(includeDefaultMemory, workspaceDir, nameSet),
|
||||
...resolveCustomPaths(qmdCfg?.paths, workspaceDir, nameSet),
|
||||
];
|
||||
|
||||
const resolved: ResolvedQmdConfig = {
|
||||
command: (qmdCfg?.command?.trim() || "qmd").split(/\s+/)[0] || "qmd",
|
||||
collections,
|
||||
includeDefaultMemory,
|
||||
sessions: resolveSessionConfig(qmdCfg?.sessions, workspaceDir),
|
||||
update: {
|
||||
intervalMs: resolveIntervalMs(qmdCfg?.update?.interval),
|
||||
debounceMs: resolveDebounceMs(qmdCfg?.update?.debounceMs),
|
||||
onBoot: qmdCfg?.update?.onBoot !== false,
|
||||
embedIntervalMs: resolveEmbedIntervalMs(qmdCfg?.update?.embedInterval),
|
||||
},
|
||||
limits: resolveLimits(qmdCfg?.limits),
|
||||
scope: qmdCfg?.scope ?? DEFAULT_QMD_SCOPE,
|
||||
};
|
||||
|
||||
return {
|
||||
backend: "qmd",
|
||||
citations,
|
||||
qmd: resolved,
|
||||
};
|
||||
}
|
||||
+6
-1
@@ -1,2 +1,7 @@
|
||||
export type { MemoryIndexManager, MemorySearchResult } from "./manager.js";
|
||||
export { MemoryIndexManager } from "./manager.js";
|
||||
export type {
|
||||
MemoryEmbeddingProbeResult,
|
||||
MemorySearchManager,
|
||||
MemorySearchResult,
|
||||
} from "./types.js";
|
||||
export { getMemorySearchManager, type MemorySearchManagerResult } from "./search-manager.js";
|
||||
+13
-53
@@ -45,17 +45,14 @@ import { searchKeyword, searchVector } from "./manager-search.js";
|
||||
import { ensureMemoryIndexSchema } from "./memory-schema.js";
|
||||
import { requireNodeSqlite } from "./sqlite.js";
|
||||
import { loadSqliteVecExtension } from "./sqlite-vec.js";
|
||||
|
||||
type MemorySource = "memory" | "sessions";
|
||||
|
||||
export type MemorySearchResult = {
|
||||
path: string;
|
||||
startLine: number;
|
||||
endLine: number;
|
||||
score: number;
|
||||
snippet: string;
|
||||
source: MemorySource;
|
||||
};
|
||||
import type {
|
||||
MemoryEmbeddingProbeResult,
|
||||
MemoryProviderStatus,
|
||||
MemorySearchManager,
|
||||
MemorySearchResult,
|
||||
MemorySource,
|
||||
MemorySyncProgressUpdate,
|
||||
} from "./types.js";
|
||||
|
||||
type MemoryIndexMeta = {
|
||||
model: string;
|
||||
@@ -75,12 +72,6 @@ type SessionFileEntry = {
|
||||
content: string;
|
||||
};
|
||||
|
||||
type MemorySyncProgressUpdate = {
|
||||
completed: number;
|
||||
total: number;
|
||||
label?: string;
|
||||
};
|
||||
|
||||
type MemorySyncProgressState = {
|
||||
completed: number;
|
||||
total: number;
|
||||
@@ -115,7 +106,7 @@ const INDEX_CACHE = new Map<string, MemoryIndexManager>();
|
||||
const vectorToBlob = (embedding: number[]): Buffer =>
|
||||
Buffer.from(new Float32Array(embedding).buffer);
|
||||
|
||||
export class MemoryIndexManager {
|
||||
export class MemoryIndexManager implements MemorySearchManager {
|
||||
private readonly cacheKey: string;
|
||||
private readonly cfg: MoltbotConfig;
|
||||
private readonly agentId: string;
|
||||
@@ -415,39 +406,7 @@ export class MemoryIndexManager {
|
||||
return { text: slice.join("\n"), path: relPath };
|
||||
}
|
||||
|
||||
status(): {
|
||||
files: number;
|
||||
chunks: number;
|
||||
dirty: boolean;
|
||||
workspaceDir: string;
|
||||
dbPath: string;
|
||||
provider: string;
|
||||
model: string;
|
||||
requestedProvider: string;
|
||||
sources: MemorySource[];
|
||||
sourceCounts: Array<{ source: MemorySource; files: number; chunks: number }>;
|
||||
cache?: { enabled: boolean; entries?: number; maxEntries?: number };
|
||||
fts?: { enabled: boolean; available: boolean; error?: string };
|
||||
fallback?: { from: string; reason?: string };
|
||||
vector?: {
|
||||
enabled: boolean;
|
||||
available?: boolean;
|
||||
extensionPath?: string;
|
||||
loadError?: string;
|
||||
dims?: number;
|
||||
};
|
||||
batch?: {
|
||||
enabled: boolean;
|
||||
failures: number;
|
||||
limit: number;
|
||||
wait: boolean;
|
||||
concurrency: number;
|
||||
pollIntervalMs: number;
|
||||
timeoutMs: number;
|
||||
lastError?: string;
|
||||
lastProvider?: string;
|
||||
};
|
||||
} {
|
||||
status(): MemoryProviderStatus {
|
||||
const sourceFilter = this.buildSourceFilter();
|
||||
const files = this.db
|
||||
.prepare(`SELECT COUNT(*) as c FROM files WHERE 1=1${sourceFilter.sql}`)
|
||||
@@ -489,9 +448,10 @@ export class MemoryIndexManager {
|
||||
return sources.map((source) => ({ source, ...bySource.get(source)! }));
|
||||
})();
|
||||
return {
|
||||
backend: "builtin",
|
||||
files: files?.c ?? 0,
|
||||
chunks: chunks?.c ?? 0,
|
||||
dirty: this.dirty,
|
||||
dirty: this.dirty || this.sessionsDirty,
|
||||
workspaceDir: this.workspaceDir,
|
||||
dbPath: this.settings.store.path,
|
||||
provider: this.provider.id,
|
||||
@@ -545,7 +505,7 @@ export class MemoryIndexManager {
|
||||
return this.ensureVectorReady();
|
||||
}
|
||||
|
||||
async probeEmbeddingAvailability(): Promise<{ ok: boolean; error?: string }> {
|
||||
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
|
||||
try {
|
||||
await this.embedBatchWithRetry(["ping"]);
|
||||
return { ok: true };
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { EventEmitter } from "node:events";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("node:child_process", () => {
|
||||
const spawn = vi.fn((_cmd: string, _args: string[]) => {
|
||||
const stdout = new EventEmitter();
|
||||
const stderr = new EventEmitter();
|
||||
const child = new EventEmitter() as {
|
||||
stdout: EventEmitter;
|
||||
stderr: EventEmitter;
|
||||
kill: () => void;
|
||||
emit: (event: string, code: number) => boolean;
|
||||
};
|
||||
child.stdout = stdout;
|
||||
child.stderr = stderr;
|
||||
child.kill = () => {
|
||||
child.emit("close", 0);
|
||||
};
|
||||
setImmediate(() => {
|
||||
stdout.emit("data", "");
|
||||
stderr.emit("data", "");
|
||||
child.emit("close", 0);
|
||||
});
|
||||
return child;
|
||||
});
|
||||
return { spawn };
|
||||
});
|
||||
|
||||
import { spawn as mockedSpawn } from "node:child_process";
|
||||
import type { MoltbotConfig } from "../config/config.js";
|
||||
import { resolveMemoryBackendConfig } from "./backend-config.js";
|
||||
import { QmdMemoryManager } from "./qmd-manager.js";
|
||||
|
||||
const spawnMock = mockedSpawn as unknown as vi.Mock;
|
||||
|
||||
describe("QmdMemoryManager", () => {
|
||||
let tmpRoot: string;
|
||||
let workspaceDir: string;
|
||||
let stateDir: string;
|
||||
let cfg: MoltbotConfig;
|
||||
const agentId = "main";
|
||||
|
||||
beforeEach(async () => {
|
||||
spawnMock.mockClear();
|
||||
tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), "qmd-manager-test-"));
|
||||
workspaceDir = path.join(tmpRoot, "workspace");
|
||||
await fs.mkdir(workspaceDir, { recursive: true });
|
||||
stateDir = path.join(tmpRoot, "state");
|
||||
await fs.mkdir(stateDir, { recursive: true });
|
||||
process.env.MOLTBOT_STATE_DIR = stateDir;
|
||||
cfg = {
|
||||
agents: {
|
||||
list: [{ id: agentId, default: true, workspace: workspaceDir }],
|
||||
},
|
||||
memory: {
|
||||
backend: "qmd",
|
||||
qmd: {
|
||||
includeDefaultMemory: false,
|
||||
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
|
||||
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
|
||||
},
|
||||
},
|
||||
} as MoltbotConfig;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
delete process.env.MOLTBOT_STATE_DIR;
|
||||
await fs.rm(tmpRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("debounces back-to-back sync calls", async () => {
|
||||
const resolved = resolveMemoryBackendConfig({ cfg, agentId });
|
||||
const manager = await QmdMemoryManager.create({ cfg, agentId, resolved });
|
||||
expect(manager).toBeTruthy();
|
||||
if (!manager) throw new Error("manager missing");
|
||||
|
||||
const baselineCalls = spawnMock.mock.calls.length;
|
||||
|
||||
await manager.sync({ reason: "manual" });
|
||||
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 2);
|
||||
|
||||
await manager.sync({ reason: "manual-again" });
|
||||
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 2);
|
||||
|
||||
(manager as unknown as { lastUpdateAt: number | null }).lastUpdateAt =
|
||||
Date.now() - (resolved.qmd?.update.debounceMs ?? 0) - 10;
|
||||
|
||||
await manager.sync({ reason: "after-wait" });
|
||||
// By default we refresh embeddings less frequently than index updates.
|
||||
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 3);
|
||||
|
||||
await manager.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,684 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import type { MoltbotConfig } from "../config/config.js";
|
||||
import { resolveStateDir } from "../config/paths.js";
|
||||
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
|
||||
import { createSubsystemLogger } from "../logging/subsystem.js";
|
||||
import {
|
||||
listSessionFilesForAgent,
|
||||
buildSessionEntry,
|
||||
type SessionFileEntry,
|
||||
} from "./session-files.js";
|
||||
import { requireNodeSqlite } from "./sqlite.js";
|
||||
import type {
|
||||
MemoryEmbeddingProbeResult,
|
||||
MemoryProviderStatus,
|
||||
MemorySearchManager,
|
||||
MemorySearchResult,
|
||||
MemorySource,
|
||||
MemorySyncProgressUpdate,
|
||||
} from "./types.js";
|
||||
|
||||
type SqliteDatabase = import("node:sqlite").DatabaseSync;
|
||||
import type { ResolvedMemoryBackendConfig, ResolvedQmdConfig } from "./backend-config.js";
|
||||
|
||||
const log = createSubsystemLogger("memory");
|
||||
|
||||
const SNIPPET_HEADER_RE = /@@\s*-([0-9]+),([0-9]+)/;
|
||||
|
||||
type QmdQueryResult = {
|
||||
docid?: string;
|
||||
score?: number;
|
||||
file?: string;
|
||||
snippet?: string;
|
||||
body?: string;
|
||||
};
|
||||
|
||||
type CollectionRoot = {
|
||||
path: string;
|
||||
kind: MemorySource;
|
||||
};
|
||||
|
||||
type SessionExporterConfig = {
|
||||
dir: string;
|
||||
retentionMs?: number;
|
||||
collectionName: string;
|
||||
};
|
||||
|
||||
export class QmdMemoryManager implements MemorySearchManager {
|
||||
static async create(params: {
|
||||
cfg: MoltbotConfig;
|
||||
agentId: string;
|
||||
resolved: ResolvedMemoryBackendConfig;
|
||||
}): Promise<QmdMemoryManager | null> {
|
||||
const resolved = params.resolved.qmd;
|
||||
if (!resolved) return null;
|
||||
const manager = new QmdMemoryManager({ cfg: params.cfg, agentId: params.agentId, resolved });
|
||||
await manager.initialize();
|
||||
return manager;
|
||||
}
|
||||
|
||||
private readonly cfg: MoltbotConfig;
|
||||
private readonly agentId: string;
|
||||
private readonly qmd: ResolvedQmdConfig;
|
||||
private readonly workspaceDir: string;
|
||||
private readonly stateDir: string;
|
||||
private readonly agentStateDir: string;
|
||||
private readonly qmdDir: string;
|
||||
private readonly xdgConfigHome: string;
|
||||
private readonly xdgCacheHome: string;
|
||||
private readonly indexPath: string;
|
||||
private readonly env: NodeJS.ProcessEnv;
|
||||
private readonly collectionRoots = new Map<string, CollectionRoot>();
|
||||
private readonly sources = new Set<MemorySource>();
|
||||
private readonly docPathCache = new Map<
|
||||
string,
|
||||
{ rel: string; abs: string; source: MemorySource }
|
||||
>();
|
||||
private readonly sessionExporter: SessionExporterConfig | null;
|
||||
private updateTimer: NodeJS.Timeout | null = null;
|
||||
private pendingUpdate: Promise<void> | null = null;
|
||||
private closed = false;
|
||||
private db: SqliteDatabase | null = null;
|
||||
private lastUpdateAt: number | null = null;
|
||||
private lastEmbedAt: number | null = null;
|
||||
|
||||
private constructor(params: {
|
||||
cfg: MoltbotConfig;
|
||||
agentId: string;
|
||||
resolved: ResolvedQmdConfig;
|
||||
}) {
|
||||
this.cfg = params.cfg;
|
||||
this.agentId = params.agentId;
|
||||
this.qmd = params.resolved;
|
||||
this.workspaceDir = resolveAgentWorkspaceDir(params.cfg, params.agentId);
|
||||
this.stateDir = resolveStateDir(process.env, os.homedir);
|
||||
this.agentStateDir = path.join(this.stateDir, "agents", this.agentId);
|
||||
this.qmdDir = path.join(this.agentStateDir, "qmd");
|
||||
// QMD uses XDG base dirs for its internal state.
|
||||
// Collections are managed via `qmd collection add` and stored inside the index DB.
|
||||
// - config: $XDG_CONFIG_HOME (contexts, etc.)
|
||||
// - cache: $XDG_CACHE_HOME/qmd/index.sqlite
|
||||
this.xdgConfigHome = path.join(this.qmdDir, "xdg-config");
|
||||
this.xdgCacheHome = path.join(this.qmdDir, "xdg-cache");
|
||||
this.indexPath = path.join(this.xdgCacheHome, "qmd", "index.sqlite");
|
||||
|
||||
this.env = {
|
||||
...process.env,
|
||||
XDG_CONFIG_HOME: this.xdgConfigHome,
|
||||
XDG_CACHE_HOME: this.xdgCacheHome,
|
||||
NO_COLOR: "1",
|
||||
};
|
||||
this.sessionExporter = this.qmd.sessions.enabled
|
||||
? {
|
||||
dir: this.qmd.sessions.exportDir ?? path.join(this.qmdDir, "sessions"),
|
||||
retentionMs: this.qmd.sessions.retentionDays
|
||||
? this.qmd.sessions.retentionDays * 24 * 60 * 60 * 1000
|
||||
: undefined,
|
||||
collectionName: this.pickSessionCollectionName(),
|
||||
}
|
||||
: null;
|
||||
if (this.sessionExporter) {
|
||||
this.qmd.collections = [
|
||||
...this.qmd.collections,
|
||||
{
|
||||
name: this.sessionExporter.collectionName,
|
||||
path: this.sessionExporter.dir,
|
||||
pattern: "**/*.md",
|
||||
kind: "sessions",
|
||||
},
|
||||
];
|
||||
}
|
||||
}
|
||||
|
||||
private async initialize(): Promise<void> {
|
||||
await fs.mkdir(this.xdgConfigHome, { recursive: true });
|
||||
await fs.mkdir(this.xdgCacheHome, { recursive: true });
|
||||
await fs.mkdir(path.dirname(this.indexPath), { recursive: true });
|
||||
|
||||
this.bootstrapCollections();
|
||||
await this.ensureCollections();
|
||||
|
||||
if (this.qmd.update.onBoot) {
|
||||
await this.runUpdate("boot", true);
|
||||
}
|
||||
if (this.qmd.update.intervalMs > 0) {
|
||||
this.updateTimer = setInterval(() => {
|
||||
void this.runUpdate("interval").catch((err) => {
|
||||
log.warn(`qmd update failed (${String(err)})`);
|
||||
});
|
||||
}, this.qmd.update.intervalMs);
|
||||
}
|
||||
}
|
||||
|
||||
private bootstrapCollections(): void {
|
||||
this.collectionRoots.clear();
|
||||
this.sources.clear();
|
||||
for (const collection of this.qmd.collections) {
|
||||
const kind: MemorySource = collection.kind === "sessions" ? "sessions" : "memory";
|
||||
this.collectionRoots.set(collection.name, { path: collection.path, kind });
|
||||
this.sources.add(kind);
|
||||
}
|
||||
}
|
||||
|
||||
private async ensureCollections(): Promise<void> {
|
||||
// QMD collections are persisted inside the index database and must be created
|
||||
// via the CLI. Prefer listing existing collections when supported, otherwise
|
||||
// fall back to best-effort idempotent `qmd collection add`.
|
||||
const existing = new Set<string>();
|
||||
try {
|
||||
const result = await this.runQmd(["collection", "list", "--json"]);
|
||||
const parsed = JSON.parse(result.stdout) as unknown;
|
||||
if (Array.isArray(parsed)) {
|
||||
for (const entry of parsed) {
|
||||
if (typeof entry === "string") existing.add(entry);
|
||||
else if (entry && typeof entry === "object") {
|
||||
const name = (entry as { name?: unknown }).name;
|
||||
if (typeof name === "string") existing.add(name);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// ignore; older qmd versions might not support list --json.
|
||||
}
|
||||
|
||||
for (const collection of this.qmd.collections) {
|
||||
if (existing.has(collection.name)) continue;
|
||||
try {
|
||||
await this.runQmd([
|
||||
"collection",
|
||||
"add",
|
||||
collection.path,
|
||||
"--name",
|
||||
collection.name,
|
||||
"--mask",
|
||||
collection.pattern,
|
||||
]);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
// Idempotency: qmd exits non-zero if the collection name already exists.
|
||||
if (message.toLowerCase().includes("already exists")) continue;
|
||||
if (message.toLowerCase().includes("exists")) continue;
|
||||
log.warn(`qmd collection add failed for ${collection.name}: ${message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async search(
|
||||
query: string,
|
||||
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
|
||||
): Promise<MemorySearchResult[]> {
|
||||
if (!this.isScopeAllowed(opts?.sessionKey)) return [];
|
||||
const trimmed = query.trim();
|
||||
if (!trimmed) return [];
|
||||
await this.pendingUpdate?.catch(() => undefined);
|
||||
const limit = Math.min(
|
||||
this.qmd.limits.maxResults,
|
||||
opts?.maxResults ?? this.qmd.limits.maxResults,
|
||||
);
|
||||
const args = ["query", trimmed, "--json", "-n", String(limit)];
|
||||
let stdout: string;
|
||||
try {
|
||||
const result = await this.runQmd(args, { timeoutMs: this.qmd.limits.timeoutMs });
|
||||
stdout = result.stdout;
|
||||
} catch (err) {
|
||||
log.warn(`qmd query failed: ${String(err)}`);
|
||||
throw err instanceof Error ? err : new Error(String(err));
|
||||
}
|
||||
let parsed: QmdQueryResult[] = [];
|
||||
try {
|
||||
parsed = JSON.parse(stdout);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
log.warn(`qmd query returned invalid JSON: ${message}`);
|
||||
throw new Error(`qmd query returned invalid JSON: ${message}`);
|
||||
}
|
||||
const results: MemorySearchResult[] = [];
|
||||
for (const entry of parsed) {
|
||||
const doc = await this.resolveDocLocation(entry.docid);
|
||||
if (!doc) continue;
|
||||
const snippet = entry.snippet?.slice(0, this.qmd.limits.maxSnippetChars) ?? "";
|
||||
const lines = this.extractSnippetLines(snippet);
|
||||
const score = typeof entry.score === "number" ? entry.score : 0;
|
||||
const minScore = opts?.minScore ?? 0;
|
||||
if (score < minScore) continue;
|
||||
results.push({
|
||||
path: doc.rel,
|
||||
startLine: lines.startLine,
|
||||
endLine: lines.endLine,
|
||||
score,
|
||||
snippet,
|
||||
source: doc.source,
|
||||
});
|
||||
}
|
||||
return this.clampResultsByInjectedChars(results.slice(0, limit));
|
||||
}
|
||||
|
||||
async sync(params?: {
|
||||
reason?: string;
|
||||
force?: boolean;
|
||||
progress?: (update: MemorySyncProgressUpdate) => void;
|
||||
}): Promise<void> {
|
||||
if (params?.progress) {
|
||||
params.progress({ completed: 0, total: 1, label: "Updating QMD index…" });
|
||||
}
|
||||
await this.runUpdate(params?.reason ?? "manual", params?.force);
|
||||
if (params?.progress) {
|
||||
params.progress({ completed: 1, total: 1, label: "QMD index updated" });
|
||||
}
|
||||
}
|
||||
|
||||
async readFile(params: {
|
||||
relPath: string;
|
||||
from?: number;
|
||||
lines?: number;
|
||||
}): Promise<{ text: string; path: string }> {
|
||||
const relPath = params.relPath?.trim();
|
||||
if (!relPath) throw new Error("path required");
|
||||
const absPath = this.resolveReadPath(relPath);
|
||||
const content = await fs.readFile(absPath, "utf-8");
|
||||
if (!params.from && !params.lines) {
|
||||
return { text: content, path: relPath };
|
||||
}
|
||||
const lines = content.split("\n");
|
||||
const start = Math.max(1, params.from ?? 1);
|
||||
const count = Math.max(1, params.lines ?? lines.length);
|
||||
const slice = lines.slice(start - 1, start - 1 + count);
|
||||
return { text: slice.join("\n"), path: relPath };
|
||||
}
|
||||
|
||||
status(): MemoryProviderStatus {
|
||||
const counts = this.readCounts();
|
||||
return {
|
||||
backend: "qmd",
|
||||
provider: "qmd",
|
||||
model: "qmd",
|
||||
requestedProvider: "qmd",
|
||||
files: counts.totalDocuments,
|
||||
chunks: counts.totalDocuments,
|
||||
dirty: false,
|
||||
workspaceDir: this.workspaceDir,
|
||||
dbPath: this.indexPath,
|
||||
sources: Array.from(this.sources),
|
||||
sourceCounts: counts.sourceCounts,
|
||||
vector: { enabled: true, available: true },
|
||||
batch: {
|
||||
enabled: false,
|
||||
failures: 0,
|
||||
limit: 0,
|
||||
wait: false,
|
||||
concurrency: 0,
|
||||
pollIntervalMs: 0,
|
||||
timeoutMs: 0,
|
||||
},
|
||||
custom: {
|
||||
qmd: {
|
||||
collections: this.qmd.collections.length,
|
||||
lastUpdateAt: this.lastUpdateAt,
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
async probeVectorAvailability(): Promise<boolean> {
|
||||
return true;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
if (this.updateTimer) {
|
||||
clearInterval(this.updateTimer);
|
||||
this.updateTimer = null;
|
||||
}
|
||||
await this.pendingUpdate?.catch(() => undefined);
|
||||
if (this.db) {
|
||||
this.db.close();
|
||||
this.db = null;
|
||||
}
|
||||
}
|
||||
|
||||
private async runUpdate(reason: string, force?: boolean): Promise<void> {
|
||||
if (this.pendingUpdate && !force) return this.pendingUpdate;
|
||||
if (this.shouldSkipUpdate(force)) {
|
||||
return;
|
||||
}
|
||||
const run = async () => {
|
||||
if (this.sessionExporter) {
|
||||
await this.exportSessions();
|
||||
}
|
||||
await this.runQmd(["update"], { timeoutMs: 120_000 });
|
||||
const embedIntervalMs = this.qmd.update.embedIntervalMs;
|
||||
const shouldEmbed =
|
||||
Boolean(force) ||
|
||||
this.lastEmbedAt === null ||
|
||||
(embedIntervalMs > 0 && Date.now() - this.lastEmbedAt > embedIntervalMs);
|
||||
if (shouldEmbed) {
|
||||
try {
|
||||
await this.runQmd(["embed"], { timeoutMs: 120_000 });
|
||||
this.lastEmbedAt = Date.now();
|
||||
} catch (err) {
|
||||
log.warn(`qmd embed failed (${reason}): ${String(err)}`);
|
||||
}
|
||||
}
|
||||
this.lastUpdateAt = Date.now();
|
||||
this.docPathCache.clear();
|
||||
};
|
||||
this.pendingUpdate = run().finally(() => {
|
||||
this.pendingUpdate = null;
|
||||
});
|
||||
await this.pendingUpdate;
|
||||
}
|
||||
|
||||
private async runQmd(
|
||||
args: string[],
|
||||
opts?: { timeoutMs?: number },
|
||||
): Promise<{ stdout: string; stderr: string }> {
|
||||
return await new Promise((resolve, reject) => {
|
||||
const child = spawn(this.qmd.command, args, {
|
||||
env: this.env,
|
||||
cwd: this.workspaceDir,
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const timer = opts?.timeoutMs
|
||||
? setTimeout(() => {
|
||||
child.kill("SIGKILL");
|
||||
reject(new Error(`qmd ${args.join(" ")} timed out after ${opts.timeoutMs}ms`));
|
||||
}, opts.timeoutMs)
|
||||
: null;
|
||||
child.stdout.on("data", (data) => {
|
||||
stdout += data.toString();
|
||||
});
|
||||
child.stderr.on("data", (data) => {
|
||||
stderr += data.toString();
|
||||
});
|
||||
child.on("error", (err) => {
|
||||
if (timer) clearTimeout(timer);
|
||||
reject(err);
|
||||
});
|
||||
child.on("close", (code) => {
|
||||
if (timer) clearTimeout(timer);
|
||||
if (code === 0) {
|
||||
resolve({ stdout, stderr });
|
||||
} else {
|
||||
reject(new Error(`qmd ${args.join(" ")} failed (code ${code}): ${stderr || stdout}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
private ensureDb(): SqliteDatabase {
|
||||
if (this.db) return this.db;
|
||||
const { DatabaseSync } = requireNodeSqlite();
|
||||
this.db = new DatabaseSync(this.indexPath, { readOnly: true });
|
||||
return this.db;
|
||||
}
|
||||
|
||||
private async exportSessions(): Promise<void> {
|
||||
if (!this.sessionExporter) return;
|
||||
const exportDir = this.sessionExporter.dir;
|
||||
await fs.mkdir(exportDir, { recursive: true });
|
||||
const files = await listSessionFilesForAgent(this.agentId);
|
||||
const keep = new Set<string>();
|
||||
const cutoff = this.sessionExporter.retentionMs
|
||||
? Date.now() - this.sessionExporter.retentionMs
|
||||
: null;
|
||||
for (const sessionFile of files) {
|
||||
const entry = await buildSessionEntry(sessionFile);
|
||||
if (!entry) continue;
|
||||
if (cutoff && entry.mtimeMs < cutoff) continue;
|
||||
const target = path.join(exportDir, `${path.basename(sessionFile, ".jsonl")}.md`);
|
||||
await fs.writeFile(target, this.renderSessionMarkdown(entry), "utf-8");
|
||||
keep.add(target);
|
||||
}
|
||||
const exported = await fs.readdir(exportDir).catch(() => []);
|
||||
for (const name of exported) {
|
||||
if (!name.endsWith(".md")) continue;
|
||||
const full = path.join(exportDir, name);
|
||||
if (!keep.has(full)) {
|
||||
await fs.rm(full, { force: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private renderSessionMarkdown(entry: SessionFileEntry): string {
|
||||
const header = `# Session ${path.basename(entry.absPath, path.extname(entry.absPath))}`;
|
||||
const body = entry.content?.trim().length ? entry.content.trim() : "(empty)";
|
||||
return `${header}\n\n${body}\n`;
|
||||
}
|
||||
|
||||
private pickSessionCollectionName(): string {
|
||||
const existing = new Set(this.qmd.collections.map((collection) => collection.name));
|
||||
if (!existing.has("sessions")) return "sessions";
|
||||
let counter = 2;
|
||||
let candidate = `sessions-${counter}`;
|
||||
while (existing.has(candidate)) {
|
||||
counter += 1;
|
||||
candidate = `sessions-${counter}`;
|
||||
}
|
||||
return candidate;
|
||||
}
|
||||
|
||||
private async resolveDocLocation(
|
||||
docid?: string,
|
||||
): Promise<{ rel: string; abs: string; source: MemorySource } | null> {
|
||||
if (!docid) return null;
|
||||
const normalized = docid.startsWith("#") ? docid.slice(1) : docid;
|
||||
if (!normalized) return null;
|
||||
const cached = this.docPathCache.get(normalized);
|
||||
if (cached) return cached;
|
||||
const db = this.ensureDb();
|
||||
const row = db
|
||||
.prepare("SELECT collection, path FROM documents WHERE hash LIKE ? AND active = 1 LIMIT 1")
|
||||
.get(`${normalized}%`) as { collection: string; path: string } | undefined;
|
||||
if (!row) return null;
|
||||
const location = this.toDocLocation(row.collection, row.path);
|
||||
if (!location) return null;
|
||||
this.docPathCache.set(normalized, location);
|
||||
return location;
|
||||
}
|
||||
|
||||
private extractSnippetLines(snippet: string): { startLine: number; endLine: number } {
|
||||
const match = SNIPPET_HEADER_RE.exec(snippet);
|
||||
if (match) {
|
||||
const start = Number(match[1]);
|
||||
const count = Number(match[2]);
|
||||
if (Number.isFinite(start) && Number.isFinite(count)) {
|
||||
return { startLine: start, endLine: start + count - 1 };
|
||||
}
|
||||
}
|
||||
const lines = snippet.split("\n").length;
|
||||
return { startLine: 1, endLine: lines };
|
||||
}
|
||||
|
||||
private readCounts(): {
|
||||
totalDocuments: number;
|
||||
sourceCounts: Array<{ source: MemorySource; files: number; chunks: number }>;
|
||||
} {
|
||||
try {
|
||||
const db = this.ensureDb();
|
||||
const rows = db
|
||||
.prepare(
|
||||
"SELECT collection, COUNT(*) as c FROM documents WHERE active = 1 GROUP BY collection",
|
||||
)
|
||||
.all() as Array<{ collection: string; c: number }>;
|
||||
const bySource = new Map<MemorySource, { files: number; chunks: number }>();
|
||||
for (const source of this.sources) {
|
||||
bySource.set(source, { files: 0, chunks: 0 });
|
||||
}
|
||||
let total = 0;
|
||||
for (const row of rows) {
|
||||
const root = this.collectionRoots.get(row.collection);
|
||||
const source = root?.kind ?? "memory";
|
||||
const entry = bySource.get(source) ?? { files: 0, chunks: 0 };
|
||||
entry.files += row.c ?? 0;
|
||||
entry.chunks += row.c ?? 0;
|
||||
bySource.set(source, entry);
|
||||
total += row.c ?? 0;
|
||||
}
|
||||
return {
|
||||
totalDocuments: total,
|
||||
sourceCounts: Array.from(bySource.entries()).map(([source, value]) => ({
|
||||
source,
|
||||
files: value.files,
|
||||
chunks: value.chunks,
|
||||
})),
|
||||
};
|
||||
} catch (err) {
|
||||
log.warn(`failed to read qmd index stats: ${String(err)}`);
|
||||
return {
|
||||
totalDocuments: 0,
|
||||
sourceCounts: Array.from(this.sources).map((source) => ({ source, files: 0, chunks: 0 })),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
private isScopeAllowed(sessionKey?: string): boolean {
|
||||
const scope = this.qmd.scope;
|
||||
if (!scope) return true;
|
||||
const channel = this.deriveChannelFromKey(sessionKey);
|
||||
const chatType = this.deriveChatTypeFromKey(sessionKey);
|
||||
const normalizedKey = sessionKey ?? "";
|
||||
for (const rule of scope.rules ?? []) {
|
||||
if (!rule) continue;
|
||||
const match = rule.match ?? {};
|
||||
if (match.channel && match.channel !== channel) continue;
|
||||
if (match.chatType && match.chatType !== chatType) continue;
|
||||
if (match.keyPrefix && !normalizedKey.startsWith(match.keyPrefix)) continue;
|
||||
return rule.action === "allow";
|
||||
}
|
||||
const fallback = scope.default ?? "allow";
|
||||
return fallback === "allow";
|
||||
}
|
||||
|
||||
private deriveChannelFromKey(key?: string) {
|
||||
if (!key) return undefined;
|
||||
const parts = key.split(":").filter(Boolean);
|
||||
if (parts.length >= 3 && (parts[1] === "group" || parts[1] === "channel")) {
|
||||
return parts[0]?.toLowerCase();
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
private deriveChatTypeFromKey(key?: string) {
|
||||
if (!key) return undefined;
|
||||
if (key.includes(":group:")) return "group";
|
||||
if (key.includes(":channel:")) return "channel";
|
||||
return "direct";
|
||||
}
|
||||
|
||||
private toDocLocation(
|
||||
collection: string,
|
||||
collectionRelativePath: string,
|
||||
): { rel: string; abs: string; source: MemorySource } | null {
|
||||
const root = this.collectionRoots.get(collection);
|
||||
if (!root) return null;
|
||||
const normalizedRelative = collectionRelativePath.replace(/\\/g, "/");
|
||||
const absPath = path.normalize(path.resolve(root.path, collectionRelativePath));
|
||||
const relativeToWorkspace = path.relative(this.workspaceDir, absPath);
|
||||
const relPath = this.buildSearchPath(
|
||||
collection,
|
||||
normalizedRelative,
|
||||
relativeToWorkspace,
|
||||
absPath,
|
||||
);
|
||||
return { rel: relPath, abs: absPath, source: root.kind };
|
||||
}
|
||||
|
||||
private buildSearchPath(
|
||||
collection: string,
|
||||
collectionRelativePath: string,
|
||||
relativeToWorkspace: string,
|
||||
absPath: string,
|
||||
): string {
|
||||
const insideWorkspace = this.isInsideWorkspace(relativeToWorkspace);
|
||||
if (insideWorkspace) {
|
||||
const normalized = relativeToWorkspace.replace(/\\/g, "/");
|
||||
if (!normalized) return path.basename(absPath);
|
||||
return normalized;
|
||||
}
|
||||
const sanitized = collectionRelativePath.replace(/^\/+/, "");
|
||||
return `qmd/${collection}/${sanitized}`;
|
||||
}
|
||||
|
||||
private isInsideWorkspace(relativePath: string): boolean {
|
||||
if (!relativePath) return true;
|
||||
if (relativePath.startsWith("..")) return false;
|
||||
if (relativePath.startsWith(`..${path.sep}`)) return false;
|
||||
return !path.isAbsolute(relativePath);
|
||||
}
|
||||
|
||||
private resolveReadPath(relPath: string): string {
|
||||
if (relPath.startsWith("qmd/")) {
|
||||
const [, collection, ...rest] = relPath.split("/");
|
||||
if (!collection || rest.length === 0) {
|
||||
throw new Error("invalid qmd path");
|
||||
}
|
||||
const root = this.collectionRoots.get(collection);
|
||||
if (!root) throw new Error(`unknown qmd collection: ${collection}`);
|
||||
const joined = rest.join("/");
|
||||
const resolved = path.resolve(root.path, joined);
|
||||
if (!this.isWithinRoot(root.path, resolved)) {
|
||||
throw new Error("qmd path escapes collection");
|
||||
}
|
||||
return resolved;
|
||||
}
|
||||
const absPath = path.resolve(this.workspaceDir, relPath);
|
||||
if (!this.isWithinWorkspace(absPath)) {
|
||||
throw new Error("path escapes workspace");
|
||||
}
|
||||
return absPath;
|
||||
}
|
||||
|
||||
private isWithinWorkspace(absPath: string): boolean {
|
||||
const normalizedWorkspace = this.workspaceDir.endsWith(path.sep)
|
||||
? this.workspaceDir
|
||||
: `${this.workspaceDir}${path.sep}`;
|
||||
if (absPath === this.workspaceDir) return true;
|
||||
const candidate = absPath.endsWith(path.sep) ? absPath : `${absPath}${path.sep}`;
|
||||
return candidate.startsWith(normalizedWorkspace);
|
||||
}
|
||||
|
||||
private isWithinRoot(root: string, candidate: string): boolean {
|
||||
const normalizedRoot = root.endsWith(path.sep) ? root : `${root}${path.sep}`;
|
||||
if (candidate === root) return true;
|
||||
const next = candidate.endsWith(path.sep) ? candidate : `${candidate}${path.sep}`;
|
||||
return next.startsWith(normalizedRoot);
|
||||
}
|
||||
|
||||
private clampResultsByInjectedChars(results: MemorySearchResult[]): MemorySearchResult[] {
|
||||
const budget = this.qmd.limits.maxInjectedChars;
|
||||
if (!budget || budget <= 0) return results;
|
||||
let remaining = budget;
|
||||
const clamped: MemorySearchResult[] = [];
|
||||
for (const entry of results) {
|
||||
if (remaining <= 0) break;
|
||||
const snippet = entry.snippet ?? "";
|
||||
if (snippet.length <= remaining) {
|
||||
clamped.push(entry);
|
||||
remaining -= snippet.length;
|
||||
} else {
|
||||
const trimmed = snippet.slice(0, Math.max(0, remaining));
|
||||
clamped.push({ ...entry, snippet: trimmed });
|
||||
break;
|
||||
}
|
||||
}
|
||||
return clamped;
|
||||
}
|
||||
|
||||
private shouldSkipUpdate(force?: boolean): boolean {
|
||||
if (force) return false;
|
||||
const debounceMs = this.qmd.update.debounceMs;
|
||||
if (debounceMs <= 0) return false;
|
||||
if (!this.lastUpdateAt) return false;
|
||||
return Date.now() - this.lastUpdateAt < debounceMs;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mockPrimary = {
|
||||
search: vi.fn(async () => []),
|
||||
readFile: vi.fn(async () => ({ text: "", path: "MEMORY.md" })),
|
||||
status: vi.fn(() => ({
|
||||
backend: "qmd" as const,
|
||||
provider: "qmd",
|
||||
model: "qmd",
|
||||
requestedProvider: "qmd",
|
||||
files: 0,
|
||||
chunks: 0,
|
||||
dirty: false,
|
||||
workspaceDir: "/tmp",
|
||||
dbPath: "/tmp/index.sqlite",
|
||||
sources: ["memory" as const],
|
||||
sourceCounts: [{ source: "memory" as const, files: 0, chunks: 0 }],
|
||||
})),
|
||||
sync: vi.fn(async () => {}),
|
||||
probeEmbeddingAvailability: vi.fn(async () => ({ ok: true })),
|
||||
probeVectorAvailability: vi.fn(async () => true),
|
||||
close: vi.fn(async () => {}),
|
||||
};
|
||||
|
||||
vi.mock("./qmd-manager.js", () => ({
|
||||
QmdMemoryManager: {
|
||||
create: vi.fn(async () => mockPrimary),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock("./manager.js", () => ({
|
||||
MemoryIndexManager: {
|
||||
get: vi.fn(async () => null),
|
||||
},
|
||||
}));
|
||||
|
||||
import { QmdMemoryManager } from "./qmd-manager.js";
|
||||
import { getMemorySearchManager } from "./search-manager.js";
|
||||
|
||||
beforeEach(() => {
|
||||
mockPrimary.search.mockClear();
|
||||
mockPrimary.readFile.mockClear();
|
||||
mockPrimary.status.mockClear();
|
||||
mockPrimary.sync.mockClear();
|
||||
mockPrimary.probeEmbeddingAvailability.mockClear();
|
||||
mockPrimary.probeVectorAvailability.mockClear();
|
||||
mockPrimary.close.mockClear();
|
||||
QmdMemoryManager.create.mockClear();
|
||||
});
|
||||
|
||||
describe("getMemorySearchManager caching", () => {
|
||||
it("reuses the same QMD manager instance for repeated calls", async () => {
|
||||
const cfg = {
|
||||
memory: { backend: "qmd", qmd: {} },
|
||||
agents: { list: [{ id: "main", default: true, workspace: "/tmp/workspace" }] },
|
||||
} as const;
|
||||
|
||||
const first = await getMemorySearchManager({ cfg, agentId: "main" });
|
||||
const second = await getMemorySearchManager({ cfg, agentId: "main" });
|
||||
|
||||
expect(first.manager).toBe(second.manager);
|
||||
// eslint-disable-next-line @typescript-eslint/unbound-method
|
||||
expect(QmdMemoryManager.create).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
@@ -1,8 +1,18 @@
|
||||
import { createSubsystemLogger } from "../logging/subsystem.js";
|
||||
import type { MoltbotConfig } from "../config/config.js";
|
||||
import type { MemoryIndexManager } from "./manager.js";
|
||||
import { resolveMemoryBackendConfig } from "./backend-config.js";
|
||||
import type { ResolvedQmdConfig } from "./backend-config.js";
|
||||
import type {
|
||||
MemoryEmbeddingProbeResult,
|
||||
MemorySearchManager,
|
||||
MemorySyncProgressUpdate,
|
||||
} from "./types.js";
|
||||
|
||||
const log = createSubsystemLogger("memory");
|
||||
const QMD_MANAGER_CACHE = new Map<string, MemorySearchManager>();
|
||||
|
||||
export type MemorySearchManagerResult = {
|
||||
manager: MemoryIndexManager | null;
|
||||
manager: MemorySearchManager | null;
|
||||
error?: string;
|
||||
};
|
||||
|
||||
@@ -10,6 +20,40 @@ export async function getMemorySearchManager(params: {
|
||||
cfg: MoltbotConfig;
|
||||
agentId: string;
|
||||
}): Promise<MemorySearchManagerResult> {
|
||||
const resolved = resolveMemoryBackendConfig(params);
|
||||
if (resolved.backend === "qmd" && resolved.qmd) {
|
||||
const cacheKey = buildQmdCacheKey(params.agentId, resolved.qmd);
|
||||
const cached = QMD_MANAGER_CACHE.get(cacheKey);
|
||||
if (cached) {
|
||||
return { manager: cached };
|
||||
}
|
||||
try {
|
||||
const { QmdMemoryManager } = await import("./qmd-manager.js");
|
||||
const primary = await QmdMemoryManager.create({
|
||||
cfg: params.cfg,
|
||||
agentId: params.agentId,
|
||||
resolved,
|
||||
});
|
||||
if (primary) {
|
||||
const wrapper = new FallbackMemoryManager(
|
||||
{
|
||||
primary,
|
||||
fallbackFactory: async () => {
|
||||
const { MemoryIndexManager } = await import("./manager.js");
|
||||
return await MemoryIndexManager.get(params);
|
||||
},
|
||||
},
|
||||
() => QMD_MANAGER_CACHE.delete(cacheKey),
|
||||
);
|
||||
QMD_MANAGER_CACHE.set(cacheKey, wrapper);
|
||||
return { manager: wrapper };
|
||||
}
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
log.warn(`qmd memory unavailable; falling back to builtin: ${message}`);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
const { MemoryIndexManager } = await import("./manager.js");
|
||||
const manager = await MemoryIndexManager.get(params);
|
||||
@@ -19,3 +63,131 @@ export async function getMemorySearchManager(params: {
|
||||
return { manager: null, error: message };
|
||||
}
|
||||
}
|
||||
|
||||
class FallbackMemoryManager implements MemorySearchManager {
|
||||
private fallback: MemorySearchManager | null = null;
|
||||
private primaryFailed = false;
|
||||
private lastError?: string;
|
||||
|
||||
constructor(
|
||||
private readonly deps: {
|
||||
primary: MemorySearchManager;
|
||||
fallbackFactory: () => Promise<MemorySearchManager | null>;
|
||||
},
|
||||
private readonly onClose?: () => void,
|
||||
) {}
|
||||
|
||||
async search(
|
||||
query: string,
|
||||
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
|
||||
) {
|
||||
if (!this.primaryFailed) {
|
||||
try {
|
||||
return await this.deps.primary.search(query, opts);
|
||||
} catch (err) {
|
||||
this.primaryFailed = true;
|
||||
this.lastError = err instanceof Error ? err.message : String(err);
|
||||
log.warn(`qmd memory failed; switching to builtin index: ${this.lastError}`);
|
||||
await this.deps.primary.close?.().catch(() => {});
|
||||
}
|
||||
}
|
||||
const fallback = await this.ensureFallback();
|
||||
if (fallback) {
|
||||
return await fallback.search(query, opts);
|
||||
}
|
||||
throw new Error(this.lastError ?? "memory search unavailable");
|
||||
}
|
||||
|
||||
async readFile(params: { relPath: string; from?: number; lines?: number }) {
|
||||
if (!this.primaryFailed) {
|
||||
return await this.deps.primary.readFile(params);
|
||||
}
|
||||
const fallback = await this.ensureFallback();
|
||||
if (fallback) {
|
||||
return await fallback.readFile(params);
|
||||
}
|
||||
throw new Error(this.lastError ?? "memory read unavailable");
|
||||
}
|
||||
|
||||
status() {
|
||||
if (!this.primaryFailed) {
|
||||
return this.deps.primary.status();
|
||||
}
|
||||
const fallbackStatus = this.fallback?.status();
|
||||
const fallbackInfo = { from: "qmd", reason: this.lastError ?? "unknown" };
|
||||
if (fallbackStatus) {
|
||||
const custom = fallbackStatus.custom ?? {};
|
||||
return {
|
||||
...fallbackStatus,
|
||||
fallback: fallbackInfo,
|
||||
custom: {
|
||||
...custom,
|
||||
fallback: { disabled: true, reason: this.lastError ?? "unknown" },
|
||||
},
|
||||
};
|
||||
}
|
||||
const primaryStatus = this.deps.primary.status();
|
||||
const custom = primaryStatus.custom ?? {};
|
||||
return {
|
||||
...primaryStatus,
|
||||
fallback: fallbackInfo,
|
||||
custom: {
|
||||
...custom,
|
||||
fallback: { disabled: true, reason: this.lastError ?? "unknown" },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async sync(params?: {
|
||||
reason?: string;
|
||||
force?: boolean;
|
||||
progress?: (update: MemorySyncProgressUpdate) => void;
|
||||
}) {
|
||||
if (!this.primaryFailed) {
|
||||
await this.deps.primary.sync?.(params);
|
||||
return;
|
||||
}
|
||||
const fallback = await this.ensureFallback();
|
||||
await fallback?.sync?.(params);
|
||||
}
|
||||
|
||||
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
|
||||
if (!this.primaryFailed) {
|
||||
return await this.deps.primary.probeEmbeddingAvailability();
|
||||
}
|
||||
const fallback = await this.ensureFallback();
|
||||
if (fallback) {
|
||||
return await fallback.probeEmbeddingAvailability();
|
||||
}
|
||||
return { ok: false, error: this.lastError ?? "memory embeddings unavailable" };
|
||||
}
|
||||
|
||||
async probeVectorAvailability() {
|
||||
if (!this.primaryFailed) {
|
||||
return await this.deps.primary.probeVectorAvailability();
|
||||
}
|
||||
const fallback = await this.ensureFallback();
|
||||
return (await fallback?.probeVectorAvailability()) ?? false;
|
||||
}
|
||||
|
||||
async close() {
|
||||
await this.deps.primary.close?.();
|
||||
await this.fallback?.close?.();
|
||||
this.onClose?.();
|
||||
}
|
||||
|
||||
private async ensureFallback(): Promise<MemorySearchManager | null> {
|
||||
if (this.fallback) return this.fallback;
|
||||
const fallback = await this.deps.fallbackFactory();
|
||||
if (!fallback) {
|
||||
log.warn("memory fallback requested but builtin index is unavailable");
|
||||
return null;
|
||||
}
|
||||
this.fallback = fallback;
|
||||
return this.fallback;
|
||||
}
|
||||
}
|
||||
|
||||
function buildQmdCacheKey(agentId: string, config: ResolvedQmdConfig): string {
|
||||
return `${agentId}:${JSON.stringify(config)}`;
|
||||
}
|
||||
@@ -3,6 +3,7 @@ import path from "node:path";
|
||||
|
||||
import { resolveSessionTranscriptsDirForAgent } from "../config/sessions/paths.js";
|
||||
import { createSubsystemLogger } from "../logging/subsystem.js";
|
||||
import { redactSensitiveText } from "../logging/redact.js";
|
||||
import { hashText } from "./internal.js";
|
||||
|
||||
const log = createSubsystemLogger("memory");
|
||||
@@ -87,8 +88,9 @@ export async function buildSessionEntry(absPath: string): Promise<SessionFileEnt
|
||||
if (message.role !== "user" && message.role !== "assistant") continue;
|
||||
const text = extractSessionText(message.content);
|
||||
if (!text) continue;
|
||||
const safe = redactSensitiveText(text, { mode: "tools" });
|
||||
const label = message.role === "user" ? "User" : "Assistant";
|
||||
collected.push(`${label}: ${text}`);
|
||||
collected.push(`${label}: ${safe}`);
|
||||
}
|
||||
const content = collected.join("\n");
|
||||
return {
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
export type MemorySource = "memory" | "sessions";
|
||||
|
||||
export type MemorySearchResult = {
|
||||
path: string;
|
||||
startLine: number;
|
||||
endLine: number;
|
||||
score: number;
|
||||
snippet: string;
|
||||
source: MemorySource;
|
||||
citation?: string;
|
||||
};
|
||||
|
||||
export type MemoryEmbeddingProbeResult = {
|
||||
ok: boolean;
|
||||
error?: string;
|
||||
};
|
||||
|
||||
export type MemorySyncProgressUpdate = {
|
||||
completed: number;
|
||||
total: number;
|
||||
label?: string;
|
||||
};
|
||||
|
||||
export type MemoryProviderStatus = {
|
||||
backend: "builtin" | "qmd";
|
||||
provider: string;
|
||||
model?: string;
|
||||
requestedProvider?: string;
|
||||
files?: number;
|
||||
chunks?: number;
|
||||
dirty?: boolean;
|
||||
workspaceDir?: string;
|
||||
dbPath?: string;
|
||||
sources?: MemorySource[];
|
||||
sourceCounts?: Array<{ source: MemorySource; files: number; chunks: number }>;
|
||||
cache?: { enabled: boolean; entries?: number; maxEntries?: number };
|
||||
fts?: { enabled: boolean; available: boolean; error?: string };
|
||||
fallback?: { from: string; reason?: string };
|
||||
vector?: {
|
||||
enabled: boolean;
|
||||
available?: boolean;
|
||||
extensionPath?: string;
|
||||
loadError?: string;
|
||||
dims?: number;
|
||||
};
|
||||
batch?: {
|
||||
enabled: boolean;
|
||||
failures: number;
|
||||
limit: number;
|
||||
wait: boolean;
|
||||
concurrency: number;
|
||||
pollIntervalMs: number;
|
||||
timeoutMs: number;
|
||||
lastError?: string;
|
||||
lastProvider?: string;
|
||||
};
|
||||
custom?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
export interface MemorySearchManager {
|
||||
search(
|
||||
query: string,
|
||||
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
|
||||
): Promise<MemorySearchResult[]>;
|
||||
readFile(params: {
|
||||
relPath: string;
|
||||
from?: number;
|
||||
lines?: number;
|
||||
}): Promise<{ text: string; path: string }>;
|
||||
status(): MemoryProviderStatus;
|
||||
sync?(params?: {
|
||||
reason?: string;
|
||||
force?: boolean;
|
||||
progress?: (update: MemorySyncProgressUpdate) => void;
|
||||
}): Promise<void>;
|
||||
probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult>;
|
||||
probeVectorAvailability(): Promise<boolean>;
|
||||
close?(): Promise<void>;
|
||||
}
|
||||
Reference in new issue
Block a user