Memory System

N.E.K.O.'s memory is a per-character pipeline, not a single vector database. It keeps a bounded working context for the next conversation, a chronological record of turns, extracted facts, higher-level reflections, and durable persona knowledge. These layers have different write paths, retention rules, and prompt-injection behavior.

This page describes the current runtime contract. Thresholds, model tiers, and maintenance intervals are tuning details defined in config/memory_settings.py; they may change without altering the architecture described here.

Conceptual model

Layer	Purpose	Primary representation	Included in a new-dialog context?
Working context	The memory text prepared for the next LLM session	Generated by GET /new_dialog/{lanlan_name}	This is the resulting context, not a separate database
Recent memory	A bounded mixture of recent turns and an in-place summary memo	recent.json	Yes
Facts	Atomic observations extracted from conversations and tagged by subject, importance, and event time	facts.json	No; used for reflection synthesis and explicit recall
Reflections	Higher-level interpretations synthesized from groups of facts	reflections.json	Pending and active confirmed reflections are considered for the rendered context
Persona	Durable knowledge about the master, the character, and their relationship	persona.json	Yes, subject to rendering budgets and suppression rules

The raw chronological store, time_indexed.db, supports fact extraction, temporal scans, and conversation-gap calculations. It is a backing journal rather than another prompt layer: raw historical turns are not automatically copied into every LLM prompt.

All of these stores are scoped by character (lanlan_name). The default layout is under the configured memory root, but per-character recent-history and SQLite paths can be overridden by configuration.

Write path: conversation to long-term memory

1. Persist the turn

The main server sends completed conversation data to the internal memory server on port 48912. Depending on the session lifecycle, the connector uses POST /cache/{lanlan_name}, POST /process/{lanlan_name}, POST /renew/{lanlan_name}, or POST /settle/{lanlan_name}.

The foreground persistence path does two important writes:

• append the turn to the character's recent history;
• append the original messages, with a timestamp and conversation ID, to the time_indexed_original table in time_indexed.db.

It also records a replayable post-turn operation in outbox.ndjson. Expensive extraction and review work is scheduled in the background so an LLM failure does not need to block the active conversation.

2. Compress recent memory in place

memory/recent.py::CompressedRecentHistoryManager keeps recent context bounded. When the recent history exceeds its configured threshold, older entries are summarized and replaced by a SystemMessage memo at the front of recent.json; the newest turns remain verbatim. Oversized inputs use a bounded map-reduce path, and a final hard cap prevents an indefinitely failing summarizer from growing the prompt without limit.

The summary model is selected from the configured summary model tier. The result remains in recent memory.

Legacy compressed table

SQLite may still contain a time_indexed_compressed table for schema and migration compatibility, but the current runtime does not write new summaries to it. Facts and reflections replaced that old long-term abstraction path. Do not build new code against retrieve_summary_by_timeframe() or treat the compressed table as an active store.

3. Extract and deduplicate facts

Fact extraction converts conversation windows into atomic records such as a user preference, an event, or a relationship observation. The extractor preserves low-importance facts for auditability; consumers decide which importance levels they need.

New facts are deduplicated before persistence. Exact/hash and text-search checks are always available. When local embeddings are available, a background worker can identify likely paraphrases for later LLM arbitration. Facts already consumed by a reflection are marked absorbed; old absorbed facts move from facts.json to facts_archive.json rather than being discarded.

Fact scheduling depends on the powerful-memory switch described below. In either mode, persisted facts remain available to reflection synthesis and explicit recall.

4. Synthesize reflections

memory/reflection/ periodically groups enough unabsorbed facts into a higher-level reflection. The reflection ID is derived from its source fact IDs, so retrying the same batch is idempotent. A successful synthesis writes a pending reflection and marks its source facts as absorbed.

Reflections carry their own evidence, entity, lifecycle status, and optional event-time ontology. A reflection can remain tentative, become confirmed, be promoted or merged into persona, be denied, or eventually be archived. Reflection synthesis runs as a memory-server background service and does not depend on the memory-browser page or proactive-chat frontend being open.

5. Promote durable knowledge into persona

Persona entries are grouped by entity. The built-in entities are master, neko, and relationship, while the storage format permits additional entity sections. Character-card-derived entries are marked as protected and are not removed by evidence decay or archive sweeps.

With powerful memory enabled, promotion is evidence-driven. A confirmed reflection that reaches the promotion threshold goes through a correction-tier merge decision: the result may become a new persona entry, merge into existing persona knowledge, be rejected, or queue a contradiction for later resolution. With powerful memory disabled, a configurable age-based fallback confirms and promotes reflections without the merge LLM.

Recall subsystem and automatic context

Automatic context for a new dialog

GET /new_dialog/{lanlan_name} constructs the memory section used for a new LLM session. It waits for any in-progress settle operation, then renders:

1. protected and active persona entries, within the persona token budget;
2. pending and active confirmed reflections, within the reflection token budget;
3. the recent-history memo and recent raw turns;
4. current-time and long-conversation-gap hints used by the dialog prompt.

The renderer separates tentative and confirmed reflections, moves temporally stale confirmed reflections into a past-memory section, and marks repeatedly mentioned persona items as temporarily not to be raised proactively.

This path does not run a semantic search over all facts or raw conversations. It also does not dump the complete fact store into the prompt.

User-facing hybrid recall

memory/hybrid_recall.py is the user-facing retrieval backend. The main conversation path registers the built-in recall_memory tool in main_logic/core/tool_calling.py; its handler calls POST /query_memory/{lanlan_name}. The QQ auto-reply plugin also calls that endpoint directly with the incoming message and injects at most five rendered hits into its reply context.

The conversation model may supply a natural-language query, a time expression, or both:

Tool arguments	Retrieval behavior
query only	BM25 over active facts, active reflections, and archived facts, plus optional cosine retrieval over active facts and reflections; fuse both ranked lists with reciprocal-rank fusion
time only	Facts, reflections, and archived facts nearest to the parsed event-time window
query and time	Hard-filter by the event-time window, then run semantic retrieval inside that window

For this user-facing pool, persona is intentionally excluded because it is already rendered into ordinary dialog context. The shared hard filter removes negative-score, suppressed, malformed, and terminal reflection entries. The query path takes up to four BM25 and four cosine hits by default, fuses them with RRF, and returns at most eight unique results. It does not add another LLM reranking request.

The HTTP endpoint returns structured rows, but the main conversation handler renders them as localized Markdown bullets before sending tool output to the model. Memory text is not translated; each bullet includes its tier/entity and, when present, the event date and relative age. If the memory-server request fails, the handler returns an empty-result message so the conversation can continue.

Internal maintenance relevance recall

memory/recall.py::MemoryRecallReranker is a separate background subsystem. It does not answer the recall_memory tool:

Consumer	Candidate pool and query	Ranking and fallback
Stage-2 evidence-signal detection in memory/facts.py	Confirmed/promoted reflections plus non-protected persona; query texts are newly extracted facts	Shared hard filter, vector coarse rank to three times the budget, then an optional LLM rerank to the 30-entry observation budget. Without vectors, it keeps hard-filtered entries in evidence-score order and does not call the reranker LLM.
Reflection synthesis in memory/reflection/synthesis.py	Absorbed facts queried separately by each unabsorbed fact	Per-query cosine top 3, then round-robin union/dedup with a total cap of 20. Without ready, valid embeddings it returns no related-context anchors rather than injecting unrelated high-score facts.

The distinction matters: persona is excluded only from user-facing hybrid recall, while non-protected persona is deliberately part of Stage-2 maintenance recall.

The shared embedding service in memory/embeddings.py is optional and local, using the CPU ONNX execution provider. Its failure modes differ by caller: user-facing recall becomes BM25-only, Stage-2 maintenance recall falls back to evidence-score order, and reflection-anchor recall becomes an empty related-context block.

NEKO_DISABLE_BUILTIN_TOOLS=1 removes the built-in schema from main conversation sessions for diagnostics. It does not disable POST /query_memory, QQ auto-reply recall, or the internal maintenance reranker.

Evidence and reflection lifecycle

Evidence is stored as independent reinforcement and disputation values. Their effective values decay at read time using separate clocks, and the net score determines whether an entry is still tentative, confirmed, eligible for promotion, or an archive candidate. Thresholds and half-lives are centralized in config/memory_settings.py.

Evidence can come from several paths:

• a newly extracted fact reinforces or negates an existing reflection or persona entry;
• the user confirms, denies, or ignores a reflection that the character surfaced;
• a negative-keyword hit is checked against candidate memories before applying a rebuttal signal;
• a periodic rebuttal scan checks new user messages against active confirmed reflections.

Direct user feedback has a different weight from inferred fact-to-memory relationships. Promotion and archival decisions therefore use the accumulated evidence rather than treating any single LLM extraction as immediately authoritative.

The lifecycle is approximately:

facts
  -> pending reflection
  -> confirmed reflection
  -> promoted or merged persona knowledge

negative evidence
  -> denied / below-zero entry
  -> archive candidate
  -> sharded archive after the configured persistence period

Mention suppression is separate from evidence. It temporarily prevents the character from proactively repeating the same memory too often; it does not delete the underlying entry.

Powerful-memory modes

The powerful_memory_enabled field in core_config.json is hot-read by the memory server and exposed through GET and POST /api/memory/powerful_memory_config on the main server. Missing configuration defaults to enabled.

Capability	Enabled	Disabled
Fact extraction	Batched Stage 1 extraction after enough user turns or an idle trigger; Stage 2 maps new facts to evidence signals	Per-turn Stage 1 fallback keeps basic fact accumulation working; Stage 2 is paused
AI-disclosure facts	Periodic AI-aware extraction from user-engaged windows	Paused
Reflection synthesis	Runs	Runs
Feedback on surfaced reflections	Runs	Runs
Periodic rebuttal and negative-keyword target checks	Runs	Paused
Promotion	Evidence score plus merge decision	Configurable age-based confirmation and promotion, without merge LLM
Persona contradictions, vector-assisted fact dedup, persona/reflection refinement	Runs when work and dependencies are available	Queued or paused until re-enabled
Recent compression and recent-history review	Runs; review has its own recent_memory_auto_review switch	Runs; review has its own switch
Explicit recall, archive sweep, migrations, schema recheck	Available independently of this switch	Available independently of this switch

Switching from enabled to disabled resets the age anchor of currently confirmed reflections before saving the new setting, preventing old entries from being promoted immediately by the time-driven fallback.

Background maintenance

The memory-server startup hook scans pending outbox records and spawns replay tasks, then begins event reconciliation and one-shot schema/archive migrations. The current runtime does not await the spawned outbox tasks before those later steps, so outbox handlers can overlap reconciliation, migrations, and the earliest staggered-loop work. Do not rely on a strict recovery order or assume an outbox side effect is visible before reconciliation. The long-running jobs include:

• batched fact and evidence extraction;
• confirmed-reflection rebuttal checks;
• score-driven or time-driven auto-promotion;
• recent compression, recent-history review, fact dedup, and contradiction resolution during idle periods;
• reflection synthesis;
• persona and reflection refinement;
• below-zero evidence tracking and archive sweeps;
• slow migration of legacy fact/reflection event-time schemas;
• optional embedding warmup and backfill.

Exact intervals and batch sizes are operational tuning, not public API guarantees.

Storage and recovery

The default per-character directory contains the following durable data. Some files or directories are created lazily only when that feature first has work to persist.

Path under memory/<character>/	Role
recent.json	Recent turns plus the current summary memo
recent_meta.json	Recent-summary staleness metadata
time_indexed.db	Raw chronological conversation rows in time_indexed_original; the legacy compressed table may also exist
facts.json, facts_archive.json	Active facts and older absorbed facts
reflections.json, surfaced.json	Active reflections and user-feedback/surfacing state
reflection_archive/	Sharded archived reflections
persona.json, persona_corrections.json	Persona view and queued contradiction decisions
persona_archive/	Sharded archived persona entries
cursors.json	Durable progress positions for periodic scans
outbox.ndjson	Pending/done records for replayable background operations
events.ndjson, events_applied.json	Ordered state-transition journal and reconciliation sentinel

JSON view files remain editable state, so this is not full event sourcing. For event-backed transitions, the runtime appends the event before updating the view; startup reconciliation can replay an event whose view write did not complete. Outbox handlers use at-least-once delivery and must be idempotent.

Writes are atomic where a full JSON view is replaced, and per-character locks serialize conflicting mutations. Background LLM failures use bounded retries/backoff and progress markers; a failed maintenance task should degrade or retry without blocking normal chat.

Privacy and failure behavior

• Memory data is stored locally by default, but memory processing is not necessarily local. Summary, extraction, reflection, promotion, review, and correction tasks use the configured model providers. Relevant conversation or memory text is sent to those providers when those tasks run.
• Explicit recall results are returned to the active conversation model, so the selected chat provider receives those recalled snippets as tool output.
• The main conversation tool handler's normal INFO recall logs contain metadata such as mode, hit count, and elapsed time, not the raw query or recalled text. Diagnostic DEBUG logging can contain the raw query and tool arguments.
• Vector inference is local and optional. Losing vector support does not disable facts, reflections, persona, BM25 recall, or time recall.
• A corrupt or unavailable optional archive degrades to the active store. A recall error returns no hits. A failed summary leaves the uncompressed recent history available and is retried or bounded by the hard cap.
• During storage-location selection, migration, or recovery, the memory server can enter a limited mode and return 409 for memory operations until storage is safe to use.

Review UI and interfaces

Open http://localhost:48911/memory_browser to browse and edit recent conversation memory and to configure recent-memory review and powerful memory. The current browser API reads and writes recent.json; it is not a general editor for facts.json, reflections.json, or persona.json.

For the user-facing main-server routes, see the Memory REST API. For service-to-service details, see the Memory Server API. Internal memory-server paths such as /cache, /process, /settle, /new_dialog, and /query_memory are implementation interfaces between N.E.K.O. processes, not externally supported public endpoints.

For the user-level boundary between local storage and provider processing, read Where does N.E.K.O send conversations and memory? and Can N.E.K.O run completely offline?.