Dreaming — Memory Consolidation

Engrava’s dreaming extension provides periodic memory consolidation: it evaluates stored thoughts, scores them against configurable signals, and promotes the most important ones by setting their priority to P1.

Dreaming runs outside the normal CRUD path — the consumer decides when to invoke run_consolidation() (after N cycles, in a cron job, or manually).

Quick Start

from engrava.config import DreamingConfig, DreamingGates
from engrava.extensions.dreaming import DreamingExtension

config = DreamingConfig(
    enabled=True,
    promote_threshold=0.55,
    gates=DreamingGates(
        allow_zero_confirmation=True,
        min_age_cycles=1,
    ),
)
ext = DreamingExtension(config=config)

# After ingesting thoughts into `store`:
result = await ext.run_consolidation(store, current_cycle=1)
print(f"Promoted {result.promoted_count} thoughts")

Gates

Before a thought is scored against the promotion threshold, it must pass all active gates. Gates are cheap boolean checks that filter out clearly ineligible candidates.

allow_zero_confirmation

When True (default), the confirmation gate is skipped entirely. This is essential for single-write batch-ingest scenarios where thoughts are stored once and never confirmed — without this flag, no thought would ever pass the confirmation gate and dreaming would be effectively dead.

Set to False only when your application explicitly tracks confirmations and you want to require at least min_confirmations experience-based validations before a thought is eligible for promotion.

Signals

Signals compute a score in [0.0, 1.0] for each candidate thought. The weighted sum of all signal scores is compared against promote_threshold.

Custom signals can be provided via DreamingSignalProtocol:

class MySignal:
    def __call__(self, thought: ThoughtRecord, ctx: DreamingContext) -> float:
        return 0.42

ext = DreamingExtension(
    config=config,
    custom_signals={"my_signal": MySignal()},
)

Priority Signal in Search

After dreaming promotes thoughts to P1, the hybrid search search_hybrid() can use priority as a 4th scoring signal alongside FTS5, vector similarity, and recency.

The priority signal maps each thought’s Priority enum to a boost multiplier:

The default priority weight is 0.05 (5% of the total score). Configure it via SearchConfig:

search:
  default_priority_weight: 0.05
  priority_boost_p1: 1.0
  priority_boost_p2: 0.6
  priority_boost_p3: 0.3
  priority_boost_p4: 0.0

To disable the priority signal entirely, set default_priority_weight: 0.0.

Edge Creation

When dreaming promotes a thought to P1, it can also create ASSOCIATED edges connecting the promoted thought to its nearest neighbours by embedding similarity. This persists the dream’s structural knowledge in the graph so it survives application restarts.

Edges are created with source=KnowledgeSource.DREAMING for attribution.

Configuration

extensions:
  dreaming:
    edges:
      enabled: true          # create edges on promotion (default: true)
      top_k: 1               # max neighbours per promoted thought
      min_similarity: 0.7    # cosine threshold for edge creation
      edge_weight_factor: 0.5  # edge.weight = factor * similarity

Idempotence

Re-running run_consolidation() on the same data does not create duplicate edges. Before creating an edge, the extension checks whether the promoted thought already has any edge connecting it to the candidate neighbour (regardless of type).

Edge Weight Formula

edge.weight = edge_weight_factor * cosine_similarity

With the default edge_weight_factor=0.5 and min_similarity=0.7, practical edge weights range from 0.35 to 0.50.

Graph-Aware Search

After dream-created edges exist in the graph, search_hybrid() can use them as a 5th scoring signal (graph signal). The signal uses 1-hop-weighted neighbour boost: if a candidate thought’s graph neighbours also match the query, the candidate receives a boost proportional to the neighbour’s score and the connecting edge weight.

Algorithm

For each candidate C in the fusion pool:
  neighbours = get_edges(C, direction="BOTH")   # then cap to max_neighbors
  For each (edge, neighbour):
    neighbour_base = max(fts[neighbour], vector[neighbour])
    boost[C] += edge.weight * neighbour_base * graph_edge_decay
final_score[C] += graph_weight * boost[C]

Configuration

search:
  default_graph_weight: 0.0     # opt-in (0.0 = disabled)
  graph_edge_decay: 0.5         # decay factor for 1-hop distance
  max_neighbors_per_candidate: 5  # safety cap

Per-query override:

result = await store.search_hybrid(
    "python async",
    graph_weight=0.1,
    graph_edge_decay=0.3,
)

The graph signal is opt-in (default_graph_weight=0.0). When the weight is 0.0, no graph queries are made and there is zero performance impact.

See Hybrid Search for the full signal model and per-query override reference.

Reflections (Meta-Consolidation)

run_consolidation() runs a third phase after promotion and edge creation: it clusters semantically related thoughts and creates ThoughtType.REFLECTION meta-thoughts that aggregate each cluster.

What is a REFLECTION?

A REFLECTION is a first-class ThoughtRecord that represents a higher-order abstraction over a cluster of related thoughts:

No LLM is involved — content is purely structural (keyword frequency counts from member text, centroid from member vectors).

How Clustering Works

Two algorithms are available via DreamingGates.cluster_algorithm:

"lpa" (default) — Label Propagation Algorithm

• Operates over the ASSOCIATED dream-edge graph built in phase 2.
• Deterministic via seeded PRNG (seed=42 by default).
• O(E * iterations), no external dependencies.
• Works when the graph is dense enough to form communities.

"agglomerative" — cosine-similarity single-linkage

• Operates over all active ACTIVE thoughts, independent of graph edges.
• Intended for sparse-graph / first-run scenarios where LPA finds no clusters.
• Nodes whose cosine similarity >= cluster_similarity_threshold are merged via Union-Find.
• Use when you want clustering before dreams have built up a graph.

Idempotence

Before creating a REFLECTION, the extension derives a 16-hex content-hash from the sorted member IDs and checks whether any REFLECTION with source = "dreaming:<hash>" already exists. Re-running run_consolidation() on unchanged data creates zero duplicate REFLECTIONs.

Configuration

extensions:
  dreaming:
    gates:
      min_cluster_size: 3             # min members for a reflection to be created
      cluster_similarity_threshold: 0.7  # cosine threshold (agglomerative only)
      cluster_algorithm: lpa          # "lpa" or "agglomerative"
      enable_reflections: true        # set to false to skip phase 3 entirely

ConsolidationResult Fields

result = await ext.run_consolidation(store, current_cycle=42)
print(result.promoted_count)       # thoughts promoted to P1
print(result.edges_created)        # ASSOCIATED edges created
print(result.reflections_created)  # new REFLECTION thoughts created

Querying Reflections

See Hybrid Search — “Querying Reflections” section.

Configuration Reference

See Configuration for the full YAML reference of DreamingGates, EdgeCreationConfig, and SearchConfig fields.