Dreaming — Memory Consolidation

Engrava’s dreaming extension provides periodic memory consolidation: it evaluates stored thoughts, scores them against configurable signals, and promotes the most important ones by setting their priority to P1.

Dreaming runs outside the normal CRUD path — the consumer decides when to invoke run_consolidation() (after N cycles, in a cron job, or manually).

Quick Start

from engrava.config import DreamingConfig, DreamingGates
from engrava.extensions.dreaming import DreamingExtension

config = DreamingConfig(
    enabled=True,
    promote_threshold=0.55,
    gates=DreamingGates(
        allow_zero_confirmation=True,
        min_age_cycles=1,
    ),
)
ext = DreamingExtension(config=config)

# After ingesting thoughts into `store`:
result = await ext.run_consolidation(store, current_cycle=1)
print(f"Promoted {result.promoted_count} thoughts")

Gates

Before a thought is scored against the promotion threshold, it must pass all active gates. Gates are cheap boolean checks that filter out clearly ineligible candidates.

The candidate pool itself is bounded: promotion scores the first candidates_limit (default 200) ACTIVE thoughts, not the whole store.

Promotion is capped twice

max_promoted_per_run is not the only ceiling. Two fields on DreamingConfig (not DreamingGates) constrain promotion further — one a population-level cap no individual thought can pass, the other a per-thought type filter:

Either cap can bind first. The slot budget has a floor of one, so a store under 20 thoughts can end up above the fraction; and the cap only withholds new promotions, never demotes, so a store already over it stays over.

Read result.promotion_capped narrowly: it is set when a thought scoring above the threshold finds no free slot, which happens before its type is checked against promote_targets. So it means the cap stopped a high-scoring thought, not that the cap is why a promotable thought went unpromoted.

allow_zero_confirmation

When True (default), the confirmation gate is skipped entirely. This is essential for single-write batch-ingest scenarios where thoughts are stored once and never confirmed — without this flag, no thought would ever pass the confirmation gate and dreaming would be effectively dead.

Set to False only when your application explicitly tracks confirmations and you want to require at least min_confirmations experience-based validations before a thought is eligible for promotion.

Signals

Signals compute a score in [0.0, 1.0] for each candidate thought. The score compared against promote_threshold is a weighted average over the signals active for that run, not a weighted sum over all six.

The six default weights deliberately sum to more than 1.0. Before scoring, every default signal is tested once for activeness; an inactive signal is dropped and its weight redistributed over the rest, which are renormalised to sum to 1.0. What activeness means differs per signal:

So action_outcome is inactive in a store that records no action outcomes, and the other five renormalise exactly as they did before the signal existed. But note what the table does not say: no signal asks whether its values vary. What switches a signal off is an absent data source — no confirmations, no accesses, no outcome scores — not a constant one: a pool where every candidate carries the same non-default value keeps the signal active. The two cycle-based signals turn on the run’s arguments rather than on the pool at all.

The consequence for tuning: the configured map is a set of relative priorities, not a probability distribution. Weights hand-set to sum to 1.0 are still renormalised over the active subset, so the weighting that actually ran is result.active_signal_weights, and the dropped names are in result.flat_signals. Activeness is decided once per run over the whole pool, never per thought; custom signals have no introspectable data source and count as always active. When no signal is active every score is 0.0 and nothing is promoted.

Custom signals can be provided via DreamingSignalProtocol:

class MySignal:
    def __call__(self, thought: ThoughtRecord, ctx: DreamingContext) -> float:
        return 0.42

ext = DreamingExtension(
    config=config,
    custom_signals={"my_signal": MySignal()},
)

Priority Signal in Search

After dreaming promotes thoughts to P1, the hybrid search search_hybrid() can use priority as a 4th scoring signal alongside FTS5, vector similarity, and recency.

The priority signal maps each thought’s Priority enum to a boost multiplier:

The default priority weight is 0.05 (5% of the total score). Configure it via SearchConfig:

search:
  default_priority_weight: 0.05
  priority_boost_p1: 1.0
  priority_boost_p2: 0.6
  priority_boost_p3: 0.3
  priority_boost_p4: 0.0

To disable the priority signal entirely, set default_priority_weight: 0.0.

Edge Creation

When dreaming promotes a thought to P1, it can also create ASSOCIATED edges connecting the promoted thought to its nearest neighbours by embedding similarity. This persists the dream’s structural knowledge in the graph so it survives application restarts.

Edges are created with source=KnowledgeSource.DREAMING for attribution.

Configuration

extensions:
  dreaming:
    edges:
      enabled: true          # create edges on promotion (default: true)
      top_k: 1               # max neighbours per promoted thought
      min_similarity: 0.7    # cosine threshold for edge creation
      edge_weight_factor: 0.5  # edge.weight = factor * similarity

Idempotence

Re-running run_consolidation() on the same data does not create duplicate edges. Before creating an edge, the extension checks whether the promoted thought already has any edge connecting it to the candidate neighbour (regardless of type).

Edge Weight Formula

edge.weight = edge_weight_factor * cosine_similarity

With the default edge_weight_factor=0.5 and min_similarity=0.7, practical edge weights range from 0.35 to 0.50.

Graph-Aware Search

After dream-created edges exist in the graph, search_hybrid() can use them as a 5th scoring signal (graph signal). The signal uses 1-hop-weighted neighbour boost: if a candidate thought’s graph neighbours also match the query, the candidate receives a boost proportional to the neighbour’s score and the connecting edge weight.

Algorithm

For each candidate C in the fusion pool:
  neighbours = get_edges(C, direction="BOTH")   # then cap to max_neighbors
  For each (edge, neighbour):
    # 0.0 unless the neighbour is itself in the pool
    neighbour_base = candidate_scores.get(neighbour, 0.0)
    boost[C] += edge.weight * neighbour_base * graph_edge_decay
final_score[C] += graph_weight * boost[C]

candidate_scores holds the semantic-only base score (max(fts, vector)) for each thought in the fusion pool, so an edge to a thought that did not reach the pool contributes nothing.

Configuration

search:
  default_graph_weight: 0.0     # opt-in (0.0 = disabled)
  graph_edge_decay: 0.5         # decay factor for 1-hop distance
  max_neighbors_per_candidate: 5  # safety cap

Per-query override:

result = await store.search_hybrid(
    "python async",
    graph_weight=0.1,
    graph_edge_decay=0.3,
)

The graph signal is opt-in (default_graph_weight=0.0). When the weight is 0.0, no graph queries are made and there is zero performance impact.

See Hybrid Search for the full signal model and per-query override reference.

Reflections (Meta-Consolidation)

run_consolidation() runs a third phase after promotion and edge creation: it clusters semantically related thoughts and creates ThoughtType.REFLECTION meta-thoughts that aggregate each cluster.

The phase is on by default, but several guards can end it having created nothing — none of them raises. See Why a run creates no REFLECTION below.

What is a REFLECTION?

A REFLECTION is a first-class ThoughtRecord that represents a higher-order abstraction over a cluster of related thoughts:

No language-model call is involved — content is purely structural (keyword frequency counts from member text, centroid from member vectors).

How Clustering Works

Two algorithms are available via DreamingGates.cluster_algorithm:

"lpa" (default) — Label Propagation Algorithm

• Operates over the ASSOCIATED dream-edge graph built in phase 2.
• Deterministic via seeded PRNG (seed=42 by default).
• O(E * iterations), no external dependencies.
• Works when the graph is dense enough to form communities.

"agglomerative" — cosine-similarity single-linkage

• Operates over ACTIVE thoughts whose type is listed in cluster_allowed_types (default: OBSERVATION only), bounded by candidates_limit (default 200) per listed type — so a multi-type cluster_allowed_types raises the ceiling to candidates_limit times the number of types — independent of graph edges.
• Intended for sparse-graph / first-run scenarios where LPA finds no clusters.
• Nodes whose cosine similarity >= cluster_similarity_threshold are merged via Union-Find.
• Use when you want clustering before dreams have built up a graph.

Idempotence

Before creating a REFLECTION, the extension derives a 16-hex content-hash from the sorted member IDs that survived the eligibility filter — not from the raw cluster — and checks whether any REFLECTION with source = "dreaming:<hash>" already exists. Re-running run_consolidation() on unchanged data creates zero duplicate REFLECTIONs. The filtered subset is what feeds the hash, the centroid, the content payload and the CONSOLIDATED_FROM edges, which is why the same cluster scanned under different eligibility configuration legitimately produces a new REFLECTION.

Why a run creates no REFLECTION

Clustering and REFLECTION creation are guarded at several points. None of them raises, so an empty reflections_created is the only signal at the call site.

• enable_reflections: false skips phase 3 outright.
• clustering_min_new_candidates (default 50) skips clustering entirely when the eligible ACTIVE population has not grown by at least 50 since the last run that cleared this guard — a skipped run leaves the baseline where it was. The count is held in memory, so the first run after construction always proceeds; set the field to 0 to disable it. Promotion still executes.
• The pool is empty. Agglomerative reads ACTIVE thoughts of cluster_allowed_types (default OBSERVATION), so a store of BELIEFs or NOTEs yields nothing. LPA instead reads the ASSOCIATED dream-edge graph and returns nothing while that graph is empty, unless cold_start_clustering (default false) is enabled to fall through to the agglomerative path.
• Size bounds. min_cluster_size (default 3) drops clusters that are too small; max_cluster_size (default 200) drops oversized ones whole, leaving their members ungrouped.
• Per-member eligibility. The metadata filter runs again on the resolved members and min_cluster_size is re-checked against the filtered subset. Only excluded_content_types (default [code]) filters anything under the default configuration, and only for thoughts that carry structured metadata.
• cluster_quality_gating_enabled (default true) rejects a cluster on the first failed content check: duplicate member content, persona-only share, contradictory members, low cohesion, mixed external source, and inconsistent named entities. A separately switchable check on generic keyphrases runs once the content payload is built.
• No member embeddings. The centroid is the mean of member vectors, so a cluster whose members have none is skipped.

Configuration

extensions:
  dreaming:
    gates:
      min_cluster_size: 3             # min members for a reflection to be created
      cluster_similarity_threshold: 0.7  # cosine threshold (agglomerative only)
      cluster_algorithm: lpa          # "lpa" or "agglomerative"
      enable_reflections: true        # set to false to skip phase 3 entirely
      cold_start_clustering: false    # opt-in: LPA falls back to agglomerative
                                      #   clustering when the edge graph is empty
      cluster_allowed_types: [OBSERVATION]
      clustering_min_new_candidates: 50
      max_cluster_size: 200           # null disables the upper bound
      cluster_quality_gating_enabled: true

ConsolidationResult Fields

result = await ext.run_consolidation(store, current_cycle=42)
print(result.promoted_count)       # thoughts promoted to P1
print(result.edges_created)        # ASSOCIATED edges created
print(result.reflections_created)  # new REFLECTION thoughts created

# Diagnostics for a run that promoted less than you expected
print(result.promotion_capped)     # True if the fraction cap stopped a high-scoring thought
print(result.p1_fraction_after)    # P1 share of the corpus, measured at promotion time
print(result.active_signal_weights)  # effective weights after redistribution
print(result.flat_signals)         # signals dropped as inactive this run

Querying Reflections

See Hybrid Search — “Querying Reflections” section.

A consolidation cycle can also run memory hygiene

The store-level store.consolidate() convenience wrapper does one more thing than DreamingExtension.run_consolidation(): when a hygiene policy is configured with enabled: true and the cycle satisfies its cadence (current_cycle % check_every_n_cycles == 0), one Memory Hygiene pass runs at the end of the cycle, after promotion and the orphan-reflection sweep. That pass can archive cold, low-value thoughts, so consolidation is not purely additive on a store with hygiene enabled.

Hygiene is off by default and has to be deliberately enabled, and an explicit run_hygiene() call bypasses the cadence entirely. See Configuration → hygiene_policy for the policy fields and the safety gates on the destructive path.

Configuration Reference

See Configuration for the full YAML reference of DreamingGates, EdgeCreationConfig, and SearchConfig fields.