Hybrid Search

engrava’s search_hybrid() combines up to five scoring signals into a single ranked result list.

Signal Model

Default weights sum to 1.0. When a signal is unavailable (e.g. neither recency reference nor a cycle provider can resolve recency, no embeddings → vector skipped), its weight is redistributed proportionally across active signals.

Keyword query syntax (FTS)

The keyword signal — and the search_fts() method that exposes it directly — runs your text against an SQLite FTS5 index. engrava normalises the query before handing it to FTS5, with two modes that switch automatically on what you type:

Bare queries are matched with OR. A plain natural-language query like what was my sister doing is treated as a bag of words joined with OR, so a document matches when it shares any word. BM25’s IDF weighting then ranks the documents that share the most distinctive words first, so common function words (what, was, my) carry little weight and need no stopword list or stemmer — this works in any language.

# Bare query -> OR-matched: finds docs sharing any content word, best-ranked first
hits = await store.search_fts("what was my sister doing", top_k=10)

Expert syntax is preserved unchanged. If your query uses FTS5 operators, it is passed through as written:

• quoted phrases — "machine learning" matches the exact phrase;
• uppercase booleans — AND, OR, NOT (must be uppercase) compose terms, e.g. python AND NOT snake;
• prefix — a trailing * does prefix matching, e.g. neur*;
• column filters — essence: and content: restrict a term to that column, e.g. content:berlin.

Punctuation never raises. Unsafe characters split a token into separate terms rather than breaking the query: a contraction like sister's becomes sister OR s, so it still matches a stored sister's dog. Pasting a URL or a timestamp is safe too — only the real essence: / content: column filters are honoured, so http://example.com and 12:30 are treated as ordinary search terms. When a normalised full-text expression is a genuinely malformed FTS5 query, engrava logs a warning, increments the read-only fts_match_failure_count counter, and retries once through the bare normalisation (unsafe characters dropped, wildcards collapsed to legal prefixes, any exposed AND/OR/NOT phrase-quoted so FTS5 cannot read it as an operator), which is always a valid MATCH; the FTS arm returns that query’s matches — an empty set when the sanitised query matches nothing.

Graceful Degradation

• FTS5 unavailable or empty query → FTS skipped.
• query_vector is None and no embedding provider → vector skipped.
• No explicit current_cycle, no configured cycle provider, and no recency_now → recency skipped.
• priority_weight is 0.0 → priority skipped.
• graph_weight is 0.0 → graph skipped, zero overhead.
• All signals disabled → a query-less window of up to top_k rows ordered by updated_cycle, honouring the same filters and archived exclusion as the arms. With no cycle reference to decay against every row scores a flat 0.0; a current_cycle supplied at recency_weight=0.0 still yields cycle-decayed scores, though the order is the same either way because the window is already ordered by updated_cycle and the decay rises with it. When only FTS and vector are unusable, that same path pre-orders the window by whichever recency axis is still active.

Two recency axes

Recency ranks along two separately-typed axes, and a query picks exactly one. Passing neither explicit reference uses a configured cycle provider when one is present; without one, recency stays off.

Both use the same exponential half-life decay; only the clock differs.

Why the second axis exists. A consumer with no cadence writes everything at cycle 0, where the cognitive-cycle axis is degenerate: every row has the same age, so recency ranks nothing. The transaction-time axis makes “recently written” rankable on exactly that store — the common case for externally-written memories.

# Rank by how recently each memory was written, relative to a caller-supplied
# "now", with a 24-hour freshness half-life.
from datetime import UTC, datetime

result = await store.search_hybrid(
    "incident timeline",
    recency_now=datetime.now(UTC).isoformat(),   # the caller owns "now"
    recency_weight=0.4,
    recency_now_half_life=86400,                 # seconds; default 604800 (7 days)
)

The rules worth knowing before you wire it:

• The caller owns “now”. recency_now is a caller-supplied ISO-8601 instant; engrava’s core reads no wall clock when ranking, so retrieval stays deterministic and replayable — the same store plus the same recency_now yields the same ranking.
• Naive values are read as UTC. The host timezone is never consulted.
• Malformed input raises — once the axis is on. With a recency_now given, a malformed instant, or a non-positive recency_now_half_life, raises InvalidRecencyArgumentError. Both checks sit behind recency_now, so a recency_now_half_life passed without one is not validated and not applied: it is silently ignored, along with the axis it was meant to tune.
• A row with no usable timestamp scores minimum. If its updated_at / created_at is missing or malformed it is treated as maximally old rather than dropped.
• Explicit beats passive. An explicit recency_now takes precedence over a configured cycle_provider — the provider is not consulted. Supplying both explicit references, current_cycle and recency_now, raises RecencyModeConflictError.
• The units never mix. A wall-clock age is never subtracted from a cognitive cycle. To use transaction-time recency on a store that has a cycle provider, pass recency_now and omit an explicit current_cycle.

Archived thoughts leave default retrieval

Thoughts whose lifecycle_status is ARCHIVED are excluded from the default retrieval candidate set — on search_hybrid(), recall(), search_fts(), search_similar(), and the query-less fallback alike. This is an eligibility filter in the same class as the existing expired-row filter, not a scoring change: a thought you archived stops surfacing without being deleted.

# Search the archive for this call only — the rows stay ARCHIVED.
result = await store.search_hybrid("old incident", include_archived=True)

# Or bring one back for good.
restored = await store.restore_thought(thought_id)

Two boundaries are worth knowing before you rely on it:

• It is not a global filter. list_thoughts() and count_thoughts() stay lifecycle-neutral — they still return archived rows unless you filter them explicitly with lifecycle_status=.
• It does not cover retired reflections. A REFLECTION retired because its whole source cluster left ACTIVE is held out by a separate freshness rule, so it stays excluded even under include_archived=True.

Graph-Aware Ranking

The graph signal uses 1-hop-weighted neighbour boost. If a candidate thought’s graph neighbours also match the query, the candidate receives a boost proportional to the neighbour’s semantic score and the connecting edge weight.

Algorithm

For each candidate C in the fusion pool:
  neighbours = get_edges(C, direction="BOTH")   # then cap to max_neighbors
               ordered by edge.weight DESC (deterministic)
  For each (edge, neighbour):
    neighbour_base = max(fts_score[neighbour], vector_score[neighbour])
    boost[C] += edge.weight * neighbour_base * graph_edge_decay
final_score[C] += graph_weight * boost[C]

Key properties:

• Only semantic scores propagate — priority, recency, and graph scores are excluded from neighbour_base to prevent hub-cascade effects.
• No new candidates — graph signal re-ranks existing results; it does not add thoughts to the result set.
• Deterministic — neighbours are sorted by edge.weight DESC before the max_neighbors cap is applied.

Configuration

search:
  default_graph_weight: 0.0          # opt-in (0.0 = disabled)
  graph_edge_decay: 0.5              # decay factor for 1-hop distance
  max_neighbors_per_candidate: 5     # safety cap

Per-query override:

result = await store.search_hybrid(
    "python async",
    graph_weight=0.1,
    graph_edge_decay=0.3,
)

Performance

When graph_weight=0.0 (default), no graph queries are executed and there is zero performance impact. When active, the implementation issues a small number of batched SQL queries (one per ~450 candidates) and caps each candidate to max_neighbors_per_candidate neighbours.

The cap bounds the boost loop, not the work before it. Each candidate’s full adjacency list is sorted by edge weight and only then truncated, so per-candidate work is O(d log d) in that candidate’s edge degree d, with the boost accumulation itself O(max_neighbors_per_candidate). The cap is a relevance control and a bound on scoring; it is not what keeps a heavily-connected hub cheap to rank.

Observability

When the graph signal contributes to at least one candidate, "graph" appears in HybridSearchResult.backends_used.

Per-Query Overrides

All weights can be overridden per call via keyword arguments:

result = await store.search_hybrid(
    "quantum computing",
    query_vector=embedding,
    fts_weight=0.4,
    vector_weight=0.4,
    recency_weight=0.1,
    priority_weight=0.05,
    graph_weight=0.05,
    current_cycle=42,
)

Configuration Reference

See Configuration for the full YAML reference of SearchConfig fields.

Querying Reflections

After DreamingExtension.run_consolidation() runs its clustering phase, ThoughtType.REFLECTION meta-thoughts exist in the store. Four controls govern how hybrid search handles them — and one of the four is on by default, so read reflection_topk_cap below before tuning the other three.

include_reflections (default True)

When False, REFLECTION thoughts are excluded from search_hybrid() results. Useful when you want raw observations / insights without higher-order aggregates:

result = await store.search_hybrid(
    "machine learning",
    query_vector=embedding,
    include_reflections=False,
)

reflection_boost (default SearchConfig.reflection_boost = 1.0)

When REFLECTIONs are included, their final score is multiplied by this factor. The default 1.0 leaves REFLECTIONs on equal footing with regular thoughts while scoring; raise it above 1.0 to give high-level abstractions a modest upranking so they surface for broad queries without dominating narrow ones. Equal scoring is not the last word on the result window: reflection_topk_cap runs afterwards and can evict a REFLECTION that scored its way in.

# Stronger boost -- reflections rank near the top for broad queries
result = await store.search_hybrid(
    "patterns in memory",
    query_vector=embedding,
    reflection_boost=1.5,
)

# Disable boost -- reflections compete on equal footing
result = await store.search_hybrid(
    "specific fact",
    query_vector=embedding,
    reflection_boost=1.0,
)

Configure the default in YAML:

search:
  reflection_boost: 1.0   # applies when reflection_boost not overridden per-call

reflection_topk_cap (default SearchConfig.reflection_topk_cap = 0.3)

This one is active unless you turn it off. It bounds the fraction of the final top-K window REFLECTIONs may occupy, and it is applied after fusion, boosting and de-fragmentation — so it reshapes a window the scores had already decided.

At the default 0.3 with top_k=10, REFLECTIONs get at most int(10 * 0.3) = 3 slots. If more than that survive into the window, the lowest-scoring excess are removed and the freed slots are backfilled with off-list non-REFLECTION candidates from just below the cut. The eviction is unconditional: if there are not enough off-list candidates to backfill with, the excess REFLECTIONs still go, the window comes back shorter, and engrava logs a warning.

search:
  reflection_topk_cap: 0.3   # fraction of top_k; 1.0 disables the cap

Unlike include_reflections and reflection_boost, this is configuration only — search_hybrid() takes no per-call override, so a store built without a SearchConfig runs on the 0.3 default. Set it to 1.0 to let REFLECTIONs hold as many slots as they score into.

Every query reports what the cap did:

result = await store.search_hybrid("patterns in memory", query_vector=embedding)
result.reflections_evicted   # 0 => the ranked results are exactly what scoring produced

search_reflections_only()

Convenience helper that returns only REFLECTION thoughts, scored by cosine similarity to the query vector (plus optional recency blend). Designed for queries like “what themes exist in my memory?”:

result = await store.search_reflections_only(
    "recurring ideas about learning",
    query_vector=embedding,
    top_k=5,
    current_cycle=42,   # optional recency blend
)
for thought_id, score in result.results:
    ref = await store.get_thought(thought_id)
    print(ref.content)  # JSON with member_ids + keywords

Key difference from search_hybrid(include_reflections=True): search_reflections_only() fetches all REFLECTIONs directly from the store (no pagination gap) and scores them purely by cosine similarity to the query. It does not compete against regular thoughts for result slots.