> built 2026-10-14 21:13 UTC from 537ca94 (master) · ir 0.1.48. Details: build_info.json # index.html.md # ir `ir` is an information-retrieval substrate for agentic systems: one “find the relevant things in this corpus” contract over skills, packages, documents, sessions, MCP tools or any mapping. It retrieves candidates, commits to a small high-precision subset (or abstains), and loads each committed item’s payload only when asked. [For developers with a pulse →]() ## For agents: the agentic layer **Skills.** `ir` ships two [Agent Skills](https://agentskills.io): | Skill | Use it to | Real files | |-------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------| | `ir-search` | search a user’s `ir` corpora and review the hits for real relevance | `ir/data/skills/ir-search/` (ships with `pip install ir`) | | `ir-dev-workflow` | work on `ir` itself: module map, tests, conventions, the ir/raglab boundary | `skills/ir-dev-workflow/` (repository only) | ```bash gh skill install i2mint/ir ir-search --agent claude-code # GitHub CLI with the `skill` command npx skills add i2mint/ir --skill ir-search # or the skills CLI ``` The consumer skill also travels inside the installed package. Find it with `python -c "import ir, os; print(os.path.join(ir.__path__[0], 'data', 'skills'))"`, then link it with the [`skill`](https://github.com/thorwhalen/skill) package (`skill link-skills `) or copy the folder into `~/.claude/skills/`. In a clone of this repository, Claude Code finds both skills through the relative symlinks in `.claude/skills/` (on Windows, clone with `git config core.symlinks true` and symlink permission, or install the skills as above). **Subagent and instructions.** `.claude/agents/ir-discover-agent.md` is a reviewing search subagent: it writes several queries, reads and grades the candidates, reformulates, and returns only the verified few. `.claude/CLAUDE.md` holds the project instructions a coding agent needs, including the ir/raglab boundary. **What an agent can do with `ir`:** - Give itself one search tool over a large catalogue (skills, MCP tools, packages, documents) instead of loading every schema into context. - Commit to the few results worth acting on, or abstain when nothing fits (`min_score=`, or `min_score="auto"` with a calibrated floor). - Search several corpora in one call (`ir.discover(["skills", "packages"], query)`) and get JSON-ready results (`.to_dict()`, `ir.tools.search`). A minimal runnable example, offline and with no model download (the `light` embedder is a numpy-only hashing embedder): ```python import ir from ir.store import CorpusStore tools = [ {"name": "fetch_url", "description": "Fetch a web page over HTTP and return its text"}, {"name": "send_email", "description": "Send an email message to a recipient"}, {"name": "list_files", "description": "List the files in a directory"}, ] source = ir.CorpusSource.from_mcp_tools(tools) # embeds name + description only corpus = ir.build(source, store=CorpusStore.memory(), embedder="light") result = ir.discover(corpus, "fetch the text of a web page", mode="dense") assert [item.name for item in result.results] == ["fetch_url"] nothing = ir.discover(corpus, "bake a chocolate cake", mode="dense", min_score=0.3) assert nothing.abstained # no tool clears the floor, so the agent is told so ``` > ### Looking for a search *agent*? > **`ir` is a retrieval *substrate*, not a search agent — it has no agent of its own.** `ir.discover` is a deterministic, single-shot `retrieve → select → disclose` pipeline: it embeds your query *verbatim* (no LLM query rewriting) and commits via a score rule (no LLM reviewing the results). There is no planner and no *back-edge* (evaluate → reformulate → search again) — and that back-edge is precisely what makes a system an agent. > If you want the **agent** — a planner, an LLM formulator that rewrites queries, and the evaluator→reformulate loop over one or more `ir` corpora — reach for **[`raglab`](https://github.com/thorwhalen/raglab)**, which is **built on top of `ir`** (`raglab` imports `ir`; `ir` never imports `raglab`, so it stays installable-light and offline-by-default). > Prefer to drive it from **Claude Code** instead of a coded agent? `ir` ships a reviewing **search subagent** and an `ir-search` **skill** (see [the agentic layer]()) — a lightweight “raglab-in-instructions” that formulates queries, *reads and judges* candidates against your real intent, filters, and reformulates, all on your Claude subscription (see [`ir_10`]()). ## Quickstart ```bash pip install ir ``` ```python import ir # Define a corpus, build the index (incremental), then discover. source = ir.CorpusSource.from_skills() # your agent skills (read through `priv`) # Other sources: from_packages(), from_md_reports(), from_claude_sessions(), # from_files(root), from_mapping(docs, name=...), from_mcp_tools(tools), # from_records(fetcher="mypkg.recall:records") # records another package owns corpus = ir.build(source) # embed + persist under XDG dirs result = ir.discover(corpus, "how do I deploy the app to the server") for item in result.results: print(item.score, item.name) # the committed few (or result.abstained) print(result.to_dict()) # JSON-serializable (qh / HTTP ready) ``` ## Install `ir` is **light by default** — `numpy` + [`dol`](https://github.com/i2mint/dol) for storage, plus [`ef`](https://github.com/thorwhalen/ef) / [`vd`](https://github.com/i2mint/vd) for embedding and lexical/hybrid retrieval. Python ≥ 3.10. Notes for the default (semantic) path: - The default embedder is `all-MiniLM-L6-v2` (384-dim, via `sentence-transformers`), **downloaded on first build** (needs network) and cached under `~/.cache/ir`. For a fast, offline, dependency-light run — tests, CI, quick experiments — pass `embedder="light"` (a numpy-only hashing embedder): `ir.build(source, embedder="light")`. - `ir` sets `USE_TF=0` on import so `transformers` does not pull in TensorFlow (which crashes on some numpy ABIs); import `ir` before anything that imports `transformers`. - Case generation (`ir.eval_gen`) and the optional LLM selector need an LLM via [`aix`](https://github.com/thorwhalen/aix) — install the extra, `pip install "ir[llm]"`. Scoring and evaluation themselves stay offline. The LLM is an *injected callable* at every seam, so it can run on a Claude subscription instead of a metered key — see [`ir_10`]() (MCP sampling for connectors / the Agent SDK for Claude Code). ## Install on a fresh server A machine with only a system `python3` cannot run `ir` (no `numpy`, `dol`, `ef`, `vd`, `cw`). One command gives you an `ir` on `PATH`, in its own isolated environment: ```bash uv tool install ir # puts `ir` on PATH (~/.local/bin) ir discover packages "cache" ``` Or install into an environment you own: `uv pip install --python /bin/python ir` (a distro-managed system Python refuses this; use `uv tool install`). If you run `ir` where its dependencies are missing, it now says which ones and prints these commands, instead of a bare `ModuleNotFoundError` from `ir/base.py`. **Where the index comes from.** Nothing is shipped or synced: each machine builds its own indexes from what is on that machine. The first `ir discover` on a machine that never built one prints `corpus 'packages' is empty; build it first: ir build packages`. Run it once (`ir build skills`, `ir build packages`, …); later builds are incremental, and `ir maintain --all` (or `ir schedule`) keeps them fresh. Indexes live under `~/.local/share/ir`. **How long a first build takes.** With the light embedder (no model download), `ir build packages` over about 2,800 records took 12 s on a small server. With the default MiniLM embedder the first build also downloads the model (about 90 MB) and embeds on CPU, so budget minutes (not measured here); `uv pip install --python 'ef[sentence-transformers]'` enables it. Without that package `ir build` warns and builds with the light embedder. A corpus is pinned to the embedder it was built with (its `embedder_id`, shown by `ir info`), so installing or removing `sentence-transformers` later does not break querying. To switch embedders, re-embed with `ir build NAME`. ## The pipeline `ir` is a six-stage pipeline, each stage a small, swappable seam: | Stage | Entry point | What it does | |--------------|----------------|------------------------------------------------------------------------------------------------------------------| | **source** | `CorpusSource` | what is in the corpus + what counts as stale | | **index** | `ir.build` | decompose artifacts into embeddable *surfaces*, embed, persist (incremental, idempotent) | | **retrieve** | `ir.search` | hard metadata filter + `dense` / `lexical` / `hybrid` ranking | | **expand** | `ir.expand` | stitch a hit’s stored neighborhood (sentence window / whole artifact) into the passage downstream reads — opt-in | | **select** | `ir.select` | commit to a distractor-robust subset, or abstain | | **disclose** | `ir.disclose` | load the heavy payload (SKILL.md body, package pointer, file text) for committed items — append-only | `ir.discover` chains retrieve → select → disclose into the single agent-callable (and `qh`-exposable) tool. Pass a **list** of corpus names for single-shot *federated* discovery across several corpora: ```python ir.discover(["skills", "packages"], "deploy the app") # fan-out → fuse → select ir.discover( ["skills", "packages"], q, min_score="auto" ) # gate each source on its own floor ``` Each source is searched and gated on its **own** calibrated abstention floor *before* any merging; the survivors then rank-fuse (weighted RRF via `ir.fuse_hits`) — raw scores never cross a source boundary, because scores from different corpora / embedders / modes live on incommensurable scales. Every hit carries its corpus name as `hit.source`, so same-id artifacts from different corpora stay distinct, attributable results. The caller names the sources; `ir` never chooses the set (source planning belongs to the agent layer — see [`raglab`](https://github.com/thorwhalen/raglab)). ### Retrieve ```python hits = ir.search(corpus, "deploy app", mode="hybrid") # dense | lexical | hybrid (RRF) ``` Dense is exact brute-force cosine; `lexical` is Okapi BM25; `hybrid` fuses both by Reciprocal Rank Fusion (the strongest default for short, identifier-heavy capability text). Lexical/hybrid reuse [`vd`](https://github.com/i2mint/vd); dense needs only numpy. Hybrid has a second fusion, `fusion="blend"` — a magnitude-preserving score blend instead of rank-RRF. RRF discards score magnitude, which is exactly what abstention calibration needs, so `blend` separates in-scope from out-of-scope queries far better (and even beats dense); the tradeoff is lower lexical recall on terse corpora, so RRF stays the default. Use `blend` when abstention matters — see [`ir_08`](). `surface_weights={"deps": 2.0}` multiplies each surface-hit’s first-stage (fused) score by its kind’s weight before the per-artifact collapse, so a high-precision surface such as `Package(embed_deps=True)`’s `deps` can outrank prose chunks. The default weighs every surface equally. Weighted scores need their own abstention floor. To sweep weights on one corpus, map it under several labels in `ir.eval.compare_indexings(..., search_kw_by_label={"deps x2": {"surface_weights": {"deps": 2.0}}})`. #### Precision stage: cross-encoder rerank (opt-in) First-stage retrieval is tuned for recall; a cross-encoder re-scores the top `fetch_k` candidates against the query *jointly*, which suppresses jargon-sharing distractors. Single-shot and agent-free (the evaluator back-edge stays in `raglab`): ```python hits = ir.search( corpus, "audio DAG pipelines", mode="hybrid", rerank="cross-encoder", # ef's cross-encoder, loaded lazily rerank_text="artifact", ) # read the whole artifact, not the matched surface ``` `pip install 'ir[rerank]'` brings `sentence-transformers`; without it the search warns once and returns the first-stage ranking. `rerank=` also takes any `ef.Reranker` callable (a stub, a hosted model), and `rerank_text=` a `(corpus, meta) -> str` callable. The `"cross-encoder"` spec squashes the model’s raw logits through a sigmoid into (0, 1), so relative selectors keep working. A reranked hit’s score is the reranker’s, on its own scale, so its abstention floor is calibrated and stored separately (a callable you pass needs a stable `rerank_id` attribute for that): pass the same `rerank=` / `rerank_text=` to `ir.eval.calibrate_min_score` and `ir.discover(..., min_score="auto")` (see `ir.rerank`). ### Expand ```python passage = ir.expand(hit, corpus) # ±1 chunk window (default) passage = ir.expand(hit, corpus, policy=ir.parent_policy()) # the whole artifact ``` The matched chunk is rarely the unit you want to read, and the whole document is usually too much. `ir.expand` stitches a hit’s *stored sibling segments* into a mid-granularity `Passage` (retrieve → expand → rerank), with chunker overlap deduped and the hit’s identity and score untouched. Policies are injectable (`NeighborhoodPolicy`); `sentence_window_policy(k)` and `parent_policy()` ship. It also works through the disclosure seam: ```python ir.disclose(sel, expand=ir.sentence_window_policy(2), corpus=corpus) ir.discover( "skills", q, expand=ir.sentence_window_policy() ) # passages on committed items ``` Each `Disclosure` then carries the stitched text as `.passage` (additive; `summary` stays the matched surface, `body` stays the pointer’s payload). ### Select ```python sel = ir.select(hits) # conservative default: stay within rel of top, cap at max_k sel = ir.select(hits, min_score=0.4) # opt in to abstention ("nothing applies") sel = ir.select( hits, strategy="score_gap" ) # elbow cut, or "top_k" / "rel_threshold" / a callable ``` The abstention floor is mode-specific (dense cosine, BM25, and RRF live on different scales), so rather than guess `min_score`, **calibrate** it from a case file and let `discover` load it: ```python ev.calibrate_min_score( corpus, cases, mode="dense", persist=True ) # learn + store the floor ir.discover( corpus, query, mode="dense", min_score="auto" ) # abstain by the calibrated floor ``` Calibration separates in-scope from out-of-scope query top-scores and picks the floor that best splits them — see [`ir_07`](); it works best on `dense` / `lexical` (hybrid’s RRF scores barely separate). `min_score` defaults to `None` (never abstain), so abstention stays fully opt-in. The conservative defaults (`max_k=3`, `rel=0.9`) are tuned, not guessed — see [`ir_06`](); re-tune for your own corpus with `ev.sweep_selector` / `ir sweep-select`. Selection is *relative* (ratios to the top score), so one selector works across `dense` / `hybrid` / `lexical` whose absolute scales differ by orders of magnitude. The result carries auditable `signals` and a `reason` — no opaque “confidence” float. An optional LLM selector (`make_llm_selector`, lazy on [`aix`](https://github.com/thorwhalen/aix), injectable for tests) falls back to the heuristic on any failure. ### Disclose ```python payloads = ir.disclose(sel, level="body") # "metadata" (no I/O) | "body" | "bundled" ``` Disclosure is a *pure* read that follows the pointer already stored on each hit (`skill_path` / `path`); it never mutates the ranked hits and tolerates a stale pointer. Keeping the agent’s context append-only (to protect the prompt cache) is then the caller’s discipline — `ir` hands back additive payloads. ### Graph & traverse (opt-in) Artifacts refer to each other — a package depends on packages, a skill has a parent. `ir` models those as a **semantic link graph**: a typed-edge `links` view on the store, populated at build time by an `EdgeExtractor`. ```python corpus = ir.build( source, edge_extractor=ir.default_edge_extractor ) # deps→REF, parent→PARENT graph = ir.CorpusGraph(corpus) graph.neighbors("contaix", edge_type="REF") # the package's dependencies ``` `ir.traverse` walks that structure at query time under a pluggable `WalkPolicy` (*score frontier → select → expand → stop*). **Safety is the operator’s**: a visited-set, depth cap, and node budget live in `traverse` itself, so even a cyclic graph and a never-stopping policy terminate. The shipped `collapsed_tree_policy` is pure-vector — it routes a query that matches an artifact’s *summary* down to that artifact’s best *chunk*: ```python hits = ir.traverse(query, corpus, policy=ir.collapsed_tree_policy()) ``` Dependencies are also a ranking signal, seeded by the caller. `graph.reverse_neighbors("networkx")` lists a library’s dependents and `graph.fan_in(...)` counts them. `ir.reverse_dependency_policy(seeds=[...])` walks those reverse edges, so a package that depends on a domain library but never says so in prose is still found. At ranking time, `ir.search(..., dep_boost=ir.dependency_evidence_boost({"networkx"}, weight=0.02))` adds score to candidates whose `deps` include a listed library. `ir` never chooses the libraries: that is the agent layer’s planning decision. ```python policy = ir.reverse_dependency_policy(seeds=["networkx"], fan_in_weight=0.1) walked = ir.traverse(query, corpus, policy=policy) hits = ir.fuse_hits({"flat": ir.search(corpus, query, mode="hybrid"), "deps": walked}) ``` The *summary* a query routes on can be an artifact’s own short field — or an **LLM-authored synopsis**. `ir.with_synopsis` wraps any indexing strategy to add one `synopsis` surface per artifact at build time (the document-summary-index pattern: build-time cost, ≈free at query time), and that synopsis becomes the collapsed-tree router: ```python strat = ir.with_synopsis( ir.Chunked(), synthesize=my_summarizer ) # or default (lazy aix) corpus = ir.build(ir.CorpusSource.from_mapping(docs, name="d", strategy=strat)) hits = ir.traverse( q, corpus, policy=ir.collapsed_tree_policy() ) # routes via the synopsis ``` `synthesize` is injectable (a test double or your own summarizer); omitted, it is built lazily on [`aix`](https://github.com/thorwhalen/aix) so `import ir` stays offline. Synopses are derived state with a stamped synthesizer identity, so a prompt/model change re-synthesizes only the affected artifacts on the next incremental `build` — no silent staleness. **Pay only where it helps.** `when=` gates synthesis per artifact (default `"always"`). `ir.thin_description_predicate` selects packages whose description is empty or short — their README is the only topical signal — so the LLM runs on those few and is never called for the well-described rest: ```python strat = ir.with_synopsis(ir.Package(), when=ir.thin_description_predicate) # threshold: when=ir.make_thin_description_predicate(min_chars=80) ``` The predicate’s identity (`when_id`) is part of the strategy id, so changing it re-processes the corpus and re-synthesizes every artifact the new gate selects. **Flat top-k stays the default** — `traverse` is opt-in, and a policy earns its keep only by beating flat+rerank on your eval set (a strong flat retriever wins simple lookup; graph methods cost far more). Results are ordinary `SearchHit`s with additive `metadata["walk_depth"]` / `["seed"]` provenance, so `select` / `disclose` compose unchanged. This is the **semantic link graph** (cyclic, query-time) — distinct from `ef.artifact_graph` (the acyclic build-time derivation DAG). ## Evaluation `ir.eval` scores discovery quality offline (reusing [`ef`](https://github.com/thorwhalen/ef)’s retrieval metrics): ```python from ir import eval as ev cases = ev.load_cases("skills_eval.jsonl") # query + gold artifact_id(s) ev.evaluate_discovery( corpus, cases, mode="hybrid" ) # recall@k / NDCG@k / MRR / MAP + failure taxonomy ev.evaluate_selection( corpus, cases, strategy="conservative" ) # conditional commit rate + selection P/R/F1 ev.sweep_selector(corpus, cases) # tune max_k × rel; .best() / .frontier() / .table() ev.distractor_robustness_curve(source.scope, probes) # accuracy vs catalog size ``` `evaluate_selection`’s headline is the **conditional commit rate** — the selection decision *isolated* from retrieval (did the selector keep the gold, *given* retrieval surfaced it?). `sweep_selector` scores a whole `max_k × rel` grid against the cases off **one** retrieval pass, so the selector defaults can be read off the data (`.best()`) rather than guessed. Generate cases by back-translation with `ir.eval_gen` (needs an LLM; scoring stays offline). ## CLI ```bash ir build skills # build/update a preset corpus ir search skills "deploy the app" # rank candidates (retrieval only) ir search skills "deploy the app" --rerank cross-encoder # + cross-encoder precision stage (ir[rerank]) ir discover skills "deploy the app" # retrieve -> select ir discover skills "deploy the app" --disclose # + load bodies ir discover skills "deploy the app" --min-score auto # + calibrated abstention ir build sessions # index recent Claude Code sessions (turn pairs) ir search sessions "numpy abi error" --mode lexical # find past sessions ir ls # list corpora + record counts ir coverage reports # disk-vs-index coverage (silent-gap detector) ir info skills # config, stats, policy, calibrated floors ir maintain --all # run due background work (idempotent; cron-friendly) ir schedule # install/inspect the OS job that runs it ir register notes files --root ~/notes --pattern '.*\.md$' # register a custom corpus ir rm notes # unregister (keeps built data) ir eval-gen skills skills_eval.jsonl # generate eval cases (needs aix/LLM) ir eval skills skills_eval.jsonl # score retrieval on a case file ir eval-select skills skills_eval.jsonl # score the selection stage ir sweep-select skills skills_eval.jsonl # tune the selector (max_k × rel) on your corpus ir calibrate-min-score skills skills_eval.jsonl --persist # calibrate the abstention floor ``` ## Keeping an index fresh `ir maintain` does the work that a corpus’s policy says is due — an incremental rebuild when reindex is due, LLM synopses inside their downtime window — and no-ops everything that is not. It is idempotent, so it is safe to run as often as you like. Something has to *call* it, though, and a corpus that nobody rebuilds keeps answering searches confidently from months-old content. `ir schedule` installs that caller — a launchd agent on macOS, a crontab block elsewhere — and then lets you operate it: ```bash ir schedule # ensure a schedule exists (creates one if there is none) ir schedule --status # what is scheduled, when it last ran, and whether the OS loaded it ir schedule --every 30m # change the interval (15m / 2h / 1d, or plain minutes) ir schedule --restart # reload it after upgrading ir or switching interpreter ir schedule --remove # stop it and delete the definition ir schedule --dry-run # print the plist/crontab that would be written ``` Bare `ir schedule` is idempotent in the way that matters: it creates a schedule when there is none, and when one already exists it reports it and the commands above rather than silently reinstalling over an interval you tuned. The job is pinned to the absolute interpreter that installed it, because neither launchd nor cron resolves `ir` on your shell’s `PATH`, and it carries the environment ir reads (`$PP`, `$PTH_FILEPATH`, `IR_*`, `XDG_*`) — launchd starts jobs with a near-empty environment, and without those the `packages` and `reports` corpora fail every time the job fires while it still exits 0. **An existing definition is data, not a template.** Only a fresh install snapshots your shell; `--every` and `--restart` carry the stored environment forward untouched, so operating a working schedule from a shell that happens to lack `$PP` never silently re-points it at a different corpus store. To re-snapshot deliberately: `ir schedule --remove && ir schedule`. `--status` is the health check. It reports the last run and the last log line, and flags three failures that otherwise look like success: an interpreter that no longer exists (a rebuilt pyenv version or recreated venv leaves a job that fires on time and does nothing), a job whose environment is missing something your shell has, and a non-zero exit status from the last run. `--restart` re-pins the interpreter. **`ir` still does not run a scheduler.** It writes a definition, hands it to the OS, and exits: no daemon, no in-process timer. launchd/cron remain the executor, exactly as `ir.policy` describes. ## For developers with a pulse **Set up and run the tests.** The suite is hermetic (light embedder, in-memory stores, no network) and doctests count: ```bash git clone https://github.com/i2mint/ir && cd ir uv venv -q .venv && . .venv/bin/activate uv pip install -q -e ".[dev]" python -m pytest -q --doctest-modules ir tests ruff check ir && ruff format --check ir ``` **Contributing.** The `ir-dev-workflow` skill (`skills/ir-dev-workflow/SKILL.md`) is also the human contributor guide: the module map, the CLI grammar golden, the conventions review enforces, and the ir/raglab boundary. New arguments default to today’s behaviour. Don’t bump the version: CI does that when a pull request merges to `master`, and the merge publishes to PyPI. `thorwhalen/raglab` depends on `ir`, so keep its tests green. **Design.** The design is grounded in a set of capability-discovery research reports and eval-run findings under `misc/docs/` (`ir_01`–`ir_10`): the single-search-tool pattern, indexing & embedding strategy, evaluation, the `ef` + `vd` reuse analysis, the dense-vs-lexical-vs-hybrid eval, selector tuning, abstention-floor calibration, magnitude-preserving fusion, the composable search-agent architecture (`ir_09`) and LLM routing (`ir_10`). `ir` is light by default (numpy / `dol`) and reuses the ecosystem (`ef`, `vd`, and `aix` for the opt-in LLM seams) only where it composes cleanly. **Where to ask.** Open an issue on [i2mint/ir](https://github.com/i2mint/ir/issues). Issue #1 is the roadmap and epic #38 tracks the agentic-search direction.

This documentation as a single file: ir.md (Markdown, for agents).

# _autosummary/ir.base.html.md # ir.base Core data model for `ir`. Retrieval in `ir` flows through four small, explicit types: - [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) — a logical item in a corpus (a file, a skill, a package). Opaque `raw` payload plus `metadata`. - [`Surface`](_autosummary/ir.base.html.md#ir.base.Surface) — one *embeddable unit* derived from an artifact. A single artifact may yield several heterogeneous surfaces (a short description, an AI-authored synopsis, a list of problem classes, body chunks). The artifact→surfaces decomposition is the job of an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy). - [`IndexPlan`](_autosummary/ir.base.html.md#ir.base.IndexPlan) — a strategy’s output for one artifact: the `filter_fields` (hard-filterable metadata, *not* embedded) and the list of surfaces (embedded). - [`Record`](_autosummary/ir.base.html.md#ir.base.Record) — a stored, embedded surface (one row in the index; maps directly to a `vd` `Document`). - [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) — a scored record returned by retrieval, with a helper to collapse multiple surface-hits of the same artifact. The split between **filter_fields** (metadata you filter on) and **surfaces** (text you embed) is deliberate and central: good retrieval is hard metadata filtering *and* semantic ranking, not only embeddings. ### Module Attributes | [`FilterFields`](_autosummary/ir.base.html.md#ir.base.FilterFields) | Non-embedded, hard-filterable metadata for an artifact (name, owner, tags). | |-----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| | [`POINTER_KEYS`](_autosummary/ir.base.html.md#ir.base.POINTER_KEYS) | Metadata keys checked, in order, for a disclosure *pointer* — the file/dir whose contents are an artifact's body. | ### Functions | [`best_per_artifact`](_autosummary/ir.base.html.md#ir.base.best_per_artifact)(hits) | Collapse hits to the highest-scoring surface per artifact. | |----------------------------------------------------------------------------|-----------------------------------------------------------------------------| | [`ledger_key`](_autosummary/ir.base.html.md#ir.base.ledger_key)(artifact_id) | The ledger key under which an artifact's entry is filed. | | [`storage_key`](_autosummary/ir.base.html.md#ir.base.storage_key)(\*parts) | Stable, filesystem-safe id from arbitrary string parts (truncated SHA-256). | | [`tag_source`](_autosummary/ir.base.html.md#ir.base.tag_source)(hits, source) | Stamp *source* on every hit that doesn't already carry one. | ### Classes | [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)(id, raw[, metadata]) | A logical corpus item before decomposition into surfaces. | |----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| | [`IndexPlan`](_autosummary/ir.base.html.md#ir.base.IndexPlan)([filter_fields, surfaces]) | An [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)'s output for one artifact. | | [`Record`](_autosummary/ir.base.html.md#ir.base.Record)(id, artifact_id, surface_kind, ...[, ...]) | A stored, embedded surface — one row of the index. | | [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)(artifact_id, surface_kind, score, text) | A scored record returned by retrieval (higher score = closer). | | [`Surface`](_autosummary/ir.base.html.md#ir.base.Surface)(artifact_id, kind, text[, ...]) | One embeddable unit derived from an artifact. | ### *class* ir.base.Artifact(id, raw, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A logical corpus item before decomposition into surfaces. ### ir.base.FilterFields Non-embedded, hard-filterable metadata for an artifact (name, owner, tags). alias of [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### *class* ir.base.IndexPlan(filter_fields=, surfaces=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)’s output for one artifact. ### ir.base.POINTER_KEYS *= ('skill_path', 'path')* Metadata keys checked, in order, for a disclosure *pointer* — the file/dir whose contents are an artifact’s body. A [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) is a *pointer + snippet* (ir_09 §5): `text` is the snippet, the pointer is the key a resource store dereferences to the full payload. Skills stamp `skill_path`; packages / reports / files stamp `path` (see [`ir.sources`](_autosummary/ir.sources.html.md#module-ir.sources)). [`ir.select`](_autosummary/ir.html.md#ir.select) re-exports this for disclosure. ### *class* ir.base.Record(id, artifact_id, surface_kind, surface_index, text, vector, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A stored, embedded surface — one row of the index. #### *static* make_id(artifact_id, surface_kind, surface_index) Deterministic storage id for a surface of an artifact. `surface_index` is the surface’s **plan-global** position — its enumeration index across *all* surfaces of the artifact’s [`IndexPlan`](_autosummary/ir.base.html.md#ir.base.IndexPlan), regardless of kind — as assigned by [`ir.index.build()`](_autosummary/ir.index.html.md#ir.index.build). On multi-kind strategies it therefore differs from per-kind counters like `metadata["chunk_index"]` (e.g. [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package): the `description` surface takes position 0, shifting `readme_chunk` *j* to `surface_index` `j+1` — and the offset is plan-dependent, since empty surfaces are dropped). Ids of already-built corpora are a stability contract: never re-derive a sibling’s id from a per-kind index — address siblings through the ledger via [`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### *class* ir.base.SearchHit(artifact_id, surface_kind, score, text, metadata=, source=None, surface_index=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A scored record returned by retrieval (higher score = closer). Maps onto ir_09’s `Result`: `text` is the snippet, `score` the rank score, `metadata` the meta, and [`pointer`](_autosummary/ir.base.html.md#ir.base.SearchHit.pointer) the key into a resource store (ir_09 §5). [`to_dict()`](_autosummary/ir.base.html.md#ir.base.SearchHit.to_dict) is the serialization-clean form for a cross-process / subagent boundary (no numpy scalars leak). `source` is the corpus/source name the hit came from (`None` when unattributed — e.g. an ad-hoc corpus without a name). It is a first-class field, not a metadata key, because `metadata` is the strategy-owned hard-filter namespace and provenance is structural: artifact identity is only unique *within* a source, so any cross-source operation keys on `(source, artifact_id)` (see [`best_per_artifact()`](_autosummary/ir.base.html.md#ir.base.best_per_artifact)). `surface_index` is the stored `Record.surface_index` of the hit’s surface — its plan-global position among the artifact’s surfaces — so a hit can name *which* surface of its artifact it is (the prerequisite for sibling addressing and context expansion). `None` when unknown (e.g. a hand-built hit). It is **not** the per-kind `metadata["chunk_index"]`; see [`Record.make_id()`](_autosummary/ir.base.html.md#ir.base.Record.make_id) for why the two differ on multi-kind strategies. #### *property* pointer *: [str](https://docs.python.org/3/builtins/stdtypes.html#str) | [None](https://docs.python.org/3/builtins/constants.html#None)* The disclosure pointer on this hit, if any (see [`POINTER_KEYS`](_autosummary/ir.base.html.md#ir.base.POINTER_KEYS)). #### to_dict() JSON-serializable form (`score` cast to a Python `float`). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.base.Surface(artifact_id, kind, text, granularity='document', metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One embeddable unit derived from an artifact. `kind` names the surface type (e.g. `"description"`, `"synopsis"`, `"problem_class"`, `"chunk"`) so a query can match the *right part* of an artifact. `granularity` is a coarse hint (`"document"` / `"chunk"` / `"field"`). `metadata` is surface-local (e.g. chunk offsets). ### ir.base.best_per_artifact(hits) Collapse hits to the highest-scoring surface per artifact. Returns the surviving hits sorted by score (descending). Identity is `(source, artifact_id)`: the same id in two different sources names two different artifacts (the skills-corpus “dol” is not the packages-corpus “dol”), so cross-source input never collapses them. Single-source input (all hits sharing one `source`, or all `None`) behaves exactly as an id-keyed collapse. Note the raw-score comparison and the final sort assume one score scale — sound within a source; for mixed-source hits use [`ir.retrieve.fuse_hits()`](_autosummary/ir.retrieve.html.md#ir.retrieve.fuse_hits), which only compares scores within a source. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] ### ir.base.ledger_key(artifact_id) The ledger key under which an artifact’s entry is filed. The single source of truth for ledger keying — [`ir.index.build()`](_autosummary/ir.index.html.md#ir.index.build) writes entries under this key and readers (e.g. [`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)) resolve them with it, so the two can never drift apart. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.base.storage_key(\*parts) Stable, filesystem-safe id from arbitrary string parts (truncated SHA-256). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.base.tag_source(hits, source) Stamp *source* on every hit that doesn’t already carry one. Existing tags win: a hit already attributed to a corpus keeps that attribution (so re-tagging under a different registry key cannot double-count one corpus as two sources). A `None` source is the untagged pseudo-source — hits pass through unattributed. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] # _autosummary/ir.cli.html.md # ir.cli Command-line surface for `ir` (dispatched by cw). Commands operate on **named** corpora from the registry (see [`ir.registry`](_autosummary/ir.registry.html.md#module-ir.registry)): ```default ir build skills # build/update the skills preset corpus ir search skills "deploy app" # rank candidates (retrieval only) ir discover skills "deploy app" # retrieve -> commit to a high-precision subset ir discover skills "deploy app" --disclose # + load each selected item's body ir ls # list corpora + record counts ir coverage reports # disk-vs-index coverage (silent-gap detector) ir info packages # config + stats for a corpus ir maintain --all # run due background work (idempotent) ir schedule # install/inspect the OS job that runs it ir register notes files --root ~/notes --pattern '.*\.md$' ir rm notes # unregister (keeps built data) ir eval-gen skills skills_eval.jsonl --k 5 # generate cases (needs aix/LLM) ir eval skills skills_eval.jsonl --mode hybrid # score retrieval on a case file ir eval-select skills skills_eval.jsonl # score the selection stage ir sweep-select skills skills_eval.jsonl # tune max_k/rel for the selector ir calibrate-min-score skills skills_eval.jsonl --persist # abstention floor ir discover skills "deploy app" --min-score auto # use the calibrated floor ``` ### Functions | [`build`](_autosummary/ir.cli.html.md#ir.cli.build)(name, \*[, embedder, full]) | Build or incrementally update a registered (or preset) corpus. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| | [`calibrate_min_score`](_autosummary/ir.cli.html.md#ir.cli.calibrate_min_score)(name, cases, \*[, mode, ...]) | Calibrate the absolute abstention min_score floor for a corpus + mode. | | [`coverage`](_autosummary/ir.cli.html.md#ir.cli.coverage)([name]) | Report disk-vs-index coverage for a reports corpus (silent-gap detector). | | [`discover`](_autosummary/ir.cli.html.md#ir.cli.discover)(name, query, \*[, k, mode, ...]) | Search a corpus, commit to a distractor-robust subset, and show it. | | [`eval`](_autosummary/ir.cli.html.md#ir.cli.eval)(name, cases, \*[, mode, k]) | Score a built corpus's retrieval against a DiscoveryCase JSONL file. | | [`eval_gen`](_autosummary/ir.cli.html.md#ir.cli.eval_gen)(name, out, \*[, k, abstention_frac, ...]) | Generate an eval-case file for a corpus by back-translation (needs aix/LLM). | | [`eval_select`](_autosummary/ir.cli.html.md#ir.cli.eval_select)(name, cases, \*[, strategy, ...]) | Score a selector against a DiscoveryCase JSONL file (selection quality). | | [`info`](_autosummary/ir.cli.html.md#ir.cli.info)(name) | Show a corpus's stored config, stats, policy, and any abstention floors. | | [`ls`](_autosummary/ir.cli.html.md#ir.cli.ls)() | List registered corpora with their kind, embedder, and record count. | | [`maintain`](_autosummary/ir.cli.html.md#ir.cli.maintain)([name, all, dry_run]) | Run due background work: incremental reindex, synopsis in its downtime window. | | [`register`](_autosummary/ir.cli.html.md#ir.cli.register)(name, kind, \*[, root, pattern, ...]) | Register a named corpus. | | [`rm`](_autosummary/ir.cli.html.md#ir.cli.rm)(name) | Unregister a corpus (does not delete its built data). | | [`schedule`](_autosummary/ir.cli.html.md#ir.cli.schedule)(\*[, every, status, restart, ...]) | Install, inspect, and operate the OS job that runs `ir maintain`. | | [`search`](_autosummary/ir.cli.html.md#ir.cli.search)(name, query, \*[, k, mode, fusion, ...]) | Search a built corpus and print the top-k hits. | | [`sweep_select`](_autosummary/ir.cli.html.md#ir.cli.sweep_select)(name, cases, \*[, strategy, ...]) | Sweep selector knobs (max_k × rel) against a case file; print the grid + best. | ### ir.cli.build(name, , embedder=None, full=True) Build or incrementally update a registered (or preset) corpus. ### ir.cli.calibrate_min_score(name, cases, , mode='hybrid', k=10, sensitivity_weight=0.5, persist=False, fusion='rrf', rerank=None, rerank_text='surface') Calibrate the absolute abstention min_score floor for a corpus + mode. Separates in-scope (gold-bearing) from out-of-scope (empty-gold) query top scores and reports the floor that best splits them, with sensitivity / specificity / Youden’s J. `cases` must include BOTH gold-bearing and abstention cases — generate them with `ir eval-gen` (it adds an abstention slice). `--persist` stores the floor so `ir discover ... --min-score auto` will abstain by it. `--sensitivity-weight` (0..1, balanced default); lower it to abstain more readily (precision-leaning). For hybrid, calibrate with `--fusion blend` (rank-based RRF barely separates — see ir_08) and query under the same fusion. mode: dense | lexical | hybrid. fusion (hybrid only): rrf | blend. `--rerank cross-encoder` calibrates the floor for a reranked search (its own floor, stored apart from the plain one); query with the same `--rerank` / `--rerank-text`. ### ir.cli.coverage(name='reports') Report disk-vs-index coverage for a reports corpus (silent-gap detector). Independently walks every `*/*/docs` and `*/*/misc/docs` tree on disk and diffs the includable reports against what is actually indexed, listing any file that is on disk but missing from the index (rebuild with `ir build` to fix). Catches the ingestion gaps that back-translation evals structurally cannot — a doc that was never indexed can never be an eval gold id. ### ir.cli.discover(name, query, , k=10, mode='hybrid', strategy='conservative', disclose=False, min_score=None, fusion='rrf', rerank=None, rerank_text='surface') Search a corpus, commit to a distractor-robust subset, and show it. Retrieves `k` candidates, then *selects* the few high-precision results an agent should act on (or abstains). `--disclose` additionally loads each selected item’s body (SKILL.md / file text) via its stored pointer. `--min-score auto` turns on absolute abstention using the floor calibrated by `ir calibrate-min-score` (or pass a float). mode: dense | lexical | hybrid. fusion (hybrid only): rrf (default) | blend (magnitude-preserving; pair with –min-score for abstention — see ir_08). strategy: conservative | top_k | rel_threshold | score_gap. `--rerank cross-encoder` adds the calibrated cross-encoder precision stage (`--rerank-text` picks what it reads); with `--min-score auto` it uses the floor calibrated under the same `--rerank` / `--rerank-text`. ### ir.cli.eval(name, cases, , mode='hybrid', k=10) Score a built corpus’s retrieval against a DiscoveryCase JSONL file. cases: path to a JSONL file of cases (see [`ir.eval`](_autosummary/ir.eval.html.md#module-ir.eval)); each line is a `{"query": ..., "gold": [artifact_id, ...]}` record (empty `gold` = an abstention case). Prints recall@k / NDCG@k / MRR / MAP plus the failure-mode taxonomy. mode: dense | lexical | hybrid. ### ir.cli.eval_gen(name, out, , k=5, abstention_frac=0.15, max_artifacts=None) Generate an eval-case file for a corpus by back-translation (needs aix/LLM). Writes a DiscoveryCase JSONL set (gold cases + an abstention slice) for the registered corpus *name* to *out*, stamping a corpus-signature into the header so the frozen file can be checked against the live corpus later. This command calls an LLM via aix; scoring it afterwards (`ir eval`) is offline. ### ir.cli.eval_select(name, cases, , strategy='conservative', mode='hybrid', k=10, max_k=3, rel=0.9, min_score=None) Score a selector against a DiscoveryCase JSONL file (selection quality). Reports the conditional commit rate (the selection decision isolated from retrieval) plus selection precision / recall / F1 and abstention accuracy. strategy: conservative | top_k | rel_threshold | score_gap. mode: dense | lexical | hybrid. max_k / rel / min_score tune the commit (see `ir.select`); `ir sweep-select` sweeps them to find good values. ### ir.cli.info(name) Show a corpus’s stored config, stats, policy, and any abstention floors. ### ir.cli.ls() List registered corpora with their kind, embedder, and record count. ### ir.cli.maintain(name=None, , all=False, dry_run=False) Run due background work: incremental reindex, synopsis in its downtime window. With a name, maintains that corpus; with –all (or no name), every registered corpus. Idempotent and safe to schedule (cron/launchd): it no-ops what is not due. –dry-run reports what would run without doing it. ### ir.cli.register(name, kind, , root=None, pattern=None, embedder='default', reindex_on=None, every_hours=None, synopsis=False) Register a named corpus. kind: skills | packages | reports | files. Background-work policy (optional; smart per-kind defaults otherwise — see `ir.policy`): reindex_on (source-change | interval | manual), every_hours (for interval), synopsis (enable LLM synopses, run only in the policy’s downtime window by `ir maintain`). ### ir.cli.rm(name) Unregister a corpus (does not delete its built data). ### ir.cli.schedule(, every=None, status=False, restart=False, remove=False, dry_run=False, backend=None) Install, inspect, and operate the OS job that runs `ir maintain`. Bare `ir schedule` is idempotent: it creates the schedule when there is none, and when there already is one it reports it (interval, definition, whether the OS has it loaded, last run) plus how to operate it, rather than silently reinstalling over a schedule you tuned. An existing definition’s environment is carried forward by every operation, so operating a working schedule from a shell that happens to lack $PP never silently re-points it at a different corpus store. every: interval as 15m / 2h / 1d, or a plain number of minutes. status: report only; never mutates. restart: reload the definition and re-pin it to this interpreter. remove: stop the schedule and delete its definition. dry_run: print the definition that would be written, and write nothing. backend: launchd | cron (default: whichever this machine provides). ### ir.cli.search(name, query, , k=10, mode='dense', fusion='rrf', rerank=None, rerank_text='surface') Search a built corpus and print the top-k hits. mode: dense (cosine) | lexical (BM25) | hybrid (dense + BM25 fused). fusion (hybrid only): rrf (rank-based, default) | blend (magnitude-preserving; better abstention separability — see ir_08). `--rerank cross-encoder` adds the cross-encoder precision stage (needs the `ir[rerank]` extra; without it the search warns and keeps the first-stage ranking); `--rerank-text` picks what it reads: surface (default) | artifact. Reranked scores are on the reranker’s own scale. ### ir.cli.sweep_select(name, cases, , strategy='conservative', mode='hybrid', k=10, objective='selection_f1') Sweep selector knobs (max_k × rel) against a case file; print the grid + best. Retrieves once per case and reuses the candidates across the whole grid, so the sweep is cheap. Prints a table (one row per setting, best objective first) and the winning max_k / rel. Use it to pick `ir.select` defaults empirically. objective: selection_f1 | selection_precision | selection_recall | conditional_commit_rate | mean_selected_size. mode: dense | lexical | hybrid. # _autosummary/ir.config.html.md # ir.config Filesystem locations and process-wide defaults for `ir`. `ir` separates three kinds of on-disk state, each under an XDG-standard base (overridable per-install via `IR_CONFIG_DIR` / `IR_DATA_DIR` / `IR_CACHE_DIR`, then `XDG_CONFIG_HOME` / `XDG_DATA_HOME` / `XDG_CACHE_HOME`, then `~/.config` / `~/.local/share` / `~/.cache`): - **config** (`~/.config/ir`) — the named-corpus registry and user settings. - **data** (`~/.local/share/ir`) — durable corpus stores (record metadata, vectors, ledgers). The source of truth; losing it means rebuilding the index. - **cache** (`~/.cache/ir`) — regenerable derived data, chiefly the embedding cache keyed by `(model, content_hash)`. Every path is a plain [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) created on demand, so callers can treat the directories as guaranteed to exist. ### Functions | [`cache_dir`](_autosummary/ir.config.html.md#ir.config.cache_dir)() | Regenerable cache directory (default `~/.cache/ir`). | |---------------------------------------------------------------------------------|--------------------------------------------------------------------| | [`config_dir`](_autosummary/ir.config.html.md#ir.config.config_dir)() | User configuration directory (default `~/.config/ir`). | | [`corpus_dir`](_autosummary/ir.config.html.md#ir.config.corpus_dir)(name) | Durable directory holding one corpus's stores. | | [`data_dir`](_autosummary/ir.config.html.md#ir.config.data_dir)() | Durable data directory (default `~/.local/share/ir`). | | [`embeddings_cache_dir`](_autosummary/ir.config.html.md#ir.config.embeddings_cache_dir)(model_id) | Cache directory for one embedding model's vectors. | | [`registry_path`](_autosummary/ir.config.html.md#ir.config.registry_path)() | JSON file mapping registered corpus names to their build settings. | | [`safe_name`](_autosummary/ir.config.html.md#ir.config.safe_name)(name) | Filesystem-safe slug for a corpus/model identifier. | ### ir.config.cache_dir() Regenerable cache directory (default `~/.cache/ir`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.config_dir() User configuration directory (default `~/.config/ir`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.corpus_dir(name) Durable directory holding one corpus’s stores. * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.data_dir() Durable data directory (default `~/.local/share/ir`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.embeddings_cache_dir(model_id) Cache directory for one embedding model’s vectors. * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.registry_path() JSON file mapping registered corpus names to their build settings. * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.config.safe_name(name) Filesystem-safe slug for a corpus/model identifier. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ir.coverage.html.md # ir.coverage Corpus-coverage diagnostic — what’s on disk vs. what’s actually indexed. Retrieval can only return what was indexed, and an ingestion gap is *silent*: a whole folder of reports can be missing from the index with no error (a non-recursive glob once dropped every doc in a `research/` / `decisions/` subtree — 29% of the corpus — invisibly). Back-translation evals cannot catch this: their gold is drawn from what is *already* indexed, so an un-indexed doc can never lower a metric. Coverage therefore needs its own check. [`reports_coverage()`](_autosummary/ir.coverage.html.md#ir.coverage.reports_coverage) **independently** walks the reports doc-trees on disk (the raw `*/*/docs` and `*/*/misc/docs` files, before any filtering), classifies each file with the same inclusion SSOT the ingestion walk uses ([`ir.sources.report_exclude_reason()`](_autosummary/ir.sources.html.md#ir.sources.report_exclude_reason)), and diffs the *should-be-indexed* set against the built corpus’s ledger — surfacing exactly the on-disk-but-absent reports that a search would silently miss. ### Functions | [`reports_coverage`](_autosummary/ir.coverage.html.md#ir.coverage.reports_coverage)([name, projects_root, ...]) | Diff the reports docs tree on disk against the built `name` corpus. | |-------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------| ### Classes | [`CoverageReport`](_autosummary/ir.coverage.html.md#ir.coverage.CoverageReport)(name, on_disk, indexed, excluded) | Disk-vs-index coverage for a filesystem-backed corpus. | |-----------------------------------------------------------------------------------------------------|----------------------------------------------------------| ### *class* ir.coverage.CoverageReport(name, on_disk, indexed, excluded, missing=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Disk-vs-index coverage for a filesystem-backed corpus. #### *property* coverage_ratio *: [float](https://docs.python.org/3/builtins/functions.html#float)* Fraction of should-index files that are actually indexed (1.0 = full). #### *property* should_index *: [int](https://docs.python.org/3/builtins/functions.html#int)* On-disk files that pass the inclusion rule (indexed + missing). ### ir.coverage.reports_coverage(name='reports', , projects_root=None, indexed_ids=None, exclude_dirs=frozenset({'.git', '.hg', '.ipynb_checkpoints', '.obsidian', '.svn', '.tox', '.venv', '_\_pycache_\_', 'build', 'dist', 'node_modules', 'site-packages', 'venv'})) Diff the reports docs tree on disk against the built `name` corpus. Walks every `*/*/docs` and `*/*/misc/docs` folder recursively (raw, no filtering), classifies each `*.md` with the shared inclusion SSOT, and reports how many *includable* reports are indexed vs. silently missing. A `coverage_ratio` below 1.0 means on-disk reports are absent from the index — rebuild with `ir build `. `indexed_ids` (the built corpus’s artifact ids) is read from the registered `name` corpus by default; pass it explicitly (with `projects_root`) to diff an arbitrary tree against an arbitrary index without opening a corpus. * **Return type:** [`CoverageReport`](_autosummary/ir.coverage.html.md#ir.coverage.CoverageReport) # _autosummary/ir.embed.html.md # ir.embed Embedder resolution for `ir` — a decent local default, a light fallback. `ir` favors a *decent local* embedding so retrieval works offline and tests need no API keys, while keeping a *light* (numpy-only) option for when semantic power is not what’s under test. - `"default"` / `"local"` / `"minilm"` → `all-MiniLM-L6-v2` (384-dim) via `ef.embedder_adapters.sentence_transformers_embedder()` (`normalize=True`), wrapped in `ef.CachedEmbedder` over a `dol` cache under `~/.cache/ir/embeddings/`. If `sentence-transformers` is unavailable, it degrades to the hashing embedder with a warning. - `"light"` / `"hashing"` → `ef.HashingEmbedder` (numpy only). - any other string → treated as a sentence-transformers model name. - a callable / existing `Embedder` → passed through `ef.as_embedder`. [`make_embedder()`](_autosummary/ir.embed.html.md#ir.embed.make_embedder) returns `(embedder, embedder_id)`; the id pins the model in the corpus ledger so a model change triggers a re-embed (the SSOT discipline that keeps the index from silently drifting). Importing this module sets `USE_TF=0` so `sentence-transformers` (via `transformers`) does not import TensorFlow, which crashes on this stack’s numpy ABI. Import `ir` before anything that imports `transformers`. ### Functions | [`make_embedder`](_autosummary/ir.embed.html.md#ir.embed.make_embedder)([spec, cache, strict]) | Resolve *spec* to `(embedder, embedder_id)`. | |-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------| | [`make_embedder_for_id`](_autosummary/ir.embed.html.md#ir.embed.make_embedder_for_id)(embedder_id, \*[, cache]) | Re-create the embedder a corpus was built with, from its stored id. | ### Exceptions | [`EmbedderError`](_autosummary/ir.embed.html.md#ir.embed.EmbedderError) | A corpus cannot be queried with the embedder at hand. | |----------------------------------------------------------------------|-------------------------------------------------------------| | [`EmbedderMismatch`](_autosummary/ir.embed.html.md#ir.embed.EmbedderMismatch) | The query embedding's dimension differs from the index's. | | [`EmbedderUnavailable`](_autosummary/ir.embed.html.md#ir.embed.EmbedderUnavailable) | The embedder a corpus was built with cannot be loaded here. | ### *exception* ir.embed.EmbedderError Bases: [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) A corpus cannot be queried with the embedder at hand. Subclasses [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) (the type earlier versions raised) so existing `except ValueError` callers keep working. The message is always actionable: it says what is wrong and the command that fixes it. ### *exception* ir.embed.EmbedderMismatch Bases: [`EmbedderError`](_autosummary/ir.embed.html.md#ir.embed.EmbedderError) The query embedding’s dimension differs from the index’s. ### *exception* ir.embed.EmbedderUnavailable Bases: [`EmbedderError`](_autosummary/ir.embed.html.md#ir.embed.EmbedderError) The embedder a corpus was built with cannot be loaded here. ### ir.embed.make_embedder(spec='default', , cache=True, strict=False) Resolve *spec* to `(embedder, embedder_id)`. `embedder` is a batch callable `Iterable[str] -> ndarray(n, dim)` that also accepts `input_type=` (`"query"` / `"document"`). With `strict=True` a local model that cannot be loaded raises [`EmbedderUnavailable`](_autosummary/ir.embed.html.md#ir.embed.EmbedderUnavailable) instead of degrading to the hashing embedder (used when reopening a corpus, whose vectors only match its own model). * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.embed.make_embedder_for_id(embedder_id, , cache=True) Re-create the embedder a corpus was built with, from its stored id. The ids [`make_embedder()`](_autosummary/ir.embed.html.md#ir.embed.make_embedder) emits are invertible: `hashing__dim` and `st__`. Returns `None` for an id that is not one of those forms (e.g. `"custom"`, or empty), so the caller falls back to the stored spec. Never degrades silently: an unavailable model raises [`EmbedderUnavailable`](_autosummary/ir.embed.html.md#ir.embed.EmbedderUnavailable), and an id of one of those forms that does not parse raises [`EmbedderError`](_autosummary/ir.embed.html.md#ir.embed.EmbedderError) (falling back to the spec there is the drift this exists to prevent). This is what pins a corpus to its own embedder, so the *machine* querying it (with or without `sentence-transformers`) cannot change the vector space. # _autosummary/ir.eval.html.md # ir.eval Capability-discovery evaluation — measuring how well a corpus is *found*. `ir.eval` answers the question that the rest of `ir` raises: \*does retrieval actually surface the right capability?\* It is a thin, deterministic scoring layer over the substrate already in place — `ir.retrieve.search` for ranking and `ef.evaluation` for the metric mathematics — plus the genuinely new pieces a *capability-discovery* eval needs that generic retrieval evaluation does not. ## Design stance (retrieval-centric, reuse `ef`) `ir`’s corpora are **documents** — skills, packages, reports — each a name + description (+ body), joined on a stable `artifact_id`. They are *not* a typed function-calling registry with argument signatures. So the eval that fits is **retrieval of the right artifact**, not argument-dict scoring: - **recall@k / NDCG@k / MRR / MAP** — computed by `ef.evaluation`’s pure, tested primitives (`ef.evaluation.recall_at_k()`, … ) and driver (`ef.evaluation.evaluate_retrieval()`). `ir.eval` does not reimplement them; it adapts an `ir` corpus to `ef`’s retriever contract and reads back a `ef.evaluation.RetrievalEvalReport`. - **distractor-robustness curve** — retrieval accuracy as the catalog grows (1 gold + N−1 distractors). The single most diagnostic capability-discovery metric, and one `ef` does not provide. - **failure-mode taxonomy** — every gold case is classed `hit_rank_1` / `surfaced_low_rank` / `retrieval_miss` so a headline number decomposes into *why* it is what it is. - **abstention** — an optional score-threshold proxy for “no artifact applies” cases. True abstention is a *selection* concern (a selector commits or refuses); here it is a retrieval-side diagnostic, clearly labelled as such. The unit of evaluation is a [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase): an intent (`query`) and the `gold` `artifact_id`s that should answer it (empty `gold` = an abstention case). Cases are plain data — [`save_cases()`](_autosummary/ir.eval.html.md#ir.eval.save_cases) / [`load_cases()`](_autosummary/ir.eval.html.md#ir.eval.load_cases) round-trip them as JSONL so an evaluation set is a committable, reproducible fixture (the corpora themselves are machine-specific and live, so freezing the cases is what makes a run repeatable; [`validate_cases()`](_autosummary/ir.eval.html.md#ir.eval.validate_cases) flags gold ids that have since drifted out of the corpus). Case *generation* (back-translation from a corpus with name-masking) needs an LLM and is deliberately **out of scope here** — this module scores a given set of cases offline, with no network and no model download. (Dense scoring needs only numpy; `lexical` / `hybrid` ranking additionally need the optional `vd` dependency — a hybrid run that silently fell back to dense because `vd` was absent is flagged in the report.) Quick start: ```default import ir from ir import eval as ev corpus = ir.build(ir.CorpusSource.from_skills()) # or ir.open_corpus("skills") cases = ev.load_cases("skills_eval.jsonl") report = ev.evaluate_discovery(corpus, cases, mode="hybrid") print(report) # NDCG@10 + taxonomy # Just the BEIR-shaped metrics (pure ef), e.g. to A/B dense vs hybrid: ev.retrieval_report(corpus, cases, mode="dense").primary # mean NDCG@10 ev.retrieval_report(corpus, cases, mode="hybrid").primary ``` ### Module Attributes | [`DFLT_K_VALUES`](_autosummary/ir.eval.html.md#ir.eval.DFLT_K_VALUES) | Rank cutoffs reported by default. | |--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`DFLT_METRICS`](_autosummary/ir.eval.html.md#ir.eval.DFLT_METRICS) | Metrics reported by default — the `ef.evaluation` retrieval metric names. | | [`DFLT_MODE`](_autosummary/ir.eval.html.md#ir.eval.DFLT_MODE) | Default ranking mode for the eval adapter (hybrid is `ir`'s strongest). | | [`DFLT_DISTRACTOR_SIZES`](_autosummary/ir.eval.html.md#ir.eval.DFLT_DISTRACTOR_SIZES) | Catalog sizes swept by [`distractor_robustness_curve()`](_autosummary/ir.eval.html.md#ir.eval.distractor_robustness_curve) (RAG-MCP-style). | | [`DFLT_CURVE_MODE`](_autosummary/ir.eval.html.md#ir.eval.DFLT_CURVE_MODE) | Defaults for [`distractor_robustness_curve()`](_autosummary/ir.eval.html.md#ir.eval.distractor_robustness_curve) — deliberately *not* `DFLT_MODE`: the curve builds throwaway in-memory sub-corpora, so it favours the fast, always-offline dense + hashing path. | | [`DFLT_SWEEP_MAX_K`](_autosummary/ir.eval.html.md#ir.eval.DFLT_SWEEP_MAX_K) | `max_k` values swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) (the commit-size cap). | | [`DFLT_SWEEP_REL`](_autosummary/ir.eval.html.md#ir.eval.DFLT_SWEEP_REL) | `rel` values swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) (relative-to-top keep band). | | [`DFLT_SWEEP_MIN_SCORE`](_autosummary/ir.eval.html.md#ir.eval.DFLT_SWEEP_MIN_SCORE) | `min_score` floors swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector). | | [`DFLT_SWEEP_OBJECTIVE`](_autosummary/ir.eval.html.md#ir.eval.DFLT_SWEEP_OBJECTIVE) | Default objective [`SelectionSweep.best()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.best) optimizes — the standard precision/recall balance. | | [`SWEEP_METRICS`](_autosummary/ir.eval.html.md#ir.eval.SWEEP_METRICS) | The scalar selection metrics a sweep can optimize / rank by. | | [`DFLT_CALIB_SENSITIVITY_WEIGHT`](_autosummary/ir.eval.html.md#ir.eval.DFLT_CALIB_SENSITIVITY_WEIGHT) | Weight on *sensitivity* (catch true positives) vs *specificity* (reject out-of-scope queries) when [`calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) picks the abstention floor: `objective = w·sensitivity + (1−w)·specificity`. | | [`RELEVANCE_LEVELS`](_autosummary/ir.eval.html.md#ir.eval.RELEVANCE_LEVELS) | Graded relevance levels for the package-relevance harness, weakest -> strongest. | | [`LEVEL_GAINS`](_autosummary/ir.eval.html.md#ir.eval.LEVEL_GAINS) | Graded relevance gains per level — fed straight to `ef.evaluation`'s graded nDCG. | ### Functions | [`as_doc_retriever`](_autosummary/ir.eval.html.md#ir.eval.as_doc_retriever)(corpus, \*[, mode, surfaces]) | Adapt an `ir` corpus to `ef`'s retriever contract. | |-----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`calibrate_min_score`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score)(corpus, cases, \*[, ...]) | Calibrate the absolute abstention `min_score` floor for one `mode`. | | [`compare_indexings`](_autosummary/ir.eval.html.md#ir.eval.compare_indexings)(corpora, cases, \*, themes, ...) | A/B (or N-way) regression gate over indexing / embedder configurations. | | [`corpus_artifact_ids`](_autosummary/ir.eval.html.md#ir.eval.corpus_artifact_ids)(corpus) | The set of `artifact_id`s present in a built corpus. | | [`derive_named_sets`](_autosummary/ir.eval.html.md#ir.eval.derive_named_sets)(cases, theme, \*, ...[, gains]) | Derive the [`NamedSets`](_autosummary/ir.eval.html.md#ir.eval.NamedSets) for `theme` deterministically from labels. | | [`distractor_curve_from_cases`](_autosummary/ir.eval.html.md#ir.eval.distractor_curve_from_cases)(source, cases, ...) | Convenience: a distractor curve over a source's scope from single-gold cases. | | [`distractor_robustness_curve`](_autosummary/ir.eval.html.md#ir.eval.distractor_robustness_curve)(scope, probes, \*) | Retrieval accuracy as the catalog grows — the needle-in-a-haystack curve. | | [`evaluate_discovery`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery)(corpus, cases, \*[, ...]) | Score a corpus against `cases` — metrics, failure taxonomy, abstention. | | [`evaluate_named_sets`](_autosummary/ir.eval.html.md#ir.eval.evaluate_named_sets)(corpus, named_sets[, ...]) | Score one theme's named distractor / hard-positive sets with a single probe. | | [`evaluate_selection`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection)(corpus, cases, \*[, ...]) | Score a selector against `cases` — selection quality, isolated. | | [`fp_rate_on_distractors`](_autosummary/ir.eval.html.md#ir.eval.fp_rate_on_distractors)(ranking, ...) | Fraction of named distractors that appear in the top-`k` of `ranking`. | | [`level_histogram`](_autosummary/ir.eval.html.md#ir.eval.level_histogram)(cases, theme) | Count of artifacts at each level for `theme` (every level represented). | | [`load_calibration`](_autosummary/ir.eval.html.md#ir.eval.load_calibration)(corpus, mode, \*[, rerank, ...]) | Load a persisted [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) for `(corpus, mode)`. | | [`load_cases`](_autosummary/ir.eval.html.md#ir.eval.load_cases)(path) | Read [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s from a JSONL file (skips a `__meta__` header). | | [`load_package_cases`](_autosummary/ir.eval.html.md#ir.eval.load_package_cases)(path) | Read [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)s from JSONL (skips a `__meta__` header). | | [`read_package_meta`](_autosummary/ir.eval.html.md#ir.eval.read_package_meta)(path) | Return the `__meta__` header dict from a package-cases JSONL (`{}` if absent). | | [`recall_on_hard_positives`](_autosummary/ir.eval.html.md#ir.eval.recall_on_hard_positives)(ranking, ...) | Fraction of named hard-positives in the top-`k` (duplicate ids counted once). | | [`retrieval_report`](_autosummary/ir.eval.html.md#ir.eval.retrieval_report)(corpus, cases, \*[, ...]) | Score a corpus's retrieval against the cases — the pure-`ef` path. | | [`save_cases`](_autosummary/ir.eval.html.md#ir.eval.save_cases)(cases, path, \*[, meta]) | Write `cases` to a JSONL file (one case per line). | | [`save_package_cases`](_autosummary/ir.eval.html.md#ir.eval.save_package_cases)(cases, path, \*[, meta]) | Write [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)s to JSONL (mirrors [`save_cases()`](_autosummary/ir.eval.html.md#ir.eval.save_cases)). | | [`sweep_selector`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector)(corpus, cases, \*[, strategy, ...]) | Tune the selector: score a grid of commit knobs against the cases. | | [`to_graded_qrels`](_autosummary/ir.eval.html.md#ir.eval.to_graded_qrels)(cases, theme, \*[, probe, ...]) | Build `ef`'s **graded** `(queries, qrels)` for one `theme`. | | [`to_qrels`](_autosummary/ir.eval.html.md#ir.eval.to_qrels)(cases, \*[, grade]) | Build `ef`'s `(queries, qrels)` from the gold-bearing cases. | | [`validate_cases`](_autosummary/ir.eval.html.md#ir.eval.validate_cases)(corpus, cases) | Gold ids a case references that are absent from the corpus. | ### Classes | [`ComparisonReport`](_autosummary/ir.eval.html.md#ir.eval.ComparisonReport)(k, themes, labels, ...) | The outcome of [`compare_indexings()`](_autosummary/ir.eval.html.md#ir.eval.compare_indexings) — an N-way indexing/embedder bake-off. | |----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| | [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)(query[, gold, corpus, ...]) | One discovery probe: an intent and the artifact(s) that should answer it. | | [`DiscoveryReport`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryReport)(retrieval, failure_classes, ...) | The outcome of [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery). | | [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration)(corpus_name, mode, k, ...) | A calibrated absolute abstention floor for one `(corpus, mode)`. | | [`NamedSetReport`](_autosummary/ir.eval.html.md#ir.eval.NamedSetReport)(theme, k, fp_rate, ...[, ...]) | The outcome of [`evaluate_named_sets()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_named_sets) for one theme. | | [`NamedSets`](_autosummary/ir.eval.html.md#ir.eval.NamedSets)(theme[, distractors, hard_positives]) | The two named diagnostic sets for one theme. | | [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)(artifact_id[, labels, ...]) | One artifact's graded relevance to one or more themes. | | [`SelectionGridPoint`](_autosummary/ir.eval.html.md#ir.eval.SelectionGridPoint)(max_k, rel, min_score, report) | One cell of a selector sweep: a parameter setting and the report it scored. | | [`SelectionReport`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport)(selection_precision, ...[, ...]) | The outcome of [`evaluate_selection()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection) — selection quality, isolated. | | [`SelectionSweep`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep)(points, objective, strategy, ...) | The outcome of [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) — a grid of selector settings scored. | ### *class* ir.eval.ComparisonReport(k, themes, labels, baseline, metrics, deltas) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of [`compare_indexings()`](_autosummary/ir.eval.html.md#ir.eval.compare_indexings) — an N-way indexing/embedder bake-off. #### k the rank cutoff all `@k` metrics use. ### themes / labels the themes scored and the corpus labels compared. #### baseline the label every [`regressions()`](_autosummary/ir.eval.html.md#ir.eval.ComparisonReport.regressions) check compares against (the first entry of `corpora`). #### metrics `{label: {theme: {"ndcg", "fp_rate"?, "hard_positive_recall"?}}}`. #### deltas `{theme: {artifact_id: {"role", "by_label": {label: {"rank", "score"}}}}}` for every named FP/FN id — the per-package effect. #### regressions(, threshold=0, baseline=None) Named packages that got WORSE than `baseline` — drives a pytest gate. For a `hard_positive`, worse = its rank dropped (grew larger) by more than `threshold` positions. For a `distractor`, worse = its rank rose (grew smaller, more prominent) by more than `threshold`. A package absent from a ranking is treated as rank `inf` (worst), so a vanished hard-positive and a newly-appearing distractor both register. Returns one dict per regressing `(theme, id, label)`. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] #### to_dict() JSON-serializable form — the qh / HTTP surface. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.eval.DFLT_CALIB_SENSITIVITY_WEIGHT *= 0.5* Weight on *sensitivity* (catch true positives) vs *specificity* (reject out-of-scope queries) when [`calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) picks the abstention floor: `objective = w·sensitivity + (1−w)·specificity`. `0.5` is balanced (its argmax matches Youden’s J / balanced accuracy). Lower it to lean *precision* — abstain more readily — per ir_01 §3’s stance that a padded commit is the costlier failure for an agent surface than a dropped gold. ### ir.eval.DFLT_CURVE_MODE *= 'dense'* Defaults for [`distractor_robustness_curve()`](_autosummary/ir.eval.html.md#ir.eval.distractor_robustness_curve) — deliberately *not* `DFLT_MODE`: the curve builds throwaway in-memory sub-corpora, so it favours the fast, always-offline dense + hashing path. Pass `mode="hybrid"` / `embedder="default"` to measure the production configuration instead. ### ir.eval.DFLT_DISTRACTOR_SIZES *= (1, 4, 8, 16, 32, 64, 128)* Catalog sizes swept by [`distractor_robustness_curve()`](_autosummary/ir.eval.html.md#ir.eval.distractor_robustness_curve) (RAG-MCP-style). ### ir.eval.DFLT_K_VALUES *= (1, 5, 10)* Rank cutoffs reported by default. ### ir.eval.DFLT_METRICS *= ('ndcg', 'recall', 'precision', 'mrr', 'map')* Metrics reported by default — the `ef.evaluation` retrieval metric names. ### ir.eval.DFLT_MODE *= 'hybrid'* Default ranking mode for the eval adapter (hybrid is `ir`’s strongest). ### ir.eval.DFLT_SWEEP_MAX_K *= (1, 2, 3, 5, 8)* `max_k` values swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) (the commit-size cap). ### ir.eval.DFLT_SWEEP_MIN_SCORE *= (None,)* `min_score` floors swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector). Default `(None,)` — the absolute floor is mode-specific (cosine / RRF / BM25 scales differ), so it is left off unless the caller supplies calibrated values to sweep. ### ir.eval.DFLT_SWEEP_OBJECTIVE *= 'selection_f1'* Default objective [`SelectionSweep.best()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.best) optimizes — the standard precision/recall balance. `mean_selected_size` is the only metric minimized; every other is maximized (see `_SWEEP_MINIMIZE`). ### ir.eval.DFLT_SWEEP_REL *= (0.4, 0.5, 0.6, 0.7, 0.8, 0.9)* `rel` values swept by [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) (relative-to-top keep band). ### *class* ir.eval.DiscoveryCase(query, gold=(), corpus=None, source_id=None, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One discovery probe: an intent and the artifact(s) that should answer it. #### query the user intent / search query. #### gold the `artifact_id`s that correctly answer `query` (one for a single-answer case, several when capabilities overlap). An **empty** tuple marks an *abstention* case — no artifact applies. #### corpus optional name of the corpus the case targets (provenance / for multi-corpus case files). #### source_id optional `artifact_id` the query was generated from (back-translation provenance). #### metadata free-form per-case metadata (difficulty, generator, …). #### *classmethod* from_dict(d) Inverse of [`to_dict()`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase.to_dict); `gold` may be a string or a list. * **Return type:** [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase) #### *property* gold_is_none *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* True when no artifact should match — an abstention case. #### to_dict() JSON-serializable form (omitting empty optional fields). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.DiscoveryReport(retrieval, failure_classes, n_cases, n_gold, n_abstention, primary_k=10, abstention_accuracy=None, mode='hybrid', vd_degraded=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery). Bundles the standard retrieval metrics with the capability-discovery extras: #### retrieval the `ef.evaluation.RetrievalEvalReport` over the gold-bearing cases. #### failure_classes counts per failure class (`hit_rank_1` / `surfaced_low_rank` / `retrieval_miss` for gold cases; `abstention_ok` / `false_action` / `abstention_unscored` for abstention cases). ### n_cases / n_gold / n_abstention case counts. #### primary_k the cutoff `primary` reports at (default 10). #### abstention_accuracy accuracy on the abstention slice when an `abstain_threshold` was supplied, else `None`. #### mode the ranking mode the report was produced with — recorded so a number is never mistaken for a different configuration’s. #### vd_degraded True when `mode` was `lexical` / `hybrid` but `vd` was unavailable, so ranking silently fell back to dense (the scores are *not* the requested configuration’s). #### *property* primary *: [float](https://docs.python.org/3/builtins/functions.html#float) | [None](https://docs.python.org/3/builtins/constants.html#None)* The headline number — mean NDCG@\`\`primary_k\`\` (`None` if not computed). ### ir.eval.LEVEL_GAINS *= {'core': 3.0, 'none': 0.0, 'strong': 2.0, 'tangential': 0.0, 'uses-tools': 1.0}* Graded relevance gains per level — fed straight to `ef.evaluation`’s graded nDCG. `uses-tools` is a *weak* positive (a package that merely uses the relevant tooling); `tangential`/`none` are non-relevant (gain `0`). These gains are the knob that lets nDCG rank a core-first ordering strictly above a uses-tools-first one at identical recall. ### *class* ir.eval.MinScoreCalibration(corpus_name, mode, k, min_score, sensitivity, specificity, youden_j, balanced_accuracy, separable, sensitivity_weight, n_positive, n_abstention, n_retrieval_miss, positive_scores, abstention_scores, grid, reason, embedder_id=None, strategy='conservative', vd_degraded=False, rerank=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A calibrated absolute abstention floor for one `(corpus, mode)`. Produced by [`calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score). `min_score` is the floor to pass to `ir.select.select()` / [`ir.discover()`](_autosummary/ir.html.md#ir.discover) (`discover(min_score="auto")` loads it): the conservative selector abstains when even the top hit scores below it. `None` means no floor could be calibrated (one of the two query classes was missing — see [`reason`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration.reason)); the selector then never abstains by absolute score, exactly as before. The quality numbers treat `top_score ≥ min_score → commit` as a binary classifier over in-scope (label *commit*) vs out-of-scope (label *abstain*) queries: #### min_score the calibrated floor (`None` if not calibratable). #### sensitivity TPR — fraction of in-scope queries kept (`top ≥ floor`). #### specificity TNR — fraction of out-of-scope queries abstained (`top < floor`). #### youden_j `sensitivity + specificity − 1` (0 = no separation, 1 = perfect). #### balanced_accuracy `(sensitivity + specificity) / 2`. #### separable whether the two score distributions are perfectly separable (some floor reaches `youden_j == 1`). #### sensitivity_weight the `w` used to pick the floor (see [`DFLT_CALIB_SENSITIVITY_WEIGHT`](_autosummary/ir.eval.html.md#ir.eval.DFLT_CALIB_SENSITIVITY_WEIGHT)). #### n_positive in-scope cases used (gold-bearing **and** gold-retrieved). #### n_abstention out-of-scope (empty-gold) cases used. #### n_retrieval_miss gold-bearing cases whose gold never reached the `k` candidates — excluded (a *retrieval* failure, not an abstention signal), mirroring [`evaluate_selection()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection)’s conditioning. ### positive_scores / abstention_scores `{min, median, max}` summaries of each class’s top-score distribution (the auditable evidence the floor sits between them). #### grid every evaluated floor, `{min_score, sensitivity, specificity, youden_j, objective}` — the full sweep behind the choice. #### reason why `min_score` is what it is (`"calibrated"`, `"no_positive_cases"`, `"no_abstention_cases"`, `"no_cases"`). ### corpus_name / mode / k / embedder_id the configuration scored — and the `embedder_id` stamp lets a consumer detect a stale floor after a rebuild with a different embedder. #### vd_degraded True when `mode` was `lexical` / `hybrid` but `vd` was unavailable, so ranking fell back to dense and the floor is on the *dense* scale, not the requested mode’s — recalibrate once `vd` is installed. #### *classmethod* from_dict(d) Inverse of [`to_dict()`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration.to_dict) (tolerant of a persisted record’s defaults). * **Return type:** [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) #### save(corpus) Persist this calibration on `corpus` (keyed by mode); returns self. Stored in the corpus’s `calibration` view (not its build `config`), so it survives reopen and is loaded by `discover(min_score="auto")`. The floor is machine-local (derived from a live corpus + embedder) and is never committed to source control. * **Return type:** [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) #### to_dict() JSON-serializable form (for persistence and the qh / HTTP surface). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.NamedSetReport(theme, k, fp_rate, hard_positive_recall, distractors_seen=(), hard_positives_missed=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of [`evaluate_named_sets()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_named_sets) for one theme. ### theme / k the theme and the rank cutoff scored. #### fp_rate distractor false-positive rate at `k` (lower is better). #### hard_positive_recall recall over the hard-positive set at `k` (higher is better). #### distractors_seen the distractor ids actually in the top-`k` — so the `fp_rate` is auditable back to specific packages. #### hard_positives_missed the hard-positive ids NOT in the top-`k`. #### to_dict() JSON-serializable form. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.NamedSets(theme, distractors=(), hard_positives=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The two named diagnostic sets for one theme. #### theme the theme these sets diagnose. #### distractors ids that are NOT relevant (`none`/`tangential`) yet rank prominently — every appearance in the top-`k` is a false positive. #### hard_positives ids that ARE relevant (`core`/`strong`) but hard to rank (e.g. thin-description packages) — recall on these is the headline gap a fix must close. #### *classmethod* from_dict(d) Inverse of [`to_dict()`](_autosummary/ir.eval.html.md#ir.eval.NamedSets.to_dict). * **Return type:** [`NamedSets`](_autosummary/ir.eval.html.md#ir.eval.NamedSets) #### to_dict() JSON-serializable form. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.PackageRelevanceCase(artifact_id, labels=, evidence=, observed=, thin_description=False, is_distractor=, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One artifact’s graded relevance to one or more themes. Unlike [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase) (one query -> gold ids, flat-binary gold), a case here is one **artifact** carrying its graded label per theme. The per-theme *probe text* (the query each theme is scored with) is **not** on the case — it lives in the JSONL `__meta__` header (see [`save_package_cases()`](_autosummary/ir.eval.html.md#ir.eval.save_package_cases)), alongside the corpus signature ([`ir.eval_gen.corpus_signature()`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.corpus_signature)), so a frozen label set pins to the corpus snapshot it was judged against. #### artifact_id the package id (matches the corpus’ `artifact_id`). #### labels `{theme: level}` with each level in [`RELEVANCE_LEVELS`](_autosummary/ir.eval.html.md#ir.eval.RELEVANCE_LEVELS). #### evidence `{theme: short reason}` — the human-auditable *why*. #### observed `{theme: score}` the ranking gave this artifact in the frozen run (optional; used to *derive* the named distractor set). #### thin_description True when the package had an empty/near-empty pyproject description (used to derive the hard-positive set). #### is_distractor optional cached `{theme: bool}`; when absent it is derived (see [`derive_named_sets()`](_autosummary/ir.eval.html.md#ir.eval.derive_named_sets)). #### metadata free-form per-case metadata. #### *classmethod* from_dict(d) Inverse of [`to_dict()`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase.to_dict); rejects any level outside [`RELEVANCE_LEVELS`](_autosummary/ir.eval.html.md#ir.eval.RELEVANCE_LEVELS). * **Return type:** [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase) #### gain(theme, , gains={'core': 3.0, 'none': 0.0, 'strong': 2.0, 'tangential': 0.0, 'uses-tools': 1.0}) The graded gain for `theme` under `gains` (default [`LEVEL_GAINS`](_autosummary/ir.eval.html.md#ir.eval.LEVEL_GAINS)). A level missing from `gains` (e.g. a partial caller-supplied mapping) defaults to `0.0` — the non-relevant convention — rather than raising. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### level(theme) This artifact’s graded level for `theme` (`"none"` if unlabeled). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) #### to_dict() JSON-serializable form (omitting empty optional fields). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.eval.RELEVANCE_LEVELS *= ('none', 'tangential', 'uses-tools', 'strong', 'core')* Graded relevance levels for the package-relevance harness, weakest -> strongest. The SSOT consumed by [`LEVEL_GAINS`](_autosummary/ir.eval.html.md#ir.eval.LEVEL_GAINS), [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase), and [`to_graded_qrels()`](_autosummary/ir.eval.html.md#ir.eval.to_graded_qrels). ### ir.eval.SWEEP_METRICS *= ('selection_f1', 'selection_precision', 'selection_recall', 'conditional_commit_rate', 'abstention_accuracy', 'mean_selected_size')* The scalar selection metrics a sweep can optimize / rank by. ### *class* ir.eval.SelectionGridPoint(max_k, rel, min_score, report) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One cell of a selector sweep: a parameter setting and the report it scored. ### max_k / rel / min_score the swept selection parameters at this cell. #### report the [`SelectionReport`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport) for those parameters. #### *property* params *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]* The swept parameters as a plain dict (`max_k` / `rel` / `min_score`). #### to_dict() JSON-serializable form: the params plus the full report dict. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.SelectionReport(selection_precision, selection_recall, selection_f1, conditional_commit_rate, n_gold_retrieved, abstention_accuracy, mean_selected_size, n_cases, n_gold, n_abstention, strategy='conservative', mode='hybrid', k=10) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of [`evaluate_selection()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection) — selection quality, isolated. Where [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery) scores the *ranking*, this scores the *commit*: given the same `k` candidates, what subset did the selector keep? The headline is [`conditional_commit_rate`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport.conditional_commit_rate) — accuracy of the selection decision **conditioned on retrieval having surfaced the gold** — which is the one number that separates a selection failure from a retrieval failure. Every selection-quality metric shares that conditioning: precision, recall and F1 are computed **only over the** [`n_gold_retrieved`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport.n_gold_retrieved) **cases** whose gold reached the `k` candidates, so a retrieval miss is never charged to the selector (it is [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery)’s to report). #### selection_precision mean over committed gold-retrieved cases of `|selected ∩ gold| / |selected|` (`None` if nothing was ever committed) — how clean the committed sets are. #### selection_recall mean over gold-retrieved cases of `|selected ∩ gold| / |gold ∩ retrieved|` — of the gold the selector was actually shown, how much it kept (denominator is the *retrievable* gold, so retrieval misses do not depress it). #### selection_f1 mean per-case F1 over gold-retrieved cases (an empty commit scores 0). #### conditional_commit_rate among gold cases whose gold was in the `k` candidates, the fraction where the selector committed to ≥1 gold — the selection decision isolated from retrieval recall (`None` if retrieval never surfaced any gold). #### n_gold_retrieved the shared denominator of every selection-quality metric above (cases whose gold reached the `k` candidates). #### abstention_accuracy fraction of abstention cases (empty gold) the selector correctly committed nothing to (`None` if none). #### mean_selected_size mean committed-set size over gold-retrieved cases. ### n_cases / n_gold / n_abstention case counts. ### strategy / mode / k the configuration scored (so a number is never mistaken for a different setup’s). #### to_dict() JSON-serializable form (for the qh / HTTP surface). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.eval.SelectionSweep(points, objective, strategy, mode, k, n_cases, n_gold, n_abstention, n_gold_retrieved) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of [`sweep_selector()`](_autosummary/ir.eval.html.md#ir.eval.sweep_selector) — a grid of selector settings scored. Every point shares the same cases, retrieval `mode` / candidate depth `k` and `strategy`; they differ only in the swept selection knobs (`max_k`, `rel`, `min_score`). [`best()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.best) picks a single setting by an objective; [`frontier()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.frontier) returns the precision/recall Pareto front for the honest trade-off view; [`table()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.table) renders the whole grid. #### points one [`SelectionGridPoint`](_autosummary/ir.eval.html.md#ir.eval.SelectionGridPoint) per grid cell (grid order). #### objective the default metric [`best()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.best) optimizes. ### strategy / mode / k the shared configuration the grid was scored under. ### n_cases / n_gold / n_abstention / n_gold_retrieved case counts (echoed from the reports; selection metrics share the `n_gold_retrieved` denominator, so it is surfaced here too). #### best(metric=None, , minimize=None) The grid point that optimizes `metric` (default [`objective`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.objective)). Ties are broken toward the *cheaper, tighter* commit — smaller `mean_selected_size`, then smaller `max_k`, then larger `rel` — encoding the capability-discovery stance that fewer, higher-precision commits beat more (ir_01 §3). `minimize` is inferred from the metric (only `mean_selected_size` minimizes) but can be forced. * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the sweep is empty or `metric` is unknown. * **Return type:** [`SelectionGridPoint`](_autosummary/ir.eval.html.md#ir.eval.SelectionGridPoint) #### frontier(, x='selection_recall', y='selection_precision') The Pareto-optimal points trading `x` off against `y` (both maximized). A point is on the frontier when no other point is at least as good on both axes and strictly better on one. `mean_selected_size` on an axis is read as “smaller is better” automatically. Returned sorted by `x` ascending. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SelectionGridPoint`](_autosummary/ir.eval.html.md#ir.eval.SelectionGridPoint)] #### table(, sort_by=None) A human-readable grid table, one row per setting (best objective first). `sort_by` defaults to [`objective`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.objective). `min_score` is shown only when any cell sets one (an all-`None` column is elided as noise). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) #### to_dict() JSON-serializable form: the grid, the best point, and the config. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.eval.as_doc_retriever(corpus, , mode='hybrid', surfaces=None, \*\*search_kw) Adapt an `ir` corpus to `ef`’s retriever contract. Returns a callable `retriever(query, *, limit=10) -> [artifact_id, …]` — bare doc-id strings, best-ranked first, one per artifact. That is exactly what `ef.evaluation.evaluate_retrieval()` consumes (it maps a bare string straight to a document id), so the returned callable can be handed to `ef` with no further adaptation. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – an [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) or a registered corpus *name*. * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode — `"dense"` / `"lexical"` / `"hybrid"`. * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict to these surface kinds (e.g. `{"description"}`). * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword (`filter`, `rrf_k`, `rerank`, `bm25`, …). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.eval.calibrate_min_score(corpus, cases, , mode='hybrid', k=10, surfaces=None, sensitivity_weight=0.5, floor_grid=None, persist=False, \*\*search_kw) Calibrate the absolute abstention `min_score` floor for one `mode`. Retrieves `k` candidates per case **once** (reusing [`evaluate_selection()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection)’s retrieval pass), then treats floor-picking as a 1-D separation between the *in-scope* top scores (gold-bearing cases whose gold reached the candidates) and the *out-of-scope* top scores (empty-gold abstention cases). The floor that best separates them — by `w·sensitivity + (1−w)·specificity` — is the calibrated `min_score`. `cases` must contain **both** gold-bearing and abstention cases; with only one class the floor is undefined (`min_score=None`, `reason` records which class was missing). Generate abstention cases with [`ir.eval_gen`](_autosummary/ir.eval_gen.html.md#module-ir.eval_gen) (its `abstention` slice) — this module does not synthesize them. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – an [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) or a registered corpus *name*. * **cases** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)]) – mixed [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s (gold-bearing + abstention). * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode to calibrate (a floor is mode-specific). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – candidate depth retrieved before selection (match your `discover` k). * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict retrieval to these surface kinds. * **sensitivity_weight** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – `w` in the pick objective (see [`DFLT_CALIB_SENSITIVITY_WEIGHT`](_autosummary/ir.eval.html.md#ir.eval.DFLT_CALIB_SENSITIVITY_WEIGHT)); `0.5` is balanced. * **floor_grid** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`float`](https://docs.python.org/3/builtins/functions.html#float)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – candidate floors to evaluate; default derives them from the observed top scores (midpoints between adjacent values + the commit-all / abstain-all extremes), which is exact for a 1-D split. * **persist** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when True, [`MinScoreCalibration.save()`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration.save) the result on the corpus so `discover(min_score="auto")` will load it. * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword. * **Return type:** [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) * **Returns:** a [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration). ### ir.eval.compare_indexings(corpora, cases, , themes, probes, k=20, named_sets=None, mode='hybrid', rank_depth=1000, gains={'core': 3.0, 'none': 0.0, 'strong': 2.0, 'tangential': 0.0, 'uses-tools': 1.0}, surfaces=None, search_kw_by_label=None, \*\*search_kw) A/B (or N-way) regression gate over indexing / embedder configurations. `corpora` maps a label -> a built corpus (an `ef` instruction-tuned-embedder corpus or an `ir` deps-as-text corpus is just another entry — the harness is embedder-agnostic). For each `(label, theme)` it computes graded nDCG@k (via [`to_graded_qrels()`](_autosummary/ir.eval.html.md#ir.eval.to_graded_qrels) + `ef.evaluation.ndcg_at_k`), the named-set FP-rate / hard-positive recall@k (when `named_sets` is given), and the rank+score of every named FP/FN id — so a change’s effect is quantified *per package*. `probes` supplies the per-theme query text (normally loaded from the JSONL `__meta__` via [`read_package_meta()`](_autosummary/ir.eval.html.md#ir.eval.read_package_meta)). The first label is the baseline that [`ComparisonReport.regressions()`](_autosummary/ir.eval.html.md#ir.eval.ComparisonReport.regressions) compares against. Each ranking is fetched once to depth `rank_depth` (deep enough that even a buried named id gets a real rank) and sliced at `k` for the `@k` metrics. `search_kw_by_label` overrides search settings per label, on top of the shared `mode` / `surfaces` / `**search_kw`: map one corpus under several labels to compare *search* configurations rather than indexings, e.g. `{"deps x2": {"surface_weights": {"deps": 2.0}}}`. Any [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword except `k` (the depth is `rank_depth`) and `per_artifact` (rankings are per artifact). * **Return type:** [`ComparisonReport`](_autosummary/ir.eval.html.md#ir.eval.ComparisonReport) ### ir.eval.corpus_artifact_ids(corpus) The set of `artifact_id`s present in a built corpus. * **Return type:** [`set`](https://docs.python.org/3/builtins/stdtypes.html#set)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.eval.derive_named_sets(cases, theme, , observed_floor, gains={'core': 3.0, 'none': 0.0, 'strong': 2.0, 'tangential': 0.0, 'uses-tools': 1.0}) Derive the [`NamedSets`](_autosummary/ir.eval.html.md#ir.eval.NamedSets) for `theme` deterministically from labels. `distractors` = non-relevant (`none`/`tangential`) packages whose frozen `observed` score for the theme is at or above `observed_floor` (they ranked high enough to pollute the top-`k`) — unless a case carries an explicit cached `is_distractor[theme]`, which then overrides the observed-floor rule. `hard_positives` = relevant (`core`/`strong`) packages flagged `thin_description` (the ones a description+README index struggles to surface). Both lists are de-duplicated and sorted, so the result is a stable, committable artifact. * **Return type:** [`NamedSets`](_autosummary/ir.eval.html.md#ir.eval.NamedSets) ### ir.eval.distractor_curve_from_cases(source, cases, \*\*kwargs) Convenience: a distractor curve over a source’s scope from single-gold cases. Probes are the single-gold cases (`(query, gold[0])`); the source’s `scope` is the distractor pool and its `indexing_strategy` is reused unless a `strategy` is passed in `kwargs`. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`float`](https://docs.python.org/3/builtins/functions.html#float)] ### ir.eval.distractor_robustness_curve(scope, probes, , strategy=None, embedder='light', mode='dense', sizes=(1, 4, 8, 16, 32, 64, 128), trials=20, seed=0, k=1) Retrieval accuracy as the catalog grows — the needle-in-a-haystack curve. For each catalog size `N` and each `(query, gold_id)` probe, build many `N`-artifact sub-corpora (the gold artifact plus `N−1` distractors sampled from `scope`) and measure how often the gold artifact lands in the top `k`. The accuracy at each `N` is averaged over the probes and over `trials` random distractor draws. `N=1` is the trivial anchor (gold alone) and is always 1.0; a steep decline as `N` grows is the signature of retrieval that does not discriminate. Sub-corpora are built in-memory with the `"light"` (numpy-only) embedder by default, so the curve is fast and offline; pass `embedder="default"` (and `mode="hybrid"`) to measure the production configuration. Sizes larger than the corpus are **capped** at `len(scope)` (the largest catalog the pool can fill) and de-duplicated, so the returned keys are the catalog sizes actually built — the x-axis never claims an `N` larger than the corpus could supply. * **Parameters:** * **scope** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the full `{artifact_id: raw}` pool to draw the catalog from. * **probes** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – `(query, gold_id)` pairs; `gold_id` must be a key of `scope`. * **strategy** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) to index with (default [`WholeText`](_autosummary/ir.strategy.html.md#ir.strategy.WholeText)). * **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder spec for the sub-corpora. * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode for the probe searches. * **sizes** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – the catalog sizes to sweep (capped at `len(scope)`, deduped). * **trials** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – random distractor draws averaged per size (a size that consumes the whole pool has no sampling freedom, so it runs a single trial). * **seed** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – base RNG seed (reproducible; varied per size). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – the top-`k` the gold must reach to count as a hit. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`float`](https://docs.python.org/3/builtins/functions.html#float)] * **Returns:** `{N: accuracy}` for each *achievable* `N` (the requested sizes capped at `len(scope)` and de-duplicated). ### ir.eval.evaluate_discovery(corpus, cases, , k_values=(1, 5, 10), primary_k=10, metrics=('ndcg', 'recall', 'precision', 'mrr', 'map'), mode='hybrid', surfaces=None, abstain_threshold=None, \*\*search_kw) Score a corpus against `cases` — metrics, failure taxonomy, abstention. Runs retrieval **once per case** and, from each ranking, computes the `ef.evaluation` metric primitives (so the numbers match the pure-`ef` path) *and* the capability-discovery extras in one pass. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – an [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) or a registered corpus *name*. * **k_values** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – rank cutoffs to report (`primary_k` is always included). * **primary_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – the cutoff the headline `primary` reports at. * **metrics** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – which `ef` metrics to compute (`ndcg` / `recall` / `precision` / `mrr` / `map`). * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode passed to [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict to these surface kinds. * **abstain_threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – if given, an abstention case is scored *correct* when the top hit’s score is below this threshold (a retrieval-side proxy for “no artifact applies”); leave `None` to only count abstention cases without scoring them. * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword. * **Return type:** [`DiscoveryReport`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryReport) * **Returns:** a [`DiscoveryReport`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryReport). If no case carries gold (an empty or all-abstention set) the retrieval metrics are omitted — `report.primary` is `None` — but the taxonomy and abstention counts are still returned (unlike [`retrieval_report()`](_autosummary/ir.eval.html.md#ir.eval.retrieval_report), which raises `ValueError` on that input). ### ir.eval.evaluate_named_sets(corpus, named_sets, theme=None, , probe, mode='hybrid', k=10, surfaces=None, \*\*search_kw) Score one theme’s named distractor / hard-positive sets with a single probe. Runs the theme `probe` once, takes the per-artifact ranking, and reports the distractor false-positive rate and hard-positive recall at `k` — plus the distractor ids actually seen in the top-`k` and the hard-positives missed, so a number is always auditable back to specific packages. `theme` defaults to `named_sets.theme` (the only thing it labels); pass it only to override. * **Return type:** [`NamedSetReport`](_autosummary/ir.eval.html.md#ir.eval.NamedSetReport) ### ir.eval.evaluate_selection(corpus, cases, , strategy='conservative', mode='hybrid', k=10, surfaces=None, max_k=3, rel=0.9, gap_ratio=0.5, min_score=None, \*\*search_kw) Score a selector against `cases` — selection quality, isolated. Retrieves `k` candidates per case (the window the selector sees), commits with `ir.select.select()`, and scores the *commit*. The key number, [`SelectionReport.conditional_commit_rate`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport.conditional_commit_rate), conditions on retrieval having surfaced the gold among those `k` — so a low value means the *selector* dropped a gold it was shown, not that retrieval missed it. `k` is the candidate window; hold it equal to the `k` used with [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery) to compare the two stages on the same footing. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – an [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) or a registered corpus *name*. * **cases** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)]) – the [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s (gold-bearing and/or abstention). * **strategy** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – selection strategy (see `ir.select.select()`). * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode for retrieval. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – candidate depth retrieved before selection. * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict retrieval to these surface kinds. * **max_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – selection parameters. * **rel** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – selection parameters. * **gap_ratio** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – selection parameters. * **min_score** ([`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – selection parameters. * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword. * **Return type:** [`SelectionReport`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport) * **Returns:** a [`SelectionReport`](_autosummary/ir.eval.html.md#ir.eval.SelectionReport). ### ir.eval.fp_rate_on_distractors(ranking, distractor_ids, , k) Fraction of named distractors that appear in the top-`k` of `ranking`. A distractor surfacing in the committed top-`k` is a false positive; this is the rate over the named set (`0.0` when the set is empty). Duplicate ids are counted once. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ir.eval.level_histogram(cases, theme) Count of artifacts at each level for `theme` (every level represented). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`int`](https://docs.python.org/3/builtins/functions.html#int)] ### ir.eval.load_calibration(corpus, mode, , rerank=None, rerank_text='surface', rerank_max_chars=2000) Load a persisted [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) for `(corpus, mode)`. `rerank` / `rerank_text` select the floor calibrated for a reranked search (same values as [`ir.search()`](_autosummary/ir.html.md#ir.search)); the default is the first-stage floor. Returns `None` when no calibration has been stored for that mode. Does not validate the embedder stamp — that staleness check lives at the `discover(min_score="auto")` boundary, which knows the live corpus’s embedder id. * **Return type:** [`MinScoreCalibration`](_autosummary/ir.eval.html.md#ir.eval.MinScoreCalibration) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.eval.load_cases(path) Read [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s from a JSONL file (skips a `__meta__` header). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)] ### ir.eval.load_package_cases(path) Read [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)s from JSONL (skips a `__meta__` header). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)] ### ir.eval.read_package_meta(path) Return the `__meta__` header dict from a package-cases JSONL (`{}` if absent). The header carries the per-theme probe text (`probes`) and the corpus signature the labels were frozen against — everything [`compare_indexings()`](_autosummary/ir.eval.html.md#ir.eval.compare_indexings) needs that is not on a per-artifact case. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.eval.recall_on_hard_positives(ranking, hard_positive_ids, , k) Fraction of named hard-positives in the top-`k` (duplicate ids counted once). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ir.eval.retrieval_report(corpus, cases, , k_values=(1, 5, 10), metrics=('ndcg', 'recall', 'precision', 'mrr', 'map'), mode='hybrid', surfaces=None, limit=None, \*\*search_kw) Score a corpus’s retrieval against the cases — the pure-`ef` path. A thin wrapper that builds the retriever adapter and hands it, with the cases’ `(queries, qrels)`, to `ef.evaluation.evaluate_retrieval()`. Returns its `ef.evaluation.RetrievalEvalReport` (`.primary` is mean [NDCG@10](mailto:NDCG@10)). Use this to A/B configurations — dense vs hybrid, with vs without a reranker — on the standard BEIR/MTEB metrics. For the richer report (taxonomy + abstention) use [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if no case carries gold (`ef` requires at least one positively-judged query). [`evaluate_discovery()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_discovery) instead tolerates an all-abstention set — the two paths differ only on that degenerate input. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ir.eval.save_cases(cases, path, , meta=None) Write `cases` to a JSONL file (one case per line). An optional `meta` mapping is written as a leading `{"__meta__": …}` line — the natural home for a corpus-version anchor that pins the cases to the corpus snapshot they were generated against. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.eval.save_package_cases(cases, path, , meta=None) Write [`PackageRelevanceCase`](_autosummary/ir.eval.html.md#ir.eval.PackageRelevanceCase)s to JSONL (mirrors [`save_cases()`](_autosummary/ir.eval.html.md#ir.eval.save_cases)). `meta` is written as a leading `{"__meta__": …}` line — the home for the per-theme probe text (`probes`) and the corpus signature ([`ir.eval_gen.corpus_signature()`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.corpus_signature)) that pin the labels to a corpus snapshot. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.eval.sweep_selector(corpus, cases, , strategy='conservative', mode='hybrid', k=10, surfaces=None, max_k_grid=(1, 2, 3, 5, 8), rel_grid=(0.4, 0.5, 0.6, 0.7, 0.8, 0.9), min_score_grid=(None,), gap_ratio=0.5, objective='selection_f1', \*\*search_kw) Tune the selector: score a grid of commit knobs against the cases. The selection defaults (`DFLT_MAX_K`, `DFLT_REL_THRESHOLD`) were tuned on real corpora (see ir_06), not guessed. This sweeps `max_k × rel × min_score` and scores each cell with [`evaluate_selection()`](_autosummary/ir.eval.html.md#ir.eval.evaluate_selection)’s exact metric, so the right defaults for a *given* corpus can be read off empirically (precision vs recall vs commit size) rather than assumed. Retrieval is run **once per case** (selector parameters do not change the ranking) and the cached candidates are reused across the whole grid, so a 6×5 grid costs one retrieval pass, not thirty. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – an [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) or a registered corpus *name*. * **cases** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)]) – the [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s to score against. * **strategy** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the selection strategy held fixed across the grid (the swept knobs are `conservative`’s; a non-`conservative` strategy that ignores some knobs simply yields duplicate rows). * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode for retrieval. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – candidate depth retrieved before selection (the selector’s window). * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict retrieval to these surface kinds. * **min_score_grid** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – the axes of the sweep. * **gap_ratio** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – held fixed (used only by the `score_gap` strategy). * **objective** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the default metric [`SelectionSweep.best()`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep.best) optimizes. * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword. * **Return type:** [`SelectionSweep`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep) * **Returns:** a [`SelectionSweep`](_autosummary/ir.eval.html.md#ir.eval.SelectionSweep) (`.best()` / `.frontier()` / `.table()` / `.to_dict()`). ### ir.eval.to_graded_qrels(cases, theme, , probe=None, gains={'core': 3.0, 'none': 0.0, 'strong': 2.0, 'tangential': 0.0, 'uses-tools': 1.0}, query_id=None) Build `ef`’s **graded** `(queries, qrels)` for one `theme`. Unlike [`to_qrels()`](_autosummary/ir.eval.html.md#ir.eval.to_qrels) (which hardcodes grade `1`), every positive artifact is judged at its graded [`LEVEL_GAINS`](_autosummary/ir.eval.html.md#ir.eval.LEVEL_GAINS) value, so graded gains reach `ef.evaluation`’s nDCG with zero `ef` change. There is exactly **one** query per theme (the theme *probe*); `qrels` maps it to `{artifact_id: gain}` for every artifact with a positive gain. Non-positive (`none`/`tangential`) artifacts are omitted, matching `ef`’s judged-positive qrels convention. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]] ### ir.eval.to_qrels(cases, , grade=1.0) Build `ef`’s `(queries, qrels)` from the gold-bearing cases. Abstention cases (empty `gold`) are skipped — `ef`’s retrieval metrics require at least one positively-judged document per query. Query ids are derived from each case’s position so they stay stable across the gap left by skipped abstention cases. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]] ### ir.eval.validate_cases(corpus, cases) Gold ids a case references that are absent from the corpus. Returns `{case_index: [missing_artifact_id, …]}` for every case whose gold has drifted out of the corpus (an empty dict means the cases are aligned). The eval corpora are live and machine-specific, so a committed case file can quietly fall out of sync; run this before trusting a report. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] # _autosummary/ir.eval_gen.html.md # ir.eval_gen LLM-backed generation of evaluation cases for [`ir.eval`](_autosummary/ir.eval.html.md#module-ir.eval). This is the **build-time** companion to the offline scoring harness: it turns a corpus into a set of [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)s by *back-translation* — given a capability’s description, ask an LLM for the user intents that should route to it. The artifact’s id is the free ground-truth label. Two ideas make the generated set honest: - **Name masking.** The artifact’s *name* is stripped from the description *before* it is shown to the generator (and any output that still leaks the name is dropped). Otherwise a lexical retriever (the BM25 leg of hybrid) would trivially match query→gold on surface name overlap and inflate scores. Many real descriptions contain their own name, so masking the **input** — not just filtering the output — is what matters. - **An abstention slice.** A fraction of cases are “no artifact applies” intents (empty `gold`), so the eval can measure correct refusal, not just hits. The LLM is **injected** (`query_generator` / `abstention_generator` callables), so the generation *logic* — masking, gold assignment, the leakage guard, the abstention fraction — is fully testable with a deterministic stub and no network. The default generators are built lazily on `aix` (`aix.prompt_func`, the multi-provider LLM facade), so `import ir.eval_gen` stays cheap and offline; `aix` is only imported when you actually generate with the real LLM. The output is plain [`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase) data — freeze it with [`ir.eval.save_cases()`](_autosummary/ir.eval.html.md#ir.eval.save_cases) (stamping [`corpus_signature()`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.corpus_signature) into the `__meta__` header) and score it with [`ir.eval`](_autosummary/ir.eval.html.md#module-ir.eval). Generation needs a model; scoring never does. Quick start: ```default import ir from ir import eval_gen as eg source = ir.CorpusSource.from_skills() cases = eg.build_eval_set(source, k=5, corpus_name="skills") # uses aix from ir.eval import save_cases save_cases(cases, "skills_eval.jsonl", meta={"corpus": "skills", "corpus_signature": eg.corpus_signature(source)}) ``` ### Module Attributes | [`DFLT_QUERIES_PER_ARTIFACT`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.DFLT_QUERIES_PER_ARTIFACT) | Queries generated per artifact by default. | |------------------------------------------------------------------------------|---------------------------------------------------------------------------| | [`DFLT_ABSTENTION_FRAC`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.DFLT_ABSTENTION_FRAC) | Target share of the case set that is abstention ("no artifact applies"). | | [`DFLT_MIN_DESCRIPTION_CHARS`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.DFLT_MIN_DESCRIPTION_CHARS) | Minimum description length (chars) for an artifact to be back-translated. | | [`NAME_PLACEHOLDER`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.NAME_PLACEHOLDER) | What a masked name is replaced with in a description / query. | | [`DFLT_ABSTENTION_THEME`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.DFLT_ABSTENTION_THEME) | Default theme used when generating abstention ("out of scope") intents. | | [`QueryGenerator`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.QueryGenerator) | `(description, *, n) -> list[str]` (n candidate intents). | | [`AbstentionGenerator`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.AbstentionGenerator) | `(*, n, theme) -> list[str]` (n out-of-scope intents). | ### Functions | [`build_eval_set`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.build_eval_set)(source, \*[, k, ...]) | Generate a full eval set — gold cases plus an abstention slice. | |-----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------| | [`corpus_signature`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.corpus_signature)(source_or_corpus) | A short, order-independent hash of a corpus's artifact ids. | | [`generate_abstention_cases`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.generate_abstention_cases)(n, \*[, generator, ...]) | Generate `n` abstention cases — out-of-scope intents (empty `gold`). | | [`generate_cases`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.generate_cases)(source, \*[, k, mask_names, ...]) | Back-translate a corpus source into gold-bearing `DiscoveryCase`s. | | [`make_default_abstention_generator`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.make_default_abstention_generator)(\*[, prompt]) | Build the default abstention generator on `aix` (lazy import). | | [`make_default_query_generator`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.make_default_query_generator)(\*[, prompt]) | Build the default back-translation generator on `aix` (lazy import). | | [`mask_name`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.mask_name)(text, name, \*[, placeholder]) | Replace occurrences of `name` in `text` with `placeholder`. | ### ir.eval_gen.AbstentionGenerator `(*, n, theme) -> list[str]` (n out-of-scope intents). * **Type:** An abstention generator alias of `Callable`[[…], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.eval_gen.DFLT_ABSTENTION_FRAC *= 0.15* Target share of the case set that is abstention (“no artifact applies”). ### ir.eval_gen.DFLT_ABSTENTION_THEME *= 'software developer tools'* Default theme used when generating abstention (“out of scope”) intents. ### ir.eval_gen.DFLT_MIN_DESCRIPTION_CHARS *= 20* Minimum description length (chars) for an artifact to be back-translated. ### ir.eval_gen.DFLT_QUERIES_PER_ARTIFACT *= 5* Queries generated per artifact by default. ### ir.eval_gen.NAME_PLACEHOLDER *= 'this capability'* What a masked name is replaced with in a description / query. ### ir.eval_gen.QueryGenerator `(description, *, n) -> list[str]` (n candidate intents). * **Type:** A query generator alias of `Callable`[[…], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.eval_gen.build_eval_set(source, , k=5, abstention_frac=0.15, query_generator=None, abstention_generator=None, theme='software developer tools', corpus_name=None, \*\*gen_kwargs) Generate a full eval set — gold cases plus an abstention slice. The abstention count is chosen so abstention cases make up (at least) `abstention_frac` of the returned set: `ceil(frac * G / (1 - frac))` for `G` gold cases. Extra `gen_kwargs` flow to [`generate_cases()`](_autosummary/ir.eval_gen.html.md#ir.eval_gen.generate_cases) (`mask_names`, `min_chars`, `max_artifacts`, `describe`). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `abstention_frac` is outside `[0, 1)` (`frac=0` means no abstention slice) or `k` is less than 1. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)] ### ir.eval_gen.corpus_signature(source_or_corpus) A short, order-independent hash of a corpus’s artifact ids. Stamp this into a case file’s `__meta__` header so a frozen eval set can be checked against the (live, machine-specific) corpus it was generated from. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.eval_gen.generate_abstention_cases(n, , generator=None, theme='software developer tools', corpus_name=None) Generate `n` abstention cases — out-of-scope intents (empty `gold`). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)] ### ir.eval_gen.generate_cases(source, , k=5, mask_names=True, query_generator=None, describe=None, min_chars=20, max_artifacts=None, corpus_name=None) Back-translate a corpus source into gold-bearing `DiscoveryCase`s. For each artifact in `source.scope` (id → raw), extract a description, mask the artifact’s name out of it, ask `query_generator` for `k` user intents, and emit one case per surviving intent (gold = the artifact id). Intents that still leak the name are dropped; artifacts whose description is shorter than `min_chars` are skipped (and the count is warned, never silently dropped). * **Parameters:** * **source** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) (anything with `.items()`). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – intents to request per artifact. * **mask_names** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – scrub the artifact name from the description before generating, and drop any generated intent that still contains it. * **query_generator** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]]) – `(description, *, n) -> [intent, …]`. Defaults to the `aix`-backed back-translator (built lazily; needs a model). * **describe** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – `raw -> description` (default: the `description` / `text` field, else the joined string fields). * **min_chars** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – skip artifacts whose description is shorter than this. * **max_artifacts** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – cap how many artifacts to process (for a quick/cheap run); when set, artifacts are taken in sorted-id order so the subset is deterministic even for filesystem-ordered (`dol`-backed) scopes. * **corpus_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – stamped on each case’s `corpus` field. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`DiscoveryCase`](_autosummary/ir.eval.html.md#ir.eval.DiscoveryCase)] * **Returns:** the generated gold cases. * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `k` is less than 1. ### ir.eval_gen.make_default_abstention_generator(, prompt='You are generating "no applicable tool" cases for a tool-retrieval evaluation.\\\\n\\\\nThe tool catalog is about: {theme}.\\\\n\\\\nWrite {n} natural, plausible user requests that such a catalog should NOT be able\\\\nto satisfy because they fall outside its scope.\\\\nOne request per line. No numbering, no quotes, no extra commentary.\\\\n', \*\*prompt_function_kwargs) Build the default abstention generator on `aix` (lazy import). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.eval_gen.make_default_query_generator(, prompt='You are generating evaluation data for a tool-retrieval system.\\\\n\\\\nBelow is a description of a capability (its name has been hidden on purpose):\\\\n\\\\n{description}\\\\n\\\\nWrite {n} natural, varied user requests that this capability should handle.\\\\nRules:\\\\n- Do NOT mention any tool, function, skill, or package name.\\\\n- Vary phrasing, specificity, and the implied (not explicit) parameters.\\\\n- One request per line. No numbering, no quotes, no extra commentary.\\\\n', \*\*prompt_function_kwargs) Build the default back-translation generator on `aix` (lazy import). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.eval_gen.mask_name(text, name, , placeholder='this capability') Replace occurrences of `name` in `text` with `placeholder`. Matches the name as a contiguous phrase tolerant of the separator between its tokens (`"ci-advisor"` / `"ci advisor"` / `"ci_advisor"` all match), case-insensitively and whole-word (a short token like `"ci"` does not blast through `"specific"`). If `placeholder` would itself match the name, a neutral alternate is used so the masked text is not self-referential. Used to scrub the artifact name out of a description *before* it is generated from. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ir.formulate.html.md # ir.formulate Query formulation — the Formulator seam (ir_09 §3). Turn a user query into one or more concrete low-level queries *before* retrieval: rewrite, expand, paraphrase, HyDE. This is the single most-acknowledged gap in ir’s standalone retrieval — the raw query is otherwise embedded as-is — and a good formulation lifts recall on short, identifier-heavy capability text with **no agent and no loop**. The seam is **opt-in and identity by default**: `ir.search(corpus, q)` with no `formulate=` embeds `q` verbatim (exactly today’s behavior). A formulator returns a `str` (one query) or a sequence of strings (multi-query fan-out); when it returns several, [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) runs each and fuses the candidate lists (best surface per artifact across all queries). LOAD-BEARING BOUNDARY: a [`Formulator`](_autosummary/ir.formulate.html.md#ir.formulate.Formulator) returns **queries, never SubTasks**. Decomposing a goal into sub-tasks + source selection is the *Planner’s* job — that lives in the agent layer (`raglab`), not here. [`make_llm_formulator()`](_autosummary/ir.formulate.html.md#ir.formulate.make_llm_formulator) mirrors `ir.select.make_llm_selector()`: an injectable `rewriter` callable, built lazily on `aix` when omitted (so importing ir stays offline), falling back to identity on any failure — a formulator must never make retrieval *worse* than the raw query. ### Module Attributes | [`Formulator`](_autosummary/ir.formulate.html.md#ir.formulate.Formulator) | a query string -> one query (`str`) or several (`Sequence[str]`). | |---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------| | [`FORMULATION_PROMPT`](_autosummary/ir.formulate.html.md#ir.formulate.FORMULATION_PROMPT) | Default prompt for [`make_llm_formulator()`](_autosummary/ir.formulate.html.md#ir.formulate.make_llm_formulator) — diverse paraphrases for recall. | ### Functions | [`identity_formulator`](_autosummary/ir.formulate.html.md#ir.formulate.identity_formulator)(query) | The default formulator: return the query unchanged (embed it verbatim). | |---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| | [`make_llm_formulator`](_autosummary/ir.formulate.html.md#ir.formulate.make_llm_formulator)(\*[, rewriter, prompt, ...]) | An LLM-backed [`Formulator`](_autosummary/ir.formulate.html.md#ir.formulate.Formulator) (rewrite / expand / multi-query). | ### ir.formulate.FORMULATION_PROMPT *= 'Rewrite the search query into {n} short, diverse alternative search queries that would retrieve the same target documents: fix typos, expand jargon, and add synonyms, but keep each a terse search phrase. One query per line, no numbering.\\n\\nQuery: {query}'* Default prompt for [`make_llm_formulator()`](_autosummary/ir.formulate.html.md#ir.formulate.make_llm_formulator) — diverse paraphrases for recall. ### ir.formulate.Formulator a query string -> one query (`str`) or several (`Sequence[str]`). Identity by default; an LLM rewriter / HyDE / multi-query producer when injected. * **Type:** A formulator alias of `Callable`[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], `str | Sequence[str]`] ### ir.formulate.identity_formulator(query) The default formulator: return the query unchanged (embed it verbatim). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.formulate.make_llm_formulator(, rewriter=None, prompt='Rewrite the search query into {n} short, diverse alternative search queries that would retrieve the same target documents: fix typos, expand jargon, and add synonyms, but keep each a terse search phrase. One query per line, no numbering.\\\\n\\\\nQuery: {query}', n=3, fallback=None, \*\*prompt_function_kwargs) An LLM-backed [`Formulator`](_autosummary/ir.formulate.html.md#ir.formulate.Formulator) (rewrite / expand / multi-query). `rewriter` is an injectable `query -> str | [str, ...]` callable (a test double, or your own router); when omitted it is built lazily on `aix` (`aix.prompt_func`), so importing this module stays offline. `n` is the multi-query fan-out width. Any error or empty reply falls back to `fallback` (default: [`identity_formulator()`](_autosummary/ir.formulate.html.md#ir.formulate.identity_formulator)). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] # _autosummary/ir.graph.html.md # ir.graph The semantic link graph — typed edges between artifacts (report 12). ir indexes artifacts that *refer to each other*: a package depends on other packages, a skill belongs to a parent, a chunk follows the previous one, a report cites another. This module models those references as \*\*typed, directed edges\*\* between nodes identified by `(source, artifact_id)` (ir’s canonical node identity since 0.1.16), queryable at retrieval time — the substrate the traversal operator (`traverse`, follow-up) walks. Two pieces: - the [`GraphStore`](_autosummary/ir.graph.html.md#ir.graph.GraphStore) **protocol** — a minimal `__getitem__` + `neighbors` structural contract, so a traversal operator binds to *any* conforming store, not only ir’s; - [`CorpusGraph`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph) — the corpus-backed implementation. Node payloads are the artifact’s stored records (via [`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)); neighbors come from the `links` view on [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore). Edges are **regenerable derived state** (persisted in the `links` view, like `calibration`) — never part of build identity. They are ingested at build time by an injectable [`EdgeExtractor`](_autosummary/ir.graph.html.md#ir.graph.EdgeExtractor); the shipped [`default_edge_extractor()`](_autosummary/ir.graph.html.md#ir.graph.default_edge_extractor) turns the edge data already latent in the index (`Package` `deps` → `REF`, `Skill` `parent` → `PARENT`) into edges. Vocabulary (documented, **not** enforced): [`NEXT`](_autosummary/ir.graph.html.md#ir.graph.NEXT) / [`PREV`](_autosummary/ir.graph.html.md#ir.graph.PREV) / [`PARENT`](_autosummary/ir.graph.html.md#ir.graph.PARENT) / [`CHILD`](_autosummary/ir.graph.html.md#ir.graph.CHILD) / [`REF`](_autosummary/ir.graph.html.md#ir.graph.REF). `NEXT` / `PREV` are *not* materialized — they are derivable from the ledger’s surface order (#44); the view stores only what isn’t derivable. **Naming guard** (ADR #43): this is the **semantic link graph** — directed, **possibly cyclic** (citations, cross-references), traversed at query time. It is *not* `ef.artifact_graph`, which is an acyclic build-time derivation/lineage DAG. The id conventions are kept compatible so a later unification stays open, but the two are distinct substrates. ### Module Attributes | [`NEXT`](_autosummary/ir.graph.html.md#ir.graph.NEXT) | Next sibling surface in sequence (derivable from the ledger — not stored). | |----------------------------------------------------------------|--------------------------------------------------------------------------------| | [`PREV`](_autosummary/ir.graph.html.md#ir.graph.PREV) | Previous sibling surface in sequence (derivable from the ledger — not stored). | | [`PARENT`](_autosummary/ir.graph.html.md#ir.graph.PARENT) | a child points at its parent (e.g. a skill → its package). | | [`CHILD`](_autosummary/ir.graph.html.md#ir.graph.CHILD) | a parent points at a child. | | [`REF`](_autosummary/ir.graph.html.md#ir.graph.REF) | A generic reference / dependency / citation (e.g. a package → its deps). | | [`NodeId`](_autosummary/ir.graph.html.md#ir.graph.NodeId) | `artifact_id` within one corpus, or `(source, artifact_id)` across corpora. | | [`EdgeExtractor`](_autosummary/ir.graph.html.md#ir.graph.EdgeExtractor) | `(artifact_id, filter_fields) -> {edge_type: [target]}`. | ### Functions | [`canonical_node_id`](_autosummary/ir.graph.html.md#ir.graph.canonical_node_id)(target, \*, source) | Canonicalize a neighbor *target* to a `(source, artifact_id)` node id. | |-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------| | [`default_edge_extractor`](_autosummary/ir.graph.html.md#ir.graph.default_edge_extractor)(artifact_id, ...) | Edges latent in the standard filter fields: `deps` → REF, `parent` → PARENT. | | [`normalize_dist_name`](_autosummary/ir.graph.html.md#ir.graph.normalize_dist_name)(name) | A dependency spec's bare name, PEP 503-normalized (`-`/`_`/`.` -> `-`). | ### Classes | [`CorpusGraph`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph)(store_or_corpus) | A [`GraphStore`](_autosummary/ir.graph.html.md#ir.graph.GraphStore) over one corpus — artifact nodes, `links` edges. | |---------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | [`GraphStore`](_autosummary/ir.graph.html.md#ir.graph.GraphStore)(\*args, \*\*kwargs) | Structural contract a traversal operator binds to — node + neighbors. | ### ir.graph.CHILD *= 'CHILD'* a parent points at a child. * **Type:** Containment down ### *class* ir.graph.CorpusGraph(store_or_corpus) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A [`GraphStore`](_autosummary/ir.graph.html.md#ir.graph.GraphStore) over one corpus — artifact nodes, `links` edges. `node_id` is an `artifact_id`. `graph[aid]` is the artifact’s stored records (its scorable surfaces, in plan order); `graph.neighbors(aid, edge_type=...)` reads the corpus store’s `links` view. Single-corpus, so it resolves **intra-corpus** targets; cross-corpus `[source, artifact_id]` targets are returned by [`neighbors()`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.neighbors) verbatim but `__getitem__` only dereferences ids in *this* corpus (federated traversal is a follow-up). #### edge_types(node_id) The edge types present on *node_id* (`[]` if none). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### fan_in(node_id, , edge_type='REF') How many artifacts point at *node_id* by *edge_type* (its in-degree). With `REF` dependency edges this is the number of dependents, a PageRank-style centrality prior: a widely depended-on library is central to the corpus. * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) #### max_fan_in(, edge_type='REF') The largest [`fan_in()`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.fan_in) over all targets (`0` without edges). * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) #### neighbors(node_id, , edge_type=None) Outgoing neighbor ids of *node_id*, optionally of one *edge_type*. Returns target ids in stored form (a bare `artifact_id`, whose source is *this* graph’s [`source`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.source); or a `[source, artifact_id]` list for a cross-corpus edge), de-duplicated with first-seen order preserved. An artifact with no edges — or a store without a links view — yields `[]`. Pass a canonical `(source, artifact_id)` to [`canonical_node_id()`](_autosummary/ir.graph.html.md#ir.graph.canonical_node_id) for a traversal’s visited-set. *node_id* is an intra-corpus `artifact_id` (a `str`); a cross-corpus target fetched from another graph is out of contract here (it has no edges *in this corpus*). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) #### reverse_neighbors(node_id, , edge_type='REF') The artifact ids whose *edge_type* edges point **at** *node_id*. The reverse of [`neighbors()`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.neighbors): with the default [`REF`](_autosummary/ir.graph.html.md#ir.graph.REF) edges of [`default_edge_extractor()`](_autosummary/ir.graph.html.md#ir.graph.default_edge_extractor) (`package -> its deps`), this lists a library’s dependents. *node_id* may name an artifact of this corpus or a target outside it (a third-party dependency). For `REF` edges, string ids match through [`normalize_dist_name()`](_autosummary/ir.graph.html.md#ir.graph.normalize_dist_name) (case, and `-`/`_`/`.`, are ignored), so `"scikit_learn"` finds the dependents of `"scikit-learn"`; other edge types match ids exactly. A cross-corpus target is a `(source, artifact_id)` pair, matched exactly. Order is first-seen over the links view, de-duplicated. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### source The corpus name, when known — the `source` half of this graph’s node identities (`None` for a bare store). ### ir.graph.EdgeExtractor `(artifact_id, filter_fields) -> {edge_type: [target]}`. An injectable build-time seam (like [`Formulator`](_autosummary/ir.formulate.html.md#ir.formulate.Formulator)) that turns an artifact’s filter fields into outgoing edges. `build(..., edge_extractor=...)` runs it; [`default_edge_extractor()`](_autosummary/ir.graph.html.md#ir.graph.default_edge_extractor) is shipped. * **Type:** An edge extractor alias of `Callable`[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]], `Mapping[str, list]`] ### *class* ir.graph.GraphStore(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Structural contract a traversal operator binds to — node + neighbors. Deliberately minimal (two methods) so it is satisfied by ir’s [`CorpusGraph`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph) *and* by any external graph: `__getitem__` resolves a node id to its scorable payload, `neighbors` lists adjacent node ids (optionally of one edge type). Granularity-agnostic on purpose — an artifact graph and a surface-level tree are both ``` `` ``` GraphStore\`\`s. `runtime_checkable` makes `isinstance(x, GraphStore)` a *structural* check on attribute **names** only (not signatures) — enough to tell a conforming adapter from an arbitrary object, but it cannot validate that `neighbors` takes the right arguments; treat it as a smoke check. ### ir.graph.NEXT *= 'NEXT'* Next sibling surface in sequence (derivable from the ledger — not stored). ### ir.graph.NodeId *= 'str | tuple[str, str]'* `artifact_id` within one corpus, or `(source, artifact_id)` across corpora. The visited-set of a traversal keys on this. * **Type:** A node’s identity ### ir.graph.PARENT *= 'PARENT'* a child points at its parent (e.g. a skill → its package). * **Type:** Containment up ### ir.graph.PREV *= 'PREV'* Previous sibling surface in sequence (derivable from the ledger — not stored). ### ir.graph.REF *= 'REF'* A generic reference / dependency / citation (e.g. a package → its deps). ### ir.graph.canonical_node_id(target, , source) Canonicalize a neighbor *target* to a `(source, artifact_id)` node id. The repo’s node identity is `(source, artifact_id)` — the key a traversal visited-set must use so the same id in two corpora stays two nodes. [`CorpusGraph.neighbors()`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.neighbors) returns targets in stored form: a bare `artifact_id` (implicitly in *source*, the graph it came from) or a `[source, artifact_id]` cross-corpus pair. This resolves either to the canonical tuple. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ```pycon >>> canonical_node_id("dol", source="packages") ('packages', 'dol') >>> canonical_node_id(["skills", "deploy"], source="packages") ('skills', 'deploy') ``` ### ir.graph.default_edge_extractor(artifact_id, filter_fields) Edges latent in the standard filter fields: `deps` → REF, `parent` → PARENT. - `deps` (Package) → [`REF`](_autosummary/ir.graph.html.md#ir.graph.REF) edges to each dependency’s bare name (version specifiers / extras / markers stripped). Self-edges and blanks are dropped. - `parent` (Skill) → a single [`PARENT`](_autosummary/ir.graph.html.md#ir.graph.PARENT) edge. A package whose `deps` name other packages in the same corpus gets intra-corpus REF edges; third-party deps become REF edges to ids not in the corpus (harmless — [`CorpusGraph.neighbors()`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph.neighbors) lists them, and a traversal simply finds no node to expand). Self-edges are dropped case-insensitively (`_dep_name` lower-cases, so a package `"AA"` depending on `"aa"` is recognized as a self-reference). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)] ### ir.graph.normalize_dist_name(name) A dependency spec’s bare name, PEP 503-normalized (`-`/`_`/`.` -> `-`). The one comparison key for library names: the reverse dependency index and [`ir.dependency_evidence_boost()`](_autosummary/ir.html.md#ir.dependency_evidence_boost) both match through it, so `"scikit_learn"` finds a package that depends on `"Scikit-Learn>=1"`. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> normalize_dist_name("Scikit_Learn[extra]>=1.2") 'scikit-learn' ``` # _autosummary/ir.html.md # ir `ir` — an information-retrieval substrate for agentic systems. One uniform “find the relevant things in this corpus” contract that scales from an ad-hoc search over an ephemeral list to a maintained search engine. Retrieval is the core; selection/expansion/reranking/generation are layered on top. Quick start: ```default import ir # Define a corpus source (abstract strategy + parameters, smart defaults): source = ir.CorpusSource.from_md_reports() # project docs/ reports corpus = ir.build(source) # index (incremental) hits = ir.search(corpus, "how do I deploy the app") # ranked SearchHits # Light, dependency-free embedding for fast tests: corpus = ir.build(source, embedder="light") ``` A corpus source is defined by a `scope` (what is in the corpus), a `change_signal` (what counts as stale), an `indexing_strategy` (how a raw item becomes filter fields + embeddable surfaces), and an `embedder`. The default embedder is a decent *local* model (`all-MiniLM-L6-v2`); `"light"` selects a numpy-only hashing embedder. Data persists under XDG dirs through a `dol` repository layer. `ir` is a retrieval **substrate, not a search agent** — it has no agent of its own. [`ir.discover()`](_autosummary/ir.html.md#ir.discover) is a deterministic single-shot `retrieve → select → disclose` pipeline: the query is embedded verbatim (no LLM query rewriting) and committed by a score rule (no LLM reviewing results), with no planner and no *back-edge* (evaluate → reformulate → search again). The agent layer — planner, LLM query formulator, and the evaluator→reformulate loop — lives in **raglab** ([https://github.com/thorwhalen/raglab](https://github.com/thorwhalen/raglab)), which is built *on top of* `ir`: `raglab` imports `ir`; `ir` never imports `raglab`. To drive an agentic search from Claude Code instead of a coded agent, see the `ir-search` skill and the reviewing search subagent shipped under `.claude/`. ### Functions | [`resolve_reranker`](_autosummary/ir.html.md#ir.resolve_reranker)(spec) | The reranker `spec` names, or `None` when there is none to apply. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| | [`calibration_key`](_autosummary/ir.html.md#ir.calibration_key)(mode[, tag]) | The key an abstention floor is stored and loaded under. | | [`search_tool`](_autosummary/ir.html.md#ir.search_tool)(query, \*, corpus[, k, mode, ...]) | Search a named `ir` corpus and return a JSON-serializable result dict. | | [`make_search`](_autosummary/ir.html.md#ir.make_search)(corpus, \*[, name, description, ...]) | Return a **corpus-bound** `search(query, k=...) -> dict` tool. | | [`with_synopsis`](_autosummary/ir.html.md#ir.with_synopsis)(strategy, \*[, synthesize, ...]) | Wrap *strategy* to add one LLM-derived `synopsis` surface per artifact. | | [`make_thin_description_predicate`](_autosummary/ir.html.md#ir.make_thin_description_predicate)(\*[, ...]) | A `Predicate` true when the artifact's `field` is empty or thin. | | `thin_description_predicate`(artifact) | Ready-made gate: synthesize only for packages with an empty / short description. | | [`make_llm_synthesizer`](_autosummary/ir.html.md#ir.make_llm_synthesizer)(\*[, summarize, prompt, ...]) | An LLM-backed `Synthesizer` ([`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → synopsis). | | [`build`](_autosummary/ir.html.md#ir.build)(source, \*[, store, embedder, full, ...]) | Build or incrementally update *source* into a [`Corpus`](_autosummary/ir.html.md#ir.Corpus). | | [`open_corpus`](_autosummary/ir.html.md#ir.open_corpus)(name, \*[, embedder]) | Reopen a previously built corpus by name. | | [`reports_coverage`](_autosummary/ir.html.md#ir.reports_coverage)([name, projects_root, ...]) | Diff the reports docs tree on disk against the built `name` corpus. | | [`search`](_autosummary/ir.html.md#ir.search)(corpus, query, \*\*kwargs) | Search a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) (or a corpus *name*, reopened lazily). | | [`as_retriever`](_autosummary/ir.html.md#ir.as_retriever)(corpus_or_name, \*\*search_defaults) | Bind ONE corpus to the uniform `Retriever` contract. | | [`retrievers`](_autosummary/ir.html.md#ir.retrievers)(\*\*search_defaults) | A lazy `Mapping[name, Retriever]` view over the registry (ir_09 §8). | | [`retriever_for`](_autosummary/ir.html.md#ir.retriever_for)(name, \*\*search_defaults) | A [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) bound to the registered corpus *name*. | | [`fuse_hits`](_autosummary/ir.html.md#ir.fuse_hits)(hits_by_source, \*[, rrf_k, ...]) | Merge per-source ranked hit lists into one ranking — by rank, not score. | | [`dependency_evidence_boost`](_autosummary/ir.html.md#ir.dependency_evidence_boost)(relevant_libs, \*, ...) | A `dep_boost` for [`search()`](_autosummary/ir.html.md#ir.search): reward dependencies on *relevant_libs*. | | [`records_for_artifact`](_autosummary/ir.html.md#ir.records_for_artifact)(store_or_corpus, ...[, ...]) | All stored records of *artifact_id*, ordered by `surface_index`. | | [`default_edge_extractor`](_autosummary/ir.html.md#ir.default_edge_extractor)(artifact_id, ...) | Edges latent in the standard filter fields: `deps` → REF, `parent` → PARENT. | | [`canonical_node_id`](_autosummary/ir.html.md#ir.canonical_node_id)(target, \*, source) | Canonicalize a neighbor *target* to a `(source, artifact_id)` node id. | | [`traverse`](_autosummary/ir.html.md#ir.traverse)(query, store, \*, policy[, ...]) | Walk *store* from *query* under *policy*, returning the top-*k* hits. | | [`collapsed_tree_policy`](_autosummary/ir.html.md#ir.collapsed_tree_policy)(\*[, summary_kinds, ...]) | The pure-vector summary-routing / collapsed-tree [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy). | | [`reverse_dependency_policy`](_autosummary/ir.html.md#ir.reverse_dependency_policy)(\*, seeds[, ...]) | A [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy) that walks dependency edges backwards from *seeds*. | | [`expand`](_autosummary/ir.html.md#ir.expand)(hit, corpus, \*[, policy]) | Expand *hit* into a [`Passage`](_autosummary/ir.html.md#ir.Passage) of its neighborhood in *corpus*. | | [`sentence_window_policy`](_autosummary/ir.html.md#ir.sentence_window_policy)([k]) | ±\*k\* same-kind neighbors around the seed (NEXT/PREV expansion). | | [`parent_policy`](_autosummary/ir.html.md#ir.parent_policy)() | The whole artifact (small-to-big): every stored surface, plan order. | | [`tag_source`](_autosummary/ir.html.md#ir.tag_source)(hits, source) | Stamp *source* on every hit that doesn't already carry one. | | [`make_llm_formulator`](_autosummary/ir.html.md#ir.make_llm_formulator)(\*[, rewriter, prompt, ...]) | An LLM-backed `Formulator` (rewrite / expand / multi-query). | | [`select`](_autosummary/ir.html.md#ir.select)(hits, \*[, strategy, max_k, rel, ...]) | Commit to a subset of ranked `hits` — the selection stage. | | [`disclose`](_autosummary/ir.html.md#ir.disclose)(selection, \*[, level, loader, ...]) | Reveal the payload of each selected hit at `level` — append-only, pure. | | [`discover`](_autosummary/ir.html.md#ir.discover)(corpus, query, \*[, k, mode, ...]) | Find and commit to the capabilities for `query` — the one search tool. | | [`register`](_autosummary/ir.html.md#ir.register)(name, kind, \*[, embedder, ...]) | Register (or overwrite) a named corpus definition. | | [`corpora`](_autosummary/ir.html.md#ir.corpora)() | All registered corpus definitions, keyed by name. | | [`build_corpus`](_autosummary/ir.html.md#ir.build_corpus)(name, \*\*kwargs) | Build (or update) a registered/preset corpus by *name*; returns a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus). | | [`maintain`](_autosummary/ir.html.md#ir.maintain)([name, all, now, dry_run]) | Run due background work for one corpus (*name*) or every registered one (*all*). | | [`maintain_corpus`](_autosummary/ir.html.md#ir.maintain_corpus)(name, \*[, now, dry_run, full]) | Do the due background work for one corpus (idempotent). | | [`resolve_policy`](_autosummary/ir.html.md#ir.resolve_policy)(entry) | The effective policy for a registry `entry`: entry over kind over global. | | [`default_policy_for_kind`](_autosummary/ir.html.md#ir.default_policy_for_kind)(kind) | The smart-default [`MaintenancePolicy`](_autosummary/ir.html.md#ir.MaintenancePolicy) for a corpus `kind`. | | [`policy_for`](_autosummary/ir.html.md#ir.policy_for)(name) | The effective [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for corpus *name*. | ### Classes | [`Artifact`](_autosummary/ir.html.md#ir.Artifact)(id, raw[, metadata]) | A logical corpus item before decomposition into surfaces. | |----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| | [`Surface`](_autosummary/ir.html.md#ir.Surface)(artifact_id, kind, text[, ...]) | One embeddable unit derived from an artifact. | | [`Record`](_autosummary/ir.html.md#ir.Record)(id, artifact_id, surface_kind, ...[, ...]) | A stored, embedded surface — one row of the index. | | [`SearchHit`](_autosummary/ir.html.md#ir.SearchHit)(artifact_id, surface_kind, score, text) | A scored record returned by retrieval (higher score = closer). | | [`IndexPlan`](_autosummary/ir.html.md#ir.IndexPlan)([filter_fields, surfaces]) | An [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)'s output for one artifact. | | [`IndexingStrategy`](_autosummary/ir.html.md#ir.IndexingStrategy)(\*args, \*\*kwargs) | Decompose one artifact into filter fields + embeddable surfaces. | | [`WholeText`](_autosummary/ir.html.md#ir.WholeText)(\*[, text_key, kind]) | One surface = the entire text. | | [`Chunked`](_autosummary/ir.html.md#ir.Chunked)(\*[, chunk_size, overlap, text_key, ...]) | Split the artifact's text into overlapping chunk surfaces. | | [`Skill`](_autosummary/ir.html.md#ir.Skill)() | Capability strategy: embed `name + description` only. | | [`Package`](_autosummary/ir.html.md#ir.Package)(\*[, chunk_size, overlap, ...]) | Package strategy: `name + description` surface plus README chunks. | | [`ClaudeTurn`](_autosummary/ir.html.md#ir.ClaudeTurn)(\*[, include_full]) | Index a Claude Code session turn-pair: user prompt + assistant summary. | | [`CorpusSource`](_autosummary/ir.html.md#ir.CorpusSource)(name, scope[, ...]) | A corpus definition: scope + change signal + strategy + embedder. | | [`CorpusStore`](_autosummary/ir.html.md#ir.CorpusStore)(meta, vectors, ledger, config[, ...]) | Repository bundling the meta/vectors/ledger/config views of one corpus. | | [`Corpus`](_autosummary/ir.html.md#ir.Corpus)(name, store, embedder, embedder_id) | A built, queryable corpus: a store plus its embedder. | | [`CoverageReport`](_autosummary/ir.html.md#ir.CoverageReport)(name, on_disk, indexed, excluded) | Disk-vs-index coverage for a filesystem-backed corpus. | | [`GraphStore`](_autosummary/ir.html.md#ir.GraphStore)(\*args, \*\*kwargs) | Structural contract a traversal operator binds to — node + neighbors. | | [`CorpusGraph`](_autosummary/ir.html.md#ir.CorpusGraph)(store_or_corpus) | A [`GraphStore`](_autosummary/ir.html.md#ir.GraphStore) over one corpus — artifact nodes, `links` edges. | | [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy)(\*args, \*\*kwargs) | The pluggable strategy of a walk — graph semantics, not safety. | | [`WalkState`](_autosummary/ir.html.md#ir.WalkState)(query, max_depth, budget[, ...]) | The operator-owned state of one [`traverse()`](_autosummary/ir.html.md#ir.traverse) call — the safety home. | | [`Passage`](_autosummary/ir.html.md#ir.Passage)(artifact_id, surface_kind, score, text) | An expanded hit: the seed's identity + the stitched neighborhood text. | | [`Selection`](_autosummary/ir.html.md#ir.Selection)(selected, candidates, abstained, ...) | A selector's commitment: the chosen subset of a ranked candidate list. | | [`Disclosure`](_autosummary/ir.html.md#ir.Disclosure)(artifact_id, level, name, score, ...) | The progressively-disclosed payload for one selected artifact. | | [`DiscoveryResult`](_autosummary/ir.html.md#ir.DiscoveryResult)(query, mode, strategy, ...) | The result of [`discover()`](_autosummary/ir.html.md#ir.discover) — retrieve → select → (optional) disclose. | | [`MaintenanceResult`](_autosummary/ir.html.md#ir.MaintenanceResult)(name, ran, reason[, ...]) | What [`maintain_corpus()`](_autosummary/ir.html.md#ir.maintain_corpus) did (or would do) for one corpus. | | [`MaintenancePolicy`](_autosummary/ir.html.md#ir.MaintenancePolicy)([reindex, synopsis]) | The background-work policy for one corpus (reindex + synopsis). | | [`ReindexPolicy`](_autosummary/ir.html.md#ir.ReindexPolicy)([on, every_hours]) | When to (incrementally) rebuild a corpus. | | [`SynopsisPolicy`](_autosummary/ir.html.md#ir.SynopsisPolicy)([enabled, scope, ...]) | Whether/when to attach (expensive, LLM-generated) synopses. | ### *class* ir.Artifact(id, raw, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A logical corpus item before decomposition into surfaces. ### *class* ir.Chunked(, chunk_size=1200, overlap=200, text_key=None, kind='chunk', max_tokens=None, token_model=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Split the artifact’s text into overlapping chunk surfaces. By default chunks are packed to ~\`\`chunk_size\`\` **characters**. Set `max_tokens` to chunk by **tokens** instead — each chunk is bounded to `max_tokens` tokens of `token_model` (default: the default embedding model), so no chunk overflows the embedder’s sequence limit and is silently truncated. Token mode degrades to char mode with a warning if the tokenizer is unavailable. The char `chunk_size` / `overlap` still set the overlap *ratio* used in token mode. ### *class* ir.ClaudeTurn(, include_full=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Index a Claude Code session turn-pair: user prompt + assistant summary. Two surfaces by default — `user_prompt` (what the human asked) and `assistant_summary` (the assistant’s *final* end-of-turn text, the highest-signal “here’s what I did”; the deliberation before it is mostly noise) — so a query can target either side via `surfaces={"user_prompt"}` / `{"assistant_summary"}`. With `include_full=True` a third `assistant_full` surface (all the turn’s assistant natural-language text) trades noise for recall — off by default. Session / project / time / model / tool-use become hard-filter fields. The raw artifact is a turn-pair record (see `priv.claude_transcripts.turn_pair_records()`): a mapping with `user_prompt` / `assistant_summary` / `assistant_full` plus metadata. ### *class* ir.Corpus(name, store, embedder, embedder_id) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A built, queryable corpus: a store plus its embedder. #### search(query, \*\*kwargs) Search this corpus for *query*. `**kwargs` (`k` / `mode` / `filter` / `surfaces` / `per_artifact` / …) are forwarded to [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). ### *class* ir.CorpusGraph(store_or_corpus) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A [`GraphStore`](_autosummary/ir.html.md#ir.GraphStore) over one corpus — artifact nodes, `links` edges. `node_id` is an `artifact_id`. `graph[aid]` is the artifact’s stored records (its scorable surfaces, in plan order); `graph.neighbors(aid, edge_type=...)` reads the corpus store’s `links` view. Single-corpus, so it resolves **intra-corpus** targets; cross-corpus `[source, artifact_id]` targets are returned by [`neighbors()`](_autosummary/ir.html.md#ir.CorpusGraph.neighbors) verbatim but `__getitem__` only dereferences ids in *this* corpus (federated traversal is a follow-up). #### edge_types(node_id) The edge types present on *node_id* (`[]` if none). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### fan_in(node_id, , edge_type='REF') How many artifacts point at *node_id* by *edge_type* (its in-degree). With `REF` dependency edges this is the number of dependents, a PageRank-style centrality prior: a widely depended-on library is central to the corpus. * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) #### max_fan_in(, edge_type='REF') The largest [`fan_in()`](_autosummary/ir.html.md#ir.CorpusGraph.fan_in) over all targets (`0` without edges). * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) #### neighbors(node_id, , edge_type=None) Outgoing neighbor ids of *node_id*, optionally of one *edge_type*. Returns target ids in stored form (a bare `artifact_id`, whose source is *this* graph’s [`source`](_autosummary/ir.html.md#ir.CorpusGraph.source); or a `[source, artifact_id]` list for a cross-corpus edge), de-duplicated with first-seen order preserved. An artifact with no edges — or a store without a links view — yields `[]`. Pass a canonical `(source, artifact_id)` to [`canonical_node_id()`](_autosummary/ir.html.md#ir.canonical_node_id) for a traversal’s visited-set. *node_id* is an intra-corpus `artifact_id` (a `str`); a cross-corpus target fetched from another graph is out of contract here (it has no edges *in this corpus*). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) #### reverse_neighbors(node_id, , edge_type='REF') The artifact ids whose *edge_type* edges point **at** *node_id*. The reverse of [`neighbors()`](_autosummary/ir.html.md#ir.CorpusGraph.neighbors): with the default `REF` edges of [`default_edge_extractor()`](_autosummary/ir.html.md#ir.default_edge_extractor) (`package -> its deps`), this lists a library’s dependents. *node_id* may name an artifact of this corpus or a target outside it (a third-party dependency). For `REF` edges, string ids match through `normalize_dist_name()` (case, and `-`/`_`/`.`, are ignored), so `"scikit_learn"` finds the dependents of `"scikit-learn"`; other edge types match ids exactly. A cross-corpus target is a `(source, artifact_id)` pair, matched exactly. Order is first-seen over the links view, de-duplicated. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### source The corpus name, when known — the `source` half of this graph’s node identities (`None` for a bare store). ### *class* ir.CorpusSource(name, scope, indexing_strategy=, change_signal=, embedder='default', metadata_of=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A corpus definition: scope + change signal + strategy + embedder. #### change_signal(raw) Default change signal: a content hash of the raw payload. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) #### *classmethod* from_claude_sessions(, name='sessions', since=90, projects=None, include_full=False, include_session_title=True, max_sessions=None, root=None, fetcher=None, strategy=None, \*\*kwargs) The user’s Claude Code session transcripts as a corpus (turn pairs). Each artifact is one user→assistant turn pair; the default [`ClaudeTurn`](_autosummary/ir.strategy.html.md#ir.strategy.ClaudeTurn) strategy indexes the user prompt and the assistant’s end-of-turn summary as separate surfaces (target either with `surfaces={"user_prompt"}` / `{"assistant_summary"}`). `include_full` adds the full assistant text surface (off by default — the summary is the signal). `include_session_title` (default on) also indexes one record per session whose surface is the session’s persisted custom/AI title — a cheap “what was this session about” surface. Scope defaults to the last `since` days (a full-history build is heavy); narrow with `projects` (a cwd substring or list) and `max_sessions`. `fetcher` overrides the record source (each a mapping with `user_prompt` / `assistant_summary` / … ) — a callable (inject a test double to avoid the `priv` dependency) or a `"module:attr"` reference (`resolve_fetcher()`), the form a registry entry can carry, so another package can own the transcripts this corpus indexes. Otherwise records come from `priv.claude_transcripts.turn_pair_records()`. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_files(root, , name=None, pattern='.\*\\\\\\\\.md$', exclude=None, strategy=None, \*\*kwargs) A directory tree of text files as a corpus (lazy `dol` scope). * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_mapping(mapping, , name, strategy=None, \*\*kwargs) Any mapping `{id -> raw}` (dict, `dol` store) as a corpus. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_mcp_tools(tools, , name='tools', strategy=None, \*\*kwargs) A catalogue of MCP tools as a corpus, for tool search. Each tool is indexed on `name + description` only, through the [`Skill`](_autosummary/ir.strategy.html.md#ir.strategy.Skill) strategy: its parameter schema never enters the embedded text (ir_02 §4, “the body carries no retrieval responsibility”). The schema still drives invalidation, through `mcp_tool_signal()`, so a changed schema re-embeds the tool. Its top-level argument names ride along as the `param_names` metadata of every hit, so a caller can see what a tool takes without loading it. `tools` is an iterable of tool records, a `{name: tool}` mapping (the key names a record that has no `name`; a record’s own name wins), or a fetcher (a zero-argument callable or a `"module:attr"` reference, see `resolve_fetcher()`). A record is duck-typed (see `mcp_tool_record()`), so `ir` takes no MCP dependency; an adapter for a specific client lives with the consumer. Tool names are the artifact ids and must be unique: namespace them (`"server.tool"`) before merging several servers’ catalogues. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ```pycon >>> src = CorpusSource.from_mcp_tools( ... [{"name": "ping", "description": "Check the server is up"}] ... ) >>> plan = src.indexing_strategy.decompose("ping", src.scope["ping"]) >>> print(plan.surfaces[0].text) ping Check the server is up ``` #### *classmethod* from_md_reports(, name='reports', projects_root=None, strategy=None, recursive=True, exclude_dirs=None, \*\*kwargs) Markdown reports under projects’ `docs/` and `misc/docs/` trees. Walks each `*/*/docs` and `*/*/misc/docs` folder, and those of pocket repos (`i/_zodals//docs`, see `iter_pocket_repos()`), **recursively** (`recursive=True`, default) so reports nested one or more levels deep — `docs/research/…`, `docs/decisions/…`, `docs/adr/…` — are indexed, not just files sitting directly in the folder. Pass `recursive=False` for the old shallow (top-level-only) behavior. Excludes ALL-CAPS filenames (README/CLAUDE/MEMORY/SKILL…) and any file under a vendored/hidden directory (`exclude_dirs`, default `DFLT_EXCLUDE_DIRS`; a dotted directory is always skipped). Each record is a project-tagged document; ids are paths relative to the projects root. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_packages(, name='packages', manifest=None, readme_chars=20000, strategy=None, \*\*kwargs) The local package ecosystem, scanned from the `.pth` manifest. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_records(, name, fetcher, strategy=None, metadata_keys=None, id_key='id', text_key='text', \*\*kwargs) Records produced by a **named callable** as a corpus. The generic, *persistable* counterpart of [`from_mapping()`](_autosummary/ir.html.md#ir.CorpusSource.from_mapping): where `from_mapping` takes an in-process object (so a registry entry cannot name it), this takes a `fetcher` — a callable **or** a `"module:attr"` import reference (see `resolve_fetcher()`) — so a corpus owned by another package round-trips through `~/.config/ir/corpora.json` and rebuilds in a fresh process. Each record is a mapping; its `id_key` is the artifact id (falling back to `_` so no record is silently dropped). `metadata_keys` names the record fields lifted into the strategy’s **filter fields** — the JSON-friendly stand-in for a `metadata_of` callable, which no registry entry could carry. The default strategy chunks `text_key`. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ```pycon >>> src = CorpusSource.from_records( ... name="notes", ... fetcher=lambda: [{"id": "a", "text": "hi", "project": "p"}], ... metadata_keys=["project"], ... ) >>> src.scope["a"]["text"], src.metadata_of("a", src.scope["a"]) ('hi', {'project': 'p'}) ``` #### *classmethod* from_skills(, name='skills', filter=None, fetcher=None, strategy=None, \*\*kwargs) The agent-skills corpus, via `priv.skills_index`. `fetcher` overrides the source of skill records (each a mapping with `name`/`description`/`parent`) — a callable (inject a test double to avoid the `priv` dependency) or a `"module:attr"` reference (`resolve_fetcher()`), which is the form a registry entry can carry. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### items() Iterate `(artifact_id, raw)` pairs over the corpus scope. ### *class* ir.CorpusStore(meta, vectors, ledger, config, calibration=None, links=None, , packed_dir=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Repository bundling the meta/vectors/ledger/config views of one corpus. #### calibration_modes() The ranking modes that currently have a stored calibration. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### delete_ledger_entry(key) Remove a ledger entry; a missing key is tolerated. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### delete_links(artifact_id) Remove an artifact’s edges; a missing entry is tolerated. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### delete_record(record_id) Remove a record’s metadata + vector; a missing id is tolerated. The meta goes first, so the id stops being listed before its vector disappears. Each removal tolerates the file being gone already (another process may be deleting the same record). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_calibration(mode) The stored calibration record for ranking `mode` (`None` if absent). A deep copy, so a caller cannot mutate the nested `grid` back into the stored record (in-memory stores share their objects by reference). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_config() The persisted corpus build settings (empty dict if never written). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_ledger_entry(key) The ledger entry for *key* (`None` if absent). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_links(artifact_id) The outgoing edges of *artifact_id* — `{edge_type: [target, ...]}`. Empty dict when the artifact has no stored edges (or no links view). A copy, so a caller cannot mutate the persisted adjacency in place. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_maintenance_state() Background-work bookkeeping (e.g. `last_maintained`); `{}` if unset. Kept under a separate `config`-view key from the build settings: it is regenerable scheduler state (when `ir maintain` last ran), not part of the corpus’s build identity, so it must never clobber it. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_record(record_id) Reassemble the [`Record`](_autosummary/ir.base.html.md#ir.base.Record) for *record_id* (`KeyError` if absent). * **Return type:** [`Record`](_autosummary/ir.base.html.md#ir.base.Record) #### ledger_items() Iterate `(key, entry)` ledger pairs (the ledger may be mutated while iterating). * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### link_items() Iterate `(artifact_id, {edge_type: [target]})` adjacency pairs. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### *classmethod* local(name) File-backed store under `~/.local/share/ir/corpora/`. * **Return type:** [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) #### matrix() Return `(record_ids, normalized_matrix, metas)` for brute force. Rows are L2-normalized so cosine similarity is a dot product. Empty corpora return a `(0, 0)` matrix. Caching is two-tier: an in-process cache (invalidated on the next write) backed, for file-rooted stores, by an on-disk **packed** cache — one normalized-matrix `.npy` plus its ids/metas, written once and reloaded with a single memory-mapped read. The packed cache turns a cold reopen from a per-record vector-file storm (thousands of tiny reads) into three file reads; it is cleared by a writer’s first record write, and every record write replaces a *write stamp* that a packed set must match to be published or loaded, so a set built before any later write – by this process or another – is never served (i2mint/ir#86). Another process may be writing or deleting records while this one rebuilds. A record that vanishes or is only half-written when read is left out of the result (it is “not yet written”), with a warning; such a partial result is cached in-process but never published as the packed set, so a fresh process reads the records again. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], `ndarray`, [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### *classmethod* memory() In-memory store (no dependencies); ideal for tests. * **Return type:** [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) #### metas() Return `(record_ids, metas)` **without** loading any vectors. The vector-free counterpart of [`matrix()`](_autosummary/ir.html.md#ir.CorpusStore.matrix), for ranking modes that score on text alone (`mode="lexical"`): they need candidate metadata (text + filter fields) but never the embedding matrix, so they must not pay its I/O. Reuses the in-process or packed cache when present; else reads only the `meta` view (not `vectors`), skipping a record that vanishes or is half-written mid-read (as [`matrix()`](_autosummary/ir.html.md#ir.CorpusStore.matrix) does). * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### put_record(record) Persist *record*’s metadata + vector, invalidating the search matrix. The vector is written **before** the meta: record ids are listed from the meta view, so a reader in another process that lists an id always finds its vector (`delete_record` removes in the reverse order). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### record_ids() Iterate the record ids currently stored. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### set_calibration(mode, record) Persist a calibration `record` for ranking `mode` (one per mode). `mode` keys a file in the calibration store, so it must be a non-empty string with no path separator (the real modes — `dense` / `lexical` / `hybrid` — already satisfy this). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_config(settings) Persist the corpus build *settings* (name / embedder spec + id). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_ledger_entry(key, entry) Write the ledger *entry* (version / embedder id / record ids) for *key*. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_links(artifact_id, edges) Persist *artifact_id*’s outgoing *edges* (`{edge_type: [target]}`). Empty edge-type lists are dropped; an empty result deletes the entry (no empty adjacency rows linger). Targets are stored verbatim — a bare `artifact_id` or a `[source, artifact_id]` pair. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_maintenance_state(state) Persist the maintenance bookkeeping for this corpus. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### *class* ir.CoverageReport(name, on_disk, indexed, excluded, missing=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Disk-vs-index coverage for a filesystem-backed corpus. #### *property* coverage_ratio *: [float](https://docs.python.org/3/builtins/functions.html#float)* Fraction of should-index files that are actually indexed (1.0 = full). #### *property* should_index *: [int](https://docs.python.org/3/builtins/functions.html#int)* On-disk files that pass the inclusion rule (indexed + missing). ### *class* ir.Disclosure(artifact_id, level, name, score, summary, body=None, pointer=None, metadata=, source=None, passage=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The progressively-disclosed payload for one selected artifact. #### artifact_id the artifact this payload belongs to. #### level how much was loaded — `"metadata"` (no I/O), `"body"` (the pointer’s full text), or `"bundled"` (body + extras). #### name a display name (the `name` filter field, else the id). #### score the selecting hit’s score. #### summary the matched surface text — always present, always cheap. #### body the full payload (SKILL.md / file text); `None` below `"body"` level or when the pointer could not be read. #### pointer the source pointer (`skill_path` / `path`) — the “package pointer” an agent follows to act; `None` if the hit has none. #### metadata the hit’s filter metadata, plus a `disclosure` note when a pointer was present but unreadable (stale/moved/deleted), and an `expansion` note when expansion was requested but not possible for this hit. #### source the corpus/source name the selecting hit came from (`None` when unattributed) — the attribution a federated caller needs to tell two same-id artifacts from different corpora apart. #### passage the expanded neighborhood text (`disclose(..., expand=...)`) — the mid-granularity payload between `summary` (the matched surface) and `body` (the pointer’s full text); `None` when expansion was not requested or not possible. Assembled from the corpus’s *stored records* (see [`ir.expand`](_autosummary/ir.html.md#ir.expand)), unlike `body`, which dereferences the pointer to an external resource. #### to_dict() JSON-serializable form (score cast to `float`). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.DiscoveryResult(query, mode, strategy, disclose_level, results, abstained, reason, n_retrieved, signals=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of [`discover()`](_autosummary/ir.html.md#ir.discover) — retrieve → select → (optional) disclose. The qh-exposable payload: [`to_dict()`](_autosummary/ir.html.md#ir.DiscoveryResult.to_dict) is fully JSON-serializable (lists of dicts, floats, strings, bools — no numpy, no objects), so a FastAPI facade can return it directly. #### *property* ids *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)[[str](https://docs.python.org/3/builtins/stdtypes.html#str)]* The committed artifact ids, best-first. #### to_dict() JSON-serializable result for the qh / HTTP surface. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.GraphStore(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Structural contract a traversal operator binds to — node + neighbors. Deliberately minimal (two methods) so it is satisfied by ir’s [`CorpusGraph`](_autosummary/ir.html.md#ir.CorpusGraph) *and* by any external graph: `__getitem__` resolves a node id to its scorable payload, `neighbors` lists adjacent node ids (optionally of one edge type). Granularity-agnostic on purpose — an artifact graph and a surface-level tree are both ``` `` ``` GraphStore\`\`s. `runtime_checkable` makes `isinstance(x, GraphStore)` a *structural* check on attribute **names** only (not signatures) — enough to tell a conforming adapter from an arbitrary object, but it cannot validate that `neighbors` takes the right arguments; treat it as a smoke check. ### *class* ir.IndexPlan(filter_fields=, surfaces=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)’s output for one artifact. ### *class* ir.IndexingStrategy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Decompose one artifact into filter fields + embeddable surfaces. ### *class* ir.MaintenancePolicy(reindex=, synopsis=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The background-work policy for one corpus (reindex + synopsis). #### merged(override) Layer an `override` dict on top of this policy (entry over defaults). * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### *class* ir.MaintenanceResult(name, ran, reason, reindex=False, synopsis=False, records=None, error=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What [`maintain_corpus()`](_autosummary/ir.html.md#ir.maintain_corpus) did (or would do) for one corpus. ### *class* ir.Package(, chunk_size=1500, overlap=200, embed_deps=False, deps_template=None, max_tokens=None, token_model=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Package strategy: `name + description` surface plus README chunks. Filter fields capture ownership (ours vs third-party), name, deps. AI synopsis / problem-class surfaces are a documented extension point. With `embed_deps=True`, `decompose` additionally emits one `Surface(kind="deps", granularity="field")` whose text is a prefix-form serialization of the **bare** dependency names (`deps_template`, default `_default_deps_text()`) — so a query for a domain matches a package by the libraries it depends on (e.g. `sentence-transformers` -> embeddings, `networkx` -> graphs), and the BM25 leg picks up exact dep-token matches. The deps bag is kept **separate from prose** (its own surface) so a rare library name is not diluted, and deps remain a filter field regardless. `embed_deps` defaults `False` (today’s behavior); it folds into the strategy id, so toggling it re-decomposes incrementally. The deps surface is appended **last**, leaving the `description` (position 0) and `readme_chunk` indices unchanged. Surface indexing: the `description` surface (kept whenever *name* or *description* is non-empty) occupies plan position 0, so `readme_chunk` *j* is stored with `Record.surface_index == j + 1` while its surface metadata says `chunk_index == j` — `surface_index` is plan-global, `chunk_index` per-kind (see [`ir.base.Record.make_id()`](_autosummary/ir.base.html.md#ir.base.Record.make_id)). Never derive sibling record ids from `chunk_index`; use the ledger ([`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)). `n_chunks` is stamped on readme chunks at decompose time, but corpora built before the stamp keep records without it until the artifact re-indexes (content / embedder / strategy change) — read it with `metadata.get("n_chunks")`. #### synopsis_fields *= ('name', 'description', 'readme')* The record fields a synopsis is written from (see [`ir.synopsis.with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis)): prose only. The record’s `path` (a local absolute path) and `deps` never go to an LLM. ### *class* ir.Passage(artifact_id, surface_kind, score, text, record_ids=(), source=None, surface_index=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An expanded hit: the seed’s identity + the stitched neighborhood text. `artifact_id` / `surface_kind` / `score` / `source` / `surface_index` are the **seed hit’s** — expansion never disturbs hit identity `(source, artifact_id)` or scores. `text` is the assembled neighborhood (overlap-deduped, plan order) and `record_ids` the ordered stored segments it was stitched from (empty when expansion degraded to the seed’s own text). #### to_dict() JSON-serializable form (`score` cast to `float`, ids as a list). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.Record(id, artifact_id, surface_kind, surface_index, text, vector, metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A stored, embedded surface — one row of the index. #### *static* make_id(artifact_id, surface_kind, surface_index) Deterministic storage id for a surface of an artifact. `surface_index` is the surface’s **plan-global** position — its enumeration index across *all* surfaces of the artifact’s [`IndexPlan`](_autosummary/ir.html.md#ir.IndexPlan), regardless of kind — as assigned by [`ir.index.build()`](_autosummary/ir.index.html.md#ir.index.build). On multi-kind strategies it therefore differs from per-kind counters like `metadata["chunk_index"]` (e.g. [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package): the `description` surface takes position 0, shifting `readme_chunk` *j* to `surface_index` `j+1` — and the offset is plan-dependent, since empty surfaces are dropped). Ids of already-built corpora are a stability contract: never re-derive a sibling’s id from a per-kind index — address siblings through the ledger via [`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### *class* ir.ReindexPolicy(on='source-change', every_hours=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) When to (incrementally) rebuild a corpus. ### *class* ir.SearchHit(artifact_id, surface_kind, score, text, metadata=, source=None, surface_index=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A scored record returned by retrieval (higher score = closer). Maps onto ir_09’s `Result`: `text` is the snippet, `score` the rank score, `metadata` the meta, and [`pointer`](_autosummary/ir.html.md#ir.SearchHit.pointer) the key into a resource store (ir_09 §5). [`to_dict()`](_autosummary/ir.html.md#ir.SearchHit.to_dict) is the serialization-clean form for a cross-process / subagent boundary (no numpy scalars leak). `source` is the corpus/source name the hit came from (`None` when unattributed — e.g. an ad-hoc corpus without a name). It is a first-class field, not a metadata key, because `metadata` is the strategy-owned hard-filter namespace and provenance is structural: artifact identity is only unique *within* a source, so any cross-source operation keys on `(source, artifact_id)` (see `best_per_artifact()`). `surface_index` is the stored `Record.surface_index` of the hit’s surface — its plan-global position among the artifact’s surfaces — so a hit can name *which* surface of its artifact it is (the prerequisite for sibling addressing and context expansion). `None` when unknown (e.g. a hand-built hit). It is **not** the per-kind `metadata["chunk_index"]`; see [`Record.make_id()`](_autosummary/ir.html.md#ir.Record.make_id) for why the two differ on multi-kind strategies. #### *property* pointer *: [str](https://docs.python.org/3/builtins/stdtypes.html#str) | [None](https://docs.python.org/3/builtins/constants.html#None)* The disclosure pointer on this hit, if any (see `POINTER_KEYS`). #### to_dict() JSON-serializable form (`score` cast to a Python `float`). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.Selection(selected, candidates, abstained, reason, signals=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A selector’s commitment: the chosen subset of a ranked candidate list. #### selected the committed hits, best-first (empty iff `abstained`). #### candidates the full ranked input, kept for provenance / audit. #### abstained True iff the selector committed to nothing by policy. #### reason which rule ended the commit (e.g. `"rel_threshold"`, `"score_gap"`, `"max_k"`, `"abstain:below_floor"`). #### signals concrete, defined numbers behind the decision (`top_score`, `n_candidates`, `n_selected`, `min_ratio`) — the auditable replacement for an opaque “confidence” float. #### *property* selected_ids *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)[[str](https://docs.python.org/3/builtins/stdtypes.html#str)]* The committed artifact ids, best-first. #### *property* sufficient *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* A model-free sufficiency *hint* for an agent’s Evaluator (ir_09 §3). `True` when this selection committed to at least one item (i.e. did not abstain). It is a **signal, not a directive**: the re-query / `refinement` decision and the loop belong to the agent layer (the back-edge, ir_09 §4) — `ir` derives this from its own outcome and never acts on it. #### to_dict() JSON-serializable form (scores cast to `float`). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ir.Skill Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Capability strategy: embed `name + description` only. The body (SKILL.md) is loaded post-selection and is *not* indexed; name and parent are filter fields. ### *class* ir.Surface(artifact_id, kind, text, granularity='document', metadata=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One embeddable unit derived from an artifact. `kind` names the surface type (e.g. `"description"`, `"synopsis"`, `"problem_class"`, `"chunk"`) so a query can match the *right part* of an artifact. `granularity` is a coarse hint (`"document"` / `"chunk"` / `"field"`). `metadata` is surface-local (e.g. chunk offsets). ### *class* ir.SynopsisPolicy(enabled=False, scope='recent', window_days=30, downtime_hours=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Whether/when to attach (expensive, LLM-generated) synopses. `downtime_hours` is a `[start, end)` pair of local-clock hours (wrapping past midnight is allowed, e.g. `(22, 6)`); `None` means “any time”. `scope="recent"` limits synthesis to artifacts whose timestamp is within `window_days` (corpora that expose a time signal); `scope="all"` synthesizes every artifact (bounded by incrementality). ### *class* ir.WalkPolicy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) The pluggable strategy of a walk — graph semantics, not safety. `seed` produces the initial frontier; `score` ranks a node against the query; `select` chooses which scored frontier nodes to commit/expand this step (beam/greedy — default: all, best-first); `expand` yields a node’s neighbors; `node_id` is the hashable visited-set key; `stop` is the injected sufficiency check; `to_hit` materializes a committed node as a [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) — or `None` for a *router-only* node (a summary that routes but is not itself a result). ### *class* ir.WalkState(query, max_depth, budget, visited=, results=, cache=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The operator-owned state of one [`traverse()`](_autosummary/ir.html.md#ir.traverse) call — the safety home. `visited` (node ids already committed), `budget`, and `max_depth` are the structural safety primitives the operator enforces; `results` are the emitted hits; `cache` is scratch space a policy may use (e.g. to embed the query once). A policy reads this but the *operator* enforces the bounds — a policy cannot opt out of termination. ### *class* ir.WholeText(, text_key=None, kind='document') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One surface = the entire text. Sensible default for a naive corpus. ### ir.as_retriever(corpus_or_name, \*\*search_defaults) Bind ONE corpus to the uniform `Retriever` contract. Returns `retrieve(query, **overrides) -> list[SearchHit]` that calls [`search()`](_autosummary/ir.html.md#ir.search) with `search_defaults` (a per-call kwarg overrides a bound default). A corpus *name* is resolved once via [`ir.open_corpus()`](_autosummary/ir.html.md#ir.open_corpus); pass an open [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) to skip that. The returned callable carries the bound corpus on `.corpus` for introspection. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]] ```pycon >>> retr = as_retriever(corpus, mode="hybrid", k=20) >>> hits = retr("how do I deploy the app") >>> hits = retr("deploy", filter={"owner": "me"}) ``` ### ir.build(source, , store=None, embedder=None, full=True, batch_size=256, edge_extractor=None) Build or incrementally update *source* into a [`Corpus`](_autosummary/ir.html.md#ir.Corpus). * **Parameters:** * **store** ([`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) * **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) * **full** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) * **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) * **edge_extractor** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – (`(artifact_id, filter_fields) -> {edge_type: [target]}`) that populates the corpus’s semantic `links` graph (see [`ir.graph`](_autosummary/ir.graph.html.md#module-ir.graph); pass [`ir.default_edge_extractor()`](_autosummary/ir.html.md#ir.default_edge_extractor) for the latent deps/parent edges). Ingest is **eager** — edges are (re)written for *every* in-scope artifact, a decompose-only pass with no embedding, so the graph never goes partially stale — while embedding stays fully incremental. Edges are derived state, **not** part of build identity. A rebuild *without* an extractor leaves existing edges untouched (they are only refreshed by re-running with one, and only cleared per artifact by the `full` prune below) — so dropping `edge_extractor` does not wipe a graph. * **Return type:** [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) ### ir.build_corpus(name, \*\*kwargs) Build (or update) a registered/preset corpus by *name*; returns a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus). `**kwargs` are forwarded to [`ir.build()`](_autosummary/ir.html.md#ir.build) — notably `store`, `embedder` (e.g. `"light"` for the numpy-only hashing embedder), `full` (prune artifacts no longer in the source), and `batch_size`. ### ir.calibration_key(mode, tag=None) The key an abstention floor is stored and loaded under. `mode` alone for a first-stage ranking (today’s behaviour, so existing floors keep working); `"+"` for a reranked one. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> calibration_key("hybrid") 'hybrid' >>> calibration_key("hybrid", "rerank-m+artifact") 'hybrid+rerank-m+artifact' ``` ### ir.canonical_node_id(target, , source) Canonicalize a neighbor *target* to a `(source, artifact_id)` node id. The repo’s node identity is `(source, artifact_id)` — the key a traversal visited-set must use so the same id in two corpora stays two nodes. [`CorpusGraph.neighbors()`](_autosummary/ir.html.md#ir.CorpusGraph.neighbors) returns targets in stored form: a bare `artifact_id` (implicitly in *source*, the graph it came from) or a `[source, artifact_id]` cross-corpus pair. This resolves either to the canonical tuple. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ```pycon >>> canonical_node_id("dol", source="packages") ('packages', 'dol') >>> canonical_node_id(["skills", "deploy"], source="packages") ('skills', 'deploy') ``` ### ir.collapsed_tree_policy(, summary_kinds=('description', 'synopsis', 'capability', 'document'), leaf_kinds=('chunk', 'readme_chunk'), seed_k=10) The pure-vector summary-routing / collapsed-tree [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy). Seeds on the top *seed_k* matches among `summary_kinds` surfaces and descends to each routed artifact’s `leaf_kinds` surfaces (the emitted results), scored by cosine to the query. No LLM in the loop. A summary surface is a *router* (suppressed from results) only when its artifact has leaf surfaces; on a single-surface corpus (WholeText `document`, Skill `capability`) the summaries are leaf-less and emitted directly, so the walk degrades to flat-over-summaries instead of returning nothing. The defaults keep `document` / `capability` in `summary_kinds` \*on purpose\* — that is what lets a WholeText / Skill corpus seed at all; the structural router check (above) is what keeps those seeds from being silently swallowed. * **Return type:** [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy) ```pycon >>> hits = traverse(q, corpus, policy=collapsed_tree_policy()) ``` ### ir.corpora() All registered corpus definitions, keyed by name. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.default_edge_extractor(artifact_id, filter_fields) Edges latent in the standard filter fields: `deps` → REF, `parent` → PARENT. - `deps` (Package) → `REF` edges to each dependency’s bare name (version specifiers / extras / markers stripped). Self-edges and blanks are dropped. - `parent` (Skill) → a single `PARENT` edge. A package whose `deps` name other packages in the same corpus gets intra-corpus REF edges; third-party deps become REF edges to ids not in the corpus (harmless — [`CorpusGraph.neighbors()`](_autosummary/ir.html.md#ir.CorpusGraph.neighbors) lists them, and a traversal simply finds no node to expand). Self-edges are dropped case-insensitively (`_dep_name` lower-cases, so a package `"AA"` depending on `"aa"` is recognized as a self-reference). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)] ### ir.default_policy_for_kind(kind) The smart-default [`MaintenancePolicy`](_autosummary/ir.html.md#ir.MaintenancePolicy) for a corpus `kind`. * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### ir.dependency_evidence_boost(relevant_libs, , weight, saturate=1, deps_field='deps') A `dep_boost` for [`search()`](_autosummary/ir.html.md#ir.search): reward dependencies on *relevant_libs*. Reads the candidate’s `deps` filter field (as [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) stores it), reduces each spec to its bare, PEP 503-normalized name, and returns `weight * min(n, saturate) / saturate` where `n` counts the dependencies found in *relevant_libs*. With the default `saturate=1` a single relevant dependency earns the full `weight`. A structural signal, cheap and model-free, that a prose reranker cannot be fooled out of: an async-task package whose README says “DAG” does not depend on graph libraries. *relevant_libs* is the **caller’s** call: which libraries matter for a goal is a planning decision that belongs to the agent layer, never to `ir`. `weight` is added to the score on the search’s scale (an RRF fused score is near `0.03`, a cosine near `0.5`, a sigmoid reranker in `(0, 1)`), so tune it per configuration. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`float`](https://docs.python.org/3/builtins/functions.html#float)] ```pycon >>> from ir.base import SearchHit >>> boost = dependency_evidence_boost({"networkx"}, weight=0.1) >>> boost(SearchHit("unbox", "description", 0.4, "", {"deps": ["networkx>=3"]})) 0.1 >>> boost(SearchHit("au", "description", 0.5, "", {"deps": ["anyio"]})) 0.0 ``` ### ir.disclose(selection, , level='body', loader=None, store=None, expand=None, corpus=None) Reveal the payload of each selected hit at `level` — append-only, pure. * **Parameters:** * **selection** ([`Selection`](_autosummary/ir.html.md#ir.Selection)) – a committed [`Selection`](_autosummary/ir.html.md#ir.Selection). * **level** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"metadata"` (no I/O — summary + pointer only), `"body"` (load the pointer’s full text), or `"bundled"` (body + extras; today the same as `"body"`, reserved for bundled scripts/references). * **loader** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)]]) – override the body resolver — `metadata -> str | None`. The default reads the `skill_path` / `path` pointer from disk and tolerates a missing target (returns `None`, never raises). * **store** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – a `ResourceStore` (`pointer -> payload` `Mapping`) to dereference instead of disk — ir_09 §5 pointer-passing over a `dol` store / URL map / blob storage. Mutually exclusive with `loader`. * **expand** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]]]) – a `NeighborhoodPolicy` to also stitch each hit’s neighborhood from the corpus’s stored records into [`Disclosure.passage`](_autosummary/ir.html.md#ir.Disclosure.passage) (see [`ir.expand`](_autosummary/ir.html.md#ir.expand)). Orthogonal to `level`, which governs *pointer* payloads: e.g. `level="metadata", expand=sentence_window_policy()` reads no pointer at all but still returns mid-granularity passages. Requires `corpus=`. * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – where `expand` finds each hit’s stored siblings — a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) / [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) / name, or, for cross-source selections, a `{source_name: corpus}` `Mapping` resolved per hit via `hit.source`. Only meaningful with `expand=`. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Disclosure`](_autosummary/ir.html.md#ir.Disclosure)] * **Returns:** one [`Disclosure`](_autosummary/ir.html.md#ir.Disclosure) per selected hit, best-first. This is a pure read: the [`Selection`](_autosummary/ir.html.md#ir.Selection) and its hits are never mutated, so a caller can disclose append-only without disturbing a cached ranked prefix. ### ir.discover(corpus, query, , k=10, mode='hybrid', strategy='conservative', disclose_level='metadata', filter=None, surfaces=None, max_k=3, rel=0.9, gap_ratio=0.5, min_score=None, merge='rrf', merge_weights=None, merge_rrf_k=None, loader=None, store=None, expand=None, \*\*search_kw) Find and commit to the capabilities for `query` — the one search tool. Retrieves `k` candidates, commits to a distractor-robust subset, and (optionally) discloses each committed item’s payload. This is the single agent-callable surface the capability-discovery research argues for: one tool that returns *few, high-precision* answers rather than a long candidate list the model must then filter under context rot. * **Parameters:** * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a built [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus), or a registered corpus **name** (resolved with [`ir.open_corpus()`](_autosummary/ir.html.md#ir.open_corpus)). Pass a *name* for the qh / HTTP surface — it is the JSON-friendly form. Pass a **list/tuple of names (or Corpus objects)** for single-shot *federated* discovery across several corpora: each is searched, per-source abstention floors gate before any merging, and the survivors are rank-fused (see `merge`). The caller names the sources explicitly; ir never chooses the set (source planning is the agent layer’s job, ir_09 §3). * **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the user intent. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – candidate depth retrieved before selection. Federated: `k` candidates are retrieved *per source*, and the fused ranking is also truncated to `k` before selection. * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – ranking mode — `"hybrid"` (default; `ir`’s strongest overall), `"dense"`, or `"lexical"`. * **strategy** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]]]) – selection strategy (see [`select()`](_autosummary/ir.html.md#ir.select)). * **disclose_level** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"metadata"` (default; cheap, no body I/O), `"body"`, or `"bundled"`. * **filter** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – retrieval constraints (forwarded to [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search)). * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – retrieval constraints (forwarded to [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search)). * **max_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – selection parameters (see [`select()`](_autosummary/ir.html.md#ir.select)). `min_score="auto"` loads the floor calibrated for this `(corpus, mode)` by [`ir.eval.calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) and persisted on the corpus — the opt-in that turns on absolute abstention; it falls back to no floor (with a warning) when no calibration is stored or it is stale (a different embedder). **Federated**: floors are per-(corpus, mode, embedder), so a single number cannot apply across corpora — pass `"auto"` (each source’s own calibrated floor), a `{name: floor_or_"auto"}` mapping, or `None`; a bare float raises. Floors gate each source on its own raw scores *before* fusion; the fused ranking is never floored (rank-fused scores are ordinal — ir_07/ir_08). * **rel** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – selection parameters (see [`select()`](_autosummary/ir.html.md#ir.select)). `min_score="auto"` loads the floor calibrated for this `(corpus, mode)` by [`ir.eval.calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) and persisted on the corpus — the opt-in that turns on absolute abstention; it falls back to no floor (with a warning) when no calibration is stored or it is stale (a different embedder). **Federated**: floors are per-(corpus, mode, embedder), so a single number cannot apply across corpora — pass `"auto"` (each source’s own calibrated floor), a `{name: floor_or_"auto"}` mapping, or `None`; a bare float raises. Floors gate each source on its own raw scores *before* fusion; the fused ranking is never floored (rank-fused scores are ordinal — ir_07/ir_08). * **gap_ratio** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – selection parameters (see [`select()`](_autosummary/ir.html.md#ir.select)). `min_score="auto"` loads the floor calibrated for this `(corpus, mode)` by [`ir.eval.calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) and persisted on the corpus — the opt-in that turns on absolute abstention; it falls back to no floor (with a warning) when no calibration is stored or it is stale (a different embedder). **Federated**: floors are per-(corpus, mode, embedder), so a single number cannot apply across corpora — pass `"auto"` (each source’s own calibrated floor), a `{name: floor_or_"auto"}` mapping, or `None`; a bare float raises. Floors gate each source on its own raw scores *before* fusion; the fused ranking is never floored (rank-fused scores are ordinal — ir_07/ir_08). * **min_score** ([`float`](https://docs.python.org/3/builtins/functions.html#float) | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float) | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – selection parameters (see [`select()`](_autosummary/ir.html.md#ir.select)). `min_score="auto"` loads the floor calibrated for this `(corpus, mode)` by [`ir.eval.calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score) and persisted on the corpus — the opt-in that turns on absolute abstention; it falls back to no floor (with a warning) when no calibration is stored or it is stale (a different embedder). **Federated**: floors are per-(corpus, mode, embedder), so a single number cannot apply across corpora — pass `"auto"` (each source’s own calibrated floor), a `{name: floor_or_"auto"}` mapping, or `None`; a bare float raises. Floors gate each source on its own raw scores *before* fusion; the fused ranking is never floored (rank-fused scores are ordinal — ir_07/ir_08). * **merge** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – federated only — how the per-source rankings combine: `"rrf"` (default; rank-based, scale-free — see [`ir.retrieve.fuse_hits()`](_autosummary/ir.retrieve.html.md#ir.retrieve.fuse_hits)), `"score"` (raw-score merge, valid only when all corpora share an embedder — verified, raises on mismatch), or a callable `{name: hits} -> hits`. * **merge_weights** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – federated only — per-source trust weights for `merge="rrf"` (default 1.0 each). * **merge_rrf_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – federated only — the cross-source RRF rank constant (default: [`DFLT_RRF_K`](_autosummary/ir.retrieve.html.md#ir.retrieve.DFLT_RRF_K); distinct from the within-corpus hybrid `rrf_k` in `search_kw`). * **loader** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)]]) – optional body resolver for disclosure (see [`disclose()`](_autosummary/ir.html.md#ir.disclose)). * **expand** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]]]) – a `NeighborhoodPolicy` — also stitch each committed hit’s neighborhood from its corpus’s stored records into [`Disclosure.passage`](_autosummary/ir.html.md#ir.Disclosure.passage) (retrieval-time context expansion, see [`ir.expand`](_autosummary/ir.html.md#ir.expand)). Works at any `disclose_level`; the federated form resolves each hit’s corpus via its `source`. * **\*\*search_kw** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – any other [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) keyword (`rrf_k`, `rerank`, `bm25`, …). * **Return type:** [`DiscoveryResult`](_autosummary/ir.html.md#ir.DiscoveryResult) * **Returns:** a [`DiscoveryResult`](_autosummary/ir.html.md#ir.DiscoveryResult) (`.to_dict()` for JSON / qh). Federated results add `signals["per_source"]` (per-corpus `n_retrieved` / `top_score` / `floor` / `abstained`) and each disclosure carries its `source`. ### ir.expand(hit, corpus, , policy=None) Expand *hit* into a [`Passage`](_autosummary/ir.html.md#ir.Passage) of its neighborhood in *corpus*. Fetches the hit’s sibling records through the ledger ([`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)), asks *policy* which to keep, and stitches them in plan order with overlap-aware dedupe. The default policy is a ±:data:`DFLT_WINDOW` sentence window; pass [`parent_policy()`](_autosummary/ir.html.md#ir.parent_policy) for the whole artifact, or any `NeighborhoodPolicy`. * **Parameters:** * **hit** ([`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)) – the seed [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) (its identity and score pass through to the [`Passage`](_autosummary/ir.html.md#ir.Passage) untouched). * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus), [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore), or corpus name — whatever [`records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact) accepts. Must be the corpus the hit came from. * **policy** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]]]) – which siblings make up the neighborhood (default: sentence window). A policy that selects nothing degrades the passage to the hit’s own text (`record_ids=()`) rather than returning nothing. * **Raises:** * [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – the corpus has no ledger entry for the hit’s artifact ([`ir.retrieve.NoLedgerEntry`](_autosummary/ir.retrieve.html.md#ir.retrieve.NoLedgerEntry)), or the ledger is stale — an entry listing records missing from the store. * **SeedNotFound** – the default window policy cannot find the seed among its artifact’s stored records (stale hit / wrong corpus). * [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – the policy returned records that are not siblings of the hit’s artifact (operator-enforced safety), or the seed hit lacks `surface_index` (hand-built hit) under the default window policy — use [`parent_policy()`](_autosummary/ir.html.md#ir.parent_policy), which needs no seed position. * **Return type:** [`Passage`](_autosummary/ir.html.md#ir.Passage) ### ir.fuse_hits(hits_by_source, , rrf_k=60, weights=None, identity=None, k=None) Merge per-source ranked hit lists into one ranking — by rank, not score. The cross-source counterpart of the within-corpus hybrid fusion: scores from different (corpus, mode, embedder) tuples live on incommensurable scales (ir_07: “a different model re-scales everything”), so \*\*raw scores never cross the source boundary\*\* — within each source they order and dedup that source’s hits (one scale, sound), and across sources only ranks interact, via weighted Reciprocal Rank Fusion: each hit contributes `weights[source] / (rrf_k + rank)`. * **Parameters:** * **hits_by_source** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]]) – `{source_name: ranked hits}`. Hits without a `source` are stamped with their mapping key (existing tags win, so one corpus bound under two keys still counts as one source). A `None` key is the *untagged pseudo-source*: its hits fuse as one rank group and stay unattributed (`source=None`). Within each list, duplicate artifacts — and, when `identity` is given, identity-duplicates — collapse to their best raw score before ranking, so a multi-query / multi-round pool can never double-count one artifact’s RRF mass. * **rrf_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – the RRF rank constant (standard default 60). * **weights** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – optional per-source trust dial (default 1.0 each) — a source’s contribution scales linearly, no score comparability needed. Keys naming sources absent from `hits_by_source` are ignored (a per-round pool may legitimately lack a configured source); callers with a closed source set should validate keys upfront, as federated [`ir.discover()`](_autosummary/ir.html.md#ir.discover) does. * **identity** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – how cross-source duplicates merge — see `Identity`. Default `None`: never; each `(source, artifact_id)` stays a distinct result. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – truncate the fused ranking to this many hits. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] * **Returns:** the fused hits, best-first. Each carries the fused score in `score` and keeps its pre-fusion magnitude as `metadata["source_score"]` (+ `"source_rank"`), so downstream consumers (abstention gates, LLM judges) never lose the per-source signal. When an `identity` merge combined several sources’ hits, `metadata["fused_sources"]` lists them and the representative hit is the one with the best rank. **Single-source input passes through with raw scores** (RRF of one list is that list’s order — same convention as the hybrid fusion’s single-channel fallback), so the fused-score rescaling only happens when there is genuinely something to fuse. The post-fusion `score` is ordinal: valid for ordering and relative cuts, meaningless against absolute floors — apply calibrated `min_score` floors per source, *before* fusing (see ir_07/ir_08 and `ir.discover`’s federated form). ### ir.maintain(name=None, , all=False, now=None, dry_run=False) Run due background work for one corpus (*name*) or every registered one (*all*). With neither, defaults to all registered corpora. Returns one [`MaintenanceResult`](_autosummary/ir.html.md#ir.MaintenanceResult) per corpus considered. **A named corpus raises; a sweep records.** Asking for one corpus by name is a direct request, and swallowing its error would turn a typo’d name into a result that reads as success to any caller written before `error` existed. A sweep is the unattended case, where the opposite is true: one corpus failing must not leave every corpus after it stale with nothing to say so. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult)] ### ir.maintain_corpus(name, , now=None, dry_run=False, full=True) Do the due background work for one corpus (idempotent). Reads the corpus’s resolved policy and its `last_maintained` time, decides whether a (synopsis-aware) reindex is due *and* permitted now, and — unless `dry_run` — runs the incremental build and records the run. * **Return type:** [`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult) ### ir.make_llm_formulator(, rewriter=None, prompt='Rewrite the search query into {n} short, diverse alternative search queries that would retrieve the same target documents: fix typos, expand jargon, and add synonyms, but keep each a terse search phrase. One query per line, no numbering.\\\\n\\\\nQuery: {query}', n=3, fallback=None, \*\*prompt_function_kwargs) An LLM-backed `Formulator` (rewrite / expand / multi-query). `rewriter` is an injectable `query -> str | [str, ...]` callable (a test double, or your own router); when omitted it is built lazily on `aix` (`aix.prompt_func`), so importing this module stays offline. `n` is the multi-query fan-out width. Any error or empty reply falls back to `fallback` (default: `identity_formulator()`). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.make_llm_synthesizer(, summarize=None, prompt='Write a concise synopsis (2-4 sentences) of the document below: what it is about and what questions it answers, so that a search over synopses can route to it. Output only the synopsis, no preamble.\\\\n\\\\nDocument:\\\\n{text}', model=None, synthesizer_id=None, text_key=None, text_fields=None, \*\*prompt_function_kwargs) An LLM-backed `Synthesizer` ([`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → synopsis). `summarize` is an injectable `text -> str` callable (a test double, or your own summarizer); when omitted it is built lazily on `aix` (`aix.prompt_func`) on the **first** synthesis and reused — so importing this module, and even constructing the synthesizer, stays offline. The artifact’s text is extracted with [`ir.strategy.text_of()`](_autosummary/ir.strategy.html.md#ir.strategy.text_of) using `text_key` — which [`with_synopsis()`](_autosummary/ir.html.md#ir.with_synopsis) threads from the inner strategy, so the synopsis summarizes the *same* field the strategy indexes. `text_fields` (mutually exclusive with `text_key`) names the fields of a mapping artifact whose text is sent to the LLM, and **only those**: without it, a mapping with no `text` / `text_key` has *every* string field joined and sent (see [`ir.strategy.text_of()`](_autosummary/ir.strategy.html.md#ir.strategy.text_of)) — including, for a [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) record, its local `path`. `with_synopsis` threads the inner strategy’s `synopsis_fields` (`Package` declares `("name", "description", "readme")`), so the Package default never leaks the path. The fields fold into the default `synthesizer_id`, so changing them re-synthesizes. An empty text, or any synthesis error, yields `""` (the surface is then skipped, never a fabricated summary). The returned callable carries a `synthesizer_id` attribute (default `"aix:{model}:{sha(prompt)[:12]}"`) that [`with_synopsis()`](_autosummary/ir.html.md#ir.with_synopsis) reads into the corpus’s `strategy_id` for staleness — a prompt or model change re-synthesizes. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.make_search(corpus, , name=None, description=None, k=8, mode='hybrid', rerank=None, rerank_text='surface') Return a **corpus-bound** `search(query, k=...) -> dict` tool. The returned function exposes only `query` (and `k`) — the corpus is fixed — so a connector built over it surfaces exactly one corpus and nothing else. Its `__name__` / `__doc__` are set so an MCP/agent host shows a clean tool name and description. Use this when wiring a single-corpus connector; use [`search()`](_autosummary/ir.html.md#ir.search) when the caller should choose the corpus. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ir.make_thin_description_predicate(, min_chars=40, field='description') A `Predicate` true when the artifact’s `field` is empty or thin. Reads `artifact.raw[field]` (a [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) record’s `description`); a raw that is not a mapping is a thin source when its text is shorter than `min_chars`. Whitespace is ignored. The returned callable carries a `when_id` stamp so the threshold folds into `strategy_id`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] ```pycon >>> thin = make_thin_description_predicate(min_chars=10) >>> thin(Artifact(id="a", raw={"name": "a", "description": ""})) True >>> thin(Artifact(id="b", raw={"name": "b", "description": "Tools to parse dates"})) False ``` ### ir.open_corpus(name, , embedder=None) Reopen a previously built corpus by name. The embedding model is **lazily** resolved (see `_LazyEmbedder`): the returned corpus knows its `embedder_id` from stored config immediately, but only loads the model when a dense/hybrid query actually embeds. So `ir ls`, `ir info`, and lexical-only search open a corpus without the model-load cost. Pass `embedder=` to override the stored spec. Without an override the corpus is \*\*pinned to the embedder it was built with\*\* (its stored `embedder_id`), not re-resolved from the spec: a spec like `"default"` means different models on machines with and without `sentence-transformers`, which would make the query vectors incompatible with the index. * **Return type:** [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) ### ir.parent_policy() The whole artifact (small-to-big): every stored surface, plan order. The mid-granularity analogue of `disclose(level="body")` — but assembled from the *indexed* segments rather than dereferencing the pointer, so it works for corpora whose artifacts have no on-disk body. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]] ### ir.policy_for(name) The effective [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for corpus *name*. Resolves the registered entry’s `maintenance` over its kind’s smart default over the global default (see [`ir.policy.resolve_policy()`](_autosummary/ir.policy.html.md#ir.policy.resolve_policy)). An unregistered name resolves to the global default policy. ### ir.records_for_artifact(store_or_corpus, artifact_id, , surface_kind=None) All stored records of *artifact_id*, ordered by `surface_index`. The sibling-addressing primitive beneath retrieval-time context expansion: a [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) names its artifact (and, via `surface_index`, which surface of it matched); this returns every surface of that artifact, in plan order, so an expansion policy can stitch neighbors / parents around the hit. Resolution is **ledger-backed only**: the artifact’s ledger entry lists its `record_ids`. Record ids are never re-derived from a per-kind index like `metadata["chunk_index"]` — on multi-kind strategies that index differs from the plan-global `surface_index` baked into the id (see [`ir.base.Record.make_id()`](_autosummary/ir.base.html.md#ir.base.Record.make_id)), so derivation would fetch wrong or missing siblings. * **Parameters:** * **store_or_corpus** – a [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore), anything carrying one as `.store` (e.g. a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus)), or a corpus *name* — resolved straight to its local store: sibling lookup never embeds, so unlike [`ir.open_corpus()`](_autosummary/ir.html.md#ir.open_corpus) no embedder is loaded. * **artifact_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the artifact whose surfaces to fetch. * **surface_kind** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict to one surface kind (e.g. `"readme_chunk"`); a known artifact with no surfaces of that kind yields `[]`. * **Raises:** * [**NoLedgerEntry**](_autosummary/ir.retrieve.html.md#ir.retrieve.NoLedgerEntry) – the ledger has no entry for *artifact_id* (an unknown artifact, or a corpus built without [`ir.index.build()`](_autosummary/ir.index.html.md#ir.index.build)’s ledger bookkeeping). A `KeyError` subclass. * [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – an entry exists but lists a record missing from the store (a stale ledger: interrupted build or out-of-band `delete_record`) — data corruption, named in the message. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)] ### ir.register(name, kind, , embedder='default', strategy=None, maintenance=None, storage=None, \*\*params) Register (or overwrite) a named corpus definition. Beyond the v1 `kind` / `embedder` / `params`, an entry may now carry (all optional, with smart per-kind defaults applied at resolution time — see [`ir.policy`](_autosummary/ir.policy.html.md#module-ir.policy)): - `strategy` — an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) (or a `{"name", "params"}` spec) persisted so the corpus’s *segmentation* is stable across rebuilds. `None` keeps the preset’s default strategy. - `maintenance` — the background-work policy dict (`reindex` / `synopsis`; validated here, see [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy)). - `storage` — the persistence backend (default `{"backend": "local"}`). Entries written by older `ir` (none of these keys) keep working unchanged. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.reports_coverage(name='reports', , projects_root=None, indexed_ids=None, exclude_dirs=frozenset({'.git', '.hg', '.ipynb_checkpoints', '.obsidian', '.svn', '.tox', '.venv', '_\_pycache_\_', 'build', 'dist', 'node_modules', 'site-packages', 'venv'})) Diff the reports docs tree on disk against the built `name` corpus. Walks every `*/*/docs` and `*/*/misc/docs` folder recursively (raw, no filtering), classifies each `*.md` with the shared inclusion SSOT, and reports how many *includable* reports are indexed vs. silently missing. A `coverage_ratio` below 1.0 means on-disk reports are absent from the index — rebuild with `ir build `. `indexed_ids` (the built corpus’s artifact ids) is read from the registered `name` corpus by default; pass it explicitly (with `projects_root`) to diff an arbitrary tree against an arbitrary index without opening a corpus. * **Return type:** [`CoverageReport`](_autosummary/ir.coverage.html.md#ir.coverage.CoverageReport) ### ir.resolve_policy(entry) The effective policy for a registry `entry`: entry over kind over global. A v1 entry (no `maintenance` key) resolves to its kind’s smart default, so existing corpora gain a sensible policy without a migration. * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### ir.resolve_reranker(spec) The reranker `spec` names, or `None` when there is none to apply. `spec` is `None` (no rerank), any `ef.Reranker` callable (returned as is), or the string `"cross-encoder"` / `"cross-encoder:"` – ef’s sentence-transformers cross-encoder, built lazily and cached per model name (loading one is the expensive part). If its optional dependency is absent this warns once and returns `None`, so the caller degrades to first-stage ranking; resolve *before* deriving a calibration key so the degraded run uses the plain key. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ```pycon >>> resolve_reranker(len) is len True >>> resolve_reranker("nope") Traceback (most recent call last): ... ValueError: unknown rerank spec 'nope'; expected a callable, 'cross-encoder' or 'cross-encoder:'. ``` ### ir.retriever_for(name, \*\*search_defaults) A [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) bound to the registered corpus *name*. Opens the corpus (it must have been built) and wraps it with [`ir.as_retriever()`](_autosummary/ir.html.md#ir.as_retriever); `search_defaults` (e.g. `mode="hybrid"`) bind to every call. ### ir.retrievers(\*\*search_defaults) A lazy `Mapping[name, Retriever]` view over the registry (ir_09 §8). The query-time projection of the build-recipe registry: each value is a ready-to-call [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever). This is the source-registry facade an orchestration layer (`raglab`) consumes — it never opens a corpus until the key is accessed, and always reflects the current `registered()` set. `search_defaults` apply to every source. * **Return type:** [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.reverse_dependency_policy(, seeds, edge_type='REF', fan_in_weight=0.0, proximity_weight=0.0) A [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy) that walks dependency edges backwards from *seeds*. *seeds* are library ids the **caller** deems relevant to the query (ir never chooses them: picking the domain libraries for a goal is the planner’s job, in the agent layer). The walk commits each seed that is an artifact of the corpus, then its dependents (the artifacts whose `edge_type` edges point at it), then theirs, up to [`traverse()`](_autosummary/ir.html.md#ir.traverse)’s `max_depth`. A seed outside the corpus (a third-party library) routes without being a result. So a package that depends on a domain library but never says so in prose is still found. Each result scores its best surface’s cosine to the query, plus `fan_in_weight` times its normalized fan-in (its share of the most depended-on target, a centrality prior), plus `proximity_weight / (1 + hops)` (closer to a seed scores higher). Both weights default to `0`, so the score is plain cosine. Hits carry `metadata["walk_depth"]`, `metadata["fan_in"]` and, for dependents, `metadata["seed"]`. The store passed to [`traverse()`](_autosummary/ir.html.md#ir.traverse) is a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) built with an edge extractor (e.g. [`ir.default_edge_extractor()`](_autosummary/ir.html.md#ir.default_edge_extractor), whose `REF` edges run from a package to its dependencies). Fuse the walk’s hits with flat search through [`ir.fuse_hits()`](_autosummary/ir.html.md#ir.fuse_hits). * **Return type:** [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy) ```pycon >>> policy = reverse_dependency_policy(seeds=["networkx"]) >>> hits = traverse(query, corpus, policy=policy) ``` ### ir.search(corpus, query, \*\*kwargs) Search a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) (or a corpus *name*, reopened lazily). Thin facade over [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search); `**kwargs` are forwarded to it — the useful ones are `k` (how many hits), `mode` (`"dense"` / `"lexical"` / `"hybrid"`), `filter` (a `vd` Mongo-style metadata filter), `surfaces` (restrict to surface kinds), and `per_artifact` (collapse to the best surface per artifact). See [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) for the full signature and defaults. ### ir.search_tool(query, , corpus, k=8, mode='hybrid', filter=None, rerank=None, rerank_text='surface') Search a named `ir` corpus and return a JSON-serializable result dict. Thin agent-callable wrapper over [`ir.discover()`](_autosummary/ir.html.md#ir.discover) — returns its `.to_dict()` (committed results, scores, disclosures), fit to hand straight back from an MCP tool or HTTP endpoint. The *corpus* is a parameter, so this single function serves any corpus. * **Parameters:** * **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the natural-language query. * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a registered corpus **name** (str), a list of names (federated search), or a built [`ir.Corpus`](_autosummary/ir.html.md#ir.Corpus). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum number of results. * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"dense"` | `"lexical"` | `"hybrid"`. * **filter** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – optional `vd` Mongo-style metadata filter (hard pre-filter). * **rerank** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – optional second-stage reranker (e.g. `"cross-encoder"`); off by default. `rerank_text` picks the text it reads. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.select(hits, , strategy='conservative', max_k=3, rel=0.9, gap_ratio=0.5, min_score=None) Commit to a subset of ranked `hits` — the selection stage. * **Parameters:** * **hits** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]) – ranked [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)s (best first), as returned by [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). * **strategy** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]]]) – `"conservative"` (the default distractor-robust commit), one of `"top_k"` / `"abs_threshold"` / `"rel_threshold"` / `"score_gap"`, or any `Selector` callable (`hits -> subset`) — e.g. one built by `make_llm_selector()`. * **max_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – never commit to more than this many (caps distractor exposure). * **rel** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – relative-to-top keep threshold for `"conservative"` / the ratio for `"rel_threshold"`. * **gap_ratio** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – score-gap elbow ratio — used by the `"score_gap"` strategy only (`"conservative"` deliberately uses `rel` alone, not an elbow; see this module’s docstring). * **min_score** ([`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – optional absolute floor; with `"conservative"` the selector abstains when even the top hit falls below it (also usable as the `"abs_threshold"` floor). * **Return type:** [`Selection`](_autosummary/ir.html.md#ir.Selection) * **Returns:** a [`Selection`](_autosummary/ir.html.md#ir.Selection). `abstained` is True iff `selected` is empty. ### ir.sentence_window_policy(k=1) ±\*k\* same-kind neighbors around the seed (NEXT/PREV expansion). The window runs over the artifact’s surfaces of the **seed’s kind only**, in plan order — a `readme_chunk` window never swallows the `description` surface. `k=0` selects just the seed’s own record. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]], [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)]] ### ir.tag_source(hits, source) Stamp *source* on every hit that doesn’t already carry one. Existing tags win: a hit already attributed to a corpus keeps that attribution (so re-tagging under a different registry key cannot double-count one corpus as two sources). A `None` source is the untagged pseudo-source — hits pass through unattributed. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] ### ir.traverse(query, store, , policy, max_depth=2, node_budget=64, k=10) Walk *store* from *query* under *policy*, returning the top-*k* hits. The loop — *score the frontier → select → commit → expand* — is the operator’s; the **safety primitives are non-negotiable and enforced here**: a node id is committed at most once (the visited-set), expansion stops at `max_depth`, and no more than `node_budget` nodes are ever committed. A policy whose `expand` cycles forever and whose `stop` never fires still terminates. * **Parameters:** * **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the user intent. * **store** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – passed to *policy* verbatim — a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) for [`collapsed_tree_policy()`](_autosummary/ir.html.md#ir.collapsed_tree_policy), a [`CorpusGraph`](_autosummary/ir.graph.html.md#ir.graph.CorpusGraph) for an artifact-link policy. The operator never inspects it. * **policy** ([`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy)) – the [`WalkPolicy`](_autosummary/ir.html.md#ir.WalkPolicy) (e.g. [`collapsed_tree_policy()`](_autosummary/ir.html.md#ir.collapsed_tree_policy)). * **max_depth** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum expansion depth from a seed (safety). * **node_budget** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum nodes committed (safety). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of hits to return. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] * **Returns:** the committed hits, best-first, top-*k* — each a [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) with `metadata["walk_depth"]` / `["seed"]`. ### ir.with_synopsis(strategy, , synthesize=None, synthesizer_id=None, synopsis_kind='synopsis', when='always', when_id=None) Wrap *strategy* to add one LLM-derived `synopsis` surface per artifact. * **Parameters:** * **strategy** ([`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)) – the inner [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) (`Chunked`, `Package`, …). Its surfaces are kept; the synopsis is prepended. * **synthesize** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – an injectable `Artifact -> str` (test double / custom summarizer). Omitted → [`make_llm_synthesizer()`](_autosummary/ir.html.md#ir.make_llm_synthesizer) (lazy `aix`). * **synthesizer_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – explicit identity stamp for staleness (recommended when injecting an unnamed callable / lambda). Omitted → the synthesizer’s own `synthesizer_id` / `__qualname__`. * **synopsis_kind** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the surface kind (default `"synopsis"`, a summary kind). * **when** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]]) – `"always"` (default: every artifact) or a predicate `Artifact -> bool`; the synthesizer is called, and a synopsis surface added, only where it returns true (e.g. `thin_description_predicate`). A predicate that raises propagates. * **when_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – explicit identity stamp for an unnamed predicate (staleness); omitted → the predicate’s `when_id` / `__qualname__`. * **Return type:** [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) * **Returns:** an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) usable anywhere a strategy is — `ir.CorpusSource.from_mapping(docs, name=..., strategy=with_synopsis(...))`. ```pycon >>> strat = with_synopsis(Chunked(), synthesize=lambda a: "a summary") ``` ### Modules | [`base`](_autosummary/ir.base.html.md#module-ir.base) | Core data model for `ir`. | |--------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| | [`cli`](_autosummary/ir.cli.html.md#module-ir.cli) | Command-line surface for `ir` (dispatched by cw). | | [`config`](_autosummary/ir.config.html.md#module-ir.config) | Filesystem locations and process-wide defaults for `ir`. | | [`coverage`](_autosummary/ir.coverage.html.md#module-ir.coverage) | Corpus-coverage diagnostic — what's on disk vs. what's actually indexed. | | [`embed`](_autosummary/ir.embed.html.md#module-ir.embed) | Embedder resolution for `ir` — a decent local default, a light fallback. | | [`eval`](_autosummary/ir.eval.html.md#module-ir.eval) | Capability-discovery evaluation — measuring how well a corpus is *found*. | | [`eval_gen`](_autosummary/ir.eval_gen.html.md#module-ir.eval_gen) | LLM-backed generation of evaluation cases for [`ir.eval`](_autosummary/ir.eval.html.md#module-ir.eval). | | [`formulate`](_autosummary/ir.formulate.html.md#module-ir.formulate) | Query formulation — the Formulator seam (ir_09 §3). | | [`graph`](_autosummary/ir.graph.html.md#module-ir.graph) | The semantic link graph — typed edges between artifacts (report 12). | | [`index`](_autosummary/ir.index.html.md#module-ir.index) | The indexing pipeline and incremental maintenance. | | [`maintenance`](_autosummary/ir.maintenance.html.md#module-ir.maintenance) | Idempotent background-work runner — `ir maintain` (issue #58). | | [`policy`](_autosummary/ir.policy.html.md#module-ir.policy) | Per-corpus policy — how a corpus is segmented/stored and what background work it gets. | | [`registry`](_autosummary/ir.registry.html.md#module-ir.registry) | Named-corpus registry — persistent, reusable corpus definitions. | | [`requirements`](_autosummary/ir.requirements.html.md#module-ir.requirements) | Dependency check for `ir`: name what is missing and how to install it. | | [`rerank`](_autosummary/ir.rerank.html.md#module-ir.rerank) | The precision stage: a cross-encoder rerank over the first-stage candidates. | | [`retrieve`](_autosummary/ir.retrieve.html.md#module-ir.retrieve) | Retrieval — hard metadata filtering + dense / lexical / hybrid ranking. | | [`schedule`](_autosummary/ir.schedule.html.md#module-ir.schedule) | Install and operate the OS job that runs `ir maintain` — `ir schedule` (issue #75). | | [`sources`](_autosummary/ir.sources.html.md#module-ir.sources) | Defining a corpus source — an abstract strategy plus parameters. | | [`store`](_autosummary/ir.store.html.md#module-ir.store) | Persistence for `ir` — the repository layer over `dol` key-value views. | | [`strategy`](_autosummary/ir.strategy.html.md#module-ir.strategy) | Indexing strategies — the "what do we index?" seam. | | [`synopsis`](_autosummary/ir.synopsis.html.md#module-ir.synopsis) | Synopsis surfaces — LLM-derived summaries as an indexed surface (report 12). | | [`tools`](_autosummary/ir.tools.html.md#module-ir.tools) | Agent-callable tool surface over `ir` — plain functions returning JSON-ready dicts, deliberately MCP/HTTP-agnostic. | # _autosummary/ir.index.html.md # ir.index The indexing pipeline and incremental maintenance. [`build()`](_autosummary/ir.index.html.md#ir.index.build) turns a [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) into a queryable [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus), persisting records through a [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore). It is **incremental and idempotent**: each artifact’s change signal is compared to the ledger, and only new, changed, or re-modeled artifacts are decomposed and embedded. Artifacts that vanished from the source are pruned (full-refresh). Re-running `build` on an unchanged source is a near-no-op. This is the light maintenance path — content-hash CRUD over a flat store, which is the right tool for the small corpora `ir` targets. The heavier content-addressed artifact-graph (`ef.artifact_graph`) is the documented upgrade for corpora large enough that recomputing surfaces is the bottleneck. ### Functions | [`build`](_autosummary/ir.index.html.md#ir.index.build)(source, \*[, store, embedder, full, ...]) | Build or incrementally update *source* into a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus). | |----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | [`open_corpus`](_autosummary/ir.index.html.md#ir.index.open_corpus)(name, \*[, embedder]) | Reopen a previously built corpus by name. | ### Classes | [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus)(name, store, embedder, embedder_id) | A built, queryable corpus: a store plus its embedder. | |-----------------------------------------------------------------------------------------------|---------------------------------------------------------| ### *class* ir.index.Corpus(name, store, embedder, embedder_id) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A built, queryable corpus: a store plus its embedder. #### search(query, \*\*kwargs) Search this corpus for *query*. `**kwargs` (`k` / `mode` / `filter` / `surfaces` / `per_artifact` / …) are forwarded to [`ir.retrieve.search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). ### ir.index.build(source, , store=None, embedder=None, full=True, batch_size=256, edge_extractor=None) Build or incrementally update *source* into a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus). * **Parameters:** * **store** ([`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) * **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) * **full** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) * **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) * **edge_extractor** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – (`(artifact_id, filter_fields) -> {edge_type: [target]}`) that populates the corpus’s semantic `links` graph (see [`ir.graph`](_autosummary/ir.graph.html.md#module-ir.graph); pass [`ir.default_edge_extractor()`](_autosummary/ir.html.md#ir.default_edge_extractor) for the latent deps/parent edges). Ingest is **eager** — edges are (re)written for *every* in-scope artifact, a decompose-only pass with no embedding, so the graph never goes partially stale — while embedding stays fully incremental. Edges are derived state, **not** part of build identity. A rebuild *without* an extractor leaves existing edges untouched (they are only refreshed by re-running with one, and only cleared per artifact by the `full` prune below) — so dropping `edge_extractor` does not wipe a graph. * **Return type:** [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) ### ir.index.open_corpus(name, , embedder=None) Reopen a previously built corpus by name. The embedding model is **lazily** resolved (see `_LazyEmbedder`): the returned corpus knows its `embedder_id` from stored config immediately, but only loads the model when a dense/hybrid query actually embeds. So `ir ls`, `ir info`, and lexical-only search open a corpus without the model-load cost. Pass `embedder=` to override the stored spec. Without an override the corpus is \*\*pinned to the embedder it was built with\*\* (its stored `embedder_id`), not re-resolved from the spec: a spec like `"default"` means different models on machines with and without `sentence-transformers`, which would make the query vectors incompatible with the index. * **Return type:** [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) # _autosummary/ir.maintenance.html.md # ir.maintenance Idempotent background-work runner — `ir maintain` (issue #58). The *executing* half of ir’s maintenance story (the declarative half is [`ir.policy`](_autosummary/ir.policy.html.md#module-ir.policy)). [`maintain()`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain) reads each corpus’s resolved [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) and does the work that is **due**: - an **incremental rebuild** when reindex is due (`source-change` is always due — the build is a near-no-op when nothing changed; `interval` only when stale; `manual` never); - when `synopsis.enabled`, the rebuild’s strategy is wrapped in [`ir.with_synopsis()`](_autosummary/ir.html.md#ir.with_synopsis), so new / changed artifacts gain an LLM synopsis — > and because that build may call an LLM, it is run \*\*only inside the policy’s > downtime window\*\*. It is safe to call as often as a scheduler likes: it no-ops when nothing is due, and records `last_maintained` so interval policies converge. ir runs the work; *scheduling* it is external — a cron / launchd entry calls `ir maintain --all` every N minutes, and the downtime window lives in the policy (data), not in ir. [`ir.schedule`](_autosummary/ir.schedule.html.md#module-ir.schedule) installs that entry for you (`ir schedule`) and reports on it; it writes a definition and exits, so the executor is still cron / launchd: ```default # what `ir schedule --every 15m` writes on a cron machine: */15 * * * * -m ir maintain --all >> ~/.cache/ir/maintain.log 2>&1 ``` Because a scheduled run has nobody watching it, [`maintain()`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain) is fault-isolated per corpus: one corpus failing is recorded on its own result and the sweep carries on, rather than aborting and leaving the corpora after it silently unmaintained. ### Module Attributes | [`DFLT_STALE_LOCK_AFTER`](_autosummary/ir.maintenance.html.md#ir.maintenance.DFLT_STALE_LOCK_AFTER) | A lock older than this is assumed to belong to a run that was killed rather than one still working. | |--------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| ### Functions | [`lock_path`](_autosummary/ir.maintenance.html.md#ir.maintenance.lock_path)() | Where the single-run lock lives (regenerable, so the cache dir). | |--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| | [`maintain`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain)([name, all, now, dry_run]) | Run due background work for one corpus (*name*) or every registered one (*all*). | | [`maintain_corpus`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain_corpus)(name, \*[, now, dry_run, full]) | Do the due background work for one corpus (idempotent). | | [`single_run`](_autosummary/ir.maintenance.html.md#ir.maintenance.single_run)(\*[, path, stale_after]) | Hold the maintenance lock, or raise [`MaintenanceBusy`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceBusy). | ### Classes | [`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult)(name, ran, reason[, ...]) | What [`maintain_corpus()`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain_corpus) did (or would do) for one corpus. | |------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| ### Exceptions | [`MaintenanceBusy`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceBusy) | Another maintenance run holds the lock. | |--------------------------------------------------------------------|-------------------------------------------| ### ir.maintenance.DFLT_STALE_LOCK_AFTER *= datetime.timedelta(seconds=21600)* A lock older than this is assumed to belong to a run that was killed rather than one still working. Generous, because a first full build of a large corpus legitimately takes a long time and reclaiming a *live* lock is the bad outcome. ### *exception* ir.maintenance.MaintenanceBusy Bases: [`RuntimeError`](https://docs.python.org/3/builtins/exceptions.html#RuntimeError) Another maintenance run holds the lock. ### *class* ir.maintenance.MaintenanceResult(name, ran, reason, reindex=False, synopsis=False, records=None, error=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What [`maintain_corpus()`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain_corpus) did (or would do) for one corpus. ### ir.maintenance.lock_path() Where the single-run lock lives (regenerable, so the cache dir). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ir.maintenance.maintain(name=None, , all=False, now=None, dry_run=False) Run due background work for one corpus (*name*) or every registered one (*all*). With neither, defaults to all registered corpora. Returns one [`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult) per corpus considered. **A named corpus raises; a sweep records.** Asking for one corpus by name is a direct request, and swallowing its error would turn a typo’d name into a result that reads as success to any caller written before `error` existed. A sweep is the unattended case, where the opposite is true: one corpus failing must not leave every corpus after it stale with nothing to say so. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult)] ### ir.maintenance.maintain_corpus(name, , now=None, dry_run=False, full=True) Do the due background work for one corpus (idempotent). Reads the corpus’s resolved policy and its `last_maintained` time, decides whether a (synopsis-aware) reindex is due *and* permitted now, and — unless `dry_run` — runs the incremental build and records the run. * **Return type:** [`MaintenanceResult`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceResult) ### ir.maintenance.single_run(, path=None, stale_after=None) Hold the maintenance lock, or raise [`MaintenanceBusy`](_autosummary/ir.maintenance.html.md#ir.maintenance.MaintenanceBusy). Two maintenance runs on one corpus store are not safe: the packed store rewrites `matrix`/`ids`/`metas` as separate files, so interleaved writers can leave a matrix whose rows no longer line up with its ids — an index that answers confidently and wrongly, with nothing raised. launchd already runs one instance per label, but that does not cover cron (which happily stacks runs) or the manual `ir maintain --all` this tool’s own menu suggests while the agent may be mid-run. A lock whose owning process is gone, or which is older than *stale_after*, is reclaimed — a killed run must not wedge maintenance forever. # _autosummary/ir.policy.html.md # ir.policy Per-corpus policy — how a corpus is segmented/stored and what background work it gets. This is the *declarative* half of ir’s maintenance story (issue #58): a corpus’s policy lives as **data** in its registry entry, and ir resolves an effective policy by layering `entry` over per-`kind` defaults over a global default. An idempotent [`ir.maintenance.maintain()`](_autosummary/ir.maintenance.html.md#ir.maintenance.maintain) reads the policy and does the due work; *scheduling* that work (cron / launchd, or an orchestration layer’s budget governor) stays **external** — ir never imports an orchestration layer, and there is no global `Settings` singleton (policy is per-corpus data, injected, not a process-wide mutable). The light path stays a one-liner: every field has a smart default, so `ir build skills` needs no policy at all. The three policy axes: - **reindex** — *when* to rebuild: `"source-change"` (default; rebuild is a near-no-op when nothing changed, so it is always safe to run), `"interval"` (only when older than `every_hours`), or `"manual"` (never automatic). - **synopsis** — *whether/when* to attach LLM synopses (the expensive, off-by-default work): only the `recent` slice, only during `downtime_hours`. Synopsis is realized as a strategy wrapper ([`ir.with_synopsis()`](_autosummary/ir.html.md#ir.with_synopsis)), so it is incremental — only new/changed artifacts are synthesized. - **storage** — the persistence backend. Today only `"local"` (the file store); a `vd.Collection` backend is the documented future seam (issue #28). ### Module Attributes | [`REINDEX_TRIGGERS`](_autosummary/ir.policy.html.md#ir.policy.REINDEX_TRIGGERS) | Accepted `reindex.on` triggers. | |---------------------------------------------------------------------|-------------------------------------------------------------------| | [`STORAGE_BACKENDS`](_autosummary/ir.policy.html.md#ir.policy.STORAGE_BACKENDS) | Storage backends ir can resolve today. | | [`GLOBAL_DEFAULT`](_autosummary/ir.policy.html.md#ir.policy.GLOBAL_DEFAULT) | The global fallback policy (any kind without a specific default). | | [`DEFAULTS_BY_KIND`](_autosummary/ir.policy.html.md#ir.policy.DEFAULTS_BY_KIND) | Per-kind maintenance defaults. | ### Functions | [`default_policy_for_kind`](_autosummary/ir.policy.html.md#ir.policy.default_policy_for_kind)(kind) | The smart-default [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for a corpus `kind`. | |-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| | [`in_downtime`](_autosummary/ir.policy.html.md#ir.policy.in_downtime)(policy, now) | Whether `now` falls inside the synopsis `downtime_hours` window. | | [`is_reindex_due`](_autosummary/ir.policy.html.md#ir.policy.is_reindex_due)(policy, last_maintained, now) | Whether a reindex is due under `policy` given the last-maintained time. | | [`resolve_policy`](_autosummary/ir.policy.html.md#ir.policy.resolve_policy)(entry) | The effective policy for a registry `entry`: entry over kind over global. | | [`resolve_storage`](_autosummary/ir.policy.html.md#ir.policy.resolve_storage)(entry) | The effective storage spec for an `entry` (default `{"backend": "local"}`). | ### Classes | [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy)([reindex, synopsis]) | The background-work policy for one corpus (reindex + synopsis). | |-------------------------------------------------------------------------------------------|-------------------------------------------------------------------| | [`ReindexPolicy`](_autosummary/ir.policy.html.md#ir.policy.ReindexPolicy)([on, every_hours]) | When to (incrementally) rebuild a corpus. | | [`SynopsisPolicy`](_autosummary/ir.policy.html.md#ir.policy.SynopsisPolicy)([enabled, scope, ...]) | Whether/when to attach (expensive, LLM-generated) synopses. | ### ir.policy.DEFAULTS_BY_KIND *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)]* *= {'files': {'reindex': {'on': 'source-change'}}, 'packages': {'reindex': {'on': 'source-change'}}, 'records': {'reindex': {'every_hours': 24, 'on': 'interval'}}, 'reports': {'reindex': {'on': 'source-change'}}, 'sessions': {'reindex': {'every_hours': 24, 'on': 'interval'}, 'synopsis': {'downtime_hours': [2, 6], 'enabled': False, 'scope': 'recent', 'window_days': 30}}, 'skills': {'reindex': {'on': 'source-change'}}}* Per-kind maintenance defaults. The rule of thumb encoded here: *small, fully-enumerable* corpora (skills/packages/reports/files) rebuild on source change (cheap, exact); *large, append-mostly, time-stamped* corpora (sessions) rebuild on an interval and keep synopsis off by default, with a downtime window ready for when it is turned on. New kinds register here. ### ir.policy.GLOBAL_DEFAULT *= MaintenancePolicy(reindex=ReindexPolicy(on='source-change', every_hours=None), synopsis=SynopsisPolicy(enabled=False, scope='recent', window_days=30, downtime_hours=None))* The global fallback policy (any kind without a specific default). ### *class* ir.policy.MaintenancePolicy(reindex=, synopsis=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The background-work policy for one corpus (reindex + synopsis). #### merged(override) Layer an `override` dict on top of this policy (entry over defaults). * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### ir.policy.REINDEX_TRIGGERS *= ('source-change', 'interval', 'manual')* Accepted `reindex.on` triggers. ### *class* ir.policy.ReindexPolicy(on='source-change', every_hours=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) When to (incrementally) rebuild a corpus. ### ir.policy.STORAGE_BACKENDS *= ('local',)* Storage backends ir can resolve today. `"vd"` (vd.Collection) is the documented future backend, gated on issue #28. ### *class* ir.policy.SynopsisPolicy(enabled=False, scope='recent', window_days=30, downtime_hours=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Whether/when to attach (expensive, LLM-generated) synopses. `downtime_hours` is a `[start, end)` pair of local-clock hours (wrapping past midnight is allowed, e.g. `(22, 6)`); `None` means “any time”. `scope="recent"` limits synthesis to artifacts whose timestamp is within `window_days` (corpora that expose a time signal); `scope="all"` synthesizes every artifact (bounded by incrementality). ### ir.policy.default_policy_for_kind(kind) The smart-default [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for a corpus `kind`. * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### ir.policy.in_downtime(policy, now) Whether `now` falls inside the synopsis `downtime_hours` window. `None` window means “any time”. A window whose start hour is greater than its end hour wraps past midnight (e.g. `(22, 6)` is 22:00–06:00). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ir.policy.is_reindex_due(policy, last_maintained, now) Whether a reindex is due under `policy` given the last-maintained time. - `source-change` is always due (the build is incremental and a no-op when nothing changed, so running it cannot do harm). - `interval` is due when never maintained or older than `every_hours`. - `manual` is never automatically due. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ir.policy.resolve_policy(entry) The effective policy for a registry `entry`: entry over kind over global. A v1 entry (no `maintenance` key) resolves to its kind’s smart default, so existing corpora gain a sensible policy without a migration. * **Return type:** [`MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) ### ir.policy.resolve_storage(entry) The effective storage spec for an `entry` (default `{"backend": "local"}`). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) # _autosummary/ir.registry.html.md # ir.registry Named-corpus registry — persistent, reusable corpus definitions. A registry entry records *how* to (re)build a corpus — its `kind` (a source preset), parameters, and embedder spec — so a corpus becomes a stable name you can build once and query across sessions. The registry is a single JSON file under the config dir (`~/.config/ir/corpora.json`). Presets map to [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) constructors: - `skills` → `CorpusSource.from_skills()` - `packages` → `CorpusSource.from_packages()` - `reports` → `CorpusSource.from_md_reports()` - `sessions` → `CorpusSource.from_claude_sessions()` - `files` → `CorpusSource.from_files()` (needs `root`; optional `pattern`) - `records` → `CorpusSource.from_records()` (needs `fetcher`, a `"module:attr"` reference; optional `metadata_keys` / `id_key` / `text_key`) — the seam for a corpus whose records another package owns Unregistered preset names (`skills`/`packages`/`reports`/`sessions`) are auto-registered with defaults on first use, so `ir build skills` just works. ### Module Attributes | [`PARAMETRIC_KINDS`](_autosummary/ir.registry.html.md#ir.registry.PARAMETRIC_KINDS) | Kinds that need parameters, so they are never auto-registered from a bare name. | |---------------------------------------------------------------------|-----------------------------------------------------------------------------------| ### Functions | [`get`](_autosummary/ir.registry.html.md#ir.registry.get)(name) | The registry entry for *name*, or `None`. | |--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | [`policy_for`](_autosummary/ir.registry.html.md#ir.registry.policy_for)(name) | The effective [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for corpus *name*. | | [`register`](_autosummary/ir.registry.html.md#ir.registry.register)(name, kind, \*[, embedder, ...]) | Register (or overwrite) a named corpus definition. | | [`registered`](_autosummary/ir.registry.html.md#ir.registry.registered)() | All registered corpus definitions, keyed by name. | | [`retriever_for`](_autosummary/ir.registry.html.md#ir.registry.retriever_for)(name, \*\*search_defaults) | A [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) bound to the registered corpus *name*. | | [`retrievers`](_autosummary/ir.registry.html.md#ir.registry.retrievers)(\*\*search_defaults) | A lazy `Mapping[name, Retriever]` view over the registry (ir_09 §8). | | [`source_for`](_autosummary/ir.registry.html.md#ir.registry.source_for)(name) | Resolve *name* to a source, auto-registering a preset if needed. | | [`source_from_entry`](_autosummary/ir.registry.html.md#ir.registry.source_from_entry)(name, entry) | Reconstruct a `CorpusSource` from a registry entry. | | [`unregister`](_autosummary/ir.registry.html.md#ir.registry.unregister)(name) | Remove *name* from the registry (does not delete built data). | ### ir.registry.PARAMETRIC_KINDS *= ('files', 'records')* Kinds that need parameters, so they are never auto-registered from a bare name. ### ir.registry.get(name) The registry entry for *name*, or `None`. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.registry.policy_for(name) The effective [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy) for corpus *name*. Resolves the registered entry’s `maintenance` over its kind’s smart default over the global default (see [`ir.policy.resolve_policy()`](_autosummary/ir.policy.html.md#ir.policy.resolve_policy)). An unregistered name resolves to the global default policy. ### ir.registry.register(name, kind, , embedder='default', strategy=None, maintenance=None, storage=None, \*\*params) Register (or overwrite) a named corpus definition. Beyond the v1 `kind` / `embedder` / `params`, an entry may now carry (all optional, with smart per-kind defaults applied at resolution time — see [`ir.policy`](_autosummary/ir.policy.html.md#module-ir.policy)): - `strategy` — an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) (or a `{"name", "params"}` spec) persisted so the corpus’s *segmentation* is stable across rebuilds. `None` keeps the preset’s default strategy. - `maintenance` — the background-work policy dict (`reindex` / `synopsis`; validated here, see [`ir.policy.MaintenancePolicy`](_autosummary/ir.policy.html.md#ir.policy.MaintenancePolicy)). - `storage` — the persistence backend (default `{"backend": "local"}`). Entries written by older `ir` (none of these keys) keep working unchanged. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.registry.registered() All registered corpus definitions, keyed by name. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.registry.retriever_for(name, \*\*search_defaults) A [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) bound to the registered corpus *name*. Opens the corpus (it must have been built) and wraps it with [`ir.as_retriever()`](_autosummary/ir.html.md#ir.as_retriever); `search_defaults` (e.g. `mode="hybrid"`) bind to every call. ### ir.registry.retrievers(\*\*search_defaults) A lazy `Mapping[name, Retriever]` view over the registry (ir_09 §8). The query-time projection of the build-recipe registry: each value is a ready-to-call [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever). This is the source-registry facade an orchestration layer (`raglab`) consumes — it never opens a corpus until the key is accessed, and always reflects the current [`registered()`](_autosummary/ir.registry.html.md#ir.registry.registered) set. `search_defaults` apply to every source. * **Return type:** [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.registry.source_for(name) Resolve *name* to a source, auto-registering a preset if needed. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ### ir.registry.source_from_entry(name, entry) Reconstruct a `CorpusSource` from a registry entry. A persisted `strategy` spec (registry v2) is reconstructed and passed to the preset constructor, so a corpus’s segmentation survives across rebuilds. A v1 entry (no `strategy`) passes `strategy=None` and keeps the preset’s default — unchanged behavior. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ### ir.registry.unregister(name) Remove *name* from the registry (does not delete built data). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) # _autosummary/ir.requirements.html.md # ir.requirements Dependency check for `ir`: name what is missing and how to install it. `ir` imports its dependencies deep inside `ir/base.py` and friends; on an interpreter that lacks them (a fresh server’s system `python3`) that surfaces as a bare `ModuleNotFoundError: No module named 'numpy'` that says nothing about `ir` or the fix. [`check_requirements()`](_autosummary/ir.requirements.html.md#ir.requirements.check_requirements) runs first, from `ir/__init__.py`, using only the standard library, and raises [`MissingRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.MissingRequirements) (a [`ModuleNotFoundError`](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError)) listing every missing import, the interpreter it was looked up in, and the install commands. It also checks the version bounds `ir` declares for them (`cw>=0.1.1,<0.2`), read from `ir`’s installed metadata, and raises [`UnsatisfiedRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.UnsatisfiedRequirements) for an installed dependency outside them, instead of an opaque error later. ### Module Attributes | [`DIST`](_autosummary/ir.requirements.html.md#ir.requirements.DIST) | the distribution whose declared requirements hold the version bounds | |-----------------------------------------------------------|------------------------------------------------------------------------| | [`REQUIRED`](_autosummary/ir.requirements.html.md#ir.requirements.REQUIRED) | import name -> distribution name, for everything `import ir` needs. | ### Functions | [`check_requirements`](_autosummary/ir.requirements.html.md#ir.requirements.check_requirements)([required, bounds]) | Raise [`MissingRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.MissingRequirements) unless every dependency is importable, then [`UnsatisfiedRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.UnsatisfiedRequirements) unless each is within the bounds `ir` declares (*bounds*: `{name: specifier}`, default read from `ir`'s metadata). | |-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`declared_bounds`](_autosummary/ir.requirements.html.md#ir.requirements.declared_bounds)([dist]) | `{normalized dependency name: specifier}` that *dist*'s installed metadata declares for this environment (extras excluded; the specifiers of repeated names are joined); empty when *dist* is not installed or has none. | | [`missing_requirements`](_autosummary/ir.requirements.html.md#ir.requirements.missing_requirements)([required]) | Distribution names in *required* whose import name cannot be found. | | [`requirements_message`](_autosummary/ir.requirements.html.md#ir.requirements.requirements_message)(missing) | The actionable message for *missing* distributions. | | [`unsatisfied_message`](_autosummary/ir.requirements.html.md#ir.requirements.unsatisfied_message)(unsatisfied) | The actionable message for *unsatisfied* `(dist, installed, specifier)`. | | [`unsatisfied_requirements`](_autosummary/ir.requirements.html.md#ir.requirements.unsatisfied_requirements)([required, bounds]) | `(distribution, installed version, specifier)` of each dependency in *required* that is installed outside its declared bounds. | | [`version_satisfies`](_autosummary/ir.requirements.html.md#ir.requirements.version_satisfies)(installed, specifier) | Whether *installed* meets *specifier* (`'>=0.1.1,<0.2'`). | ### Exceptions | [`MissingRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.MissingRequirements) | `ir`'s dependencies are missing from the running interpreter. | |--------------------------------------------------------------------------|--------------------------------------------------------------------------| | [`UnsatisfiedRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.UnsatisfiedRequirements) | A dependency is installed in a version outside the bounds `ir` declares. | ### ir.requirements.DIST *= 'ir'* the distribution whose declared requirements hold the version bounds ### *exception* ir.requirements.MissingRequirements Bases: [`ModuleNotFoundError`](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError) `ir`’s dependencies are missing from the running interpreter. ### ir.requirements.REQUIRED *: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)]* *= {'cw': 'cw', 'dol': 'dol', 'ef': 'ef', 'numpy': 'numpy', 'vd': 'vd'}* import name -> distribution name, for everything `import ir` needs. ### *exception* ir.requirements.UnsatisfiedRequirements Bases: [`ImportError`](https://docs.python.org/3/builtins/exceptions.html#ImportError) A dependency is installed in a version outside the bounds `ir` declares. ### ir.requirements.check_requirements(required={'cw': 'cw', 'dol': 'dol', 'ef': 'ef', 'numpy': 'numpy', 'vd': 'vd'}, , bounds=None) Raise [`MissingRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.MissingRequirements) unless every dependency is importable, then [`UnsatisfiedRequirements`](_autosummary/ir.requirements.html.md#ir.requirements.UnsatisfiedRequirements) unless each is within the bounds `ir` declares (*bounds*: `{name: specifier}`, default read from `ir`’s metadata). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.requirements.declared_bounds(dist='ir') `{normalized dependency name: specifier}` that *dist*’s installed metadata declares for this environment (extras excluded; the specifiers of repeated names are joined); empty when *dist* is not installed or has none. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.requirements.missing_requirements(required={'cw': 'cw', 'dol': 'dol', 'ef': 'ef', 'numpy': 'numpy', 'vd': 'vd'}) Distribution names in *required* whose import name cannot be found. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.requirements.requirements_message(missing) The actionable message for *missing* distributions. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.requirements.unsatisfied_message(unsatisfied) The actionable message for *unsatisfied* `(dist, installed, specifier)`. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.requirements.unsatisfied_requirements(required={'cw': 'cw', 'dol': 'dol', 'ef': 'ef', 'numpy': 'numpy', 'vd': 'vd'}, , bounds=None) `(distribution, installed version, specifier)` of each dependency in *required* that is installed outside its declared bounds. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] ### ir.requirements.version_satisfies(installed, specifier) Whether *installed* meets *specifier* (`'>=0.1.1,<0.2'`). Uses `packaging` when it is installed. Without it, plain releases (`1.2.3`) are compared with the standard library, and anything it cannot judge (a pre-release, `dev`, `post` or local version, an unreadable specifier) is taken to hold: this check must never block `import ir` on a doubt. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ```pycon >>> version_satisfies("0.1.6", ">=0.1.1,<0.2"), version_satisfies("0.2.0", "<0.2") (True, False) >>> version_satisfies("1.4", "~=1.2"), version_satisfies("2.0", "~=1.2") (True, False) ``` # _autosummary/ir.rerank.html.md # ir.rerank The precision stage: a cross-encoder rerank over the first-stage candidates. First-stage retrieval (dense / lexical / hybrid) is tuned for *recall*; many of its worst false positives are artefacts a joint `(query, passage)` scorer catches – a package whose single “DAG” bullet inflates its pooled-vector cosine, or a pair of siblings that share jargon rather than topic. [`ir.search()`](_autosummary/ir.html.md#ir.search) therefore takes `rerank=`: it over-fetches `fetch_k` candidates, hands them to the reranker, and keeps the top `k`. This is single-shot and agent-free: a better *one* search, not the evaluator back-edge (that belongs to `raglab`). `ir` composes `ef` here and adds no reranker class. `ef.cross_encoder_reranker` is the implementation; this module adds the three things `ef` cannot know about an `ir` corpus: - **What the reranker reads.** `rerank_text="surface"` (default) scores the matched surface, exactly as before. `"artifact"` scores the artifact’s *whole* stored text – its sibling surfaces (name + description + README chunks) joined in plan order – so a thin description is judged against the full README. A callable `(corpus, meta) -> str` replaces both. - **Opt-in, with a graceful fallback.** `rerank="cross-encoder"` (or `"cross-encoder:"`) names the ef cross-encoder without importing it, so `import ir` needs no torch. When `sentence-transformers` (the `ir[rerank]` extra) is missing, the search warns once and returns the first-stage ranking rather than failing. - **Calibration that stays honest.** A reranked hit’s score is the reranker’s, on a scale unrelated to cosine / RRF / BM25, so an abstention floor calibrated for `hybrid` would be mis-scaled if applied to it. [`calibration_key()`](_autosummary/ir.rerank.html.md#ir.rerank.calibration_key) gives a reranked configuration its own key (`hybrid+rerank-+artifact`), so `calibrate_min_score(..., rerank=...)` and `discover(..., min_score="auto", rerank=...)` read and write a floor for *that* configuration, and fall back to “no floor” (with the usual warning) rather than a wrong one. The fallback to first-stage ranking maps back to the plain key. Relative selection strategies (`rel_threshold`, `score_gap`…) compare score *ratios*, so a reranker used with them must return non-negative scores. The `"cross-encoder"` spec guarantees that: most cross-encoders (including ef’s default ms-marco model) return raw logits that run negative, so ir passes them through a sigmoid into `(0, 1)`. (A model that already applies a sigmoid would be squashed a second time, which keeps the order but compresses the scale until relative selectors can no longer tell hits apart: use such a model as a callable, `ef.cross_encoder_reranker(model)` with a `rerank_id`.) A callable you pass yourself is used as is: keep it in `[0, 1]`, or give it a stable `rerank_id` attribute so its calibrated floor is kept apart. ```pycon >>> resolve_reranker(None) is None True >>> calibration_key("hybrid", None) 'hybrid' >>> calibration_key("hybrid", rerank_tag(lambda q, s: [], "artifact")) 'hybrid+rerank-~anonymous+artifact' ``` ### Module Attributes | [`RERANK_TEXTS`](_autosummary/ir.rerank.html.md#ir.rerank.RERANK_TEXTS) | The built-in `rerank_text` choices (a callable is also accepted). | |------------------------------------------------------------------------|------------------------------------------------------------------------| | [`DFLT_RERANK_MODEL`](_autosummary/ir.rerank.html.md#ir.rerank.DFLT_RERANK_MODEL) | The cross-encoder `rerank="cross-encoder"` resolves to (ef's default). | | [`DFLT_RERANK_MAX_CHARS`](_autosummary/ir.rerank.html.md#ir.rerank.DFLT_RERANK_MAX_CHARS) | Cap on the characters of one `rerank_text="artifact"` passage. | | [`CROSS_ENCODER`](_autosummary/ir.rerank.html.md#ir.rerank.CROSS_ENCODER) | The `rerank=` string that names ef's cross-encoder. | ### Functions | [`resolve_reranker`](_autosummary/ir.rerank.html.md#ir.rerank.resolve_reranker)(spec) | The reranker `spec` names, or `None` when there is none to apply. | |-------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| | [`rerank_tag`](_autosummary/ir.rerank.html.md#ir.rerank.rerank_tag)(reranker[, rerank_text, max_chars]) | A short, key-safe identity for one reranked configuration (`None` if no rerank). | | [`is_anonymous`](_autosummary/ir.rerank.html.md#ir.rerank.is_anonymous)(tag) | Does `tag` rest on an unnamed callable (so its calibration key may collide)? | | [`spec_tag`](_autosummary/ir.rerank.html.md#ir.rerank.spec_tag)(spec[, rerank_text, max_chars]) | The tag a `rerank` spec *would* get, without loading a model; `None` if the spec cannot run here (a `"cross-encoder"` spec whose extra is not installed). | | [`calibration_key`](_autosummary/ir.rerank.html.md#ir.rerank.calibration_key)(mode[, tag]) | The key an abstention floor is stored and loaded under. | | [`passage_texts`](_autosummary/ir.rerank.html.md#ir.rerank.passage_texts)(corpus, candidates[, ...]) | The text the reranker scores for each candidate's store `meta` row. | ### ir.rerank.CROSS_ENCODER *= 'cross-encoder'* The `rerank=` string that names ef’s cross-encoder. ### ir.rerank.DFLT_RERANK_MAX_CHARS *= 2000* Cap on the characters of one `rerank_text="artifact"` passage. Cross-encoders truncate at ~512 tokens anyway; the cap keeps the pair list cheap to build and the cut at a surface boundary-agnostic, front-loaded point (name + description come first). ### ir.rerank.DFLT_RERANK_MODEL *= 'cross-encoder/ms-marco-MiniLM-L-6-v2'* The cross-encoder `rerank="cross-encoder"` resolves to (ef’s default). ### ir.rerank.RERANK_TEXTS *= ('surface', 'artifact')* The built-in `rerank_text` choices (a callable is also accepted). ### ir.rerank.calibration_key(mode, tag=None) The key an abstention floor is stored and loaded under. `mode` alone for a first-stage ranking (today’s behaviour, so existing floors keep working); `"+"` for a reranked one. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> calibration_key("hybrid") 'hybrid' >>> calibration_key("hybrid", "rerank-m+artifact") 'hybrid+rerank-m+artifact' ``` ### ir.rerank.is_anonymous(tag) Does `tag` rest on an unnamed callable (so its calibration key may collide)? * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ir.rerank.passage_texts(corpus, candidates, rerank_text='surface', , max_chars=2000) The text the reranker scores for each candidate’s store `meta` row. `"surface"` is the matched surface’s own text. `"artifact"` stitches every stored surface of the candidate’s artifact in plan order (chunk overlap deduped, as in [`ir.expand`](_autosummary/ir.html.md#ir.expand)) (siblings looked up once per artifact, through the ledger) and cuts at `max_chars`; an artifact the ledger does not know falls back to its matched surface. A callable is called as `rerank_text(corpus, meta)`. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.rerank.rerank_tag(reranker, rerank_text='surface', , max_chars=2000) A short, key-safe identity for one reranked configuration (`None` if no rerank). Built from the reranker’s identity (see `_identity()`), the `rerank_text` choice and, for `"artifact"`, a non-default `max_chars` – everything that changes the score scale. An unnamed reranker or text callable puts `~anonymous` in its tag, which ir refuses to persist a floor under ([`is_anonymous()`](_autosummary/ir.rerank.html.md#ir.rerank.is_anonymous)). A tag longer than `MAX_TAG_CHARS` is shortened with a hash of the whole, so it stays a usable file name. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.rerank.resolve_reranker(spec) The reranker `spec` names, or `None` when there is none to apply. `spec` is `None` (no rerank), any `ef.Reranker` callable (returned as is), or the string `"cross-encoder"` / `"cross-encoder:"` – ef’s sentence-transformers cross-encoder, built lazily and cached per model name (loading one is the expensive part). If its optional dependency is absent this warns once and returns `None`, so the caller degrades to first-stage ranking; resolve *before* deriving a calibration key so the degraded run uses the plain key. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ```pycon >>> resolve_reranker(len) is len True >>> resolve_reranker("nope") Traceback (most recent call last): ... ValueError: unknown rerank spec 'nope'; expected a callable, 'cross-encoder' or 'cross-encoder:'. ``` ### ir.rerank.spec_tag(spec, rerank_text='surface', , max_chars=2000) The tag a `rerank` spec *would* get, without loading a model; `None` if the spec cannot run here (a `"cross-encoder"` spec whose extra is not installed). For lookups such as [`ir.eval.load_calibration()`](_autosummary/ir.eval.html.md#ir.eval.load_calibration), which need the key, not the model. # _autosummary/ir.retrieve.html.md # ir.retrieve Retrieval — hard metadata filtering + dense / lexical / hybrid ranking. [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) embeds the query, applies a hard metadata filter to narrow the candidate set (the `vd` Mongo-style filter language — ownership, name, tags), then ranks the survivors. Three ranking modes share that one filtered candidate set: - `"dense"` (default) — exact brute-force cosine over the embedding matrix. - `"lexical"` — Okapi BM25 over the candidates’ text (`vd.bm25_lexical_search`). - `"hybrid"` — dense and lexical fused, either by Reciprocal Rank Fusion (`fusion="rrf"`, the default `vd.reciprocal_rank_fusion` rank-based fuse that sidesteps the cosine/BM25 score-scale mismatch) or by a magnitude-preserving convex blend (`fusion="blend"`) that keeps the dense cosine’s absolute scale for better abstention separability (see ir_08). Hybrid matters for short, identifier-heavy capability text (skill / package / tool names), the regime where dense-only retrieval fails silently on exact identifiers and rare terms. An optional `rerank` hook (any `ef.Reranker`) re-scores the top fused candidates with a cross-encoder; it defaults to `None` so retrieval stays offline and API-free out of the box. All ranking is exact brute force — correct and instant at `ir`’s corpus sizes — and surface-hits collapse to the best surface per artifact. Lexical and fusion reuse `vd`; if `vd` is unavailable, hybrid degrades to dense and lexical returns no results (both with a warning), so a missing optional dep never hard-fails a search. Beyond the within-corpus channel fusion above, [`fuse_hits()`](_autosummary/ir.retrieve.html.md#ir.retrieve.fuse_hits) merges ranked hit lists from **different sources** (corpora / embedders / modes) by weighted Reciprocal Rank Fusion — raw scores never cross a source boundary, only ranks do. It is the shared cross-source merge primitive consumed by federated [`ir.discover()`](_autosummary/ir.html.md#ir.discover) and by an orchestration layer’s fan-in reranker (ir_09 §3). Every search hit carries its corpus name as `source` and its surface’s plan position as `surface_index`. Two optional ranking signals act inside [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search): `surface_weights` multiplies the first-stage score per surface kind, and `dep_boost` adds a structural score per candidate after the reranker (the shipped [`dependency_evidence_boost()`](_autosummary/ir.retrieve.html.md#ir.retrieve.dependency_evidence_boost) rewards dependencies on a caller-supplied library set). Both default to off. [`records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact) is the hit-operation beneath retrieval-time context expansion: given a hit’s `artifact_id`, it returns *all* of that artifact’s stored records (its sibling surfaces), ordered — resolved through the ledger, never by re-deriving record ids. The expansion operator itself (`ir.expand.expand()` — retrieve → expand → rerank) builds on it. ### Module Attributes | [`MODES`](_autosummary/ir.retrieve.html.md#ir.retrieve.MODES) | Ranking modes accepted by [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). | |-------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`FUSIONS`](_autosummary/ir.retrieve.html.md#ir.retrieve.FUSIONS) | Hybrid fusion methods accepted by [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) (`mode="hybrid"`). | | [`DFLT_RRF_K`](_autosummary/ir.retrieve.html.md#ir.retrieve.DFLT_RRF_K) | RRF rank constant — the `k` of `1 / (k + rank)` (standard default 60). | | [`DFLT_BLEND_ALPHA`](_autosummary/ir.retrieve.html.md#ir.retrieve.DFLT_BLEND_ALPHA) | Dense weight in the `"blend"` fusion convex combination (`1-alpha` on the bounded lexical term). | | [`DFLT_BM25_SAT_K`](_autosummary/ir.retrieve.html.md#ir.retrieve.DFLT_BM25_SAT_K) | `bm25 -> bm25/(bm25+k)`, a bounded squash to `[0, 1)` that needs no *per-query* normalization (which would erase the absolute-magnitude signal abstention calibration depends on). | | [`ScoreBoost`](_autosummary/ir.retrieve.html.md#ir.retrieve.ScoreBoost) | `hit -> float`, added to each candidate's score after fusion and the reranker, before the per-artifact collapse. | | [`Identity`](_autosummary/ir.retrieve.html.md#ir.retrieve.Identity) | `None` (default — hits from different sources never merge; same id, different corpus = different artifact), the string `"pointer"` (merge hits whose [`pointer`](_autosummary/ir.base.html.md#ir.base.SearchHit.pointer) match — opt-in, for corpora that genuinely index the same files), or any `hit -> hashable` callable (a falsy key falls back to per-source identity). | | [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) | `(query, **overrides) -> list[SearchHit]` — ir_09's Retriever leaf (one query, one corpus) as a swappable callable. | ### Functions | [`as_retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.as_retriever)(corpus_or_name, \*\*search_defaults) | Bind ONE corpus to the uniform [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) contract. | |------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| | [`dependency_evidence_boost`](_autosummary/ir.retrieve.html.md#ir.retrieve.dependency_evidence_boost)(relevant_libs, \*, ...) | A `dep_boost` for [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search): reward dependencies on *relevant_libs*. | | [`fuse_hits`](_autosummary/ir.retrieve.html.md#ir.retrieve.fuse_hits)(hits_by_source, \*[, rrf_k, ...]) | Merge per-source ranked hit lists into one ranking — by rank, not score. | | [`records_for_artifact`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)(store_or_corpus, ...[, ...]) | All stored records of *artifact_id*, ordered by `surface_index`. | | [`search`](_autosummary/ir.retrieve.html.md#ir.retrieve.search)(corpus, query, \*[, k, filter, ...]) | Return the top-*k* [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) for *query*. | ### Exceptions | [`NoLedgerEntry`](_autosummary/ir.retrieve.html.md#ir.retrieve.NoLedgerEntry) | The corpus has no ledger entry for the requested artifact. | |------------------------------------------------------------------|--------------------------------------------------------------| ### ir.retrieve.DFLT_BLEND_ALPHA *= 0.5* Dense weight in the `"blend"` fusion convex combination (`1-alpha` on the bounded lexical term). `0.5` weighs the two equally. ### ir.retrieve.DFLT_BM25_SAT_K *= 8.0* `bm25 -> bm25/(bm25+k)`, a bounded squash to `[0, 1)` that needs no *per-query* normalization (which would erase the absolute-magnitude signal abstention calibration depends on). * **Type:** BM25 saturation constant for `"blend"` fusion ### ir.retrieve.DFLT_RRF_K *= 60* RRF rank constant — the `k` of `1 / (k + rank)` (standard default 60). ### ir.retrieve.FUSIONS *= ('rrf', 'blend')* Hybrid fusion methods accepted by [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) (`mode="hybrid"`). `"rrf"` is the rank-based default; `"blend"` preserves score magnitude (see `_blend_fuse()` and ir_08) for better abstention separability. ### ir.retrieve.Identity `None` (default — hits from different sources never merge; same id, different corpus = different artifact), the string `"pointer"` (merge hits whose [`pointer`](_autosummary/ir.base.html.md#ir.base.SearchHit.pointer) match — opt-in, for corpora that genuinely index the same files), or any `hit -> hashable` callable (a falsy key falls back to per-source identity). * **Type:** How cross-source duplicates are detected in [`fuse_hits()`](_autosummary/ir.retrieve.html.md#ir.retrieve.fuse_hits) alias of `Callable`[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.retrieve.MODES *= ('dense', 'lexical', 'hybrid')* Ranking modes accepted by [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search). ### *exception* ir.retrieve.NoLedgerEntry Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError) The corpus has no ledger entry for the requested artifact. The *benign* miss (unknown artifact, or a corpus built without ledger bookkeeping) — distinct from a stale/torn ledger, which raises a plain `KeyError` describing the corruption. A `KeyError` subclass so existing `except KeyError` callers keep working; the disclosure seam catches this one specifically and tolerates it per hit. ### ir.retrieve.Retriever `(query, **overrides) -> list[SearchHit]` — ir_09’s Retriever leaf (one query, one corpus) as a swappable callable. [`as_retriever()`](_autosummary/ir.retrieve.html.md#ir.retrieve.as_retriever) binds one corpus to this contract so an orchestration layer (e.g. `raglab`) can register an ir corpus as one source key without importing ir internals. * **Type:** A retriever alias of `Callable`[[…], `list[SearchHit]`] ### ir.retrieve.ScoreBoost `hit -> float`, added to each candidate’s score after fusion and the reranker, before the per-artifact collapse. See [`dependency_evidence_boost()`](_autosummary/ir.retrieve.html.md#ir.retrieve.dependency_evidence_boost). * **Type:** A ranking-time score boost alias of `Callable`[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`float`](https://docs.python.org/3/builtins/functions.html#float)] ### ir.retrieve.as_retriever(corpus_or_name, \*\*search_defaults) Bind ONE corpus to the uniform [`Retriever`](_autosummary/ir.retrieve.html.md#ir.retrieve.Retriever) contract. Returns `retrieve(query, **overrides) -> list[SearchHit]` that calls [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search) with `search_defaults` (a per-call kwarg overrides a bound default). A corpus *name* is resolved once via [`ir.open_corpus()`](_autosummary/ir.html.md#ir.open_corpus); pass an open [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus) to skip that. The returned callable carries the bound corpus on `.corpus` for introspection. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]] ```pycon >>> retr = as_retriever(corpus, mode="hybrid", k=20) >>> hits = retr("how do I deploy the app") >>> hits = retr("deploy", filter={"owner": "me"}) ``` ### ir.retrieve.dependency_evidence_boost(relevant_libs, , weight, saturate=1, deps_field='deps') A `dep_boost` for [`search()`](_autosummary/ir.retrieve.html.md#ir.retrieve.search): reward dependencies on *relevant_libs*. Reads the candidate’s `deps` filter field (as [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) stores it), reduces each spec to its bare, PEP 503-normalized name, and returns `weight * min(n, saturate) / saturate` where `n` counts the dependencies found in *relevant_libs*. With the default `saturate=1` a single relevant dependency earns the full `weight`. A structural signal, cheap and model-free, that a prose reranker cannot be fooled out of: an async-task package whose README says “DAG” does not depend on graph libraries. *relevant_libs* is the **caller’s** call: which libraries matter for a goal is a planning decision that belongs to the agent layer, never to `ir`. `weight` is added to the score on the search’s scale (an RRF fused score is near `0.03`, a cosine near `0.5`, a sigmoid reranker in `(0, 1)`), so tune it per configuration. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`float`](https://docs.python.org/3/builtins/functions.html#float)] ```pycon >>> from ir.base import SearchHit >>> boost = dependency_evidence_boost({"networkx"}, weight=0.1) >>> boost(SearchHit("unbox", "description", 0.4, "", {"deps": ["networkx>=3"]})) 0.1 >>> boost(SearchHit("au", "description", 0.5, "", {"deps": ["anyio"]})) 0.0 ``` ### ir.retrieve.fuse_hits(hits_by_source, , rrf_k=60, weights=None, identity=None, k=None) Merge per-source ranked hit lists into one ranking — by rank, not score. The cross-source counterpart of the within-corpus hybrid fusion: scores from different (corpus, mode, embedder) tuples live on incommensurable scales (ir_07: “a different model re-scales everything”), so \*\*raw scores never cross the source boundary\*\* — within each source they order and dedup that source’s hits (one scale, sound), and across sources only ranks interact, via weighted Reciprocal Rank Fusion: each hit contributes `weights[source] / (rrf_k + rank)`. * **Parameters:** * **hits_by_source** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)]]) – `{source_name: ranked hits}`. Hits without a `source` are stamped with their mapping key (existing tags win, so one corpus bound under two keys still counts as one source). A `None` key is the *untagged pseudo-source*: its hits fuse as one rank group and stay unattributed (`source=None`). Within each list, duplicate artifacts — and, when `identity` is given, identity-duplicates — collapse to their best raw score before ranking, so a multi-query / multi-round pool can never double-count one artifact’s RRF mass. * **rrf_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – the RRF rank constant (standard default 60). * **weights** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – optional per-source trust dial (default 1.0 each) — a source’s contribution scales linearly, no score comparability needed. Keys naming sources absent from `hits_by_source` are ignored (a per-round pool may legitimately lack a configured source); callers with a closed source set should validate keys upfront, as federated [`ir.discover()`](_autosummary/ir.html.md#ir.discover) does. * **identity** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – how cross-source duplicates merge — see [`Identity`](_autosummary/ir.retrieve.html.md#ir.retrieve.Identity). Default `None`: never; each `(source, artifact_id)` stays a distinct result. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – truncate the fused ranking to this many hits. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] * **Returns:** the fused hits, best-first. Each carries the fused score in `score` and keeps its pre-fusion magnitude as `metadata["source_score"]` (+ `"source_rank"`), so downstream consumers (abstention gates, LLM judges) never lose the per-source signal. When an `identity` merge combined several sources’ hits, `metadata["fused_sources"]` lists them and the representative hit is the one with the best rank. **Single-source input passes through with raw scores** (RRF of one list is that list’s order — same convention as the hybrid fusion’s single-channel fallback), so the fused-score rescaling only happens when there is genuinely something to fuse. The post-fusion `score` is ordinal: valid for ordering and relative cuts, meaningless against absolute floors — apply calibrated `min_score` floors per source, *before* fusing (see ir_07/ir_08 and `ir.discover`’s federated form). ### ir.retrieve.records_for_artifact(store_or_corpus, artifact_id, , surface_kind=None) All stored records of *artifact_id*, ordered by `surface_index`. The sibling-addressing primitive beneath retrieval-time context expansion: a [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) names its artifact (and, via `surface_index`, which surface of it matched); this returns every surface of that artifact, in plan order, so an expansion policy can stitch neighbors / parents around the hit. Resolution is **ledger-backed only**: the artifact’s ledger entry lists its `record_ids`. Record ids are never re-derived from a per-kind index like `metadata["chunk_index"]` — on multi-kind strategies that index differs from the plan-global `surface_index` baked into the id (see [`ir.base.Record.make_id()`](_autosummary/ir.base.html.md#ir.base.Record.make_id)), so derivation would fetch wrong or missing siblings. * **Parameters:** * **store_or_corpus** – a [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore), anything carrying one as `.store` (e.g. a [`Corpus`](_autosummary/ir.index.html.md#ir.index.Corpus)), or a corpus *name* — resolved straight to its local store: sibling lookup never embeds, so unlike [`ir.open_corpus()`](_autosummary/ir.html.md#ir.open_corpus) no embedder is loaded. * **artifact_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the artifact whose surfaces to fetch. * **surface_kind** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – restrict to one surface kind (e.g. `"readme_chunk"`); a known artifact with no surfaces of that kind yields `[]`. * **Raises:** * [**NoLedgerEntry**](_autosummary/ir.retrieve.html.md#ir.retrieve.NoLedgerEntry) – the ledger has no entry for *artifact_id* (an unknown artifact, or a corpus built without [`ir.index.build()`](_autosummary/ir.index.html.md#ir.index.build)’s ledger bookkeeping). A `KeyError` subclass. * [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – an entry exists but lists a record missing from the store (a stale ledger: interrupted build or out-of-band `delete_record`) — data corruption, named in the message. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Record`](_autosummary/ir.base.html.md#ir.base.Record)] ### ir.retrieve.search(corpus, query, , k=10, filter=None, surfaces=None, per_artifact=True, mode='dense', fusion='rrf', rrf_k=60, alpha=0.5, bm25_sat_k=8.0, fetch_k=None, rerank=None, rerank_text='surface', rerank_max_chars=2000, bm25=None, formulate=None, surface_weights=None, dep_boost=None) Return the top-*k* [`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit) for *query*. * **Parameters:** * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) * **filter** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) * **surfaces** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) * **per_artifact** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – (dense + BM25 fused). Hybrid is the strongest default for short, identifier-heavy text; `"dense"` is the historical behavior and is kept as the default for backward compatibility. * **fusion** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – rank-based Reciprocal Rank Fusion) or `"blend"` (magnitude-preserving convex blend; better abstention separability — see ir_08). Ignored for non-hybrid modes. * **rrf_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) * **bm25_sat_k** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) * **fetch_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – (default `max(k*5, 50)` when collapsing per artifact or reranking, else `k`). * **rerank** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – that re-scores the ranked candidates after fusion – the precision stage. The reranker sees the full candidate list (up to `fetch_k` items) and may reorder it; reranked hits carry the *reranker’s* score, which is on its own scale (see [`ir.rerank`](_autosummary/ir.rerank.html.md#module-ir.rerank), and [`ir.rerank.calibration_key()`](_autosummary/ir.rerank.html.md#ir.rerank.calibration_key) for abstention floors). Pass `"cross-encoder"` (or `"cross-encoder:"`) for ef’s sentence-transformers cross-encoder, imported lazily; when that extra (`ir[rerank]`) is missing the search warns once and keeps the first-stage ranking. Default `None` keeps retrieval offline (no model download / API call). * **rerank_text** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – (default; the matched surface’s text), `"artifact"` (the artifact’s sibling surfaces joined in plan order, so a thin description is judged against its full README), or a callable `(corpus, meta) -> str`. * **rerank_max_chars** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) * **bm25** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – (e.g. `{"k1": 1.5, "b": 0.75}`). * **formulate** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]]) – [str, …]\`\`) applied *before* retrieval — rewrite / expand / HyDE (ir_09 §3). Identity by default (embed the query verbatim). When it returns several queries, ir runs each and fuses the results (best surface per artifact). See [`ir.make_llm_formulator()`](_autosummary/ir.html.md#ir.make_llm_formulator). * **surface_weights** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – `{"deps": 2.0}` to let a high-precision `deps` surface outrank prose chunks. Each surface-hit’s first-stage score (the fused score in hybrid mode) is multiplied by its kind’s weight (default 1.0). Each channel still retrieves its own top `fetch_k`; the weights reorder those candidates (in hybrid mode, the whole fused union) before the cut to `fetch_k`, the reranker and the per-artifact collapse. Setting them makes the search over-fetch like `rerank` does, and formulated sub-searches inherit them. Kinds the corpus lacks are ignored; weights must be finite and non-negative. Meant for the non-negative scores of hybrid fusion and BM25 (a negative dense cosine moves *down* when multiplied by a weight above 1). Weighted scores are on a different scale from unweighted ones, so a calibrated floor does not apply to them (`discover(min_score="auto")` warns and skips it). Default `None` weighs every surface equally, as before. * **dep_boost** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – added to each fetched candidate’s score after fusion and the reranker, before the per-artifact collapse – a structural signal next to the reranker’s textual one. [`dependency_evidence_boost()`](_autosummary/ir.retrieve.html.md#ir.retrieve.dependency_evidence_boost) builds one from a caller-supplied library set. It reorders the fetched candidates (raise `fetch_k` to show it more); boosted scores need their own abstention floor (`discover(min_score="auto")` warns and skips it). Like `rerank` and `surface_weights`, it makes the search over-fetch (`max(k*5, 50)`) even with `per_artifact=False`, so pin `fetch_k` when comparing runs with and without it. Default `None`: no boost. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ir.base.html.md#ir.base.SearchHit)] # _autosummary/ir.schedule.html.md # ir.schedule Install and operate the OS job that runs `ir maintain` — `ir schedule` (issue #75). `ir maintain` is idempotent and cron-shaped, but something has to *call* it. This module manages that caller: it writes a **definition** (a launchd plist or a crontab line), hands it to the OS, and exits. **\`\`ir\`\` still does not run a scheduler.** The boundary from issue #58 — “ir holds the declarative policy as data and exposes an idempotent `ir maintain`; it does not run a scheduler” — is preserved exactly. There is no daemon here, no event loop, no in-process timer, and no orchestration import: launchd/cron remain the executor, and this module is the *installer* for the crontab snippet [`ir.maintenance`](_autosummary/ir.maintenance.html.md#module-ir.maintenance) already prescribes in its docstring. Turning that snippet into a command is packaging, not a change of architecture. The command is idempotent and tells you how to operate what it finds: ```default ir schedule # ensure a schedule exists; report + menu if one does ir schedule --status # report only, never mutates ir schedule --every 30m # set/change the interval ir schedule --restart # reload the definition, re-pinning the interpreter ir schedule --remove # stop it and delete the definition ir schedule --dry-run # print what would be written ``` **An existing definition is data, not a template to re-derive.** Only a *fresh* install snapshots the calling shell’s environment. Every operation on a schedule that already exists (`--every`, `--restart`) carries the stored environment forward untouched, because the alternative silently re-points a working job at a different corpus store the first time you operate it from a shell that happens to lack `$PP` — which is the exact failure this module exists to prevent. Only `--restart` re-pins the interpreter, and only because repairing a dead interpreter is what it is for. Two seams, both one keyword argument: - `backend` — which OS scheduler holds the definition. Feature-detected (`launchctl` then `crontab`), never hardcoded per-platform; `systemd --user` timers are the documented next backend. A backend owns its own definition format, its own preview, and whether “loaded” means anything for it, so adding one touches [`BACKENDS`](_autosummary/ir.schedule.html.md#ir.schedule.BACKENDS) and nothing else. - [`ScheduleSpec.args`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleSpec.args) — the command the job runs, defaulting to `-m ir maintain --all` pinned to the installing interpreter. An orchestration layer’s own entry point (#58 / ADR #43) drops in here without touching backends. Everything else — the plist keys, the crontab line syntax, interval parsing, output formatting — is written directly, on purpose. ### Module Attributes | [`DFLT_LABEL`](_autosummary/ir.schedule.html.md#ir.schedule.DFLT_LABEL) | Reverse-DNS job identifier. | |------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| | [`DFLT_EVERY_MINUTES`](_autosummary/ir.schedule.html.md#ir.schedule.DFLT_EVERY_MINUTES) | How often the job fires, in minutes. | | [`INHERITED_ENV_VARS`](_autosummary/ir.schedule.html.md#ir.schedule.INHERITED_ENV_VARS) | Variables a job must inherit to see what the shell that installed it sees. | | [`LOAD_BEARING_ENV_VARS`](_autosummary/ir.schedule.html.md#ir.schedule.LOAD_BEARING_ENV_VARS) | Of those, the ones whose absence breaks a corpus outright rather than merely relocating it. | | [`BACKENDS`](_autosummary/ir.schedule.html.md#ir.schedule.BACKENDS) | Backends in preference order. | ### Functions | [`cron_expression`](_autosummary/ir.schedule.html.md#ir.schedule.cron_expression)(minutes) | The five-field cron time spec for an every-*minutes* schedule. | |----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------| | [`cron_line`](_autosummary/ir.schedule.html.md#ir.schedule.cron_line)(spec) | ir's crontab entry for *spec* — deliberately **one self-contained line**. | | [`cron_marker`](_autosummary/ir.schedule.html.md#ir.schedule.cron_marker)(label) | The trailing comment that identifies ir's crontab line. | | [`crontab_with_line`](_autosummary/ir.schedule.html.md#ir.schedule.crontab_with_line)(text, spec) | *text* with ir's line replaced by (or appended as) *spec*'s line. | | [`crontab_without_line`](_autosummary/ir.schedule.html.md#ir.schedule.crontab_without_line)(text[, label]) | *text* with ir's line removed (unchanged if there is none). | | [`ensure`](_autosummary/ir.schedule.html.md#ir.schedule.ensure)(\*[, every, backend, label]) | Make sure a schedule exists — the idempotent front door. | | [`format_every`](_autosummary/ir.schedule.html.md#ir.schedule.format_every)(minutes) | Human rendering of an interval (`90` -> `'1h30m'`). | | [`install`](_autosummary/ir.schedule.html.md#ir.schedule.install)(\*[, every, backend, label, python, ...]) | Install (or overwrite) the schedule, and return the resulting status. | | [`parse_every`](_autosummary/ir.schedule.html.md#ir.schedule.parse_every)(value) | Minutes from `"30m"` / `"2h"` / `"1d"` / `"45"` (`None` passes through). | | [`plist_bytes`](_autosummary/ir.schedule.html.md#ir.schedule.plist_bytes)(spec) | *spec* rendered as launchd plist XML. | | [`plist_dict`](_autosummary/ir.schedule.html.md#ir.schedule.plist_dict)(spec) | The launchd job description for *spec*, as a plist-ready dict. | | [`remove`](_autosummary/ir.schedule.html.md#ir.schedule.remove)(\*[, backend, label]) | Stop the schedule and delete its definition. | | [`render`](_autosummary/ir.schedule.html.md#ir.schedule.render)(state) | A human report of *state*, including how to operate what was found. | | [`resolve_backend`](_autosummary/ir.schedule.html.md#ir.schedule.resolve_backend)([backend, require_available]) | The backend named *backend*, or the first available one. | | [`restart`](_autosummary/ir.schedule.html.md#ir.schedule.restart)(\*[, backend, label]) | Reload the existing definition, re-pinning it to the current interpreter. | | [`status`](_autosummary/ir.schedule.html.md#ir.schedule.status)(\*[, backend, label]) | Report the current schedule without changing anything. | ### Classes | [`ScheduleSpec`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleSpec)([every_minutes, python, args, ...]) | What to run and how often — backend-independent. | |-----------------------------------------------------------------------------------------------------|----------------------------------------------------------------| | [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus)(installed, backend[, label, ...]) | What the scheduler holds for ir, and what just happened to it. | ### Exceptions | [`ScheduleError`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleError) | A schedule could not be read, written, or expressed on this platform. | |------------------------------------------------------------------|-------------------------------------------------------------------------| ### ir.schedule.BACKENDS *= (, )* Backends in preference order. launchd first on macOS because a LaunchAgent survives reboots and logs where the OS expects; cron is the POSIX fallback and also works on macOS if asked for by name. This tuple is the whole seam. A backend owns its definition format, its `preview`, and its `reports_loaded` answer, so adding `systemd --user` means adding a class here and nothing else — no branch in the renderer, the previewer, or the CLI. ### ir.schedule.DFLT_EVERY_MINUTES *= 60* How often the job fires, in minutes. Hourly rather than the docstring example’s 15 minutes: a `source-change` rebuild is a near-no-op only *after* it has walked the sources, and a large corpus (tens of thousands of records) makes that walk worth an hour’s spacing. `--every` moves it either way. ### ir.schedule.DFLT_LABEL *= 'com.i2mint.ir.maintain'* Reverse-DNS job identifier. Also the launchd `Label` and the token that marks ir’s line in the user crontab, so both backends find their own work and never touch anybody else’s. ### ir.schedule.INHERITED_ENV_VARS *= ('IR_CONFIG_DIR', 'IR_DATA_DIR', 'IR_CACHE_DIR', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'PP', 'PTH_FILEPATH')* Variables a job must inherit to see what the shell that installed it sees. launchd starts jobs with a near-empty environment and cron with a minimal one, so anything ir reads from the environment has to be carried into the definition or the job maintains a *different* (or empty) set of corpora. `IR_*`/`XDG_*` locate the stores ([`ir.config`](_autosummary/ir.config.html.md#module-ir.config)); `PP` and `PTH_FILEPATH` are read by [`ir.sources`](_autosummary/ir.sources.html.md#module-ir.sources) itself to find the package manifest and the projects root, and without them the `packages` and `reports` corpora fail outright — which is precisely how a scheduled job ends up “running fine” while two of three corpora go stale. ### ir.schedule.LOAD_BEARING_ENV_VARS *= ('PP', 'PTH_FILEPATH')* Of those, the ones whose absence breaks a corpus outright rather than merely relocating it. [`status()`](_autosummary/ir.schedule.html.md#ir.schedule.status) reports a job that lacks one the current shell has, because that job is failing silently right now. ### *exception* ir.schedule.ScheduleError Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception) A schedule could not be read, written, or expressed on this platform. ### *class* ir.schedule.ScheduleSpec(every_minutes=60, python=, args=('-u', '-m', 'ir', 'maintain', '--all'), label='com.i2mint.ir.maintain', log_path=, env=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What to run and how often — backend-independent. `python` is pinned to an absolute interpreter path (by default the one installing the schedule) because neither launchd nor cron inherits the shell’s `PATH`-resolved `ir`. Pinning is also what makes staleness *detectable*: [`status()`](_autosummary/ir.schedule.html.md#ir.schedule.status) can check the recorded interpreter still exists. #### args *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]* *= ('-u', '-m', 'ir', 'maintain', '--all')* Python block-buffers stdout when it is not a tty, so without it a scheduled run’s log stays empty until the process exits — exactly when you most want to watch a slow rebuild. * **Type:** `-u` matters #### *property* command *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)[[str](https://docs.python.org/3/builtins/stdtypes.html#str)]* The argv the scheduler executes. #### *property* env_dict *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)]* The job environment as a plain dict. ### *class* ir.schedule.ScheduleStatus(installed, backend, label='com.i2mint.ir.maintain', action='reported', every_minutes=None, definition=None, command=None, python=None, env=(), log_path=None, loaded=None, last_run=None, last_log_line=None, problems=(), detail='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What the scheduler holds for ir, and what just happened to it. One return type for every operation: `action` says what the call did, so a caller never has to pair a status with a separate “did it change?” flag. #### *property* changed *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* Whether this call actually mutated the schedule. #### to_dict() JSON-ready view (the shape a non-CLI surface would consume). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.schedule.cron_expression(minutes) The five-field cron time spec for an every-*minutes* schedule. cron restarts its `*/n` count each hour (and each day), so only intervals that divide evenly fire evenly. Rejecting the rest with the valid values named beats installing a job that quietly skips. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> cron_expression(15), cron_expression(60), cron_expression(360) ('*/15 * * * *', '0 * * * *', '0 */6 * * *') ``` ### ir.schedule.cron_line(spec) ir’s crontab entry for *spec* — deliberately **one self-contained line**. An earlier design used a begin/end marker block with the environment on its own `NAME=value` crontab lines. Both halves of that were wrong: - a block has an *interior*, and anything that lands in it (a job the user typed just above the end marker, or everything after a begin marker whose end marker got lost) is deleted the next time ir rewrites its block; - crontab-scope `NAME=value` lines apply to \*\*every command after them in the file\*\*, so ir’s `HOME` and `PATH` would silently attach themselves to whatever job the user appends next — and detach again the next time ir moved its block. A cron job whose environment depends on when you last ran `ir schedule` is close to undiagnosable. One line with an inline `env` prefix has no interior to corrupt and no scope beyond itself. `%` is escaped because cron reads an unescaped one as a newline that terminates the command. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.cron_marker(label) The trailing comment that identifies ir’s crontab line. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.crontab_with_line(text, spec) *text* with ir’s line replaced by (or appended as) *spec*’s line. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.crontab_without_line(text, label='com.i2mint.ir.maintain') *text* with ir’s line removed (unchanged if there is none). Only lines ir itself wrote are dropped, identified by the trailing marker. Nothing else in the file is read, moved, or rewritten. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.ensure(, every=None, backend=None, label='com.i2mint.ir.maintain') Make sure a schedule exists — the idempotent front door. Installs when there is none. When one already exists it changes nothing unless *every* differs, so running it twice is safe and running it out of habit never silently reinstalls over a schedule you tuned. * **Return type:** [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus) ### ir.schedule.format_every(minutes) Human rendering of an interval (`90` -> `'1h30m'`). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.install(, every=None, backend=None, label='com.i2mint.ir.maintain', python=None, inherit=True, dry_run=False) Install (or overwrite) the schedule, and return the resulting status. *inherit* carries an existing definition’s environment, interpreter and log path forward; pass `False` to re-snapshot them from this process. * **Return type:** [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus) ### ir.schedule.parse_every(value) Minutes from `"30m"` / `"2h"` / `"1d"` / `"45"` (`None` passes through). * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ```pycon >>> parse_every("30m"), parse_every("2h"), parse_every("1d"), parse_every(45) (30, 120, 1440, 45) ``` ### ir.schedule.plist_bytes(spec) *spec* rendered as launchd plist XML. * **Return type:** [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes) ### ir.schedule.plist_dict(spec) The launchd job description for *spec*, as a plist-ready dict. `StartInterval` (seconds) and `EnvironmentVariables` are what make the interval and the job environment readable back out of an installed job, so [`status()`](_autosummary/ir.schedule.html.md#ir.schedule.status) needs no separate bookkeeping file. `ProcessType` is deliberately `Standard` rather than `Background`: launchd throttles a `Background` job’s CPU and I/O to keep it from disrupting the user, and a corpus rebuild is exactly the file-walking, embedding-heavy work that throttling would stretch past its own interval. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ir.schedule.remove(, backend=None, label='com.i2mint.ir.maintain') Stop the schedule and delete its definition. * **Return type:** [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus) ### ir.schedule.render(state) A human report of *state*, including how to operate what was found. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.schedule.resolve_backend(backend=None, , require_available=True) The backend named *backend*, or the first available one. Naming one this machine cannot run is an error rather than a half-install: `--backend launchd` on Linux would otherwise write a plist into a directory nothing reads. Pass `require_available=False` to obtain a backend purely to render its definition format. ```pycon >>> resolve_backend("cron", require_available=False).name 'cron' ``` ### ir.schedule.restart(, backend=None, label='com.i2mint.ir.maintain') Reload the existing definition, re-pinning it to the current interpreter. This is the repair for a schedule left pointing at a rebuilt or deleted interpreter. The stored environment is carried forward untouched — restarting from a shell that lacks `$PP` must not quietly re-point a working job at a different (or empty) set of corpora. * **Return type:** [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus) ### ir.schedule.status(, backend=None, label='com.i2mint.ir.maintain') Report the current schedule without changing anything. * **Return type:** [`ScheduleStatus`](_autosummary/ir.schedule.html.md#ir.schedule.ScheduleStatus) # _autosummary/ir.sources.html.md # ir.sources Defining a corpus source — an abstract strategy plus parameters. A [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) says *what* a corpus is and *how to keep it current*, independent of how it is indexed or stored: - `scope` — a `Mapping[id -> raw]` enumerating the corpus (a dict, a `dol` file store, or any mapping). This is the “what is in the corpus” slot. - `change_signal` — `(id, raw) -> version_str`; default is the content hash. This is the “what counts as stale” slot, driving incremental reindex. - `indexing_strategy`— how a raw item becomes filter fields + surfaces. - `embedder` — embedder spec (default: the decent local model). - `metadata_of` — optional `(id, raw) -> dict` of extra filter metadata. Smart-default constructors cover the common ways to define a source: `from_mapping()`, `from_files()`, `from_md_reports()`, `from_skills()`, `from_packages()`, `from_records()`, `from_mcp_tools()`. ### Module Attributes | [`DFLT_EXCLUDE_DIRS`](_autosummary/ir.sources.html.md#ir.sources.DFLT_EXCLUDE_DIRS) | Directory names never descended into when walking a docs tree for reports. | |---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`DFLT_SESSIONS_SINCE_DAYS`](_autosummary/ir.sources.html.md#ir.sources.DFLT_SESSIONS_SINCE_DAYS) | Default look-back window (days) for [`CorpusSource.from_claude_sessions()`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource.from_claude_sessions). | | [`MCP_TOOL_PARAM_KEYS`](_autosummary/ir.sources.html.md#ir.sources.MCP_TOOL_PARAM_KEYS) | Fields a tool record may carry its parameter schema under, in lookup order: fastmcp's `parameters`, the MCP wire format's `inputSchema`, and the Anthropic API's `input_schema`. | | [`REPORT_DOC_GLOBS`](_autosummary/ir.sources.html.md#ir.sources.REPORT_DOC_GLOBS) | The two per-project doc-tree roots reports are drawn from (relative globs). | | [`REPO_DOC_SUBDIRS`](_autosummary/ir.sources.html.md#ir.sources.REPO_DOC_SUBDIRS) | The doc-tree roots inside a repo (relative to the repo folder). | ### Functions | [`content_hash_signal`](_autosummary/ir.sources.html.md#ir.sources.content_hash_signal)(artifact_id, raw) | Default change signal: a content hash of the raw payload. | |-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------| | [`iter_pocket_repos`](_autosummary/ir.sources.html.md#ir.sources.iter_pocket_repos)(root) | Yield repos one level deeper than usual: `//`. | | [`iter_report_doc_folders`](_autosummary/ir.sources.html.md#ir.sources.iter_report_doc_folders)(root) | Yield each existing report doc folder under *root*. | | [`mcp_tool_record`](_autosummary/ir.sources.html.md#ir.sources.mcp_tool_record)(tool) | Normalize one duck-typed tool to `{name, description, parameters}`. | | [`mcp_tool_signal`](_autosummary/ir.sources.html.md#ir.sources.mcp_tool_signal)(artifact_id, raw) | Change signal for a tool: SHA-256 over `(name, description, parameters)`. | | [`report_exclude_reason`](_autosummary/ir.sources.html.md#ir.sources.report_exclude_reason)(path, folder, \*[, ...]) | Why a `*.md` under *folder* is excluded from the reports corpus. | | [`resolve_fetcher`](_autosummary/ir.sources.html.md#ir.sources.resolve_fetcher)(fetcher) | Resolve a fetcher spec to a zero-argument callable returning records. | ### Classes | [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource)(name, scope[, ...]) | A corpus definition: scope + change signal + strategy + embedder. | |-------------------------------------------------------------------------------------|---------------------------------------------------------------------| ### *class* ir.sources.CorpusSource(name, scope, indexing_strategy=, change_signal=, embedder='default', metadata_of=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A corpus definition: scope + change signal + strategy + embedder. #### change_signal(raw) Default change signal: a content hash of the raw payload. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) #### *classmethod* from_claude_sessions(, name='sessions', since=90, projects=None, include_full=False, include_session_title=True, max_sessions=None, root=None, fetcher=None, strategy=None, \*\*kwargs) The user’s Claude Code session transcripts as a corpus (turn pairs). Each artifact is one user→assistant turn pair; the default [`ClaudeTurn`](_autosummary/ir.strategy.html.md#ir.strategy.ClaudeTurn) strategy indexes the user prompt and the assistant’s end-of-turn summary as separate surfaces (target either with `surfaces={"user_prompt"}` / `{"assistant_summary"}`). `include_full` adds the full assistant text surface (off by default — the summary is the signal). `include_session_title` (default on) also indexes one record per session whose surface is the session’s persisted custom/AI title — a cheap “what was this session about” surface. Scope defaults to the last `since` days (a full-history build is heavy); narrow with `projects` (a cwd substring or list) and `max_sessions`. `fetcher` overrides the record source (each a mapping with `user_prompt` / `assistant_summary` / … ) — a callable (inject a test double to avoid the `priv` dependency) or a `"module:attr"` reference ([`resolve_fetcher()`](_autosummary/ir.sources.html.md#ir.sources.resolve_fetcher)), the form a registry entry can carry, so another package can own the transcripts this corpus indexes. Otherwise records come from `priv.claude_transcripts.turn_pair_records()`. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_files(root, , name=None, pattern='.\*\\\\\\\\.md$', exclude=None, strategy=None, \*\*kwargs) A directory tree of text files as a corpus (lazy `dol` scope). * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_mapping(mapping, , name, strategy=None, \*\*kwargs) Any mapping `{id -> raw}` (dict, `dol` store) as a corpus. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_mcp_tools(tools, , name='tools', strategy=None, \*\*kwargs) A catalogue of MCP tools as a corpus, for tool search. Each tool is indexed on `name + description` only, through the [`Skill`](_autosummary/ir.strategy.html.md#ir.strategy.Skill) strategy: its parameter schema never enters the embedded text (ir_02 §4, “the body carries no retrieval responsibility”). The schema still drives invalidation, through [`mcp_tool_signal()`](_autosummary/ir.sources.html.md#ir.sources.mcp_tool_signal), so a changed schema re-embeds the tool. Its top-level argument names ride along as the `param_names` metadata of every hit, so a caller can see what a tool takes without loading it. `tools` is an iterable of tool records, a `{name: tool}` mapping (the key names a record that has no `name`; a record’s own name wins), or a fetcher (a zero-argument callable or a `"module:attr"` reference, see [`resolve_fetcher()`](_autosummary/ir.sources.html.md#ir.sources.resolve_fetcher)). A record is duck-typed (see [`mcp_tool_record()`](_autosummary/ir.sources.html.md#ir.sources.mcp_tool_record)), so `ir` takes no MCP dependency; an adapter for a specific client lives with the consumer. Tool names are the artifact ids and must be unique: namespace them (`"server.tool"`) before merging several servers’ catalogues. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ```pycon >>> src = CorpusSource.from_mcp_tools( ... [{"name": "ping", "description": "Check the server is up"}] ... ) >>> plan = src.indexing_strategy.decompose("ping", src.scope["ping"]) >>> print(plan.surfaces[0].text) ping Check the server is up ``` #### *classmethod* from_md_reports(, name='reports', projects_root=None, strategy=None, recursive=True, exclude_dirs=None, \*\*kwargs) Markdown reports under projects’ `docs/` and `misc/docs/` trees. Walks each `*/*/docs` and `*/*/misc/docs` folder, and those of pocket repos (`i/_zodals//docs`, see [`iter_pocket_repos()`](_autosummary/ir.sources.html.md#ir.sources.iter_pocket_repos)), **recursively** (`recursive=True`, default) so reports nested one or more levels deep — `docs/research/…`, `docs/decisions/…`, `docs/adr/…` — are indexed, not just files sitting directly in the folder. Pass `recursive=False` for the old shallow (top-level-only) behavior. Excludes ALL-CAPS filenames (README/CLAUDE/MEMORY/SKILL…) and any file under a vendored/hidden directory (`exclude_dirs`, default [`DFLT_EXCLUDE_DIRS`](_autosummary/ir.sources.html.md#ir.sources.DFLT_EXCLUDE_DIRS); a dotted directory is always skipped). Each record is a project-tagged document; ids are paths relative to the projects root. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_packages(, name='packages', manifest=None, readme_chars=20000, strategy=None, \*\*kwargs) The local package ecosystem, scanned from the `.pth` manifest. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### *classmethod* from_records(, name, fetcher, strategy=None, metadata_keys=None, id_key='id', text_key='text', \*\*kwargs) Records produced by a **named callable** as a corpus. The generic, *persistable* counterpart of [`from_mapping()`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource.from_mapping): where `from_mapping` takes an in-process object (so a registry entry cannot name it), this takes a `fetcher` — a callable **or** a `"module:attr"` import reference (see [`resolve_fetcher()`](_autosummary/ir.sources.html.md#ir.sources.resolve_fetcher)) — so a corpus owned by another package round-trips through `~/.config/ir/corpora.json` and rebuilds in a fresh process. Each record is a mapping; its `id_key` is the artifact id (falling back to `_` so no record is silently dropped). `metadata_keys` names the record fields lifted into the strategy’s **filter fields** — the JSON-friendly stand-in for a `metadata_of` callable, which no registry entry could carry. The default strategy chunks `text_key`. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) ```pycon >>> src = CorpusSource.from_records( ... name="notes", ... fetcher=lambda: [{"id": "a", "text": "hi", "project": "p"}], ... metadata_keys=["project"], ... ) >>> src.scope["a"]["text"], src.metadata_of("a", src.scope["a"]) ('hi', {'project': 'p'}) ``` #### *classmethod* from_skills(, name='skills', filter=None, fetcher=None, strategy=None, \*\*kwargs) The agent-skills corpus, via `priv.skills_index`. `fetcher` overrides the source of skill records (each a mapping with `name`/`description`/`parent`) — a callable (inject a test double to avoid the `priv` dependency) or a `"module:attr"` reference ([`resolve_fetcher()`](_autosummary/ir.sources.html.md#ir.sources.resolve_fetcher)), which is the form a registry entry can carry. * **Return type:** [`CorpusSource`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource) #### items() Iterate `(artifact_id, raw)` pairs over the corpus scope. ### ir.sources.DFLT_EXCLUDE_DIRS *= frozenset({'.git', '.hg', '.ipynb_checkpoints', '.obsidian', '.svn', '.tox', '.venv', '_\_pycache_\_', 'build', 'dist', 'node_modules', 'site-packages', 'venv'})* Directory names never descended into when walking a docs tree for reports. Vendored / build / hidden noise — kept as an SSOT so ingestion and any coverage diagnostic exclude exactly the same set (see `_md_in()`). ### ir.sources.DFLT_SESSIONS_SINCE_DAYS *= 90* Default look-back window (days) for [`CorpusSource.from_claude_sessions()`](_autosummary/ir.sources.html.md#ir.sources.CorpusSource.from_claude_sessions). ### ir.sources.MCP_TOOL_PARAM_KEYS *= ('parameters', 'inputSchema', 'input_schema')* Fields a tool record may carry its parameter schema under, in lookup order: fastmcp’s `parameters`, the MCP wire format’s `inputSchema`, and the Anthropic API’s `input_schema`. ### ir.sources.REPORT_DOC_GLOBS *= ('\*/\*/docs', '\*/\*/misc/docs')* The two per-project doc-tree roots reports are drawn from (relative globs). ### ir.sources.REPO_DOC_SUBDIRS *= ('docs', 'misc/docs')* The doc-tree roots inside a repo (relative to the repo folder). ### ir.sources.content_hash_signal(artifact_id, raw) Default change signal: a content hash of the raw payload. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ir.sources.iter_pocket_repos(root) Yield repos one level deeper than usual: `//`. A *pocket* is a `*/*` folder that is not itself a git repo but holds repos (`i/_zodals/zodal-groups`, `i/o/`, `i/dols/`). A child counts as a repo only when its `.git` is a **directory**: a `.git` *file* marks a worktree or submodule, whose docs duplicate a checkout indexed elsewhere. ### ir.sources.iter_report_doc_folders(root) Yield each existing report doc folder under *root*. `*/*/docs` and `*/*/misc/docs` (a repo at `/`), plus the same two folders of every pocket repo ([`iter_pocket_repos()`](_autosummary/ir.sources.html.md#ir.sources.iter_pocket_repos)). The single enumeration both ingestion and [`ir.coverage.reports_coverage()`](_autosummary/ir.coverage.html.md#ir.coverage.reports_coverage) use. ### ir.sources.mcp_tool_record(tool) Normalize one duck-typed tool to `{name, description, parameters}`. `tool` is a mapping or an object (a fastmcp or `mcp.types` tool, say) with a `name`, an optional `description`, and an optional parameter schema under any of [`MCP_TOOL_PARAM_KEYS`](_autosummary/ir.sources.html.md#ir.sources.MCP_TOOL_PARAM_KEYS). Every other field is dropped, so it neither reaches the index nor invalidates it. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ```pycon >>> mcp_tool_record({"name": "ping", "inputSchema": {"type": "object"}}) {'name': 'ping', 'description': '', 'parameters': {'type': 'object'}} ``` ### ir.sources.mcp_tool_signal(artifact_id, raw) Change signal for a tool: SHA-256 over `(name, description, parameters)`. The parameter schema takes part in the hash even though it is never embedded, so a schema change re-embeds the tool on the next build (ir_02 §6). The schema is serialized as canonical JSON (sorted keys), so key order does not count as a change. `raw` is a tool record in any form [`mcp_tool_record()`](_autosummary/ir.sources.html.md#ir.sources.mcp_tool_record) accepts. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> len(mcp_tool_signal("ping", {"name": "ping"})) 64 ``` ### ir.sources.report_exclude_reason(path, folder, , exclude_dirs=frozenset({'.git', '.hg', '.ipynb_checkpoints', '.obsidian', '.svn', '.tox', '.venv', '_\_pycache_\_', 'build', 'dist', 'node_modules', 'site-packages', 'venv'})) Why a `*.md` under *folder* is excluded from the reports corpus. The **SSOT** for “does this report file get indexed” — consumed by both the ingestion walk (`_md_in()`) and the coverage diagnostic ([`ir.coverage.reports_coverage()`](_autosummary/ir.coverage.html.md#ir.coverage.reports_coverage)) so the two can never drift. Returns `None` when the file *should* be indexed, else a short reason tag: `"allcaps"` (a README/CLAUDE/… metadata file) or `"excluded_dir"` (under a vendored/hidden subtree). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.sources.resolve_fetcher(fetcher) Resolve a fetcher spec to a zero-argument callable returning records. A callable passes through; a **string** is an import reference — `"package.module:attribute"` (preferred) or `"package.module.attribute"` — imported and returned. The string form is what makes a fetcher-backed corpus *persistable*: a registry entry is JSON, so a corpus whose records come from another package (`ir.register("mine", "records", fetcher="mypkg.recall:turn_records")`) can only survive a rebuild if the seam accepts a name rather than an object. * **Return type:** [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)] ```pycon >>> resolve_fetcher(lambda: [])() [] >>> resolve_fetcher("ir.sources:DFLT_EXCLUDE_DIRS") is DFLT_EXCLUDE_DIRS True ``` # _autosummary/ir.store.html.md # ir.store Persistence for `ir` — the repository layer over `dol` key-value views. A [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) bundles three `MutableMapping` views, so *where* and *how* data is persisted is swappable without touching the rest of `ir`: - `meta` : `record_id -> dict` (text + metadata + filter fields), JSON. - `vectors`: `record_id -> ndarray` (the embedding), numpy bytes. - `ledger` : `artifact_id -> dict` (version, embedder id, record ids) — drives incremental maintenance. - `config` : `key -> dict` (one entry: the corpus build settings). - `calibration` : `mode -> dict` (a per-ranking-mode calibrated record, today the abstention `min_score` floor from [`ir.eval.calibrate_min_score()`](_autosummary/ir.eval.html.md#ir.eval.calibrate_min_score)). Kept apart from `config` on purpose — a calibration is regenerable, derived from an eval run, and not part of the corpus’s build identity, so it must never clobber (or be clobbered by) the build settings. - `links` : `artifact_id -> {edge_type: [target, ...]}` (the semantic link graph — typed directed edges between artifacts; see [`ir.graph`](_autosummary/ir.graph.html.md#module-ir.graph)). Like `calibration` it is regenerable derived state, kept out of build identity; a target is a bare `artifact_id` (intra-corpus) or a `[source, artifact_id]` pair (cross-corpus). Optional — an absent view is simply “no edges”. The default factory [`CorpusStore.local()`](_autosummary/ir.store.html.md#ir.store.CorpusStore.local) roots all six under `~/.local/share/ir/corpora/` via `dol` file stores; [`CorpusStore.memory()`](_autosummary/ir.store.html.md#ir.store.CorpusStore.memory) gives a dependency-free in-memory store for tests. Brute-force search reads vectors into a single normalized matrix (`matrix()`), cached in-process and invalidated on writes. ### Classes | [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore)(meta, vectors, ledger, config[, ...]) | Repository bundling the meta/vectors/ledger/config views of one corpus. | |------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------| ### *class* ir.store.CorpusStore(meta, vectors, ledger, config, calibration=None, links=None, , packed_dir=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Repository bundling the meta/vectors/ledger/config views of one corpus. #### calibration_modes() The ranking modes that currently have a stored calibration. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### delete_ledger_entry(key) Remove a ledger entry; a missing key is tolerated. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### delete_links(artifact_id) Remove an artifact’s edges; a missing entry is tolerated. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### delete_record(record_id) Remove a record’s metadata + vector; a missing id is tolerated. The meta goes first, so the id stops being listed before its vector disappears. Each removal tolerates the file being gone already (another process may be deleting the same record). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_calibration(mode) The stored calibration record for ranking `mode` (`None` if absent). A deep copy, so a caller cannot mutate the nested `grid` back into the stored record (in-memory stores share their objects by reference). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_config() The persisted corpus build settings (empty dict if never written). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_ledger_entry(key) The ledger entry for *key* (`None` if absent). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None) #### get_links(artifact_id) The outgoing edges of *artifact_id* — `{edge_type: [target, ...]}`. Empty dict when the artifact has no stored edges (or no links view). A copy, so a caller cannot mutate the persisted adjacency in place. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_maintenance_state() Background-work bookkeeping (e.g. `last_maintained`); `{}` if unset. Kept under a separate `config`-view key from the build settings: it is regenerable scheduler state (when `ir maintain` last ran), not part of the corpus’s build identity, so it must never clobber it. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### get_record(record_id) Reassemble the [`Record`](_autosummary/ir.base.html.md#ir.base.Record) for *record_id* (`KeyError` if absent). * **Return type:** [`Record`](_autosummary/ir.base.html.md#ir.base.Record) #### ledger_items() Iterate `(key, entry)` ledger pairs (the ledger may be mutated while iterating). * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### link_items() Iterate `(artifact_id, {edge_type: [target]})` adjacency pairs. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### *classmethod* local(name) File-backed store under `~/.local/share/ir/corpora/`. * **Return type:** [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) #### matrix() Return `(record_ids, normalized_matrix, metas)` for brute force. Rows are L2-normalized so cosine similarity is a dot product. Empty corpora return a `(0, 0)` matrix. Caching is two-tier: an in-process cache (invalidated on the next write) backed, for file-rooted stores, by an on-disk **packed** cache — one normalized-matrix `.npy` plus its ids/metas, written once and reloaded with a single memory-mapped read. The packed cache turns a cold reopen from a per-record vector-file storm (thousands of tiny reads) into three file reads; it is cleared by a writer’s first record write, and every record write replaces a *write stamp* that a packed set must match to be published or loaded, so a set built before any later write – by this process or another – is never served (i2mint/ir#86). Another process may be writing or deleting records while this one rebuilds. A record that vanishes or is only half-written when read is left out of the result (it is “not yet written”), with a warning; such a partial result is cached in-process but never published as the packed set, so a fresh process reads the records again. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], `ndarray`, [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### *classmethod* memory() In-memory store (no dependencies); ideal for tests. * **Return type:** [`CorpusStore`](_autosummary/ir.store.html.md#ir.store.CorpusStore) #### metas() Return `(record_ids, metas)` **without** loading any vectors. The vector-free counterpart of [`matrix()`](_autosummary/ir.store.html.md#ir.store.CorpusStore.matrix), for ranking modes that score on text alone (`mode="lexical"`): they need candidate metadata (text + filter fields) but never the embedding matrix, so they must not pay its I/O. Reuses the in-process or packed cache when present; else reads only the `meta` view (not `vectors`), skipping a record that vanishes or is half-written mid-read (as [`matrix()`](_autosummary/ir.store.html.md#ir.store.CorpusStore.matrix) does). * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)]] #### put_record(record) Persist *record*’s metadata + vector, invalidating the search matrix. The vector is written **before** the meta: record ids are listed from the meta view, so a reader in another process that lists an id always finds its vector (`delete_record` removes in the reverse order). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### record_ids() Iterate the record ids currently stored. * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] #### set_calibration(mode, record) Persist a calibration `record` for ranking `mode` (one per mode). `mode` keys a file in the calibration store, so it must be a non-empty string with no path separator (the real modes — `dense` / `lexical` / `hybrid` — already satisfy this). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_config(settings) Persist the corpus build *settings* (name / embedder spec + id). * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_ledger_entry(key, entry) Write the ledger *entry* (version / embedder id / record ids) for *key*. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_links(artifact_id, edges) Persist *artifact_id*’s outgoing *edges* (`{edge_type: [target]}`). Empty edge-type lists are dropped; an empty result deletes the entry (no empty adjacency rows linger). Targets are stored verbatim — a bare `artifact_id` or a `[source, artifact_id]` pair. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) #### set_maintenance_state(state) Persist the maintenance bookkeeping for this corpus. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) # _autosummary/ir.strategy.html.md # ir.strategy Indexing strategies — the “what do we index?” seam. An [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) decomposes one artifact into an [`IndexPlan`](_autosummary/ir.base.html.md#ir.base.IndexPlan): the `filter_fields` (hard-filterable metadata, *not* embedded) and a list of [`Surface`](_autosummary/ir.base.html.md#ir.base.Surface) (embeddable units). This is the central extensibility point of `ir`: a naive corpus uses [`WholeText`](_autosummary/ir.strategy.html.md#ir.strategy.WholeText); a structured corpus (a package) decomposes into several heterogeneous surfaces so a query can match the *right part* of an artifact, and constrains candidates by metadata *before* semantic ranking. Shipped strategies: - [`WholeText`](_autosummary/ir.strategy.html.md#ir.strategy.WholeText) — one surface = the whole text. The out-of-the-box default. - [`Chunked`](_autosummary/ir.strategy.html.md#ir.strategy.Chunked) — split the text into overlapping chunks (one surface each). - [`Skill`](_autosummary/ir.strategy.html.md#ir.strategy.Skill) — embed `name + description` only (the body stays on disk, per the capability-discovery research); name/parent become filter fields. - [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) — `name + description` plus README chunks as surfaces; name/owner/deps become filter fields (AI synopsis / problem-class surfaces are a documented extension). Every strategy is a plain callable-ish object with a `decompose` method, so custom strategies need only match the [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) protocol. ### Module Attributes | [`DFLT_CHUNK_MAX_TOKENS`](_autosummary/ir.strategy.html.md#ir.strategy.DFLT_CHUNK_MAX_TOKENS) | Content-token budget for token-aware chunking under the default embedder. | |--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`STRATEGY_REGISTRY`](_autosummary/ir.strategy.html.md#ir.strategy.STRATEGY_REGISTRY) | Shipped strategies addressable by name, for persisting an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) in a registry entry and reconstructing it (#58). | ### Functions | [`strategy_from_spec`](_autosummary/ir.strategy.html.md#ir.strategy.strategy_from_spec)(spec) | Reconstruct a shipped strategy from a `{"name", "params"}` spec. | |-----------------------------------------------------------------------------|--------------------------------------------------------------------| | [`strategy_to_spec`](_autosummary/ir.strategy.html.md#ir.strategy.strategy_to_spec)(strategy) | A `{"name", "params"}` spec for a shipped, scalar-param strategy. | | [`text_of`](_autosummary/ir.strategy.html.md#ir.strategy.text_of)(raw[, text_key]) | Best-effort text extraction from a raw artifact payload. | ### Classes | [`Chunked`](_autosummary/ir.strategy.html.md#ir.strategy.Chunked)(\*[, chunk_size, overlap, text_key, ...]) | Split the artifact's text into overlapping chunk surfaces. | |------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | [`ClaudeTurn`](_autosummary/ir.strategy.html.md#ir.strategy.ClaudeTurn)(\*[, include_full]) | Index a Claude Code session turn-pair: user prompt + assistant summary. | | [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)(\*args, \*\*kwargs) | Decompose one artifact into filter fields + embeddable surfaces. | | [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package)(\*[, chunk_size, overlap, ...]) | Package strategy: `name + description` surface plus README chunks. | | [`Skill`](_autosummary/ir.strategy.html.md#ir.strategy.Skill)() | Capability strategy: embed `name + description` only. | | [`WholeText`](_autosummary/ir.strategy.html.md#ir.strategy.WholeText)(\*[, text_key, kind]) | One surface = the entire text. | ### *class* ir.strategy.Chunked(, chunk_size=1200, overlap=200, text_key=None, kind='chunk', max_tokens=None, token_model=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Split the artifact’s text into overlapping chunk surfaces. By default chunks are packed to ~\`\`chunk_size\`\` **characters**. Set `max_tokens` to chunk by **tokens** instead — each chunk is bounded to `max_tokens` tokens of `token_model` (default: the default embedding model), so no chunk overflows the embedder’s sequence limit and is silently truncated. Token mode degrades to char mode with a warning if the tokenizer is unavailable. The char `chunk_size` / `overlap` still set the overlap *ratio* used in token mode. ### *class* ir.strategy.ClaudeTurn(, include_full=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Index a Claude Code session turn-pair: user prompt + assistant summary. Two surfaces by default — `user_prompt` (what the human asked) and `assistant_summary` (the assistant’s *final* end-of-turn text, the highest-signal “here’s what I did”; the deliberation before it is mostly noise) — so a query can target either side via `surfaces={"user_prompt"}` / `{"assistant_summary"}`. With `include_full=True` a third `assistant_full` surface (all the turn’s assistant natural-language text) trades noise for recall — off by default. Session / project / time / model / tool-use become hard-filter fields. The raw artifact is a turn-pair record (see `priv.claude_transcripts.turn_pair_records()`): a mapping with `user_prompt` / `assistant_summary` / `assistant_full` plus metadata. ### ir.strategy.DFLT_CHUNK_MAX_TOKENS *= 250* Content-token budget for token-aware chunking under the default embedder. `all-MiniLM-L6-v2` caps input at 256 tokens and silently truncates beyond it. We target 250 (not 254) to leave headroom for the two special tokens (`[CLS]`/`[SEP]`) *and* the ±1–2 token drift when an offset-sliced substring is re-tokenized on its own — so an embedded chunk stays within budget and its tail is never silently truncated (see `_chunk_text()`). ### *class* ir.strategy.IndexingStrategy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Decompose one artifact into filter fields + embeddable surfaces. ### *class* ir.strategy.Package(, chunk_size=1500, overlap=200, embed_deps=False, deps_template=None, max_tokens=None, token_model=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Package strategy: `name + description` surface plus README chunks. Filter fields capture ownership (ours vs third-party), name, deps. AI synopsis / problem-class surfaces are a documented extension point. With `embed_deps=True`, `decompose` additionally emits one `Surface(kind="deps", granularity="field")` whose text is a prefix-form serialization of the **bare** dependency names (`deps_template`, default `_default_deps_text()`) — so a query for a domain matches a package by the libraries it depends on (e.g. `sentence-transformers` -> embeddings, `networkx` -> graphs), and the BM25 leg picks up exact dep-token matches. The deps bag is kept **separate from prose** (its own surface) so a rare library name is not diluted, and deps remain a filter field regardless. `embed_deps` defaults `False` (today’s behavior); it folds into the strategy id, so toggling it re-decomposes incrementally. The deps surface is appended **last**, leaving the `description` (position 0) and `readme_chunk` indices unchanged. Surface indexing: the `description` surface (kept whenever *name* or *description* is non-empty) occupies plan position 0, so `readme_chunk` *j* is stored with `Record.surface_index == j + 1` while its surface metadata says `chunk_index == j` — `surface_index` is plan-global, `chunk_index` per-kind (see [`ir.base.Record.make_id()`](_autosummary/ir.base.html.md#ir.base.Record.make_id)). Never derive sibling record ids from `chunk_index`; use the ledger ([`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact)). `n_chunks` is stamped on readme chunks at decompose time, but corpora built before the stamp keep records without it until the artifact re-indexes (content / embedder / strategy change) — read it with `metadata.get("n_chunks")`. #### synopsis_fields *= ('name', 'description', 'readme')* The record fields a synopsis is written from (see [`ir.synopsis.with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis)): prose only. The record’s `path` (a local absolute path) and `deps` never go to an LLM. ### ir.strategy.STRATEGY_REGISTRY *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [type](https://docs.python.org/3/builtins/functions.html#type)]* *= {'Chunked': , 'ClaudeTurn': , 'Package': , 'Skill': , 'WholeText': }* Shipped strategies addressable by name, for persisting an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) in a registry entry and reconstructing it (#58). New shipped strategies register here so a corpus can name its segmentation. ### *class* ir.strategy.Skill Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Capability strategy: embed `name + description` only. The body (SKILL.md) is loaded post-selection and is *not* indexed; name and parent are filter fields. ### *class* ir.strategy.WholeText(, text_key=None, kind='document') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One surface = the entire text. Sensible default for a naive corpus. ### ir.strategy.strategy_from_spec(spec) Reconstruct a shipped strategy from a `{"name", "params"}` spec. `None` (no persisted strategy) returns `None` so the caller falls back to the source preset’s default strategy — the back-compatible behavior for v1 registry entries. * **Return type:** [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ir.strategy.strategy_to_spec(strategy) A `{"name", "params"}` spec for a shipped, scalar-param strategy. Captures only scalar constructor parameters (the same identity surface `ir.index._strategy_id()` stamps), so it round-trips the shipped strategies. A custom strategy, or one wrapping another (e.g. the [`ir.with_synopsis()`](_autosummary/ir.html.md#ir.with_synopsis) wrapper), is **not** captured here — those are set programmatically at build time / by the maintenance layer, not persisted as a segmentation spec. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ir.strategy.text_of(raw, text_key=None) Best-effort text extraction from a raw artifact payload. The SSOT for turning an opaque `raw` (a `str`, a `Mapping` with a `text` field or a `text_key`, or anything else) into embeddable text — reused by the shipped strategies *and* by [`ir.synopsis.make_llm_synthesizer()`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer) so an injected-free synopsis summarizes the same text a strategy would index. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ir.synopsis.html.md # ir.synopsis Synopsis surfaces — LLM-derived summaries as an indexed surface (report 12). The document-summary-index / collapsed-tree *fuel* (ADR #43): run a summarizer over each artifact at **build time** to produce a short *synopsis*, index it as a `"synopsis"` surface, and let the collapsed-tree policy ([`ir.traverse()`](_autosummary/ir.html.md#ir.traverse)) route a synopsis match down to that artifact’s chunks. Build-time cost, ≈free at query time — and incremental, so only new / changed artifacts are re-synthesized. [`with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis) wraps *any* [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) and adds one synopsis surface per artifact: ```default strat = ir.with_synopsis(ir.Chunked(), synthesize=my_summarizer) corpus = ir.build(ir.CorpusSource.from_mapping(docs, name="d", strategy=strat)) hits = ir.traverse(q, corpus, policy=ir.collapsed_tree_policy()) # routes via synopsis ``` The synopsis is **prepended** (plan position 0) so it is the *first* summary surface — hence the collapsed-tree *router* (on `with_synopsis(Package())` the synopsis, not the terse `description`, routes). An empty synopsis is dropped, so a synth that returns `""` simply leaves the artifact with its other surfaces. `synthesize: Callable[[Artifact], str]` is **injectable** (a test double, or your own summarizer); omitted, it is built lazily on `aix` (the multi-provider LLM facade) via [`make_llm_synthesizer()`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer) (the `make_llm_*` idiom — `import ir` stays offline, `aix` is imported only on the first synthesis). **Staleness.** The wrapper exposes its identity as scalar attributes (`synthesizer_id`, `synopsis_kind`) and holds the inner strategy, so `ir.index._strategy_id()` (which recurses into nested strategies) folds both the inner strategy’s parameters *and* the synthesizer identity into the corpus’s `strategy_id`. A prompt / model change — or an inner-strategy change — therefore re-synthesizes exactly the affected artifacts, the same way an `embedder_id` change does; no silent staleness. (An injected *unnamed* synthesizer — a lambda or a local closure — has no stable identity to fold in, so `with_synopsis` **warns** and disables its staleness tracking unless you pass `synthesizer_id=`.) **Routing needs no edges.** collapsed-tree descends synopsis→chunks *within* an artifact via [`ir.retrieve.records_for_artifact()`](_autosummary/ir.retrieve.html.md#ir.retrieve.records_for_artifact) (shared `artifact_id`), so synopsis routing works with no entries in the [`ir.graph`](_autosummary/ir.graph.html.md#module-ir.graph) `links` view — that view models *cross-artifact* edges (REF / PARENT between artifacts), a different grain than surface→surface within one artifact. **Conditional synopsis.** `when=` (default `"always"`) restricts synthesis to the artifacts a predicate selects, so LLM cost is paid only where a synopsis helps — e.g. `with_synopsis(ir.Package(), when=thin_description_predicate)` synthesizes only for packages whose `[project].description` is empty or short (their README is the only topical signal), and skips the rest \*without calling the synthesizer\*. The predicate’s identity folds into `strategy_id` (`when_id`) like the synthesizer’s, so changing it re-processes the corpus (`strategy_id` is corpus-wide) and re-synthesizes every artifact the new gate selects. \*\*Caveat — `edge_extractor`.\*\* Eager edge ingest (`build(edge_extractor=...)`) calls `decompose` for *every* artifact each build, and synthesis lives in `decompose`; so combining [`with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis) with an `edge_extractor` re-runs synthesis on every rebuild. The common path — synopsis routing with no `edge_extractor` — stays fully incremental. ### Module Attributes | [`Synthesizer`](_autosummary/ir.synopsis.html.md#ir.synopsis.Synthesizer) | an [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → its synopsis text (`""` to skip — no synopsis surface is added for that artifact). | |---------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`Predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.Predicate) | an [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → whether to synthesize for it. | | [`ALWAYS`](_autosummary/ir.synopsis.html.md#ir.synopsis.ALWAYS) | `when` value meaning "synthesize for every artifact" (the default). | | [`DFLT_THIN_DESCRIPTION_CHARS`](_autosummary/ir.synopsis.html.md#ir.synopsis.DFLT_THIN_DESCRIPTION_CHARS) | Descriptions shorter than this many characters count as thin by default. | | [`SYNOPSIS_KIND`](_autosummary/ir.synopsis.html.md#ir.synopsis.SYNOPSIS_KIND) | The surface kind a synopsis is indexed under — a member of `ir.traverse.DFLT_SUMMARY_KINDS`, so collapsed-tree routes from it. | | [`SYNOPSIS_PROMPT`](_autosummary/ir.synopsis.html.md#ir.synopsis.SYNOPSIS_PROMPT) | Default prompt for [`make_llm_synthesizer()`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer) (a routing-oriented summary). | | [`TOPICAL_SYNOPSIS_PROMPT`](_autosummary/ir.synopsis.html.md#ir.synopsis.TOPICAL_SYNOPSIS_PROMPT) | A topical / classification-oriented alternative to [`SYNOPSIS_PROMPT`](_autosummary/ir.synopsis.html.md#ir.synopsis.SYNOPSIS_PROMPT), suited to thin-description packages (opt-in: `make_llm_synthesizer(prompt=...)`). | | [`thin_description_predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.thin_description_predicate)(artifact) | Ready-made gate: synthesize only for packages with an empty / short description. | ### Functions | [`make_llm_synthesizer`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer)(\*[, summarize, prompt, ...]) | An LLM-backed [`Synthesizer`](_autosummary/ir.synopsis.html.md#ir.synopsis.Synthesizer) ([`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → synopsis). | |-------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`make_thin_description_predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_thin_description_predicate)(\*[, ...]) | A [`Predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.Predicate) true when the artifact's `field` is empty or thin. | | [`thin_description_predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.thin_description_predicate)(artifact) | Ready-made gate: synthesize only for packages with an empty / short description. | | [`with_synopsis`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis)(strategy, \*[, synthesize, ...]) | Wrap *strategy* to add one LLM-derived `synopsis` surface per artifact. | ### ir.synopsis.ALWAYS *= 'always'* `when` value meaning “synthesize for every artifact” (the default). ### ir.synopsis.DFLT_THIN_DESCRIPTION_CHARS *= 40* Descriptions shorter than this many characters count as thin by default. ### ir.synopsis.Predicate an [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → whether to synthesize for it. * **Type:** A synthesis gate alias of `Callable`[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] ### ir.synopsis.SYNOPSIS_KIND *= 'synopsis'* The surface kind a synopsis is indexed under — a member of `ir.traverse.DFLT_SUMMARY_KINDS`, so collapsed-tree routes from it. ### ir.synopsis.SYNOPSIS_PROMPT *= 'Write a concise synopsis (2-4 sentences) of the document below: what it is about and what questions it answers, so that a search over synopses can route to it. Output only the synopsis, no preamble.\\n\\nDocument:\\n{text}'* Default prompt for [`make_llm_synthesizer()`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer) (a routing-oriented summary). ### ir.synopsis.Synthesizer an [`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → its synopsis text (`""` to skip — no synopsis surface is added for that artifact). * **Type:** A synthesizer alias of `Callable`[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.synopsis.TOPICAL_SYNOPSIS_PROMPT *= 'Write a concise synopsis (2-3 sentences) of the software package described below: which domain or problem class it belongs to (e.g. audio, graphs, http clients, data validation) and what it is used for, so that a search for that domain finds it. Output only the synopsis, no preamble.\\n\\nPackage:\\n{text}'* A topical / classification-oriented alternative to [`SYNOPSIS_PROMPT`](_autosummary/ir.synopsis.html.md#ir.synopsis.SYNOPSIS_PROMPT), suited to thin-description packages (opt-in: `make_llm_synthesizer(prompt=...)`). ### ir.synopsis.make_llm_synthesizer(, summarize=None, prompt='Write a concise synopsis (2-4 sentences) of the document below: what it is about and what questions it answers, so that a search over synopses can route to it. Output only the synopsis, no preamble.\\\\n\\\\nDocument:\\\\n{text}', model=None, synthesizer_id=None, text_key=None, text_fields=None, \*\*prompt_function_kwargs) An LLM-backed [`Synthesizer`](_autosummary/ir.synopsis.html.md#ir.synopsis.Synthesizer) ([`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact) → synopsis). `summarize` is an injectable `text -> str` callable (a test double, or your own summarizer); when omitted it is built lazily on `aix` (`aix.prompt_func`) on the **first** synthesis and reused — so importing this module, and even constructing the synthesizer, stays offline. The artifact’s text is extracted with [`ir.strategy.text_of()`](_autosummary/ir.strategy.html.md#ir.strategy.text_of) using `text_key` — which [`with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis) threads from the inner strategy, so the synopsis summarizes the *same* field the strategy indexes. `text_fields` (mutually exclusive with `text_key`) names the fields of a mapping artifact whose text is sent to the LLM, and **only those**: without it, a mapping with no `text` / `text_key` has *every* string field joined and sent (see [`ir.strategy.text_of()`](_autosummary/ir.strategy.html.md#ir.strategy.text_of)) — including, for a [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) record, its local `path`. `with_synopsis` threads the inner strategy’s `synopsis_fields` (`Package` declares `("name", "description", "readme")`), so the Package default never leaks the path. The fields fold into the default `synthesizer_id`, so changing them re-synthesizes. An empty text, or any synthesis error, yields `""` (the surface is then skipped, never a fabricated summary). The returned callable carries a `synthesizer_id` attribute (default `"aix:{model}:{sha(prompt)[:12]}"`) that [`with_synopsis()`](_autosummary/ir.synopsis.html.md#ir.synopsis.with_synopsis) reads into the corpus’s `strategy_id` for staleness — a prompt or model change re-synthesizes. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ir.synopsis.make_thin_description_predicate(, min_chars=40, field='description') A [`Predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.Predicate) true when the artifact’s `field` is empty or thin. Reads `artifact.raw[field]` (a [`Package`](_autosummary/ir.strategy.html.md#ir.strategy.Package) record’s `description`); a raw that is not a mapping is a thin source when its text is shorter than `min_chars`. Whitespace is ignored. The returned callable carries a `when_id` stamp so the threshold folds into `strategy_id`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)] ```pycon >>> thin = make_thin_description_predicate(min_chars=10) >>> thin(Artifact(id="a", raw={"name": "a", "description": ""})) True >>> thin(Artifact(id="b", raw={"name": "b", "description": "Tools to parse dates"})) False ``` ### ir.synopsis.thin_description_predicate(artifact) Ready-made gate: synthesize only for packages with an empty / short description. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ir.synopsis.with_synopsis(strategy, , synthesize=None, synthesizer_id=None, synopsis_kind='synopsis', when='always', when_id=None) Wrap *strategy* to add one LLM-derived `synopsis` surface per artifact. * **Parameters:** * **strategy** ([`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy)) – the inner [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) (`Chunked`, `Package`, …). Its surfaces are kept; the synopsis is prepended. * **synthesize** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – an injectable `Artifact -> str` (test double / custom summarizer). Omitted → [`make_llm_synthesizer()`](_autosummary/ir.synopsis.html.md#ir.synopsis.make_llm_synthesizer) (lazy `aix`). * **synthesizer_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – explicit identity stamp for staleness (recommended when injecting an unnamed callable / lambda). Omitted → the synthesizer’s own `synthesizer_id` / `__qualname__`. * **synopsis_kind** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the surface kind (default `"synopsis"`, a summary kind). * **when** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Artifact`](_autosummary/ir.base.html.md#ir.base.Artifact)], [`bool`](https://docs.python.org/3/builtins/functions.html#bool)]]) – `"always"` (default: every artifact) or a predicate `Artifact -> bool`; the synthesizer is called, and a synopsis surface added, only where it returns true (e.g. [`thin_description_predicate`](_autosummary/ir.synopsis.html.md#ir.synopsis.thin_description_predicate)). A predicate that raises propagates. * **when_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – explicit identity stamp for an unnamed predicate (staleness); omitted → the predicate’s `when_id` / `__qualname__`. * **Return type:** [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) * **Returns:** an [`IndexingStrategy`](_autosummary/ir.strategy.html.md#ir.strategy.IndexingStrategy) usable anywhere a strategy is — `ir.CorpusSource.from_mapping(docs, name=..., strategy=with_synopsis(...))`. ```pycon >>> strat = with_synopsis(Chunked(), synthesize=lambda a: "a summary") ``` # _autosummary/ir.tools.html.md # ir.tools Agent-callable tool surface over `ir` — plain functions returning JSON-ready dicts, deliberately MCP/HTTP-agnostic. `ir` knows nothing about MCP, HTTP, or any agent host. This module is the SSOT for “expose ir’s retrieval as a single agent-callable tool”: a wrapper (`py2mcp`, `qh`, a hand-written agent tool) references e.g. `ir.tools:search` and gets a clean JSON `dict` back. The corpus is a **parameter**, so one function serves every corpus with no per-corpus code — and [`make_search()`](_autosummary/ir.tools.html.md#ir.tools.make_search) returns a *corpus-bound* tool when a connector should expose exactly one corpus. This pairs with `ir.discover` (which `ir` already calls “the single agent-callable tool”); [`search()`](_autosummary/ir.tools.html.md#ir.tools.search) is just its JSON-returning, tool-shaped front door. ### Functions | [`search`](_autosummary/ir.tools.html.md#ir.tools.search)(query, \*, corpus[, k, mode, filter, ...]) | Search a named `ir` corpus and return a JSON-serializable result dict. | |------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------| | [`make_search`](_autosummary/ir.tools.html.md#ir.tools.make_search)(corpus, \*[, name, description, ...]) | Return a **corpus-bound** `search(query, k=...) -> dict` tool. | ### ir.tools.make_search(corpus, , name=None, description=None, k=8, mode='hybrid', rerank=None, rerank_text='surface') Return a **corpus-bound** `search(query, k=...) -> dict` tool. The returned function exposes only `query` (and `k`) — the corpus is fixed — so a connector built over it surfaces exactly one corpus and nothing else. Its `__name__` / `__doc__` are set so an MCP/agent host shows a clean tool name and description. Use this when wiring a single-corpus connector; use [`search()`](_autosummary/ir.tools.html.md#ir.tools.search) when the caller should choose the corpus. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ir.tools.search(query, , corpus, k=8, mode='hybrid', filter=None, rerank=None, rerank_text='surface') Search a named `ir` corpus and return a JSON-serializable result dict. Thin agent-callable wrapper over [`ir.discover()`](_autosummary/ir.html.md#ir.discover) — returns its `.to_dict()` (committed results, scores, disclosures), fit to hand straight back from an MCP tool or HTTP endpoint. The *corpus* is a parameter, so this single function serves any corpus. * **Parameters:** * **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the natural-language query. * **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a registered corpus **name** (str), a list of names (federated search), or a built [`ir.Corpus`](_autosummary/ir.html.md#ir.Corpus). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum number of results. * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"dense"` | `"lexical"` | `"hybrid"`. * **filter** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – optional `vd` Mongo-style metadata filter (hard pre-filter). * **rerank** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – optional second-stage reranker (e.g. `"cross-encoder"`); off by default. `rerank_text` picks the text it reads. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) # about-this-build.html.md # About this build This documentation was built on **2026-10-14 21:13 UTC** from commit 537ca94 on branch master, for **ir 0.1.48** (from pyproject.toml). #### NOTE Nothing suggests a mismatch: the tree was clean at the commit above, and the documented version is the one on PyPI. ## Source | | | |---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| | Commit | 537ca94e8b0b5149b9bd28fb7d0a9cd62a35f9cd | | Branch | master | | Tags at this commit | none | | Working tree | clean | | Remote | https://github.com/i2mint/ir | ## Continuous integration | | | |--------------|--------------------------------------------------------------------------------------------| | Repository | i2mint/ir | | Run | 38638339403 | | Ref | refs/heads/master | | Event commit | 537ca94e8b0b5149b9bd28fb7d0a9cd62a35f9cd (in the history of the built commit) | ## Tools | | | |----------|---------| | epythet | 0.2.17 | | Sphinx | 9.1.0 | | docutils | 0.22.4 | | Python | 3.12.15 | ## Configuration as resolved | | | |---------------|-------------------------------------------------------------------| | theme | auto (Sphinx theme sphinxawesome_theme) | | accent | #006e4b | | api_generator | autosummary | | ignore | tests/, scrap/, examples/ | | agent_outputs | true | | aggregates | md | | ai_artifacts | true | ## Package on PyPI Latest release: 0.1.48, the same as the documented version. ## Reproduce ```bash git clone https://github.com/i2mint/ir && cd ir git checkout 537ca94e8b0b5149b9bd28fb7d0a9cd62a35f9cd pip install "epythet==0.2.17" epythet quickstart . --ignore tests/ scrap/ examples/ ``` The same data, for machines: build_info.json (schema version 1). # ai-agents.html.md # For AI agents `ir` ships artifacts for coding agents alongside its code. This page lists them, says where each lives in the repository, and points at the machine-readable copies of this documentation. ## Skills Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel. ### `ir-search` Find specific documents, packages, skills, or past work with the `ir` retrieval engine — and have the results reviewed for real relevance before they come back. Use whenever the user wants to search/look-up over their own corpora: “find the docs/report about X”, “use ir (or ir discover) to find …”, “which of my packages does Y”, “search my reports/notes/sessions for Z”, “what do I have written about …”, “is there a doc on …”. Covers the `ir` CLI (build/search/discover), the corpus map, the mode/scoring gotchas, and when to delegate to the `ir-discover-agent` subagent for a reviewed, reformulated search vs. running a quick inline lookup. `ir` is a deterministic retrieval substrate with no agent of its own; this skill + the subagent are the Claude-Code-native way to wrap it in agentic query→review→reformulate powers (the coded equivalent is `raglab`). ```bash gh skill install i2mint/ir ir-search --agent claude-code ``` Source: [`ir/data/skills/ir-search`](https://github.com/i2mint/ir/tree/HEAD/ir/data/skills/ir-search) (bundled with the pip package). ### `ir-dev-workflow` How to work ON the `ir` package (i2mint/ir): the module map, the hermetic test recipe, the CLI grammar golden, the lint and doc rules CI enforces, the ir-vs-raglab boundary every change must respect, and how a merge publishes. Use when contributing to, fixing, reviewing, or extending `ir` itself: “add a search option to ir”, “fix this ir test”, “where does fusion happen in ir”, “update the ir CLI”, “review this ir PR”, “does this belong in ir or raglab”. Not for USING ir to search corpora (that is the `ir-search` skill). ```bash gh skill install i2mint/ir ir-dev-workflow --agent claude-code ``` Source: [`skills/ir-dev-workflow`](https://github.com/i2mint/ir/tree/HEAD/skills/ir-dev-workflow). The bundled skills are also on disk after `pip install ir`, under the package’s `data/skills/` directory; link them into an agent without network access with `skill link-skills `. ## Subagents Subagents are Markdown files with a frontmatter (`name`, `description`, `tools`) and a system prompt as the body. Copy one into your project’s `.claude/agents/` (or your agent host’s equivalent) to delegate the task it describes. ### `ir-discover-agent` A reviewing search agent over `ir` corpora (reports, packages, skills, sessions, or any registered corpus). Use it whenever the user wants to FIND specific documents, packages, skills, or past work with `ir` / `ir discover` AND wants the results actually checked for relevance before they come back — i.e. “find the docs about X”, “which of my packages does Y”, “search my reports/notes/sessions for Z”, “use ir discover to find …”. It formulates several queries, retrieves candidates, READS and JUDGES each against the real intent, reformulates when results fall short, and returns only the genuinely relevant few (or an honest “nothing sufficiently relevant”). This is a lightweight, Claude-Code-native stand-in for the coded `raglab` search agent: `ir` itself is a deterministic retrieval substrate with no agent of its own — this subagent supplies the query intelligence, the relevance review, and the reformulate loop. Source: [`.claude/agents/ir-discover-agent.md`](https://github.com/i2mint/ir/tree/HEAD/.claude/agents/ir-discover-agent.md); tools: `Bash, Read, Grep, Glob`. ## Instruction files Files agents read before working in this repository. - [`.claude/CLAUDE.md`](https://github.com/i2mint/ir/tree/HEAD/.claude/CLAUDE.md): read by Claude Code ## Machine-readable documentation This site publishes the same documentation in forms that fit an agent’s context window: - [`llms.txt`](https://i2mint.github.io/ir/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format) - [`ir.md`](https://i2mint.github.io/ir/ir.md): the whole documentation as one Markdown file - `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with `` - [`objects.inv`](https://i2mint.github.io/ir/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`) # api.html.md # API reference | [`ir`](_autosummary/ir.html.md#module-ir) | `ir` — an information-retrieval substrate for agentic systems. | |-----------------------------------------------------------------|------------------------------------------------------------------|