<!-- generated by epythet -->

# ir

`ir` is an information-retrieval substrate for agentic systems: one “find the relevant things in this corpus” contract over skills, packages, documents, sessions, MCP tools or any mapping. It retrieves candidates, commits to a small high-precision subset (or abstains), and loads each committed item’s payload only when asked.

[For developers with a pulse →]()

## For agents: the agentic layer

**Skills.** `ir` ships two [Agent Skills](https://agentskills.io):

| Skill             | Use it to                                                                   | Real files                                                |
|-------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------|
| `ir-search`       | search a user’s `ir` corpora and review the hits for real relevance         | `ir/data/skills/ir-search/` (ships with `pip install ir`) |
| `ir-dev-workflow` | work on `ir` itself: module map, tests, conventions, the ir/raglab boundary | `skills/ir-dev-workflow/` (repository only)               |
```bash
gh skill install i2mint/ir ir-search --agent claude-code        # GitHub CLI with the `skill` command
npx skills add i2mint/ir --skill ir-search                       # or the skills CLI
```

The consumer skill also travels inside the installed package. Find it with `python -c "import ir, os; print(os.path.join(ir.__path__[0], 'data', 'skills'))"`, then link it with the [`skill`](https://github.com/thorwhalen/skill) package (`skill link-skills <that path>`) or copy the folder into `~/.claude/skills/`. In a clone of this repository, Claude Code finds both skills through the relative symlinks in `.claude/skills/` (on Windows, clone with `git config core.symlinks true` and symlink permission, or install the skills as above).

**Subagent and instructions.** `.claude/agents/ir-discover-agent.md` is a reviewing search subagent: it writes several queries, reads and grades the candidates, reformulates, and returns only the verified few. `.claude/CLAUDE.md` holds the project instructions a coding agent needs, including the ir/raglab boundary.

**What an agent can do with `ir`:**

- Give itself one search tool over a large catalogue (skills, MCP tools, packages, documents) instead of loading every schema into context.
- Commit to the few results worth acting on, or abstain when nothing fits (`min_score=`, or `min_score="auto"` with a calibrated floor).
- Search several corpora in one call (`ir.discover(["skills", "packages"], query)`) and get JSON-ready results (`.to_dict()`, `ir.tools.search`).

A minimal runnable example, offline and with no model download (the `light` embedder is a numpy-only hashing embedder):

```python
import ir
from ir.store import CorpusStore

tools = [
    {"name": "fetch_url", "description": "Fetch a web page over HTTP and return its text"},
    {"name": "send_email", "description": "Send an email message to a recipient"},
    {"name": "list_files", "description": "List the files in a directory"},
]
source = ir.CorpusSource.from_mcp_tools(tools)  # embeds name + description only
corpus = ir.build(source, store=CorpusStore.memory(), embedder="light")

result = ir.discover(corpus, "fetch the text of a web page", mode="dense")
assert [item.name for item in result.results] == ["fetch_url"]

nothing = ir.discover(corpus, "bake a chocolate cake", mode="dense", min_score=0.3)
assert nothing.abstained  # no tool clears the floor, so the agent is told so
```

> ### Looking for a search *agent*?

> **`ir` is a retrieval *substrate*, not a search agent — it has no agent of its own.** `ir.discover` is a deterministic, single-shot `retrieve → select → disclose` pipeline: it embeds your query *verbatim* (no LLM query rewriting) and commits via a score rule (no LLM reviewing the results). There is no planner and no *back-edge* (evaluate → reformulate → search again) — and that back-edge is precisely what makes a system an agent.

> If you want the **agent** — a planner, an LLM formulator that rewrites queries, and the evaluator→reformulate loop over one or more `ir` corpora — reach for **[`raglab`](https://github.com/thorwhalen/raglab)**, which is **built on top of `ir`** (`raglab` imports `ir`; `ir` never imports `raglab`, so it stays installable-light and offline-by-default).

> Prefer to drive it from **Claude Code** instead of a coded agent? `ir` ships a reviewing **search subagent** and an `ir-search` **skill** (see [the agentic layer]()) — a lightweight “raglab-in-instructions” that formulates queries, *reads and judges* candidates against your real intent, filters, and reformulates, all on your Claude subscription (see [`ir_10`]()).

## Quickstart

```bash
pip install ir
```

```python
import ir

# Define a corpus, build the index (incremental), then discover.
source = ir.CorpusSource.from_skills()  # your agent skills (read through `priv`)
# Other sources: from_packages(), from_md_reports(), from_claude_sessions(),
# from_files(root), from_mapping(docs, name=...), from_mcp_tools(tools),
# from_records(fetcher="mypkg.recall:records")  # records another package owns
corpus = ir.build(source)  # embed + persist under XDG dirs
result = ir.discover(corpus, "how do I deploy the app to the server")

for item in result.results:
    print(item.score, item.name)  # the committed few (or result.abstained)
print(result.to_dict())  # JSON-serializable (qh / HTTP ready)
```

## Install

`ir` is **light by default** — `numpy` + [`dol`](https://github.com/i2mint/dol) for storage, plus [`ef`](https://github.com/thorwhalen/ef) / [`vd`](https://github.com/i2mint/vd) for embedding and lexical/hybrid retrieval. Python ≥ 3.10.

Notes for the default (semantic) path:

- The default embedder is `all-MiniLM-L6-v2` (384-dim, via `sentence-transformers`), **downloaded on first build** (needs network) and cached under `~/.cache/ir`. For a fast, offline, dependency-light run — tests, CI, quick experiments — pass `embedder="light"` (a numpy-only hashing embedder): `ir.build(source, embedder="light")`.
- `ir` sets `USE_TF=0` on import so `transformers` does not pull in TensorFlow (which crashes on some numpy ABIs); import `ir` before anything that imports `transformers`.
- Case generation (`ir.eval_gen`) and the optional LLM selector need an LLM via [`aix`](https://github.com/thorwhalen/aix) — install the extra, `pip install "ir[llm]"`. Scoring and evaluation themselves stay offline. The LLM is an *injected callable* at every seam, so it can run on a Claude subscription instead of a metered key — see [`ir_10`]() (MCP sampling for connectors / the Agent SDK for Claude Code).

## Install on a fresh server

A machine with only a system `python3` cannot run `ir` (no `numpy`, `dol`, `ef`, `vd`, `cw`). One command gives you an `ir` on `PATH`, in its own isolated environment:

```bash
uv tool install ir          # puts `ir` on PATH (~/.local/bin)
ir discover packages "cache"
```

Or install into an environment you own: `uv pip install --python <env>/bin/python ir` (a distro-managed system Python refuses this; use `uv tool install`). If you run `ir` where its dependencies are missing, it now says which ones and prints these commands, instead of a bare `ModuleNotFoundError` from `ir/base.py`.

**Where the index comes from.** Nothing is shipped or synced: each machine builds its own indexes from what is on that machine. The first `ir discover` on a machine that never built one prints `corpus 'packages' is empty; build it first: ir build packages`. Run it once (`ir build skills`, `ir build packages`, …); later builds are incremental, and `ir maintain --all` (or `ir schedule`) keeps them fresh. Indexes live under `~/.local/share/ir`.

**How long a first build takes.** With the light embedder (no model download), `ir build packages` over about 2,800 records took 12 s on a small server. With the default MiniLM embedder the first build also downloads the model (about 90 MB) and embeds on CPU, so budget minutes (not measured here); `uv pip install --python <python> 'ef[sentence-transformers]'` enables it. Without that package `ir build` warns and builds with the light embedder.

A corpus is pinned to the embedder it was built with (its `embedder_id`, shown by `ir info`), so installing or removing `sentence-transformers` later does not break querying. To switch embedders, re-embed with `ir build NAME`.

## The pipeline

`ir` is a six-stage pipeline, each stage a small, swappable seam:

| Stage        | Entry point    | What it does                                                                                                     |
|--------------|----------------|------------------------------------------------------------------------------------------------------------------|
| **source**   | `CorpusSource` | what is in the corpus + what counts as stale                                                                     |
| **index**    | `ir.build`     | decompose artifacts into embeddable *surfaces*, embed, persist (incremental, idempotent)                         |
| **retrieve** | `ir.search`    | hard metadata filter + `dense` / `lexical` / `hybrid` ranking                                                    |
| **expand**   | `ir.expand`    | stitch a hit’s stored neighborhood (sentence window / whole artifact) into the passage downstream reads — opt-in |
| **select**   | `ir.select`    | commit to a distractor-robust subset, or abstain                                                                 |
| **disclose** | `ir.disclose`  | load the heavy payload (SKILL.md body, package pointer, file text) for committed items — append-only             |

`ir.discover` chains retrieve → select → disclose into the single agent-callable (and `qh`-exposable) tool. Pass a **list** of corpus names for single-shot *federated* discovery across several corpora:

```python
ir.discover(["skills", "packages"], "deploy the app")  # fan-out → fuse → select
ir.discover(
    ["skills", "packages"], q, min_score="auto"
)  # gate each source on its own floor
```

Each source is searched and gated on its **own** calibrated abstention floor *before* any merging; the survivors then rank-fuse (weighted RRF via `ir.fuse_hits`) — raw scores never cross a source boundary, because scores from different corpora / embedders / modes live on incommensurable scales. Every hit carries its corpus name as `hit.source`, so same-id artifacts from different corpora stay distinct, attributable results. The caller names the sources; `ir` never chooses the set (source planning belongs to the agent layer — see [`raglab`](https://github.com/thorwhalen/raglab)).

### Retrieve

```python
hits = ir.search(corpus, "deploy app", mode="hybrid")  # dense | lexical | hybrid (RRF)
```

Dense is exact brute-force cosine; `lexical` is Okapi BM25; `hybrid` fuses both by Reciprocal Rank Fusion (the strongest default for short, identifier-heavy capability text). Lexical/hybrid reuse [`vd`](https://github.com/i2mint/vd); dense needs only numpy.

Hybrid has a second fusion, `fusion="blend"` — a magnitude-preserving score blend instead of rank-RRF. RRF discards score magnitude, which is exactly what abstention calibration needs, so `blend` separates in-scope from out-of-scope queries far better (and even beats dense); the tradeoff is lower lexical recall on terse corpora, so RRF stays the default. Use `blend` when abstention matters — see [`ir_08`]().

`surface_weights={"deps": 2.0}` multiplies each surface-hit’s first-stage (fused) score by its kind’s weight before the per-artifact collapse, so a high-precision surface such as `Package(embed_deps=True)`’s `deps` can outrank prose chunks. The default weighs every surface equally. Weighted scores need their own abstention floor. To sweep weights on one corpus, map it under several labels in `ir.eval.compare_indexings(..., search_kw_by_label={"deps x2": {"surface_weights": {"deps": 2.0}}})`.

#### Precision stage: cross-encoder rerank (opt-in)

First-stage retrieval is tuned for recall; a cross-encoder re-scores the top `fetch_k` candidates against the query *jointly*, which suppresses jargon-sharing distractors. Single-shot and agent-free (the evaluator back-edge stays in `raglab`):

```python
hits = ir.search(
    corpus,
    "audio DAG pipelines",
    mode="hybrid",
    rerank="cross-encoder",  # ef's cross-encoder, loaded lazily
    rerank_text="artifact",
)  # read the whole artifact, not the matched surface
```

`pip install 'ir[rerank]'` brings `sentence-transformers`; without it the search warns once and returns the first-stage ranking. `rerank=` also takes any `ef.Reranker` callable (a stub, a hosted model), and `rerank_text=` a `(corpus, meta) -> str` callable. The `"cross-encoder"` spec squashes the model’s raw logits through a sigmoid into (0, 1), so relative selectors keep working. A reranked hit’s score is the reranker’s, on its own scale, so its abstention floor is calibrated and stored separately (a callable you pass needs a stable `rerank_id` attribute for that): pass the same `rerank=` / `rerank_text=` to `ir.eval.calibrate_min_score` and `ir.discover(..., min_score="auto")` (see `ir.rerank`).

### Expand

```python
passage = ir.expand(hit, corpus)  # ±1 chunk window (default)
passage = ir.expand(hit, corpus, policy=ir.parent_policy())  # the whole artifact
```

The matched chunk is rarely the unit you want to read, and the whole document is usually too much. `ir.expand` stitches a hit’s *stored sibling segments* into a mid-granularity `Passage` (retrieve → expand → rerank), with chunker overlap deduped and the hit’s identity and score untouched. Policies are injectable (`NeighborhoodPolicy`); `sentence_window_policy(k)` and `parent_policy()` ship. It also works through the disclosure seam:

```python
ir.disclose(sel, expand=ir.sentence_window_policy(2), corpus=corpus)
ir.discover(
    "skills", q, expand=ir.sentence_window_policy()
)  # passages on committed items
```

Each `Disclosure` then carries the stitched text as `.passage` (additive; `summary` stays the matched surface, `body` stays the pointer’s payload).

### Select

```python
sel = ir.select(hits)  # conservative default: stay within rel of top, cap at max_k
sel = ir.select(hits, min_score=0.4)  # opt in to abstention ("nothing applies")
sel = ir.select(
    hits, strategy="score_gap"
)  # elbow cut, or "top_k" / "rel_threshold" / a callable
```

The abstention floor is mode-specific (dense cosine, BM25, and RRF live on different scales), so rather than guess `min_score`, **calibrate** it from a case file and let `discover` load it:

```python
ev.calibrate_min_score(
    corpus, cases, mode="dense", persist=True
)  # learn + store the floor
ir.discover(
    corpus, query, mode="dense", min_score="auto"
)  # abstain by the calibrated floor
```

Calibration separates in-scope from out-of-scope query top-scores and picks the floor that best splits them — see [`ir_07`](); it works best on `dense` / `lexical` (hybrid’s RRF scores barely separate). `min_score` defaults to `None` (never abstain), so abstention stays fully opt-in.

The conservative defaults (`max_k=3`, `rel=0.9`) are tuned, not guessed — see [`ir_06`](); re-tune for your own corpus with `ev.sweep_selector` / `ir sweep-select`.

Selection is *relative* (ratios to the top score), so one selector works across `dense` / `hybrid` / `lexical` whose absolute scales differ by orders of magnitude. The result carries auditable `signals` and a `reason` — no opaque “confidence” float. An optional LLM selector (`make_llm_selector`, lazy on [`aix`](https://github.com/thorwhalen/aix), injectable for tests) falls back to the heuristic on any failure.

### Disclose

```python
payloads = ir.disclose(sel, level="body")  # "metadata" (no I/O) | "body" | "bundled"
```

Disclosure is a *pure* read that follows the pointer already stored on each hit (`skill_path` / `path`); it never mutates the ranked hits and tolerates a stale pointer. Keeping the agent’s context append-only (to protect the prompt cache) is then the caller’s discipline — `ir` hands back additive payloads.

### Graph & traverse (opt-in)

Artifacts refer to each other — a package depends on packages, a skill has a parent. `ir` models those as a **semantic link graph**: a typed-edge `links` view on the store, populated at build time by an `EdgeExtractor`.

```python
corpus = ir.build(
    source, edge_extractor=ir.default_edge_extractor
)  # deps→REF, parent→PARENT
graph = ir.CorpusGraph(corpus)
graph.neighbors("contaix", edge_type="REF")  # the package's dependencies
```

`ir.traverse` walks that structure at query time under a pluggable `WalkPolicy` (*score frontier → select → expand → stop*). **Safety is the operator’s**: a visited-set, depth cap, and node budget live in `traverse` itself, so even a cyclic graph and a never-stopping policy terminate. The shipped `collapsed_tree_policy` is pure-vector — it routes a query that matches an artifact’s *summary* down to that artifact’s best *chunk*:

```python
hits = ir.traverse(query, corpus, policy=ir.collapsed_tree_policy())
```

Dependencies are also a ranking signal, seeded by the caller. `graph.reverse_neighbors("networkx")` lists a library’s dependents and `graph.fan_in(...)` counts them. `ir.reverse_dependency_policy(seeds=[...])` walks those reverse edges, so a package that depends on a domain library but never says so in prose is still found. At ranking time, `ir.search(..., dep_boost=ir.dependency_evidence_boost({"networkx"}, weight=0.02))` adds score to candidates whose `deps` include a listed library. `ir` never chooses the libraries: that is the agent layer’s planning decision.

```python
policy = ir.reverse_dependency_policy(seeds=["networkx"], fan_in_weight=0.1)
walked = ir.traverse(query, corpus, policy=policy)
hits = ir.fuse_hits({"flat": ir.search(corpus, query, mode="hybrid"), "deps": walked})
```

The *summary* a query routes on can be an artifact’s own short field — or an **LLM-authored synopsis**. `ir.with_synopsis` wraps any indexing strategy to add one `synopsis` surface per artifact at build time (the document-summary-index pattern: build-time cost, ≈free at query time), and that synopsis becomes the collapsed-tree router:

```python
strat = ir.with_synopsis(
    ir.Chunked(), synthesize=my_summarizer
)  # or default (lazy aix)
corpus = ir.build(ir.CorpusSource.from_mapping(docs, name="d", strategy=strat))
hits = ir.traverse(
    q, corpus, policy=ir.collapsed_tree_policy()
)  # routes via the synopsis
```

`synthesize` is injectable (a test double or your own summarizer); omitted, it is built lazily on [`aix`](https://github.com/thorwhalen/aix) so `import ir` stays offline. Synopses are derived state with a stamped synthesizer identity, so a prompt/model change re-synthesizes only the affected artifacts on the next incremental `build` — no silent staleness.

**Pay only where it helps.** `when=` gates synthesis per artifact (default `"always"`). `ir.thin_description_predicate` selects packages whose description is empty or short — their README is the only topical signal — so the LLM runs on those few and is never called for the well-described rest:

```python
strat = ir.with_synopsis(ir.Package(), when=ir.thin_description_predicate)
# threshold: when=ir.make_thin_description_predicate(min_chars=80)
```

The predicate’s identity (`when_id`) is part of the strategy id, so changing it re-processes the corpus and re-synthesizes every artifact the new gate selects.

**Flat top-k stays the default** — `traverse` is opt-in, and a policy earns its keep only by beating flat+rerank on your eval set (a strong flat retriever wins simple lookup; graph methods cost far more). Results are ordinary `SearchHit`s with additive `metadata["walk_depth"]` / `["seed"]` provenance, so `select` / `disclose` compose unchanged. This is the **semantic link graph** (cyclic, query-time) — distinct from `ef.artifact_graph` (the acyclic build-time derivation DAG).

## Evaluation

`ir.eval` scores discovery quality offline (reusing [`ef`](https://github.com/thorwhalen/ef)’s retrieval metrics):

```python
from ir import eval as ev

cases = ev.load_cases("skills_eval.jsonl")  # query + gold artifact_id(s)
ev.evaluate_discovery(
    corpus, cases, mode="hybrid"
)  # recall@k / NDCG@k / MRR / MAP + failure taxonomy
ev.evaluate_selection(
    corpus, cases, strategy="conservative"
)  # conditional commit rate + selection P/R/F1
ev.sweep_selector(corpus, cases)  # tune max_k × rel; .best() / .frontier() / .table()
ev.distractor_robustness_curve(source.scope, probes)  # accuracy vs catalog size
```

`evaluate_selection`’s headline is the **conditional commit rate** — the selection decision *isolated* from retrieval (did the selector keep the gold, *given* retrieval surfaced it?). `sweep_selector` scores a whole `max_k × rel` grid against the cases off **one** retrieval pass, so the selector defaults can be read off the data (`.best()`) rather than guessed. Generate cases by back-translation with `ir.eval_gen` (needs an LLM; scoring stays offline).

## CLI

```bash
ir build skills                          # build/update a preset corpus
ir search skills "deploy the app"        # rank candidates (retrieval only)
ir search skills "deploy the app" --rerank cross-encoder  # + cross-encoder precision stage (ir[rerank])
ir discover skills "deploy the app"      # retrieve -> select
ir discover skills "deploy the app" --disclose       # + load bodies
ir discover skills "deploy the app" --min-score auto # + calibrated abstention
ir build sessions                        # index recent Claude Code sessions (turn pairs)
ir search sessions "numpy abi error" --mode lexical   # find past sessions
ir ls                                    # list corpora + record counts
ir coverage reports                      # disk-vs-index coverage (silent-gap detector)
ir info skills                           # config, stats, policy, calibrated floors
ir maintain --all                        # run due background work (idempotent; cron-friendly)
ir schedule                              # install/inspect the OS job that runs it
ir register notes files --root ~/notes --pattern '.*\.md$'  # register a custom corpus
ir rm notes                              # unregister (keeps built data)
ir eval-gen skills skills_eval.jsonl     # generate eval cases (needs aix/LLM)
ir eval skills skills_eval.jsonl         # score retrieval on a case file
ir eval-select skills skills_eval.jsonl  # score the selection stage
ir sweep-select skills skills_eval.jsonl # tune the selector (max_k × rel) on your corpus
ir calibrate-min-score skills skills_eval.jsonl --persist  # calibrate the abstention floor
```

## Keeping an index fresh

`ir maintain` does the work that a corpus’s policy says is due — an incremental rebuild when reindex is due, LLM synopses inside their downtime window — and no-ops everything that is not. It is idempotent, so it is safe to run as often as you like.

Something has to *call* it, though, and a corpus that nobody rebuilds keeps answering searches confidently from months-old content. `ir schedule` installs that caller — a launchd agent on macOS, a crontab block elsewhere — and then lets you operate it:

```bash
ir schedule                # ensure a schedule exists (creates one if there is none)
ir schedule --status       # what is scheduled, when it last ran, and whether the OS loaded it
ir schedule --every 30m    # change the interval (15m / 2h / 1d, or plain minutes)
ir schedule --restart      # reload it after upgrading ir or switching interpreter
ir schedule --remove       # stop it and delete the definition
ir schedule --dry-run      # print the plist/crontab that would be written
```

Bare `ir schedule` is idempotent in the way that matters: it creates a schedule when there is none, and when one already exists it reports it and the commands above rather than silently reinstalling over an interval you tuned.

The job is pinned to the absolute interpreter that installed it, because neither launchd nor cron resolves `ir` on your shell’s `PATH`, and it carries the environment ir reads (`$PP`, `$PTH_FILEPATH`, `IR_*`, `XDG_*`) — launchd starts jobs with a near-empty environment, and without those the `packages` and `reports` corpora fail every time the job fires while it still exits 0.

**An existing definition is data, not a template.** Only a fresh install snapshots your shell; `--every` and `--restart` carry the stored environment forward untouched, so operating a working schedule from a shell that happens to lack `$PP` never silently re-points it at a different corpus store. To re-snapshot deliberately: `ir schedule --remove && ir schedule`.

`--status` is the health check. It reports the last run and the last log line, and flags three failures that otherwise look like success: an interpreter that no longer exists (a rebuilt pyenv version or recreated venv leaves a job that fires on time and does nothing), a job whose environment is missing something your shell has, and a non-zero exit status from the last run. `--restart` re-pins the interpreter.

**`ir` still does not run a scheduler.** It writes a definition, hands it to the OS, and exits: no daemon, no in-process timer. launchd/cron remain the executor, exactly as `ir.policy` describes.

## For developers with a pulse

**Set up and run the tests.** The suite is hermetic (light embedder, in-memory stores, no network) and doctests count:

```bash
git clone https://github.com/i2mint/ir && cd ir
uv venv -q .venv && . .venv/bin/activate
uv pip install -q -e ".[dev]"
python -m pytest -q --doctest-modules ir tests
ruff check ir && ruff format --check ir
```

**Contributing.** The `ir-dev-workflow` skill (`skills/ir-dev-workflow/SKILL.md`) is also the human contributor guide: the module map, the CLI grammar golden, the conventions review enforces, and the ir/raglab boundary. New arguments default to today’s behaviour. Don’t bump the version: CI does that when a pull request merges to `master`, and the merge publishes to PyPI. `thorwhalen/raglab` depends on `ir`, so keep its tests green.

**Design.** The design is grounded in a set of capability-discovery research reports and eval-run findings under `misc/docs/` (`ir_01`–`ir_10`): the single-search-tool pattern, indexing & embedding strategy, evaluation, the `ef` + `vd` reuse analysis, the dense-vs-lexical-vs-hybrid eval, selector tuning, abstention-floor calibration, magnitude-preserving fusion, the composable search-agent architecture (`ir_09`) and LLM routing (`ir_10`). `ir` is light by default (numpy / `dol`) and reuses the ecosystem (`ef`, `vd`, and `aix` for the opt-in LLM seams) only where it composes cleanly.

**Where to ask.** Open an issue on [i2mint/ir](https://github.com/i2mint/ir/issues). Issue #1 is the roadmap and epic #38 tracks the agentic-search direction.

<p class="epythet-aggregates">This documentation as a single file: <a href="ir.md">ir.md</a> (Markdown, for agents).</p>