ir.eval_gen¶

LLM-backed generation of evaluation cases for ir.eval.

This is the build-time companion to the offline scoring harness: it turns a corpus into a set of DiscoveryCases by back-translation — given a capability’s description, ask an LLM for the user intents that should route to it. The artifact’s id is the free ground-truth label.

Two ideas make the generated set honest:

Name masking. The artifact’s name is stripped from the description before it is shown to the generator (and any output that still leaks the name is dropped). Otherwise a lexical retriever (the BM25 leg of hybrid) would trivially match query→gold on surface name overlap and inflate scores. Many real descriptions contain their own name, so masking the input — not just filtering the output — is what matters.
An abstention slice. A fraction of cases are “no artifact applies” intents (empty gold), so the eval can measure correct refusal, not just hits.

The LLM is injected (query_generator / abstention_generator callables), so the generation logic — masking, gold assignment, the leakage guard, the abstention fraction — is fully testable with a deterministic stub and no network. The default generators are built lazily on aix (aix.prompt_func, the multi-provider LLM facade), so import ir.eval_gen stays cheap and offline; aix is only imported when you actually generate with the real LLM.

The output is plain DiscoveryCase data — freeze it with ir.eval.save_cases() (stamping corpus_signature() into the __meta__ header) and score it with ir.eval. Generation needs a model; scoring never does.

Quick start:

import ir
from ir import eval_gen as eg

source = ir.CorpusSource.from_skills()
cases = eg.build_eval_set(source, k=5, corpus_name="skills")   # uses aix
from ir.eval import save_cases
save_cases(cases, "skills_eval.jsonl",
           meta={"corpus": "skills", "corpus_signature": eg.corpus_signature(source)})

Module Attributes

`DFLT_QUERIES_PER_ARTIFACT`	Queries generated per artifact by default.
`DFLT_ABSTENTION_FRAC`	Target share of the case set that is abstention ("no artifact applies").
`DFLT_MIN_DESCRIPTION_CHARS`	Minimum description length (chars) for an artifact to be back-translated.
`NAME_PLACEHOLDER`	What a masked name is replaced with in a description / query.
`DFLT_ABSTENTION_THEME`	Default theme used when generating abstention ("out of scope") intents.
`QueryGenerator`	`(description, *, n) -> list[str]` (n candidate intents).
`AbstentionGenerator`	`(*, n, theme) -> list[str]` (n out-of-scope intents).

Functions

`build_eval_set`(source, *[, k, ...])	Generate a full eval set — gold cases plus an abstention slice.
`corpus_signature`(source_or_corpus)	A short, order-independent hash of a corpus's artifact ids.
`generate_abstention_cases`(n, *[, generator, ...])	Generate `n` abstention cases — out-of-scope intents (empty `gold`).
`generate_cases`(source, *[, k, mask_names, ...])	Back-translate a corpus source into gold-bearing `DiscoveryCase`s.
`make_default_abstention_generator`(*[, prompt])	Build the default abstention generator on `aix` (lazy import).
`make_default_query_generator`(*[, prompt])	Build the default back-translation generator on `aix` (lazy import).
`mask_name`(text, name, *[, placeholder])	Replace occurrences of `name` in `text` with `placeholder`.

ir.eval_gen.AbstentionGenerator¶

(*, n, theme) -> list[str] (n out-of-scope intents).

Type:: An abstention generator

alias of Callable[[…], Sequence[str]]

ir.eval_gen.DFLT_ABSTENTION_FRAC = 0.15¶: Target share of the case set that is abstention (“no artifact applies”).

ir.eval_gen.DFLT_ABSTENTION_THEME = 'software developer tools'¶: Default theme used when generating abstention (“out of scope”) intents.

ir.eval_gen.DFLT_MIN_DESCRIPTION_CHARS = 20¶: Minimum description length (chars) for an artifact to be back-translated.

ir.eval_gen.DFLT_QUERIES_PER_ARTIFACT = 5¶: Queries generated per artifact by default.

ir.eval_gen.NAME_PLACEHOLDER = 'this capability'¶: What a masked name is replaced with in a description / query.

ir.eval_gen.QueryGenerator¶

(description, *, n) -> list[str] (n candidate intents).

Type:: A query generator

alias of Callable[[…], Sequence[str]]

ir.eval_gen.build_eval_set(source, *, k=5, abstention_frac=0.15, query_generator=None, abstention_generator=None, theme='software developer tools', corpus_name=None, **gen_kwargs)[source]¶

Generate a full eval set — gold cases plus an abstention slice.

The abstention count is chosen so abstention cases make up (at least) abstention_frac of the returned set: ceil(frac * G / (1 - frac)) for G gold cases. Extra gen_kwargs flow to generate_cases() (mask_names, min_chars, max_artifacts, describe).

Raises:: ValueError – if abstention_frac is outside [0, 1) (frac=0 means no abstention slice) or k is less than 1.
Return type:: list[DiscoveryCase]

ir.eval_gen.corpus_signature(source_or_corpus)[source]¶

A short, order-independent hash of a corpus’s artifact ids.

Stamp this into a case file’s __meta__ header so a frozen eval set can be checked against the (live, machine-specific) corpus it was generated from.

Return type:: str

ir.eval_gen.generate_abstention_cases(n, *, generator=None, theme='software developer tools', corpus_name=None)[source]¶

Generate n abstention cases — out-of-scope intents (empty gold).

Return type:: list[DiscoveryCase]

ir.eval_gen.generate_cases(source, *, k=5, mask_names=True, query_generator=None, describe=None, min_chars=20, max_artifacts=None, corpus_name=None)[source]¶

Back-translate a corpus source into gold-bearing DiscoveryCases.

For each artifact in source.scope (id → raw), extract a description, mask the artifact’s name out of it, ask query_generator for k user intents, and emit one case per surviving intent (gold = the artifact id). Intents that still leak the name are dropped; artifacts whose description is shorter than min_chars are skipped (and the count is warned, never silently dropped).

Parameters:

source (Any) – a CorpusSource (anything with .items()).
k (int) – intents to request per artifact.
mask_names (bool) – scrub the artifact name from the description before generating, and drop any generated intent that still contains it.
query_generator (Optional[Callable[..., Sequence[str]]]) – (description, *, n) -> [intent, …]. Defaults to the aix-backed back-translator (built lazily; needs a model).
describe (Optional[Callable[[Any], str]]) – raw -> description (default: the description / text field, else the joined string fields).
min_chars (int) – skip artifacts whose description is shorter than this.
max_artifacts (int | None) – cap how many artifacts to process (for a quick/cheap run); when set, artifacts are taken in sorted-id order so the subset is deterministic even for filesystem-ordered (dol-backed) scopes.
corpus_name (str | None) – stamped on each case’s corpus field.

Return type:

list[DiscoveryCase]

Returns:

the generated gold cases.

Raises:

ValueError – if k is less than 1.

ir.eval_gen.make_default_abstention_generator(*, prompt='You are generating "no applicable tool" cases for a tool-retrieval evaluation.\\n\\nThe tool catalog is about: {theme}.\\n\\nWrite {n} natural, plausible user requests that such a catalog should NOT be able\\nto satisfy because they fall outside its scope.\\nOne request per line. No numbering, no quotes, no extra commentary.\\n', **prompt_function_kwargs)[source]¶

Build the default abstention generator on aix (lazy import).

Return type:: Callable[..., Sequence[str]]

ir.eval_gen.make_default_query_generator(*, prompt='You are generating evaluation data for a tool-retrieval system.\\n\\nBelow is a description of a capability (its name has been hidden on purpose):\\n\\n{description}\\n\\nWrite {n} natural, varied user requests that this capability should handle.\\nRules:\\n- Do NOT mention any tool, function, skill, or package name.\\n- Vary phrasing, specificity, and the implied (not explicit) parameters.\\n- One request per line. No numbering, no quotes, no extra commentary.\\n', **prompt_function_kwargs)[source]¶

Build the default back-translation generator on aix (lazy import).

Return type:: Callable[..., Sequence[str]]

ir.eval_gen.mask_name(text, name, *, placeholder='this capability')[source]¶

Replace occurrences of name in text with placeholder.

Matches the name as a contiguous phrase tolerant of the separator between its tokens ("ci-advisor" / "ci advisor" / "ci_advisor" all match), case-insensitively and whole-word (a short token like "ci" does not blast through "specific"). If placeholder would itself match the name, a neutral alternate is used so the masked text is not self-referential. Used to scrub the artifact name out of a description before it is generated from.

Return type:: str