Skip to content

recall_guard.harness.smoke

recall_guard.harness.smoke

Smoke-test gate that produces a candidate shortlist for the harness.

Implements the harness.smoke component from the honest-model-ranking design (Requirements 1.1, 1.2, 1.3, 1.4). For each candidate model the gate runs N fixed smoke prompts via core.nvidia_lm.NvidiaLM and excludes any model that times out, returns no logprobs, fails to emit a parseable Direction: value, or otherwise errors. The returned Shortlist carries one SmokeOutcome per candidate so the runner can persist the full pass/fail-reason record (1.4); this module deliberately does no I/O.

SmokeOutcome dataclass

Per-candidate smoke-test outcome.

fail_reason is None on pass and one of "timeout", "no_logprobs", "parse_failure", or "error" on fail.

Source code in recall_guard/harness/smoke.py 
34
35
36
37
38
39
40
41
42
43
44
@dataclass(frozen=True)
class SmokeOutcome:
    """Per-candidate smoke-test outcome.

    `fail_reason` is `None` on pass and one of `"timeout"`, `"no_logprobs"`,
    `"parse_failure"`, or `"error"` on fail.
    """

    model: str
    passed: bool
    fail_reason: str | None

Shortlist dataclass

Result of the smoke-test gate.

selected contains the passing models in candidate order, capped at max_size. outcomes contains one entry per candidate (regardless of pass/fail) so the runner can persist a reproducible artifact (Req 1.4).

Source code in recall_guard/harness/smoke.py 
47
48
49
50
51
52
53
54
55
56
57
@dataclass(frozen=True)
class Shortlist:
    """Result of the smoke-test gate.

    `selected` contains the passing models in candidate order, capped at
    `max_size`. `outcomes` contains one entry per candidate (regardless of
    pass/fail) so the runner can persist a reproducible artifact (Req 1.4).
    """

    selected: list[str]
    outcomes: list[SmokeOutcome]

smoke_test 

smoke_test(
    candidates,
    api_key,
    smoke_prompts,
    max_size=10,
    timeout_s=DEFAULT_TIMEOUT_S,
    lm_factory=None,
)

Run the smoke-test gate over candidates and return a Shortlist.

Parameters:

Name Type Description Default
candidates list[str]

Ordered candidate model IDs to evaluate.

 required
api_key str

NVIDIA API key forwarded to the LM factory.

 required
smoke_prompts list[str]

The fixed smoke prompts. Every candidate runs every prompt unless an exclusion fires earlier (Req 1.2).

 required
max_size int

Hard cap on Shortlist.selected (Req 1.1; default 10).

 10
timeout_s float

Per-call timeout forwarded to the LM client (Req 1.2).

 DEFAULT_TIMEOUT_S
lm_factory LMFactory | None

Optional (api_key, model, timeout_s) -> NvidiaLM factory used for test injection. Defaults to the real NvidiaLM constructor.

 None

Returns:

Type Description
Shortlist

selected capped at max_size, plus one SmokeOutcome per candidate. The function performs no I/O; the runner persists outcomes to shortlist.json (Req 1.4).
Source code in recall_guard/harness/smoke.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def smoke_test(
    candidates: list[str],
    api_key: str,
    smoke_prompts: list[str],
    max_size: int = 10,
    timeout_s: float = DEFAULT_TIMEOUT_S,
    lm_factory: LMFactory | None = None,
) -> Shortlist:
    """Run the smoke-test gate over `candidates` and return a Shortlist.

    Parameters
    ----------
    candidates:
        Ordered candidate model IDs to evaluate.
    api_key:
        NVIDIA API key forwarded to the LM factory.
    smoke_prompts:
        The fixed smoke prompts. Every candidate runs every prompt unless an
        exclusion fires earlier (Req 1.2).
    max_size:
        Hard cap on `Shortlist.selected` (Req 1.1; default 10).
    timeout_s:
        Per-call timeout forwarded to the LM client (Req 1.2).
    lm_factory:
        Optional `(api_key, model, timeout_s) -> NvidiaLM` factory used for
        test injection. Defaults to the real `NvidiaLM` constructor.

    Returns
    -------
    Shortlist
        `selected` capped at `max_size`, plus one `SmokeOutcome` per
        candidate. The function performs no I/O; the runner persists
        `outcomes` to `shortlist.json` (Req 1.4).
    """
    factory: LMFactory = lm_factory or _default_lm_factory
    outcomes: list[SmokeOutcome] = []
    selected: list[str] = []

    for model in candidates:
        outcome = _smoke_one(model, api_key, smoke_prompts, timeout_s, factory)
        outcomes.append(outcome)
        if outcome.passed and len(selected) < max_size:
            selected.append(model)

    return Shortlist(selected=selected, outcomes=outcomes)