recall_guard.harness.evaluator

recall_guard.harness.evaluator

Per-model evaluator: from raw eval set to ModelEvalResult with bootstrap CIs.

Implements the harness.evaluator component from the honest-model-ranking design (see design.md → Components and Interfaces → harness.evaluator). Satisfies Requirements 3.3, 4.3, 5.4, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3, and 10.3.

Pipeline per eval row: 1. Call model_lm.generate(prompt). Map TimeoutError to fail_reason="timeout" and RuntimeError to either "no_logprobs" (message mentions logprobs / top_logprobs) or the generic "error" bucket. Any other exception also falls into "error". 2. On success, optionally call ref_lm.generate(prompt) to obtain reference logprobs. A reference-side failure does not invalidate the row; it merely sets ref_logprobs=None. 3. Parse Direction: strictly (only -1, 0, 1) and Confidence: strictly (must be a float in [0, 1]). Either parse failure → parse_ok=False, fail_reason="parse_failure". 4. Compute MIA features, standardise against the per-model baseline, run mcs.predict_proba to get p_memorized, and apply the continuous penalty penalized_confidence = raw_confidence * (1 - p_memorized) (Req 5.4: no thresholds).

Bootstrap CIs (Req 6.1, 6.3) are computed via core.bootstrap.bootstrap_ci over parse-OK rows for accuracy and over (p_memorized, label) pairs from the supplied holdout_records for MCS-AUC. When no holdout records are available, the CI collapses to the calibrator's holdout_auc point estimate (documented fallback in design.md → Implementation Notes).

The temperature-not-honoured warning (Req 10.3) is surfaced when any LM call returns a non-None, non-zero raw_temperature_observed. All other warnings (weak-calibration, parse-unreliable, not-better-than-baseline, uncalibrated) are emitted by the ranker, not here.

Record dataclass

Per-(model, prompt) record produced by evaluate_model.

Attributes:

Name	Type	Description
model	str	NVIDIA model ID this record was scored for.
prompt_hash	str	First 16 hex chars of sha256(prompt); keeps records.jsonl readable without leaking full prompts. Uniquely keys the record together with model (design § Data Models).
parse_ok	bool	True when Direction: and Confidence: both parsed strictly and MIA features were computable.
predicted_direction	int | None	Parsed integer in {-1, 0, 1}; None when parse_ok is False.
raw_confidence	float | None	Parsed confidence in [0, 1]; None on parse failure.
penalized_confidence	float | None	raw_confidence * (1 - p_memorized) (Req 5.4); None on failure.
target_direction	int	Ground-truth direction copied from the eval row (always populated).
features_raw	MiaFeatures | None	:class:MiaFeatures instance; None when the LM call failed before feature computation could run.
features_standardised	dict[str, float | None] | None	Per-feature standardised values (z-score against the model's baseline); None whenever features_raw is None.
p_memorized	float | None	p(memorized | features) ∈ [0, 1]; None when features_raw is None.
fail_reason	str | None	One of "timeout" / "no_logprobs" / "parse_failure" / "error" on failure; None on success.
raw_response_excerpt	str | None	First ~400 chars of the raw model response when the row failed to parse. Always None for parse-OK rows (no need to bloat the artifact). Use this to inspect why a parse failed without re-running the model.

Source code in recall_guard/harness/evaluator.py

@dataclass(frozen=True)
class Record:
    """Per-(model, prompt) record produced by ``evaluate_model``.

    Attributes
    ----------
    model:
        NVIDIA model ID this record was scored for.
    prompt_hash:
        First 16 hex chars of ``sha256(prompt)``; keeps records.jsonl readable
        without leaking full prompts. Uniquely keys the record together with
        ``model`` (design § Data Models).
    parse_ok:
        ``True`` when ``Direction:`` and ``Confidence:`` both parsed strictly
        and MIA features were computable.
    predicted_direction:
        Parsed integer in ``{-1, 0, 1}``; ``None`` when ``parse_ok`` is False.
    raw_confidence:
        Parsed confidence in ``[0, 1]``; ``None`` on parse failure.
    penalized_confidence:
        ``raw_confidence * (1 - p_memorized)`` (Req 5.4); ``None`` on failure.
    target_direction:
        Ground-truth direction copied from the eval row (always populated).
    features_raw:
        :class:`MiaFeatures` instance; ``None`` when the LM call failed before
        feature computation could run.
    features_standardised:
        Per-feature standardised values (z-score against the model's baseline);
        ``None`` whenever ``features_raw`` is ``None``.
    p_memorized:
        ``p(memorized | features) ∈ [0, 1]``; ``None`` when ``features_raw``
        is ``None``.
    fail_reason:
        One of ``"timeout"`` / ``"no_logprobs"`` / ``"parse_failure"`` /
        ``"error"`` on failure; ``None`` on success.
    raw_response_excerpt:
        First ~400 chars of the raw model response when the row failed to
        parse. Always ``None`` for parse-OK rows (no need to bloat the
        artifact). Use this to inspect *why* a parse failed without re-running
        the model.
    """

    model: str
    prompt_hash: str
    parse_ok: bool
    predicted_direction: int | None
    raw_confidence: float | None
    penalized_confidence: float | None
    target_direction: int
    features_raw: MiaFeatures | None
    features_standardised: dict[str, float | None] | None
    p_memorized: float | None
    fail_reason: str | None
    raw_response_excerpt: str | None = None

CIBound dataclass

A bootstrap point estimate plus 95% percentile bounds.

Source code in recall_guard/harness/evaluator.py

@dataclass(frozen=True)
class CIBound:
    """A bootstrap point estimate plus 95% percentile bounds."""

    point: float
    lo: float
    hi: float

ModelEvalResult dataclass

Aggregate evaluation result for one model on the eval set.

Attributes:

Name	Type	Description
model	str	NVIDIA model ID.
raw_accuracy	CIBound	Bootstrap CI on predicted_direction == target_direction over parse-OK rows (Req 6.1, 7.3).
memguard_accuracy	CIBound	Same accuracy denominator as raw_accuracy: in this spec the MemGuard penalty discounts confidence but does not change the predicted direction, so the two CIs coincide. The field exists so a future confidence-thresholded variant can diverge without breaking downstream consumers.
mcs_auc	CIBound	Bootstrap CI over the (p_memorized, label) pairs in holdout_records. Falls back to mcs.holdout_auc (point only) when no holdout records are supplied (design § Implementation Notes).
parse_success_rate	float	Fraction of rows with parse_ok=True (Req 7.2). 1.0 for empty eval sets (vacuously true).
parse_failures	int	Count of rows with parse_ok=False (Req 7.1).
warnings	list[str]	Subset of {"temperature-not-honoured"}. Other warnings (weak-calibration, parse-unreliable, not-better-than-baseline, uncalibrated) are added by the ranker.
records	list[Record]	Per-row records in eval-set order.

Source code in recall_guard/harness/evaluator.py

@dataclass(frozen=True)
class ModelEvalResult:
    """Aggregate evaluation result for one model on the eval set.

    Attributes
    ----------
    model:
        NVIDIA model ID.
    raw_accuracy:
        Bootstrap CI on ``predicted_direction == target_direction`` over
        parse-OK rows (Req 6.1, 7.3).
    memguard_accuracy:
        Same accuracy denominator as ``raw_accuracy``: in this spec the
        MemGuard penalty discounts confidence but does not change the
        predicted direction, so the two CIs coincide. The field exists so a
        future confidence-thresholded variant can diverge without breaking
        downstream consumers.
    mcs_auc:
        Bootstrap CI over the ``(p_memorized, label)`` pairs in
        ``holdout_records``. Falls back to ``mcs.holdout_auc`` (point only)
        when no holdout records are supplied (design § Implementation Notes).
    parse_success_rate:
        Fraction of rows with ``parse_ok=True`` (Req 7.2). ``1.0`` for empty
        eval sets (vacuously true).
    parse_failures:
        Count of rows with ``parse_ok=False`` (Req 7.1).
    warnings:
        Subset of ``{"temperature-not-honoured"}``. Other warnings
        (weak-calibration, parse-unreliable, not-better-than-baseline,
        uncalibrated) are added by the ranker.
    records:
        Per-row records in eval-set order.
    """

    model: str
    raw_accuracy: CIBound
    memguard_accuracy: CIBound
    mcs_auc: CIBound
    parse_success_rate: float
    parse_failures: int
    warnings: list[str]
    records: list[Record]

evaluate_model

evaluate_model(
    model_lm,
    eval_set,
    baseline,
    mcs,
    ref_lm,
    holdout_records=None,
    bootstrap_n=1000,
    seed=0,
    max_workers=1,
)

Score one model against eval_set and assemble a ModelEvalResult.

With max_workers > 1 the per-row primary + reference LM calls fan out via concurrent.futures.ThreadPoolExecutor (results are paired with rows by index, so order is preserved). Post-processing (parsing, MIA feature compute, MCS scoring) runs serially.

See module docstring for the row-level pipeline. The function performs no I/O beyond the model HTTP calls; all artifact writing is owned by harness.report and harness.runner.

Source code in recall_guard/harness/evaluator.py

def evaluate_model(
    model_lm: NvidiaLM,
    eval_set: EvalSet,
    baseline: ControlBaseline,
    mcs: MCSCalibrator,
    ref_lm: NvidiaLM | None,
    holdout_records: list[Record] | None = None,
    bootstrap_n: int = 1000,
    seed: int = 0,
    max_workers: int = 1,
) -> ModelEvalResult:
    """Score one model against ``eval_set`` and assemble a ``ModelEvalResult``.

    With ``max_workers > 1`` the per-row primary + reference LM calls
    fan out via ``concurrent.futures.ThreadPoolExecutor`` (results are
    paired with rows by index, so order is preserved). Post-processing
    (parsing, MIA feature compute, MCS scoring) runs serially.

    See module docstring for the row-level pipeline. The function performs no
    I/O beyond the model HTTP calls; all artifact writing is owned by
    ``harness.report`` and ``harness.runner``.
    """
    model_id = getattr(model_lm, "model", "<unknown>")

    prompts = [row.prompt for row in eval_set.rows]
    primary_results = generate_many(model_lm, prompts, max_workers=max_workers)
    ref_results: list = (
        generate_many(ref_lm, prompts, max_workers=max_workers)
        if ref_lm is not None else [None] * len(prompts)
    )

    records: list[Record] = []
    temperature_violated = False
    for row, primary, ref_res in zip(eval_set.rows, primary_results, ref_results, strict=True):
        record, row_temp_violated = _score_row(
            model_id=model_id, row=row, primary=primary, ref_res=ref_res,
            ref_lm=ref_lm, baseline=baseline, mcs=mcs,
        )
        records.append(record)
        if row_temp_violated:
            temperature_violated = True

    n_rows = len(records)
    parse_failures = sum(1 for r in records if not r.parse_ok)
    parse_ok_records = [r for r in records if r.parse_ok]
    parse_success_rate = (
        1.0 if n_rows == 0 else (n_rows - parse_failures) / n_rows
    )

    raw_accuracy = _accuracy_ci(parse_ok_records, n=bootstrap_n, seed=seed)
    # MemGuard accuracy uses the same parse-OK denominator. The penalty
    # affects confidence only (predicted_direction is unchanged in this
    # spec), so the bootstrap statistic is identical. The dataclass field is
    # kept distinct so a future confidence-threshold variant may diverge
    # without breaking the report schema.
    memguard_accuracy = _accuracy_ci(parse_ok_records, n=bootstrap_n, seed=seed)
    mcs_auc = _mcs_auc_ci(holdout_records, mcs, n=bootstrap_n, seed=seed)

    warnings: list[str] = []
    if temperature_violated:
        warnings.append(WARNING_TEMPERATURE_NOT_HONOURED)

    return ModelEvalResult(
        model=model_id,
        raw_accuracy=raw_accuracy,
        memguard_accuracy=memguard_accuracy,
        mcs_auc=mcs_auc,
        parse_success_rate=parse_success_rate,
        parse_failures=parse_failures,
        warnings=warnings,
        records=records,
    )

compute_majority_baseline

compute_majority_baseline(
    eval_set, bootstrap_n=1000, seed=0
)

Bootstrap CI on the majority-class baseline accuracy (Req 6.2).

Returns CIBound(0.0, 0.0, 0.0) for an empty eval set.

Source code in recall_guard/harness/evaluator.py

def compute_majority_baseline(
    eval_set: EvalSet,
    bootstrap_n: int = 1000,
    seed: int = 0,
) -> CIBound:
    """Bootstrap CI on the majority-class baseline accuracy (Req 6.2).

    Returns ``CIBound(0.0, 0.0, 0.0)`` for an empty eval set.
    """
    rows = eval_set.rows
    if not rows:
        return CIBound(0.0, 0.0, 0.0)

    counts = Counter(r.target_direction for r in rows)
    majority_class, _ = counts.most_common(1)[0]
    indicators: list[int] = [1 if r.target_direction == majority_class else 0 for r in rows]

    def _mean(samples: list[int]) -> float:
        return sum(samples) / len(samples)

    point, lo, hi = bootstrap_ci(
        samples=indicators,
        statistic=_mean,
        n_resamples=bootstrap_n,
        seed=seed,
    )
    return CIBound(point=point, lo=lo, hi=hi)