recall_guard.harness.ranker

recall_guard.harness.ranker

Composite ranking + top3.md writer.

Implements the harness.ranker component from the honest-model-ranking design (see design.md → Components and Interfaces → harness.ranker). Satisfies Requirements 5.3, 6.4, 7.4, 8.1, 8.2, 8.3, 8.4.

Pipeline per ModelEvalResult:

1. Pull the three composite components (MemGuard accuracy lower CI bound, MCS-AUC point estimate, and parse-success rate) into the CompositeScore.components dict (Req 8.1).

2. Apply the four gating warnings:

3. weak-calibration when mcs_auc.point < gates["mcs_auc_min"] (Req 5.3).

4. parse-unreliable when parse_success_rate < gates["parse_min"] (Req 7.4).
5. not-better-than-baseline when memguard_accuracy.lo does not strictly exceed the majority-class upper CI bound (Req 6.4).
6. uncalibrated when the upstream evaluator/runner already flagged the result (Req 3.4 surfaced via the runner's ControlBaseline.is_calibrated check).
7. Pass through temperature-not-honoured (Req 10.3) without treating it as a gate; it is purely informational (the design.md "harness.ranker" gates table makes it explicit that only the four listed warnings block).
8. Multiplicative gate: any blocking warning sets survives_gates=False and score=0.0 (Req 8.1, design Invariants).
9. Surviving models score memguard_acc_lo * mcs_auc_point * parse_success_rate.

write_top3 (Req 8.2, 8.3, 8.4):

• Sorts by score descending, stable on ties (input order preserved).
• Top section lists at most three surviving models; gate-failed scores never appear in the top-3 ledger because they would be misleading even with a zero score.
• Whenever fewer than three models survive, a Why fewer than three models section enumerates each non-survivor and the gate(s) it failed (Req 8.3).
• A Composite score formula footer always shows the formula string and the gate thresholds so the reader can reproduce the ranking (Req 8.4).

CompositeScore dataclass

Composite rank score for one model with gate verdict + warnings.

Attributes:

Name	Type	Description
model	str	NVIDIA model ID this score belongs to.
score	float	Multiplicative composite memguard_acc_lo * mcs_auc_point * parse_success_rate if all gates pass; 0.0 otherwise (Req 8.1).
components	dict[str, float]	The three component values that fed the formula. Keys are stable: "memguard_acc_lo", "mcs_auc_point", "parse_success_rate".
survives_gates	bool	True iff none of weak-calibration, parse-unreliable, not-better-than-baseline, uncalibrated are in warnings. temperature-not-honoured does not affect this flag.
warnings	list[str]	Subset of the ranker warning vocabulary, including any informational warnings (temperature-not-honoured) passed through from the evaluator.

Source code in recall_guard/harness/ranker.py

@dataclass(frozen=True)
class CompositeScore:
    """Composite rank score for one model with gate verdict + warnings.

    Attributes
    ----------
    model:
        NVIDIA model ID this score belongs to.
    score:
        Multiplicative composite ``memguard_acc_lo * mcs_auc_point *
        parse_success_rate`` if all gates pass; ``0.0`` otherwise (Req 8.1).
    components:
        The three component values that fed the formula. Keys are stable:
        ``"memguard_acc_lo"``, ``"mcs_auc_point"``, ``"parse_success_rate"``.
    survives_gates:
        ``True`` iff none of ``weak-calibration``, ``parse-unreliable``,
        ``not-better-than-baseline``, ``uncalibrated`` are in ``warnings``.
        ``temperature-not-honoured`` does not affect this flag.
    warnings:
        Subset of the ranker warning vocabulary, including any
        informational warnings (``temperature-not-honoured``) passed through
        from the evaluator.
    """

    model: str
    score: float
    components: dict[str, float]
    survives_gates: bool
    warnings: list[str]

composite_score

composite_score(
    results,
    majority_baseline,
    formula=COMPOSITE_FORMULA,
    gates=GATES,
)

Convert per-model evaluation results into composite scores.

The formula argument is persisted alongside the gates in top3.md (Req 8.4) but is not parsed at runtime; the multiplicative formula is the only one defined in this spec. A future variant would change both the component dict keys and the formula string in lockstep.

Returns the scores in the input order of results; ordering is deferred to write_top3 (or other downstream consumers) so the canonical record stream stays aligned with the eval-set order (Req 9.3).

Source code in recall_guard/harness/ranker.py

def composite_score(
    results: list[ModelEvalResult],
    majority_baseline: CIBound,
    formula: str = COMPOSITE_FORMULA,
    gates: dict[str, float] = GATES,
) -> list[CompositeScore]:
    """Convert per-model evaluation results into composite scores.

    The ``formula`` argument is persisted alongside the gates in ``top3.md``
    (Req 8.4) but is not parsed at runtime; the multiplicative formula is the
    only one defined in this spec. A future variant would change both the
    component dict keys and the formula string in lockstep.

    Returns the scores in the *input order* of ``results``; ordering is
    deferred to ``write_top3`` (or other downstream consumers) so the
    canonical record stream stays aligned with the eval-set order (Req 9.3).
    """
    del formula  # Persisted via write_top3; not interpreted here.

    scores: list[CompositeScore] = []
    for result in results:
        components: dict[str, float] = {
            "memguard_acc_lo": float(result.memguard_accuracy.lo),
            "mcs_auc_point": float(result.mcs_auc.point),
            "parse_success_rate": float(result.parse_success_rate),
        }

        warnings = _gate_warnings(result, majority_baseline, gates)
        survives = not any(w in _BLOCKING_WARNINGS for w in warnings)

        if survives:
            score_value = (
                components["memguard_acc_lo"]
                * components["mcs_auc_point"]
                * components["parse_success_rate"]
            )
        else:
            score_value = 0.0

        scores.append(
            CompositeScore(
                model=result.model,
                score=float(score_value),
                components=components,
                survives_gates=survives,
                warnings=warnings,
            )
        )

    return scores

write_top3

write_top3(
    scores, path, formula=COMPOSITE_FORMULA, gates=GATES
)

Write top3.md to path.

The file always contains the # Top 3 Models heading and the ## Composite score formula footer; the explanatory section is included whenever fewer than three models survive (Req 8.3). The parent directory is created if missing so callers do not have to mkdir first.

Source code in recall_guard/harness/ranker.py

def write_top3(
    scores: list[CompositeScore],
    path: Path,
    formula: str = COMPOSITE_FORMULA,
    gates: dict[str, float] = GATES,
) -> None:
    """Write ``top3.md`` to ``path``.

    The file always contains the ``# Top 3 Models`` heading and the
    ``## Composite score formula`` footer; the explanatory section is
    included whenever fewer than three models survive (Req 8.3). The parent
    directory is created if missing so callers do not have to mkdir first.
    """
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)

    # Stable sort: descending by score, ties broken by input order.
    # ``sorted`` in CPython is stable, so reversing the indexed pairs is not
    # needed; we sort by negated score directly.
    indexed: list[tuple[int, CompositeScore]] = list(enumerate(scores))
    indexed.sort(key=lambda pair: -pair[1].score)
    sorted_scores = [s for _, s in indexed]

    survivors = [s for s in sorted_scores if s.survives_gates]
    nonsurvivors_in_input_order = [s for s in scores if not s.survives_gates]
    top_survivors = survivors[:3]

    lines: list[str] = ["# Top 3 Models", ""]

    if not top_survivors:
        lines.append("_No models passed all gates._")
        lines.append("")
    else:
        for rank, score in enumerate(top_survivors, start=1):
            lines.extend(_render_survivor_block(rank, score))
            lines.append("")

    if len(top_survivors) < 3:
        lines.extend(
            _render_short_list_explanation(
                total_evaluated=len(scores),
                nonsurvivors=nonsurvivors_in_input_order,
            )
        )

    lines.extend(_render_formula_footer(formula, gates))

    path.write_text("\n".join(lines), encoding="utf-8")