recall_guard.harness.report

recall_guard.harness.report

Structured report writers for the harness (Req 9.1, 9.2, 9.3, 9.4).

Implements the harness.report component from the honest-model-ranking design (see design.md → Components and Interfaces → harness.report).

Public surface

• :func:render_terminal: rich-backed table rendering of one row per shortlisted model plus a __majority_baseline__ row (Req 9.1, 9.2).
• :func:write_records: streaming JSONL writer; one JSON object per Record (Req 9.3). Memory stays bounded by writing line-by-line rather than building an in-memory list of all records first.
• :func:write_summary_csv: flat CSV with a 15-column schema plus a final __majority_baseline__ row that fills only the raw-accuracy CI cells (Req 9.3 schema half).
• :func:print_artifact_paths: final-line summary of every artifact path so the operator sees the run output up front (Req 9.4).

Design choices

• The CSV signature is (results, scores, majority, path): the design's Service Interface lists three arguments but the Req 9.3 observable in tasks.md Task 4.4 demands a majority row, so the majority CIBound is threaded through explicitly.
• Sorting in render_terminal mirrors the ranker's stable-by-input-order semantics: within equal scores the input order is preserved.
• JSON serialisation uses dataclasses.asdict for MiaFeatures so the per-record artifact stays a flat object rather than a stringified dataclass repr (audited by tests/harness/test_report.py).

render_terminal

render_terminal(results, majority, scores, console=None)

Print one table row per model plus a majority-baseline row (Req 9.1, 9.2).

Rows for surviving + non-surviving models are sorted by score descending (stable within ties on input order); the __majority_baseline__ row always renders last so a reader can compare every model against it visually.

The majority row populates only the Raw Acc CI column; the other cells are em-dashes since MemGuard accuracy, MCS-AUC, and the composite score are not defined for the baseline.

Parameters:

Name	Type	Description	Default
results	list[ModelEvalResult]	Aligned by model ID via lookup (not by index) so a missing score does not silently misalign rows.	required
scores	list[ModelEvalResult]	Aligned by model ID via lookup (not by index) so a missing score does not silently misalign rows.	required
majority	CIBound	Bootstrap CI on the majority-class baseline accuracy from compute_majority_baseline.	required
console	Console | None	Optional rich.console.Console injection point for tests; defaults to Console() (writes to stdout).	None

Source code in recall_guard/harness/report.py

def render_terminal(
    results: list[ModelEvalResult],
    majority: CIBound,
    scores: list[CompositeScore],
    console: Console | None = None,
) -> None:
    """Print one table row per model plus a majority-baseline row (Req 9.1, 9.2).

    Rows for surviving + non-surviving models are sorted by ``score``
    descending (stable within ties on input order); the
    ``__majority_baseline__`` row always renders last so a reader can compare
    every model against it visually.

    The majority row populates only the Raw Acc CI column; the other cells
    are em-dashes since MemGuard accuracy, MCS-AUC, and the composite score
    are not defined for the baseline.

    Parameters
    ----------
    results, scores:
        Aligned by model ID via lookup (not by index) so a missing score does
        not silently misalign rows.
    majority:
        Bootstrap CI on the majority-class baseline accuracy from
        ``compute_majority_baseline``.
    console:
        Optional ``rich.console.Console`` injection point for tests; defaults
        to ``Console()`` (writes to stdout).
    """
    table = Table(show_header=True, header_style="bold")
    table.add_column("Model")
    table.add_column("Raw Acc (CI)")
    table.add_column("MemGuard Acc (CI)")
    table.add_column("MCS-AUC (CI)")
    table.add_column("Parse %")
    table.add_column("Score")
    table.add_column("Warnings")

    # Stable sort: descending by score (None / missing scores treated as 0.0
    # so they fall to the bottom but stay above the majority row).
    indexed = list(enumerate(results))

    def _sort_key(pair: tuple[int, ModelEvalResult]) -> tuple[float, int]:
        idx, result = pair
        score = _score_for_model(result.model, scores)
        score_value = score.score if score is not None else 0.0
        # Negate score for descending order while keeping idx ascending for
        # stability on ties.
        return (-score_value, idx)

    indexed.sort(key=_sort_key)
    sorted_results = [r for _, r in indexed]

    for result in sorted_results:
        score = _score_for_model(result.model, scores)
        # Warnings shown in the terminal merge evaluator + ranker warnings;
        # the ranker passes through informational ones so we deduplicate while
        # preserving order.
        warning_set: list[str] = []
        for w in (result.warnings or []) + (score.warnings if score else []):
            if w not in warning_set:
                warning_set.append(w)

        table.add_row(
            result.model,
            _format_ci(result.raw_accuracy),
            _format_ci(result.memguard_accuracy),
            _format_ci(result.mcs_auc),
            _format_percent(result.parse_success_rate),
            _format_score(score),
            _format_warnings(warning_set),
        )

    # Majority-baseline row: only the Raw Acc CI is meaningful.
    em_dash = "—"
    table.add_row(
        MAJORITY_LABEL,
        _format_ci(majority),
        em_dash,
        em_dash,
        em_dash,
        em_dash,
        "",
    )

    # Auto-detect terminal width when stdout is a TTY; fall back to a wide
    # 200-col console for redirected/captured stdout (pytest, pipes) so the
    # model column and warning strings do not get truncated. Callers can
    # inject their own ``Console`` to override this.
    target = console or Console(width=shutil.get_terminal_size((200, 20)).columns)
    target.print(table)

write_records

write_records(results, path)

Stream every Record from every result to records.jsonl (Req 9.3).

Memory stays bounded for long runs because the writer opens the file once and emits one json.dumps line per record before moving to the next; no all-records list is built in memory.

The schema is documented in :func:_record_to_jsonable and audited by tests/harness/test_report.py::test_write_records_includes_all_required_fields.

Source code in recall_guard/harness/report.py

def write_records(results: Iterable[ModelEvalResult], path: Path) -> None:
    """Stream every ``Record`` from every result to ``records.jsonl`` (Req 9.3).

    Memory stays bounded for long runs because the writer opens the file once
    and emits one ``json.dumps`` line per record before moving to the next;
    no all-records list is built in memory.

    The schema is documented in :func:`_record_to_jsonable` and audited by
    ``tests/harness/test_report.py::test_write_records_includes_all_required_fields``.
    """
    target = Path(path)
    _ensure_parent(target)
    with target.open("w", encoding="utf-8") as fh:
        for result in results:
            for record in result.records:
                payload = _record_to_jsonable(record)
                fh.write(json.dumps(payload, ensure_ascii=False))
                fh.write("\n")

write_summary_csv

write_summary_csv(results, scores, majority, path)

Write one CSV row per model plus a majority-baseline row (Req 9.3).

The 15-column schema is fixed in :data:SUMMARY_CSV_COLUMNS; every CSV consumer (e.g. the qualification notebook) can rely on it.

The majority row only fills the raw-accuracy CI cells; the rest are blank because there is no MemGuard accuracy / MCS-AUC / score notion for the baseline (this matches the design's "majority row alongside model rows" interpretation).

Notes

The function signature includes majority even though the design's Service Interface lists only three arguments; Task 4.4's observable requires the majority row in the CSV, which forces the parameter through.

Source code in recall_guard/harness/report.py

def write_summary_csv(
    results: list[ModelEvalResult],
    scores: list[CompositeScore],
    majority: CIBound,
    path: Path,
) -> None:
    """Write one CSV row per model plus a majority-baseline row (Req 9.3).

    The 15-column schema is fixed in :data:`SUMMARY_CSV_COLUMNS`; every CSV
    consumer (e.g. the qualification notebook) can rely on it.

    The majority row only fills the raw-accuracy CI cells; the rest are blank
    because there is no MemGuard accuracy / MCS-AUC / score notion for the
    baseline (this matches the design's "majority row alongside model rows"
    interpretation).

    Notes
    -----
    The function signature includes ``majority`` even though the design's
    Service Interface lists only three arguments; Task 4.4's observable
    requires the majority row in the CSV, which forces the parameter through.
    """
    target = Path(path)
    _ensure_parent(target)

    with target.open("w", encoding="utf-8", newline="") as fh:
        writer = csv.DictWriter(
            fh, fieldnames=SUMMARY_CSV_COLUMNS, lineterminator="\n"
        )
        writer.writeheader()

        for result in results:
            score = _score_for_model(result.model, scores)
            writer.writerow(_result_row(result, score))

        writer.writerow(_majority_row(majority))

print_artifact_paths

print_artifact_paths(paths, console=None)

Print the final Artifacts: summary block (Req 9.4).

Each key/value pair is rendered as <name> <path> so the operator can copy paths directly out of the terminal.

Source code in recall_guard/harness/report.py

def print_artifact_paths(
    paths: dict[str, Path], console: Console | None = None
) -> None:
    """Print the final ``Artifacts:`` summary block (Req 9.4).

    Each key/value pair is rendered as ``<name>  <path>`` so the
    operator can copy paths directly out of the terminal.
    """
    target = console or Console()
    target.print("Artifacts:")
    for name, path in paths.items():
        target.print(f"  {name}\t{path}")