recall_guard.portfolio.cohens_d

recall_guard.portfolio.cohens_d

Per-(model, MIA-feature) Cohen's d artifact for cmmd-backtest.

Implements requirements 1.1, 1.2, 1.3, 1.4, 1.5, and 9.1 of cmmd-backtest: read a finished harness run's records.jsonl, split each model's parse-OK rows into IS / OOS by joining metadata.date against the model's training cutoff, and compute Cohen's d on the raw (non- standardised) value of every MIA feature. Writes cohens_d.csv and cohens_d.md into the run directory.

Design deviation: the design's compute_cohens_d signature lists only (run_dir, cutoffs_path), but records.jsonl carries prompt_hash rather than metadata.date (see recall_guard.harness.evaluator.Record). To recover the date we have to join records back to the eval set on prompt_hash, so the public signature accepts eval_path explicitly. The orchestrator (task 3.2) supplies it.

Sentrux boundaries:

• Reads records.jsonl shape produced by recall_guard.harness.report; matches the harness's own _hash_prompt convention (sha256 hex, first 16 chars). Reproduced locally to keep the portfolio layer free of upward imports.
• portfolio ↔ dataset and portfolio ↔ mia are explicitly forbidden by .sentrux/rules.toml; this module imports neither.

write_cohens_d_artifacts

write_cohens_d_artifacts(df, run_dir)

Write cohens_d.csv and cohens_d.md into run_dir.

Returns a {name: Path} map so callers (the orchestrator) can record the artifact paths in the manifest. Re-running on the same DataFrame produces byte-identical files.

Source code in recall_guard/portfolio/cohens_d.py

def write_cohens_d_artifacts(df: pd.DataFrame, run_dir: Path) -> dict[str, Path]:
    """Write ``cohens_d.csv`` and ``cohens_d.md`` into ``run_dir``.

    Returns a ``{name: Path}`` map so callers (the orchestrator) can
    record the artifact paths in the manifest. Re-running on the same
    DataFrame produces byte-identical files.
    """
    run_dir.mkdir(parents=True, exist_ok=True)
    csv_path = run_dir / "cohens_d.csv"
    md_path = run_dir / "cohens_d.md"

    # CSV: deterministic column order, fixed line terminator. We drive
    # the writer manually so NaN renders as the empty string rather than
    # the locale-dependent "nan" string pandas would emit.
    with csv_path.open("w", encoding="utf-8", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=_COLUMNS, lineterminator="\n")
        writer.writeheader()
        for _, row in df.iterrows():
            payload: dict[str, str] = {}
            for col in _COLUMNS:
                value = row[col]
                if isinstance(value, float):
                    if math.isnan(value):
                        payload[col] = ""
                    else:
                        payload[col] = f"{value:.6f}"
                else:
                    payload[col] = "" if value is None else str(value)
            writer.writerow(payload)

    md_path.write_text(_render_markdown(df, run_dir), encoding="utf-8")
    return {"cohens_d_csv": csv_path, "cohens_d_md": md_path}

compute_cohens_d

compute_cohens_d(
    run_dir,
    eval_path,
    cutoffs_path=Path("data/cutoffs.yaml"),
)

Compute per-(model, feature) Cohen's d and write artifacts.

Args: run_dir: Finished harness run directory containing records.jsonl and summary.csv. Artifacts are written here. eval_path: Path to the eval-set JSONL whose prompt strings originally fed the harness. Required because the harness's Record schema carries only prompt_hash; the date used to label IS / OOS lives on the eval row's metadata.date. cutoffs_path: Path to data/cutoffs.yaml. Models present in records.jsonl but absent here are logged as a warning and excluded from the artifact rather than crashing the run (Req 9.1 spirit).

Returns: A pandas.DataFrame with one row per (model, feature) pair and the schema documented in design.md § Cohen's d artifact schema. The CSV / MD twins are written to run_dir as a side effect.

Raises: FileNotFoundError: if records.jsonl is missing from the run directory or eval_path / cutoffs_path do not exist.

Source code in recall_guard/portfolio/cohens_d.py

def compute_cohens_d(
    run_dir: Path,
    eval_path: Path,
    cutoffs_path: Path = Path("data/cutoffs.yaml"),
) -> pd.DataFrame:
    """Compute per-(model, feature) Cohen's d and write artifacts.

    Args:
        run_dir: Finished harness run directory containing
            ``records.jsonl`` and ``summary.csv``. Artifacts are
            written here.
        eval_path: Path to the eval-set JSONL whose ``prompt`` strings
            originally fed the harness. Required because the harness's
            ``Record`` schema carries only ``prompt_hash``; the date used
            to label IS / OOS lives on the eval row's
            ``metadata.date``.
        cutoffs_path: Path to ``data/cutoffs.yaml``. Models present in
            ``records.jsonl`` but absent here are logged as a warning
            and excluded from the artifact rather than crashing the run
            (Req 9.1 spirit).

    Returns:
        A ``pandas.DataFrame`` with one row per ``(model, feature)``
        pair and the schema documented in ``design.md``
        § Cohen's d artifact schema. The CSV / MD twins are written to
        ``run_dir`` as a side effect.

    Raises:
        FileNotFoundError: if ``records.jsonl`` is missing from the run
            directory or ``eval_path`` / ``cutoffs_path`` do not exist.
    """
    run_dir = Path(run_dir)
    eval_path = Path(eval_path)
    cutoffs_path = Path(cutoffs_path)

    records_path = run_dir / "records.jsonl"
    summary_path = run_dir / "summary.csv"

    if not records_path.exists():
        raise FileNotFoundError(f"missing records.jsonl in {run_dir}")
    if not eval_path.exists():
        raise FileNotFoundError(f"missing eval set: {eval_path}")
    if not cutoffs_path.exists():
        raise FileNotFoundError(f"missing cutoffs registry: {cutoffs_path}")

    metadata_by_hash = _load_eval_metadata(eval_path)
    cutoffs = _load_cutoffs(cutoffs_path)
    auc_by_model = _load_summary_auc(summary_path)

    buckets = _collect_feature_splits(records_path, metadata_by_hash, cutoffs)

    rows: list[dict] = []
    # Stable sort: by model then by the canonical feature order.
    feature_order = {name: i for i, name in enumerate(_FEATURE_NAMES)}
    sorted_keys = sorted(
        buckets.keys(), key=lambda k: (k[0], feature_order.get(k[1], 999))
    )
    for key in sorted_keys:
        model, feature = key
        slot = buckets[key]
        rows.append(_cohens_d_row(
            model=model,
            feature=feature,
            is_values=slot["is"],
            oos_values=slot["oos"],
            mcs_auc=auc_by_model.get(model),
        ))

    df = pd.DataFrame(rows, columns=_COLUMNS)
    write_cohens_d_artifacts(df, run_dir)
    return df