`recall_guard.portfolio.backtest`

recall_guard.portfolio.backtest

Long-short cross-sectional backtest engine for the cmmd backtest.

This module takes a stream of harness-style records plus a (date x ticker) close-price matrix, builds a daily-rebalanced target-weight matrix where weight[d, t] = direction[d, t] * confidence[d, t], and runs two backtests through vectorbt:

raw_alpha: every surviving parse-OK row
cmmd: rows above the chosen p_memorized threshold removed

The two variants share one price matrix; the difference is the surviving record stream. This file owns the BacktestMetrics and BacktestResult dataclasses, plus both the compute path (run_backtest) and the artifact writer.

Layer rules and design deviation

The portfolio layer is order=1 in .sentrux/rules.toml and the harness layer is order=0. Order=1 cannot import from order=0, so this module never imports harness.evaluator.Record. Instead run_backtest accepts any record-shaped object exposing parse_ok, predicted_direction, raw_confidence, p_memorized, and prompt_hash.

The Record dataclass produced by harness.evaluator does not carry metadata.date or metadata.ticker. To build a (date x ticker) weight matrix, the caller passes prompt_metadata, mapping each record's prompt_hash to {"ticker": str, "date": str}. The orchestrator builds that mapping from the eval set; the engine stays pure compute.

Determinism

The engine is deterministic given identical records, prices, prompt_metadata, and seed. Vectorbt's portfolio construction is deterministic; the only stochastic step is the bootstrap CI on Sharpe and mean daily return, which threads seed through core.bootstrap.bootstrap_ci.

Key contracts

BacktestResult.equity_curves has columns ["raw_alpha", "cmmd", "buy_and_hold_swda"] in that exact order. The raw/cmmd curves are cumprod(1 + daily_returns), so day 0 shows the entry fee (slightly below 1.0) and the terminal value equals 1 + total_return; the fee-free buy-and-hold benchmark starts at exactly 1.0.
BacktestResult.daily_returns_bps has columns ["raw_alpha", "cmmd"] and is denominated in basis points (x10^4).
BacktestMetrics.max_drawdown_pct is signed: a negative number reports a drawdown (for example, -3.4 means -3.4%).

vectorbt 0.28 conventions

size_type='targetpercent' rebalances to the target weight on every bar. Combined with cash_sharing=True, group_by=True this gives one portfolio across all tickers; vectorbt deducts trading fees on the trade notional, which equals |Δw_t| x portfolio value at the rebalance bar.
freq='1D' sets the annualization factor (252 trading days/year) for Portfolio.sharpe_ratio().
The size matrix passed to from_orders already contains BIL's residual allocation, so vectorbt charges the BIL purchase as a real trade.

BacktestArtifactError

Bases: RuntimeError

Raised when writing the backtest artifacts to disk fails.

The writer builds every artifact in memory before touching disk and rolls back any partially-written files on failure (Req 7.6), so by the time this exception surfaces the run directory is in the same state it was in before the call.

Source code in recall_guard/portfolio/backtest.py

class BacktestArtifactError(RuntimeError):
    """Raised when writing the backtest artifacts to disk fails.

    The writer builds every artifact in memory before touching disk and
    rolls back any partially-written files on failure (Req 7.6), so by
    the time this exception surfaces the run directory is in the same
    state it was in before the call.
    """

BacktestMetrics `dataclass`

Per-variant summary statistics for one backtest run.

All fields are JSON-friendly so downstream artifact writers (task 2.5) can dump the dataclass directly. The sharpe_annualised and mean_daily_return_bps tuples are (point, lo, hi) from core.bootstrap.bootstrap_ci: point estimate first, then the 95% percentile bounds.

Source code in recall_guard/portfolio/backtest.py

@dataclass(frozen=True)
class BacktestMetrics:
    """Per-variant summary statistics for one backtest run.

    All fields are JSON-friendly so downstream artifact writers (task
    2.5) can dump the dataclass directly. The ``sharpe_annualised`` and
    ``mean_daily_return_bps`` tuples are ``(point, lo, hi)`` from
    ``core.bootstrap.bootstrap_ci``: point estimate first, then the
    95% percentile bounds.
    """

    label: str
    sharpe_annualised: tuple[float, float, float]
    mean_daily_return_bps: tuple[float, float, float]
    max_drawdown_pct: float
    total_return_pct: float
    n_trading_days: int
    n_signals_used: int
    cmmd_threshold: float | None

BacktestResult `dataclass`

Bundle of both variants and the curves needed for plotting.

equity_curves is cumprod(1 + daily_returns) per variant, so it is exactly reconstructable from daily_returns_bps (basis points, ×10⁴) and its terminal value matches total_return_pct.

Source code in recall_guard/portfolio/backtest.py

@dataclass(frozen=True)
class BacktestResult:
    """Bundle of both variants and the curves needed for plotting.

    ``equity_curves`` is ``cumprod(1 + daily_returns)`` per variant, so
    it is exactly reconstructable from ``daily_returns_bps`` (basis
    points, ×10⁴) and its terminal value matches ``total_return_pct``.
    """

    raw: BacktestMetrics
    cmmd: BacktestMetrics
    relative_sharpe_improvement: float
    equity_curves: pd.DataFrame
    daily_returns_bps: pd.DataFrame
    warnings: list[str]

run_backtest

run_backtest(
    records,
    prices,
    prompt_metadata,
    *,
    cmmd_quantile=0.8,
    fees_one_way=0.00075,
    init_cash=1.0,
    seed=0,
    bootstrap_n=1000,
)

Run the long-short backtest twice (raw + cmmd) on one price matrix.

Args: records: List of record-shaped objects (see :class:_RecordLike). Records with parse_ok=False, predicted_direction is None, raw_confidence is None, or no entry in prompt_metadata are dropped from BOTH variants (Req 9.1). prices: (date × ticker) close-price matrix. Must contain the BIL column plus at least one risk asset. The DataFrame's index is treated as the trading-day calendar; signals dated outside the index are dropped (Req 9.2). prompt_metadata: Maps each prompt_hash to a dict with "ticker" and "date" (ISO-8601) keys. Required because the harness Record schema does not carry date/ticker inline; the orchestrator builds this dict from the eval-set rows. cmmd_quantile: Quantile cut for the cmmd filter (default 0.80, i.e., drop top quintile by p_memorized). fees_one_way: One-way trading cost in fractional notional (default 0.00075 = 7.5 bps; round-trip = 15 bps per the paper). init_cash: Initial portfolio value passed to vectorbt (default 1.0 so equity curves start at 1.0). seed: Threaded through bootstrap CIs for determinism. bootstrap_n: Resamples for bootstrap_ci (default 1000).

Returns: A :class:BacktestResult with both variants populated.

Raises: ValueError: prices is empty or missing the BIL column. ValueError: cmmd_quantile outside (0, 1).

Source code in recall_guard/portfolio/backtest.py

def run_backtest(
    records: list[Any],
    prices: pd.DataFrame,
    prompt_metadata: dict[str, dict[str, str]],
    *,
    cmmd_quantile: float = 0.80,
    fees_one_way: float = 0.00075,
    init_cash: float = 1.0,
    seed: int = 0,
    bootstrap_n: int = 1000,
) -> BacktestResult:
    """Run the long-short backtest twice (raw + cmmd) on one price matrix.

    Args:
        records: List of record-shaped objects (see :class:`_RecordLike`).
            Records with ``parse_ok=False``, ``predicted_direction is
            None``, ``raw_confidence is None``, or no entry in
            ``prompt_metadata`` are dropped from BOTH variants
            (Req 9.1).
        prices: ``(date × ticker)`` close-price matrix. Must contain the
            ``BIL`` column plus at least one risk asset. The DataFrame's
            index is treated as the trading-day calendar; signals dated
            outside the index are dropped (Req 9.2).
        prompt_metadata: Maps each ``prompt_hash`` to a dict with
            ``"ticker"`` and ``"date"`` (ISO-8601) keys. Required because
            the harness ``Record`` schema does not carry date/ticker
            inline; the orchestrator builds this dict from the eval-set
            rows.
        cmmd_quantile: Quantile cut for the cmmd filter (default 0.80,
            i.e., drop top quintile by ``p_memorized``).
        fees_one_way: One-way trading cost in fractional notional
            (default 0.00075 = 7.5 bps; round-trip = 15 bps per the
            paper).
        init_cash: Initial portfolio value passed to vectorbt (default
            1.0 so equity curves start at 1.0).
        seed: Threaded through bootstrap CIs for determinism.
        bootstrap_n: Resamples for ``bootstrap_ci`` (default 1000).

    Returns:
        A :class:`BacktestResult` with both variants populated.

    Raises:
        ValueError: ``prices`` is empty or missing the BIL column.
        ValueError: ``cmmd_quantile`` outside ``(0, 1)``.
    """
    tradeable_records, cmmd_records, cmmd_threshold, warnings = _prepare_variants(
        records, prices, prompt_metadata, cmmd_quantile=cmmd_quantile
    )

    # -------- Stage 3: run both variants on the same price matrix --------
    raw_metrics, raw_returns, raw_equity = _run_one_variant(
        records=tradeable_records,
        prices=prices,
        prompt_metadata=prompt_metadata,
        label="raw_alpha",
        cmmd_threshold=None,
        fees_one_way=fees_one_way,
        init_cash=init_cash,
        seed=seed,
        bootstrap_n=bootstrap_n,
    )

    cmmd_metrics, cmmd_returns, cmmd_equity = _run_one_variant(
        records=cmmd_records,
        prices=prices,
        prompt_metadata=prompt_metadata,
        label="cmmd",
        cmmd_threshold=cmmd_threshold,
        fees_one_way=fees_one_way,
        init_cash=init_cash,
        seed=seed,
        bootstrap_n=bootstrap_n,
    )

    # -------- Stage 4: buy-and-hold benchmark + bundle --------
    return _assemble_result(
        raw_metrics=raw_metrics, raw_returns=raw_returns, raw_equity=raw_equity,
        cmmd_metrics=cmmd_metrics, cmmd_returns=cmmd_returns, cmmd_equity=cmmd_equity,
        prices=prices, warnings=warnings,
    )

write_backtest_artifacts

write_backtest_artifacts(result, run_dir)

Write the five backtest artifacts to run_dir atomically.

Builds every artifact in memory first, stages each payload to a temporary sibling file, and only after every temp file is safely on disk renames them over their targets (os.replace). If anything raises OSError (disk full, permission denied, etc.) before the publish phase, the temp files are unlinked and the pre-existing artifacts in run_dir — including those from an earlier run — are left byte-for-byte untouched (Req 7.6). The function re-raises as :class:BacktestArtifactError.

Args: result: The :class:BacktestResult to serialise. run_dir: Directory the artifacts should land in. Must exist.

Returns: {artifact_name: Path} for the five files written. Keys: backtest_summary_csv, backtest_summary_md, equity_curves_csv, equity_curves_png, daily_returns_csv, matching the manifest's backtest.artifacts block in design.md.

Raises: BacktestArtifactError: any IO failure during the write phase. The run directory is rolled back to its pre-call state.

Source code in recall_guard/portfolio/backtest.py

def write_backtest_artifacts(
    result: BacktestResult,
    run_dir: Path | str,
) -> dict[str, Path]:
    """Write the five backtest artifacts to ``run_dir`` atomically.

    Builds every artifact in memory first, stages each payload to a
    temporary sibling file, and only after every temp file is safely on
    disk renames them over their targets (``os.replace``). If anything
    raises ``OSError`` (disk full, permission denied, etc.) before the
    publish phase, the temp files are unlinked and the pre-existing
    artifacts in ``run_dir`` — including those from an earlier run —
    are left byte-for-byte untouched (Req 7.6). The function re-raises
    as :class:`BacktestArtifactError`.

    Args:
        result: The :class:`BacktestResult` to serialise.
        run_dir: Directory the artifacts should land in. Must exist.

    Returns:
        ``{artifact_name: Path}`` for the five files written. Keys:
        ``backtest_summary_csv``, ``backtest_summary_md``,
        ``equity_curves_csv``, ``equity_curves_png``,
        ``daily_returns_csv``, matching the manifest's
        ``backtest.artifacts`` block in ``design.md``.

    Raises:
        BacktestArtifactError: any IO failure during the write phase.
            The run directory is rolled back to its pre-call state.
    """
    run_path = Path(run_dir)

    # ----- in-memory build phase (any error here surfaces as a normal
    # exception; nothing has been written yet so there's no rollback).
    payloads: dict[str, tuple[Path, bytes, bool]] = {
        # key -> (path, bytes, is_text_for_logging)
        "backtest_summary_csv": (
            run_path / "backtest_summary.csv",
            _build_summary_csv(result).encode("utf-8"),
            True,
        ),
        "backtest_summary_md": (
            run_path / "backtest_summary.md",
            _build_summary_md(result).encode("utf-8"),
            True,
        ),
        "equity_curves_csv": (
            run_path / "equity_curves.csv",
            _build_equity_csv(result).encode("utf-8"),
            True,
        ),
        "equity_curves_png": (
            run_path / "equity_curves.png",
            _build_equity_png(result),
            False,
        ),
        "daily_returns_csv": (
            run_path / "daily_returns.csv",
            _build_returns_csv(result).encode("utf-8"),
            True,
        ),
    }

    # ----- atomic write phase -------------------------------------------------
    # Stage every payload to a temp sibling first; only after ALL temps are
    # on disk are they renamed over the real targets. A failure while
    # staging therefore never destroys artifacts from an earlier run.
    pending_tmp: list[Path] = []
    current: Path | None = None
    try:
        staged: list[tuple[Path, Path]] = []
        for _key, (path, blob, _is_text) in payloads.items():
            current = path
            tmp = path.with_name(f"{path.name}.tmp-{os.getpid()}")
            tmp.write_bytes(blob)
            pending_tmp.append(tmp)
            staged.append((tmp, path))
        for tmp, path in staged:
            current = path
            os.replace(tmp, path)
            pending_tmp.remove(tmp)
    except OSError as exc:
        # Roll back the temp files; pre-existing artifacts are untouched.
        for p in pending_tmp:
            try:
                p.unlink()
            except OSError:
                # Best-effort rollback; the original error is still the
                # one that matters.
                logger.warning("rollback unlink failed for %s", p)
        target = getattr(exc, "filename", None) or current or "unknown path"
        raise BacktestArtifactError(
            f"failed to write backtest artifact {target}: {exc}"
        ) from exc

    logger.info("write_backtest_artifacts done -> %s", run_path)
    return {key: payloads[key][0] for key in payloads}

recall_guard.portfolio.backtest