Skip to content

recall_guard.mia

recall_guard.mia

Public API for the mia layer of the honest-model-ranking harness.

Re-exports the consumer-facing surface (Req 12.1) so that callers (the qualification notebook, future external scripts, and the harness layers themselves) can import the MIA feature primitives, the per-model control baseline, and the per-model MCS calibrator from the package root::

from recall_guard.mia import (
    MiaFeatures, compute_mia_features, LOGPROB_FLOOR,
    ControlBaseline, build_baseline, standardise,
    MCSCalibrator, train_mcs,
)

train_mcs is the documented public name for the calibrator's training function (Req 12.1, Task 5.5 brief). The original function is defined as :func:recall_guard.mia.mcs.train; train_mcs is re-exported as an alias here so notebook code reads as "train an MCS calibrator" rather than the more ambiguous bare train. Both names point at the same callable.

LOGPROB_FLOOR module-attribute 

LOGPROB_FLOOR = -30.0

Lower bound for individual logprob values, applied before averaging.

Prevents a single -inf (or extremely negative) per-token logprob from poisoning loss / min_k / zlib_ratio / ref_delta.

ControlBaseline dataclass

Per-model baseline distribution of every MIA feature on the OOS control corpus.

Attributes:

Name Type Description
model str

The NVIDIA model ID this baseline was built for.
n_valid int

Number of control rows where model_lm.generate returned usable logprobs (i.e., did not raise TimeoutError or RuntimeError).
feature_means dict[str, float | None]

Per-feature mean across the valid rows. Keys are the five MIA feature names. feature_means["ref_delta"] is None when no reference model is configured (or every reference call failed).
feature_stds dict[str, float | None]

Per-feature standard deviation across the valid rows, floored at _STD_FLOOR. feature_stds["ref_delta"] is None whenever feature_means["ref_delta"] is None.
is_calibrated bool

True iff n_valid >= min_valid. Used by the runner to decide whether to evaluate the model or surface an uncalibrated warning.
min_valid int

The threshold used (default 50, per the Open Defaults in requirements.md).
Source code in recall_guard/mia/control.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
@dataclass(frozen=True)
class ControlBaseline:
    """Per-model baseline distribution of every MIA feature on the OOS control corpus.

    Attributes
    ----------
    model:
        The NVIDIA model ID this baseline was built for.
    n_valid:
        Number of control rows where ``model_lm.generate`` returned usable
        logprobs (i.e., did not raise ``TimeoutError`` or ``RuntimeError``).
    feature_means:
        Per-feature mean across the valid rows. Keys are the five MIA feature
        names. ``feature_means["ref_delta"]`` is ``None`` when no reference
        model is configured (or every reference call failed).
    feature_stds:
        Per-feature standard deviation across the valid rows, floored at
        ``_STD_FLOOR``. ``feature_stds["ref_delta"]`` is ``None`` whenever
        ``feature_means["ref_delta"]`` is ``None``.
    is_calibrated:
        ``True`` iff ``n_valid >= min_valid``. Used by the runner to decide
        whether to evaluate the model or surface an ``uncalibrated`` warning.
    min_valid:
        The threshold used (default 50, per the Open Defaults in
        requirements.md).
    """

    model: str
    n_valid: int
    feature_means: dict[str, float | None]
    feature_stds: dict[str, float | None]
    is_calibrated: bool
    min_valid: int

MiaFeatures dataclass

Five MIA features for one (model, prompt, response) record.

Attributes:

Name Type Description
loss float

Mean negative logprob of the realised tokens (clipped at floor). Low loss means the model found the text easy to predict, which is what stored text looks like.
min_k float

Mean of the bottom int(len * k) clipped logprobs (Min-K%). Negative; lower means more "memorized". Looks only at the hardest tokens, because that is where memorization shows first: if the model breezes through even those, it has probably seen the text.
min_k_pp float

Mean of the bottom-K per-position z-scores (Min-K%++). Same idea as min_k, but each token is graded against its own candidate distribution instead of an absolute scale.
zlib_ratio float

-sum(clipped_logprobs) / len(zlib.compress(response, 9)). 0.0 when response is empty. Dividing by the compressed size cancels plain repetitiveness; a repetitive text is cheap to predict AND cheap to compress, so what remains is the confidence the model has beyond what the text's redundancy explains.
ref_delta float | None

loss_self - loss_ref; None when ref_logprobs is None. The reference model anchors what "normal" confidence looks like for the same text, so shared easiness cancels and model-specific recall remains.
Source code in recall_guard/mia/features.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
@dataclass(frozen=True)
class MiaFeatures:
    """Five MIA features for one (model, prompt, response) record.

    Attributes
    ----------
    loss:
        Mean negative logprob of the realised tokens (clipped at floor).
        Low loss means the model found the text easy to predict, which is
        what stored text looks like.
    min_k:
        Mean of the bottom ``int(len * k)`` clipped logprobs (Min-K%).
        Negative; lower means more "memorized". Looks only at the hardest
        tokens, because that is where memorization shows first: if the
        model breezes through even those, it has probably seen the text.
    min_k_pp:
        Mean of the bottom-K per-position z-scores (Min-K%++). Same idea as
        ``min_k``, but each token is graded against its own candidate
        distribution instead of an absolute scale.
    zlib_ratio:
        ``-sum(clipped_logprobs) / len(zlib.compress(response, 9))``.
        ``0.0`` when ``response`` is empty. Dividing by the compressed size
        cancels plain repetitiveness; a repetitive text is cheap to predict
        AND cheap to compress, so what remains is the confidence the model
        has beyond what the text's redundancy explains.
    ref_delta:
        ``loss_self - loss_ref``; ``None`` when ``ref_logprobs is None``.
        The reference model anchors what "normal" confidence looks like
        for the same text, so shared easiness cancels and model-specific
        recall remains.
    """

    loss: float
    min_k: float
    min_k_pp: float
    zlib_ratio: float
    ref_delta: float | None

MCSCalibrator dataclass

Per-model logistic-regression calibrator for p(memorized | features).

Attributes:

Name Type Description
model str

The NVIDIA model ID this calibrator was trained for.
classifier LogisticRegression

The fitted sklearn.linear_model.LogisticRegression instance. sklearn estimators are mutable; frozen=True only prevents reassignment of the field reference, which is the design intent.
feature_order list[str]

Canonical order used to flatten the standardised feature dict into the classifier's input vector. Populated at train time and consumed verbatim by :meth:predict_proba so the classifier is never fed a permuted row.
holdout_auc float

ROC-AUC score of the trained classifier on the 25% held-out portion of the labelled IS/OOS corpus. Reported in the manifest and the per-model evaluation result (Req 5.2).
is_weak bool

True iff holdout_auc < min_auc at train time. Surfaced as the weak-calibration warning in top3.md (Req 5.3).
Source code in recall_guard/mia/mcs.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
@dataclass(frozen=True)
class MCSCalibrator:
    """Per-model logistic-regression calibrator for ``p(memorized | features)``.

    Attributes
    ----------
    model:
        The NVIDIA model ID this calibrator was trained for.
    classifier:
        The fitted ``sklearn.linear_model.LogisticRegression`` instance.
        sklearn estimators are mutable; ``frozen=True`` only prevents
        reassignment of the field reference, which is the design intent.
    feature_order:
        Canonical order used to flatten the standardised feature dict
        into the classifier's input vector. Populated at train time and
        consumed verbatim by :meth:`predict_proba` so the classifier is
        never fed a permuted row.
    holdout_auc:
        ROC-AUC score of the trained classifier on the 25% held-out
        portion of the labelled IS/OOS corpus. Reported in the manifest
        and the per-model evaluation result (Req 5.2).
    is_weak:
        ``True`` iff ``holdout_auc < min_auc`` at train time. Surfaced
        as the ``weak-calibration`` warning in ``top3.md`` (Req 5.3).
    """

    model: str
    classifier: LogisticRegression
    feature_order: list[str]
    holdout_auc: float
    is_weak: bool

    def predict_proba(
        self, features: MiaFeatures, baseline: ControlBaseline
    ) -> float:
        """Return the calibrated probability of "memorized" for one record.

        Standardises ``features`` against the model's ``baseline`` and
        feeds the resulting vector to the trained classifier in
        ``self.feature_order``.

        Returns
        -------
        float
            ``p(memorized | features) ∈ [0.0, 1.0]``.

        Raises
        ------
        ValueError
            If any of the four core features standardises to ``None``
            (uncalibrated baseline). A missing ``ref_delta`` does NOT
            raise: the reference feature is optional by contract, so it
            is imputed at the control-baseline mean (standardised 0.0),
            which contributes no memorization evidence either way.
        """
        standardised = standardise(features, baseline)
        if "ref_delta" in self.feature_order and standardised.get("ref_delta") is None:
            standardised = {**standardised, "ref_delta": 0.0}
        row = _row_vector(standardised, self.feature_order)
        # Estimator was trained on a 2-D matrix; predict on a 1-row matrix.
        proba = float(self.classifier.predict_proba(row.reshape(1, -1))[0, 1])
        # sklearn returns values strictly in [0, 1] for LR; clamp defensively
        # against fp64 round-off so the float postcondition holds exactly.
        if proba < 0.0:
            proba = 0.0
        elif proba > 1.0:
            proba = 1.0
        return proba

predict_proba 

predict_proba(features, baseline)

Return the calibrated probability of "memorized" for one record.

Standardises features against the model's baseline and feeds the resulting vector to the trained classifier in self.feature_order.

Returns:

Type Description
float

p(memorized | features) ∈ [0.0, 1.0].

Raises:

Type Description
ValueError

If any of the four core features standardises to None (uncalibrated baseline). A missing ref_delta does NOT raise: the reference feature is optional by contract, so it is imputed at the control-baseline mean (standardised 0.0), which contributes no memorization evidence either way.
Source code in recall_guard/mia/mcs.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
def predict_proba(
    self, features: MiaFeatures, baseline: ControlBaseline
) -> float:
    """Return the calibrated probability of "memorized" for one record.

    Standardises ``features`` against the model's ``baseline`` and
    feeds the resulting vector to the trained classifier in
    ``self.feature_order``.

    Returns
    -------
    float
        ``p(memorized | features) ∈ [0.0, 1.0]``.

    Raises
    ------
    ValueError
        If any of the four core features standardises to ``None``
        (uncalibrated baseline). A missing ``ref_delta`` does NOT
        raise: the reference feature is optional by contract, so it
        is imputed at the control-baseline mean (standardised 0.0),
        which contributes no memorization evidence either way.
    """
    standardised = standardise(features, baseline)
    if "ref_delta" in self.feature_order and standardised.get("ref_delta") is None:
        standardised = {**standardised, "ref_delta": 0.0}
    row = _row_vector(standardised, self.feature_order)
    # Estimator was trained on a 2-D matrix; predict on a 1-row matrix.
    proba = float(self.classifier.predict_proba(row.reshape(1, -1))[0, 1])
    # sklearn returns values strictly in [0, 1] for LR; clamp defensively
    # against fp64 round-off so the float postcondition holds exactly.
    if proba < 0.0:
        proba = 0.0
    elif proba > 1.0:
        proba = 1.0
    return proba

build_baseline 

build_baseline(
    model_lm,
    control_rows,
    ref_lm,
    min_valid=50,
    max_workers=1,
)

Build a per-model control-corpus baseline.

For each row in control_rows:

  • Call model_lm.generate(row.prompt). On TimeoutError or RuntimeError (e.g., missing logprobs) the row is dropped and a WARNING is logged with the row index.
  • When ref_lm is provided, also call ref_lm.generate(row.prompt). A reference-side failure does not invalidate the row; it merely sets ref_logprobs = None for that row, so the four other features still contribute to the baseline.
  • Compute :class:MiaFeatures via :func:compute_mia_features.

Per-feature mean and std are aggregated with numpy.mean and numpy.std(ddof=0). Std is floored at _STD_FLOOR to avoid div-by-zero in :func:standardise. When every valid row has ref_delta = None (because ref_lm is None or every reference call failed), the ref_delta mean and std are stored as None.

is_calibrated is set to n_valid >= min_valid.

Source code in recall_guard/mia/control.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
def build_baseline(
    model_lm: NvidiaLM,
    control_rows: list[EvalRow],
    ref_lm: NvidiaLM | None,
    min_valid: int = 50,
    max_workers: int = 1,
) -> ControlBaseline:
    """Build a per-model control-corpus baseline.

    For each row in ``control_rows``:

    - Call ``model_lm.generate(row.prompt)``. On ``TimeoutError`` or
      ``RuntimeError`` (e.g., missing logprobs) the row is dropped and a
      WARNING is logged with the row index.
    - When ``ref_lm`` is provided, also call ``ref_lm.generate(row.prompt)``.
      A reference-side failure does not invalidate the row; it merely
      sets ``ref_logprobs = None`` for that row, so the four other features
      still contribute to the baseline.
    - Compute :class:`MiaFeatures` via :func:`compute_mia_features`.

    Per-feature mean and std are aggregated with ``numpy.mean`` and
    ``numpy.std(ddof=0)``. Std is floored at ``_STD_FLOOR`` to avoid div-by-zero
    in :func:`standardise`. When every valid row has ``ref_delta = None``
    (because ``ref_lm is None`` or every reference call failed), the
    ``ref_delta`` mean and std are stored as ``None``.

    ``is_calibrated`` is set to ``n_valid >= min_valid``.
    """
    # Fan out the model + ref calls in parallel (max_workers=1 keeps the
    # original sequential ordering for tests that mock requests.post).
    prompts = [row.prompt for row in control_rows]
    primary_results = generate_many(model_lm, prompts, max_workers=max_workers)
    ref_results: list = (
        generate_many(ref_lm, prompts, max_workers=max_workers)
        if ref_lm is not None else [None] * len(prompts)
    )

    per_row_features: list[MiaFeatures] = []
    for idx, (primary, ref_res) in enumerate(zip(primary_results, ref_results, strict=True)):
        if isinstance(primary, Exception) or primary is None:
            logger.warning(
                "control baseline: skipping row %d for model %s (logprobs missing or timeout)",
                idx,
                model_lm.model,
            )
            continue
        content, logprobs = primary.content, primary.logprobs

        ref_logprobs: list[TokenLogprob] | None = None
        if ref_lm is not None:
            if isinstance(ref_res, Exception) or ref_res is None:
                logger.warning(
                    "control baseline: ref-model %s failed on row %d; "
                    "ref_delta dropped for this row",
                    ref_lm.model,
                    idx,
                )
            else:
                ref_logprobs = ref_res.logprobs

        try:
            features = compute_mia_features(content, logprobs, ref_logprobs)
        except ValueError:
            logger.warning(
                "control baseline: skipping row %d for model %s "
                "(MIA feature computation failed)",
                idx,
                model_lm.model,
            )
            continue
        per_row_features.append(features)

    n_valid = len(per_row_features)

    feature_means: dict[str, float | None] = {}
    feature_stds: dict[str, float | None] = {}

    if n_valid > 0:
        for key in ("loss", "min_k", "min_k_pp", "zlib_ratio"):
            mean, std = _aggregate_mean_std(getattr(f, key) for f in per_row_features)
            feature_means[key] = mean
            feature_stds[key] = std

        ref_values = [
            f.ref_delta for f in per_row_features if f.ref_delta is not None
        ]
        if ref_values:
            mean, std = _aggregate_mean_std(ref_values)
            feature_means["ref_delta"] = mean
            feature_stds["ref_delta"] = std
        else:
            feature_means["ref_delta"] = None
            feature_stds["ref_delta"] = None
    else:
        for key in _FEATURE_KEYS:
            feature_means[key] = None
            feature_stds[key] = None

    return ControlBaseline(
        model=model_lm.model,
        n_valid=n_valid,
        feature_means=feature_means,
        feature_stds=feature_stds,
        is_calibrated=(n_valid >= min_valid),
        min_valid=min_valid,
    )

standardise 

standardise(features, baseline)

Standardise eval-time MIA features against the model's control baseline.

For each of the four always-present features loss, min_k, min_k_pp, zlib_ratio returns (value - mean) / max(std, _STD_FLOOR).

For ref_delta returns None whenever either the baseline or the eval-time features have no reference value to standardise, i.e., the field stays "off" rather than being silently coerced to 0.0.

Source code in recall_guard/mia/control.py 
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
def standardise(
    features: MiaFeatures, baseline: ControlBaseline
) -> dict[str, float | None]:
    """Standardise eval-time MIA features against the model's control baseline.

    For each of the four always-present features ``loss``, ``min_k``,
    ``min_k_pp``, ``zlib_ratio`` returns ``(value - mean) / max(std, _STD_FLOOR)``.

    For ``ref_delta`` returns ``None`` whenever either the baseline or the
    eval-time features have no reference value to standardise, i.e., the
    field stays "off" rather than being silently coerced to ``0.0``.
    """
    out: dict[str, float | None] = {}
    for key in ("loss", "min_k", "min_k_pp", "zlib_ratio"):
        mean = baseline.feature_means[key]
        std = baseline.feature_stds[key]
        # The four core features are always populated when n_valid > 0; if a
        # caller hands us an uncalibrated baseline, fall through to None.
        if mean is None or std is None:
            out[key] = None
            continue
        divisor = std if std >= _STD_FLOOR else _STD_FLOOR
        out[key] = (float(getattr(features, key)) - float(mean)) / divisor

    ref_mean = baseline.feature_means.get("ref_delta")
    ref_std = baseline.feature_stds.get("ref_delta")
    if features.ref_delta is None or ref_mean is None or ref_std is None:
        out["ref_delta"] = None
    else:
        divisor = ref_std if ref_std >= _STD_FLOOR else _STD_FLOOR
        out["ref_delta"] = (float(features.ref_delta) - float(ref_mean)) / divisor

    return out

compute_mia_features 

compute_mia_features(
    response, logprobs, ref_logprobs, k=0.2
)

Compute the five MIA features for one record.

Parameters:

Name Type Description Default
response str

The model's emitted text. Used only for the zlib-ratio denominator.

 required
logprobs list[TokenLogprob]

Per-token logprob entries from core.nvidia_lm.NvidiaLM.generate. Must be non-empty, and each entry must carry a non-empty top_logprobs list (precondition from design).

 required
ref_logprobs list[TokenLogprob] | None

Per-token logprobs from a reference model on the same prompt; or None to disable the reference-delta feature.

 required
k float

Fraction of tokens used for the bottom-K slice in Min-K% and Min-K%++. Defaults to 0.2 (the paper's setting).

 0.2

Returns:

Type Description
MiaFeatures

Frozen dataclass with all five features.

Raises:

Type Description
ValueError

If logprobs is empty, or any entry has an empty/missing top_logprobs list.
Source code in recall_guard/mia/features.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
def compute_mia_features(
    response: str,
    logprobs: list[TokenLogprob],
    ref_logprobs: list[TokenLogprob] | None,
    k: float = 0.2,
) -> MiaFeatures:
    """Compute the five MIA features for one record.

    Parameters
    ----------
    response:
        The model's emitted text. Used only for the zlib-ratio denominator.
    logprobs:
        Per-token logprob entries from ``core.nvidia_lm.NvidiaLM.generate``.
        Must be non-empty, and each entry must carry a non-empty
        ``top_logprobs`` list (precondition from design).
    ref_logprobs:
        Per-token logprobs from a reference model on the same prompt; or
        ``None`` to disable the reference-delta feature.
    k:
        Fraction of tokens used for the bottom-K slice in Min-K% and
        Min-K%++. Defaults to 0.2 (the paper's setting).

    Returns
    -------
    MiaFeatures
        Frozen dataclass with all five features.

    Raises
    ------
    ValueError
        If ``logprobs`` is empty, or any entry has an empty/missing
        ``top_logprobs`` list.
    """
    if not logprobs:
        raise ValueError("logprobs is empty")

    clipped = _clipped_array(logprobs)
    loss_self = _loss(clipped)

    # Min-K%: bottom-K clipped logprobs
    bottom_n = _bottom_k_count(len(clipped), k)
    min_k = float(np.mean(np.sort(clipped)[:bottom_n]))

    # Min-K%++: per-position z-scores
    min_k_pp = _min_k_pp(logprobs, clipped, k)

    zlib_ratio = _zlib_ratio(response, clipped)

    if ref_logprobs is None:
        ref_delta: float | None = None
    else:
        if not ref_logprobs:
            raise ValueError("ref_logprobs is empty")
        ref_clipped = _clipped_array(ref_logprobs)
        ref_delta = loss_self - _loss(ref_clipped)

    return MiaFeatures(
        loss=loss_self,
        min_k=min_k,
        min_k_pp=min_k_pp,
        zlib_ratio=zlib_ratio,
        ref_delta=ref_delta,
    )

train_mcs 

train_mcs(
    model_lm,
    is_memorized,
    oos_control,
    baseline,
    ref_lm,
    min_auc=0.6,
    seed=0,
    max_workers=1,
)

Train the MCS classifier for one model.

Drives the LM over both labelled corpora (in parallel when max_workers > 1), fits a logistic regression on the standardised features, and reports a held-out AUC. Raises ValueError if either class ends up empty after per-row skips.

Source code in recall_guard/mia/mcs.py 
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
def train(
    model_lm: NvidiaLM,
    is_memorized: list[EvalRow],
    oos_control: list[EvalRow],
    baseline: ControlBaseline,
    ref_lm: NvidiaLM | None,
    min_auc: float = 0.6,
    seed: int = 0,
    max_workers: int = 1,
) -> MCSCalibrator:
    """Train the MCS classifier for one model.

    Drives the LM over both labelled corpora (in parallel when
    ``max_workers > 1``), fits a logistic regression on the standardised
    features, and reports a held-out AUC. Raises ``ValueError`` if either
    class ends up empty after per-row skips.
    """
    feature_order = _resolve_feature_order(baseline)
    x, y, n_valid_is, n_valid_oos = _gather_train_xy(
        model_lm=model_lm,
        is_memorized=is_memorized,
        oos_control=oos_control,
        baseline=baseline,
        ref_lm=ref_lm,
        feature_order=feature_order,
        max_workers=max_workers,
    )

    # The stratified holdout needs at least one row per class in BOTH the
    # train and holdout halves. Check up front so tiny corpora fail with a
    # clear message instead of an opaque sklearn split error.
    n_total = n_valid_is + n_valid_oos
    n_holdout = math.ceil(_HOLDOUT_FRACTION * n_total)
    if n_holdout < 2 or (n_total - n_holdout) < 2:
        raise ValueError(
            f"mcs.train: {n_total} valid rows "
            f"(n_valid_is={n_valid_is}, n_valid_oos={n_valid_oos}) cannot "
            f"support the stratified {_HOLDOUT_FRACTION:.0%} holdout split "
            f"(holdout would hold {n_holdout} row(s), need >= 2 with both "
            "classes). Provide more calibration rows."
        )

    x_train, x_holdout, y_train, y_holdout = train_test_split(
        x, y,
        test_size=_HOLDOUT_FRACTION,
        random_state=seed,
        stratify=y,
    )

    classifier = LogisticRegression(
        class_weight="balanced",
        solver="liblinear",
        random_state=seed,
    )
    classifier.fit(x_train, y_train)

    holdout_scores = classifier.predict_proba(x_holdout)[:, 1]
    holdout_auc = float(roc_auc_score(y_holdout, holdout_scores))
    is_weak = holdout_auc < float(min_auc)

    logger.info(
        "mcs.train: model=%s n_valid_is=%d n_valid_oos=%d "
        "holdout_auc=%.4f is_weak=%s",
        model_lm.model, n_valid_is, n_valid_oos, holdout_auc, is_weak,
    )

    return MCSCalibrator(
        model=model_lm.model,
        classifier=classifier,
        feature_order=feature_order,
        holdout_auc=holdout_auc,
        is_weak=is_weak,
    )