recall_guard.core.consensus
recall_guard.core.consensus
Pure statistics for reducing a set of repeated model draws.
Everything here is a deterministic function of its arguments: no I/O, no randomness, no global state, and no domain knowledge about what a draw means. That is what lets a stored draw set be replayed into a bit-identical result without re-querying the model.
The implementation is hand-rolled with the standard library alone; scipy is
intentionally avoided, matching :mod:recall_guard.core.bootstrap. Numpy
scalars and arrays are accepted -- every numeric argument is coerced with
float() before use, because repr(np.float64(0.15)) is
'np.float64(0.15)' under numpy 2 and would otherwise reach Decimal.
The normal quantile the Wilson interval needs comes from
:class:statistics.NormalDist, which agrees with the usual reference to within
5e-16 -- far inside anything that matters here.
Three choices are load-bearing and deliberately not the textbook ones, because the textbook ones were measured against real draws and failed:
- Location is never snapped to a lattice. Snapping degrades accuracy on every measured component and biases the best-estimated one systematically, because its estimate sits near a bin edge and always rounds the same way. Snapping is a reporting convention, not an estimator.
- Summation is exact. Pairwise summation makes a mean depend on array order in the last bit, and a last-bit difference changes a persisted artifact hash.
- An even-count median takes the lower order statistic, never the midpoint of two draws -- a midpoint is a value the model never emitted.
Tail
Bases: StrEnum
Which tail an interval's confidence level refers to.
This must be declared rather than assumed: the draw count needed to certify an agreement target differs substantially between the two conventions, so a feasibility check evaluated against the wrong one is meaningless.
Source code in recall_guard/core/consensus.py
48 49 50 51 52 53 54 55 56 57 | |
MultimodalVerdict
dataclass
Outcome of the separated-cluster check for one component.
separated is the gate: when it is true the component holds two clusters
with a genuine gap between them, there is no single location to estimate,
and a location estimator must not be run at all.
Source code in recall_guard/core/consensus.py
330 331 332 333 334 335 336 337 338 339 340 341 342 343 | |
wilson_interval
wilson_interval(
k,
n,
*,
confidence=0.95,
tail=Tail.TWO_SIDED,
continuity=False,
)
Score interval for a binomial proportion, or None when n is zero.
Inverting the score test rather than the Wald test keeps the interval inside
[0, 1] and, critically, non-degenerate at k == n. The Wald interval
collapses to zero width exactly there, which at high agreement is the
typical case -- it would report certainty from a couple of dozen draws.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Successes observed. Must satisfy |
required |
n
|
int
|
Draws observed. |
required |
continuity
|
bool
|
Apply the Newcombe continuity correction, widening the interval. |
False
|
Source code in recall_guard/core/consensus.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | |
smallest_certifiable_n
smallest_certifiable_n(
target,
*,
confidence=0.95,
tail=Tail.TWO_SIDED,
limit=100000,
)
Fewest unanimous draws whose interval's lower bound reaches target.
Unanimity is the best case, so this is a hard floor: below it no observed agreement can certify the target, and a configuration requesting fewer draws can never succeed no matter what the model returns. Surfacing it at construction turns a silently-unreachable setting into an error.
Source code in recall_guard/core/consensus.py
139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | |
smallest_detectable_split_n
smallest_detectable_split_n(
*,
cluster_positions,
min_cluster_draws=8,
min_cluster_density=1.5,
)
Fewest draws at which :func:detect_multimodal can flag a split.
The companion to :func:smallest_certifiable_n, and the reason both exist:
agreement precision and split detection need different sample sizes, and
sizing for the first silently under-sizes for the second. Agreement is where
the reported confidence lives; component splits are where a silently wrong
answer lives.
This is a necessary condition, not a sufficient one -- exactly as its companion is a floor under unanimity rather than a promise. Below the value returned here the density guard cannot be satisfied at all, so a split of that shape is undetectable no matter how clean the data. Above it, detection becomes possible; whether it fires still depends on sampling noise in the trough and in the cluster masses.
Measured on a corpus whose split is unambiguous at full size, detection still missed ~3.6% of 64-draw bootstrap resamples -- the closer analogue to a fresh ensemble, which draws independently rather than from a fixed pool. Reaching 99% detection on that corpus took ~128 draws.
The figure tracks parsed draws, not the configured count. The test
only ever sees replies that survived transport, parsing, and the caller's
projection. A prompt with a 5% failure rate configured at draws=64 is
really operating at n=61, where the same measurement gives 5.5% rather than
3.6% -- so size against n_parsed, not against draws.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cluster_positions
|
int
|
How many lattice positions the two clusters together occupy. Wider clusters need more draws to reach the same density. |
required |
Source code in recall_guard/core/consensus.py
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 | |
snap_to_grid
snap_to_grid(value, grid)
Round value onto a lattice of step grid, half away from zero.
Deliberately does not divide by grid. That division is inexact in
binary in a value-dependent way -- 0.85 / 0.1 is exactly 8.5 while
0.95 / 0.1 is 9.499999999999998 -- so the tie direction ends up
depending on the value rather than on the rule, and the obvious spellings
disagree with one another. Working in integer lattice units avoids it.
Source code in recall_guard/core/consensus.py
229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
grid_adherence
grid_adherence(values, grid, *, tolerance=1e-09)
Fraction of values that actually lie on the declared lattice.
Reported so a caller who declares a lattice the data does not follow finds out, instead of silently receiving mis-snapped results. A continuous quantity scores near zero here at any lattice.
A non-finite draw counts as off-lattice rather than raising: this reports on data quality, so it has to survive the bad data it exists to describe.
Source code in recall_guard/core/consensus.py
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | |
scale_floor
scale_floor(values, *, grid=None)
Robust scale estimate, floored at the lattice's resolution limit.
The floor exists because a concentrated lattice-valued component can drive the median absolute deviation to exactly zero, leaving the usual robust scale undefined. It is an identifiability floor, not an estimate of quantization noise: any true dispersion far below one lattice step produces observations on one or two lattice points and is indistinguishable from zero, so an estimate below that level carries no information.
Two caveats worth knowing before relying on it. It binds whenever
1.4826 * MAD < grid / sqrt(12) -- that is, for every MAD below
roughly 0.195 * grid, not only when the deviation is exactly zero. So on
a sharply concentrated component it can inflate a small but perfectly
well-defined estimate. And it only helps at all because the declared lattice
is coarser than the emitted one; declare the true lattice and the undefined
case returns.
Source code in recall_guard/core/consensus.py
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 | |
detect_multimodal
detect_multimodal(
values,
*,
grid,
mass_min=0.25,
trough_steps=3,
density_ratio=10.0,
min_draws=8,
min_cluster_density=1.5,
)
Detect two clusters separated by a sparse gap.
Returns None when the check could not run -- never a silent "found
nothing". There are three such cases, and a caller who needs to tell them
apart should read the lattice adherence alongside this:
- no lattice was declared (a continuous quantity has none);
- fewer than
min_drawsdraws, which cannot evidence two clusters; - the clusters are too thinly populated for a gap between them to mean anything (see below).
The rule is defined directly on the lattice rather than by a classical unimodality test. Those assume a continuous distribution, and on heavily tied lattice data they measure tie mass instead of modality -- badly enough that on the measured corpus the most sharply converged component scores as more multimodal than the genuinely split one.
The sparsity guard matters. trough_steps counts lattice steps, not
draws, so on a lattice much finer than the sampling, empty runs occur
everywhere by chance; the density test is vacuous there because an empty gap
has no peak to compare against. Measured, a unimodal normal at a 0.001
lattice flagged on every single subsample.
What separates the two regimes is not how wide the gap is but how dense the
clusters are: a real cluster stacks many draws onto few lattice positions,
while a spurious one is a scatter of singletons whose gaps are ordinary
spacing. So each side of the split must average at least
min_cluster_density draws per occupied position.
Every threshold is a parameter because all of them were tuned against a single measurement date. Detects separated clusters only: two overlapping modes with no gap between them are invisible to it, a known and accepted false-negative class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mass_min
|
float
|
Minimum share of draws each cluster must hold. |
0.25
|
trough_steps
|
int
|
Minimum width of the gap, in lattice steps. |
3
|
density_ratio
|
float
|
How much denser the taller cluster peak must be than the busiest bin inside the gap. Note this binds only when the gap is non-empty; at realistic draw counts most detections win on a completely empty gap. |
10.0
|
min_draws
|
int
|
Below this many draws the check does not run. |
8
|
min_cluster_density
|
float
|
Minimum draws per occupied lattice position within each cluster. |
1.5
|
Source code in recall_guard/core/consensus.py
366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 | |
lag_dependence
lag_dependence(labels, groups)
How much more alike draws are within a collection group than across them.
Returns None when there are fewer than two groups, or too few pairs to
compare -- never a misleading zero.
The statistic is the probability that two draws from the same group carry the same label, minus the probability for two draws from different groups. Zero means the grouping carries no information, which is what independence looks like; positive means draws collected together agree more than draws collected apart.
This exists because the reported agreement interval assumes independent draws, and that is precisely the assumption a serving stack violates -- batching, cache reuse, and node affinity all couple requests issued together. Positive dependence makes every interval narrower than its label, in the one direction that matters. Measuring it does not correct the interval; it makes the assumption falsifiable instead of merely disclaimed.
Depends only on the stored labels and group tags, never on arrival order, so it replays identically from a persisted draw set.
Source code in recall_guard/core/consensus.py
548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 | |
robust_location
robust_location(values, *, mode='median', trim=0.25)
Reduce values to one location, independent of their order.
mode is one of "mean", "median", or "trimmed".
Note what this cannot do: on a component whose draws form two separated clusters there is no single location to estimate, and no symmetric trim fraction escapes the gap between them -- trimming converges toward the median, not toward a mode. Callers must run the multimodality check first and skip this entirely for a flagged component.
Source code in recall_guard/core/consensus.py
597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 | |