Statistics, baselines, and experiments

Performance Lab preserves raw repetition evidence and computes distributions without converting missing evidence into zeroes. Every metric reports raw, included, and excluded sample counts plus minimum, median, nearest-rank p90, p95, p99, and maximum where supported.

Outliers

None is the default and keeps every valid sample. TukeyFence is an explicit reviewed policy. Changing outlier policy changes the statistical evidence and must not be treated as the same baseline.

Baselines

Baselines are immutable reviewed revisions. Acceptance, rejection, migration, selection, and rollback are recorded in append-only lifecycle history. Rollback selects an earlier authenticated revision; it does not rewrite or delete newer history.

Environment matching is enabled by default. Missing fingerprints, scenario mismatch, unit mismatch, capability failures, outlier-policy mismatch, or invalid baseline identities produce an invalid decision rather than a pass.

Budgets

Absolute budgets compare a statistic directly with a limit. Relative budgets require a valid baseline and compare percentage increase. Use percentiles for frame-time tail regressions and maximum only when the maximum itself is the intended contract.

A/B

A/B requires explicit A and B source identities from comparable evidence. Keep scene, build type, target, environment, sample policy, and repetition policy stable unless the changed factor is the experiment itself.

A-B-B-A

A-B-B-A is supported for contiguous A, B, B, A run order. The ordering contract helps expose drift; invalid or ambiguous source ordering is rejected. Do not reorder captures after the fact to manufacture the pattern.