Interactive AI testing / 03 / Budget the evidence

AI Evaluation & CI/CD Lab

Change the workload and coverage policy, then compare estimated monthly judge calls and costs in euros (EUR).

Browser-local learning exercise. All scenarios, costs and receipts are synthetic. No real model, API or service is being evaluated.

1. Tune your test policy

Move the sliders to explore a hypothetical team's workload. Judge prices, test speed and other assumptions are editable in code but do not represent any provider's live prices.

Commits / month How often fast CI runs Commits reaching PR (%) How many PR evaluations run Golden dataset cases Cases available per complete test run Full nightly runs / month Deep-judge runs over the entire dataset

Commit: fast-judge sample (%) Fast judge on the selected fraction per commit PR: fast-judge sample (%) Per-PR sample of the versioned test set Fail early, before judge (%) Assumed deterministic rejection rate (illustrative) Enable short-circuit (early exit)

Fixed illustrative assumptions: fast judge €0.002/case, deep judge €0.02/case, fast judge 3 seconds per case, 20 parallel slots and 4 seconds deterministic CI overhead. These are not benchmark results. Nightly testing stays full coverage in both strategies.

2. Inspect the modeled outcomes

Tiered policy / month€0Estimated judge spend

Difference vs full-judge baseline€00% reduction

Non-failing commit CI bound0sSimple batch model, not observed latency

Monthly costs across three pipeline phases

Stage	Tiered judge calls	Baseline judge calls	Per-run sample

3. Interpret before deciding

What this model supports

Compare explicit test frequencies, call counts, estimated costs and a simple non-failing commit duration bound as you change the policy.

What this model cannot tell you

Failure-detection rate, judge agreement, actual wall-clock latency, prompt variability and production reliability require real test runs and calibration. More sampled cases do not imply a fixed accuracy percentage.

Suggested practice: keep schema, permission and loop checks deterministic; use sampled model evaluations for frequent feedback; reserve broader, reviewable quality checks for PR and nightly jobs.