LLM regression testing checks whether a change damages behavior that the team previously accepted. It does not prove that the entire application is correct, and it is different from asking an LLM to generate software tests. A useful release gate connects particular failures to an owner and a clear next action.
Define the change being compared
Record the prompt, model, retrieval configuration, tool behavior and evaluation criteria relevant to the comparison. If several change together, the result may reveal a regression without identifying its cause. Use a bounded comparison when the team needs to diagnose which change matters.
Choose checks before observing the result
State which failures block a release and which require review. A deterministic format requirement may have an exact check, while a source-support judgment may need a rubric and reviewed examples. Do not invent a universal passing percentage or adjust the acceptance threshold after seeing an inconvenient result.
Preserve important slices
Keep cases for supported tasks, known prior failures and required escalation. Inspect critical slices separately from the overall average so frequent easy tasks do not hide a serious regression. A regression suite records known expectations; it is not an independent held-out test of every new behavior. Maintain that distinction when discussing coverage.
Account for output variability
The same request can produce different outputs. Decide whether repeat runs or another uncertainty treatment are needed for the release decision, and record the chosen settings. A single failed run may reveal a real failure case even when an average looks stable. A single successful rerun does not erase the earlier observation.
Evaluate the judge and the application separately
If a model judge scores the result, keep its prompt, model and rubric version with the run. Inspect its decisions against reviewed examples. A changed judge can move a score even when the application is unchanged, so do not attribute every score difference to the proposed release.