IEE 475: Lecture H (2026-10-27): Verification, Validation, and Calibration of Simulation Models

Overview

Ted Pavlic explains how simulation model quality depends on three linked activities: verification checks that the model was implemented correctly, validation compares simulated outputs with real-system data, and calibration iteratively improves the model. The lecture also reviews statistical tools for input modeling and validation, including power analysis, ROC curves, AIC, chi-square and KS goodness-of-fit tests, and a bank drive-through example using Arena replications.

Key takeaways

A high p-value from a small sample does not establish that an input distribution fits; limited data may simply give the goodness-of-fit test too little power.
Power equals 1 minus the type II error rate and increases with more replications or lower outcome variability, so power analysis helps determine the sample size needed to detect a meaningful effect.
For chi-square goodness-of-fit tests, subtract degrees of freedom for parameters estimated from the data, but not for parameters specified in advance as part of the hypothesis.
Use the KS test when small samples make chi-square expected-count requirements impractical; use Anderson-Darling when accurate tail behavior matters and Shapiro-Wilk when normality is the assumption under test.
Verification checks whether a model was implemented as intended, validation compares its outputs with real-system measurements, and calibration changes the model when those comparisons expose a mismatch.
In the bank example, six two-hour Arena replications produce average-delay estimates that fall below the real-system benchmark of 4.3 minutes, prompting a validation test and further model investigation.

Chapters

0:00 Input Modeling Report and Final Project Scope

Teams must submit a two-page input modeling report identifying their chosen system, anticipated inputs and outputs, and research questions.
Students can gather live data, observe systems remotely through webcams, or use public datasets; the project should fit discrete-event simulation.
Research questions based on relative comparisons, such as whether a structural change improves performance, are more defensible than precise predictions of real-world effect size.
Lab 9 covers output analysis, Lab 10 is the final lab, and the group report is due Sunday.

8:20 Why P-Values Depend on Sample Size

A low p-value is more concerning when a test has many observations, because large samples can detect very small departures from a hypothesized distribution.
A high p-value from a small sample is weak evidence of fit; the test may simply lack enough data to detect a mismatch.
For input models, small datasets may require comparing AIC, QQ plots, and PP plots rather than relying on goodness-of-fit p-values alone.
Ted Pavlic connects test sensitivity to statistical power, the probability of detecting a false null hypothesis.

18:34 Type I Error, Type II Error, and Statistical Power

Type I error, alpha, is the probability of rejecting a true null hypothesis; type II error, beta, is the probability of failing to detect a false null.
Statistical power is 1 minus beta, also called sensitivity or the true positive rate; alpha is the false positive rate.
Power increases with more samples and lower outcome variability, which makes simulation especially useful for designing controlled experiments.
A power analysis estimates how many replications are needed to detect a meaningful effect, such as a difference greater than one second.

25:19 ROC Curves Show the Alpha-Power Tradeoff

An ROC curve plots alpha against power, showing how changing a test's rejection threshold trades false positives against true positives.
With more samples, a test's ROC curve moves toward the ideal upper-left corner; with very few samples, it approaches the diagonal representing random performance.
Classification systems, including self-driving-car object detectors, use thresholds to balance missed detections against false alarms.
In machine learning, area under the ROC curve summarizes classifier performance across thresholds; this lecture focuses on selecting a practical alpha.

32:35 Maximum Likelihood, AIC, and Choosing Input Distributions

Maximum likelihood estimation chooses distribution parameters by holding observed data fixed and finding the parameters that make those data most likely.
For an exponential model, the estimated rate is the reciprocal of the sample mean interarrival time.
AIC is generally preferred in this class because Arena input models are treated as approximations; BIC applies a stronger penalty when the candidate family is assumed to contain the true process.
When candidate models have AIC values within two of the minimum, compare QQ plots and prefer a familiar, explainable distribution unless a more flexible model fits substantially better.

38:34 Chi-Square Goodness-of-Fit Binning Rules

The chi-square test divides the outcome range into bins, counts observations in each bin, and compares observed counts with expected counts.
For a uniform distribution, expected counts are equal across bins; for other distributions, calculate each bin's probability and multiply by the total sample size.
Any bin with an expected count below five must be combined with neighboring bins, while retaining as many bins as possible to preserve test power.
For unbounded distributions such as the exponential, edge bins must include observations beyond the chosen displayed range rather than discarding them.

42:05 Degrees of Freedom and Expected Counts from a CDF

For a chi-square test with n bins, begin with n minus one degrees of freedom and subtract one additional degree for each parameter estimated from the data.
Do not subtract degrees of freedom for a parameter specified in the hypothesis rather than estimated from observations.
For a continuous distribution, a bin's probability is the CDF at its upper boundary minus the CDF at its lower boundary.
Open versus closed bin endpoints do not change probabilities for continuous distributions because any single point has probability zero.

48:15 Exponential Chi-Square Example with Ten Bins

For ten bins testing an exponential distribution with a prespecified mean of two years, the degrees of freedom are nine.
If the exponential mean had instead been estimated from the data, the test would use eight degrees of freedom.
For the interval from two to three years, the expected probability is the exponential CDF at three minus the CDF at two, approximately 14.5 percent.
Multiply each bin probability by the total observations to obtain expected counts, then compare those counts with observed counts using the chi-square statistic.

52:18 Using the KS Test for Small Samples

The one-sample Kolmogorov-Smirnov test measures the maximum distance between the empirical CDF and a hypothesized distribution's CDF.
To test a general distribution, sort the observations, evaluate the hypothesized CDF at each observation, and compare those values with the empirical CDF.
The KS test is useful when samples are too small for chi-square bins to meet the expected-count threshold, but standard KS use generally assumes parameters were not estimated from the data.
When parameters have been estimated, input analyzer software may use bootstrapping to produce appropriate p-values.

56:29 Anderson-Darling, Shapiro-Wilk, and Diagnostic Plots

The Anderson-Darling test gives greater emphasis to distribution tails, while the KS test is more sensitive to differences near the middle.
The Shapiro-Wilk test specifically assesses normality, an assumption relevant to t-tests, ANOVA, and linear models.
Input analyzer tools can provide QQ and PP plots; agreement with the fitted distribution strengthens the case for a selected input model.
Choose a goodness-of-fit test based on the feature that matters: tails, central behavior, or normality.

1:00:31 Verification, Validation, and Calibration Defined

Verification asks whether the conceptual model was implemented correctly, including whether an Arena arrival rate uses the intended time units.
Validation asks whether simulated outputs resemble real-system outputs when the model uses appropriately verified inputs.
Calibration is the iterative process of changing model structure or assumptions in response to validation results.
The combined terms V&V and T&V&V describe verification, validation, and the testing processes used to support model improvement.

1:04:15 Bank Drive-Through Data for Model Validation

A bank drive-through study can collect real arrival times and teller service times as inputs, then measure waiting time and other performance outputs.
Validation compares real-system measurements, labeled Z, with simulated responses, labeled Y, for the same variables and time period.
Matching observation periods matters because bank performance can differ between morning and afternoon conditions.
Input and decision variables define the model and research question, while output variables such as mean waiting time support validation.

1:07:33 Six Arena Replications and the Next Validation Step

The example runs six Arena replications, each covering two simulated hours, and records arrivals per hour, teller service times, and average delay.
Arrival rates and service times are checked for verification, while average delay is compared with a real-system reference mean of 4.3 minutes.
The simulated delays are visibly below 4.3 minutes, suggesting a t-test would reject the hypothesis that the simulated and real means match.
The lecture leaves the choice of replication count and the response to a failed validation test for the next class.

Transcript excerpt

0:06 All right. So, um, flipping on to, you know, we're done input modeling. So, starting thinking about, uh, more of the statistical side of it. So, before we get to thinking about the outputs, uh, and analyzing the outputs, we still have to now uh, see if we've built a decent model. So, that's kind of today will be verification for how do we implement those? Did we implement the input models correctly? validation, how well does our model look relative to the real system and calibration, how do we make it better? So that's uh what we're going to be doing today. The schedule kind of looking forward um lab schedule this week all the teams are formed. So um that's great. So now work with your team to submit this one to twopage document which tells me which system that you're

0:54 you've picked uh which are the you know input models you're anticipating you're going to need to to use for that system. What are some of the outputs you're anticipating to have to measure from that system? And then what are some research questions that you think will be valid? Like why did you pick that system? What's the kind of the the thing that you're studying about that system? And then from there, show me you have access to that system. So, gather some data. Uh, and like I said, you this could be data that's gathered live. Um, so if you pick the uh the Starbucks at the MU, you know, you could go and gather some inter arrival data or something like that. Show me that you've got access to the system or uh you I've mentioned there are remote observation options, you know, webcams and things like that looking at intersections. There's also a bunch of public data sets

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ted Pavlic.