IEE 475: Lecture I (2026-10-29): Statistical Reflections

Overview

Ted Pavlic connects hypothesis-testing errors to engineering decisions, then derives how the one-sample t test supports small-sample comparisons and how statistical power determines whether a simulation study can detect meaningful differences. Using a bank drive-through example, he shows how to validate a simulation against real data, diagnose a missing source of delay, and use effect sizes and operating-characteristic curves to plan replications.

Key takeaways

Alpha and beta reflect different costs: increasing alpha can reduce missed detections but increases false positives, so engineering, exploratory, scientific, and medical contexts may rationally choose different thresholds.
The t test extends z-based inference to small samples by using a heavier-tailed null distribution whose shape depends on degrees of freedom.
Standard deviation measures variability among observations, while standard error measures uncertainty in a sample-mean estimate and shrinks as sample size increases.
A simulation that fails validation can expose omitted system interactions: the bank model underestimated delay until it included walk-in customers competing with drive-through customers for tellers.
Not rejecting a difference does not establish that a simulation is accurate; power analysis must show that the chosen replication count can detect differences stakeholders consider important.
For paired historical-input validation, test the mean of the paired output differences against zero, then use the difference standard deviation and an OC curve to determine required replications.

Chapters

0:00 Input Modeling Report, Final Project, and Upcoming Coursework

The group input modeling report is due Sunday, replaces a lab but does not count as one, and cannot be dropped.
Only one group member needs to upload the report; the grade applies to all four members.
The one-to-two-page report should show access to real system data, ideally with a pilot sample and fitted input models.
Feedback will flag projects that are too broad or already answerable with queuing theory; Labs 9 and 10 precede final-project work.

3:10 Choosing Alpha for Exploratory, Scientific, and Medical Decisions

Alpha is the tolerated Type I error rate: rejecting a true null hypothesis, illustrated with the Halloween question of ghost versus water heater.
The course uses alpha = 0.05, but exploratory engineering studies may use 0.10, 0.15, or 0.20 to avoid missing possible effects.
A one-off biological discovery or a medical decision may call for lower alpha because a false positive can be costly or difficult to correct.
A p-value measures how likely data at least as extreme as the observed data are under the null; compare it with a preselected alpha.

8:00 Type II Error, Statistical Power, and the Decision Matrix

Beta is the probability of failing to reject a false null hypothesis; power is 1 minus beta, or the true-positive rate.
An 80% power target means accepting a 20% Type II error rate.
The decision matrix distinguishes true negatives, false positives, false negatives, and true positives according to whether the null is true and whether it is rejected.
A high-power test can detect a small difference, such as normal distributions whose means are 5 and 5.5; a low-power test may detect only a much larger difference.

12:00 Overlapping Distributions Explain the Alpha-Power Tradeoff and ROC Curve

A decision threshold separates results treated as consistent with the null from results counted as detections of an alternative.
Overlap between null and alternative distributions produces false positives and false negatives; outcomes in non-overlapping tails support correct decisions.
Moving the threshold changes the tradeoff: lowering false-positive rate raises false-negative rate and lowers power.
An ROC curve plots true-positive rate against false-positive rate as the threshold sweeps across possible values.

19:11 William Sealy Gosset and the Small-Sample Student t Test

William Sealy Gosset developed the Student t test at Guinness Brewery in 1908 to compare beer batches without requiring huge numbers of samples.
Because Guinness restricted publication under his own name, Gosset published the method under the pen name Student.
A one-sample t statistic compares a sample mean with a hypothesized mean, using the unbiased sample standard deviation divided by the square root of n.
The test assumes independent, normally distributed observations; checking independence is an experimental-design issue, while normality can be assessed with tools such as Shapiro-Wilk.

27:15 Why the Standard Error of the Mean Is Not Standard Deviation

Under the null, the expected sample mean equals the population mean, so the expected numerator, sample mean minus hypothesized mean, is zero.
The standard error of the mean is the standard deviation of the sample-mean estimator, equal to population standard deviation divided by √n under the stated assumptions.
Standard deviation describes the spread of individual observations; standard error describes how much estimates of the mean vary across samples.
As n grows, the standard error shrinks toward zero even though the underlying distribution's standard deviation remains unchanged.

32:04 The t Distribution Accounts for Small Samples

The central limit theorem makes the standardized sample mean approach a standard normal distribution as sample size increases.
For small samples, estimating variability from the same data creates heavier tails than a standard normal; with two observations, the t distribution has one degree of freedom.
As degrees of freedom increase, the t distribution's tails thin and it approaches the standard normal; around 20 observations, the two are quite similar.
The t distribution fills the gap left by z tests, which require large samples for reliable normal approximations.

38:54 One-Sample Versus Two-Sample and One-Tailed Versus Two-Tailed Tests

A one-sample test compares one data-derived mean with a scalar hypothesized value; a two-sample test compares means from two data sets.
A two-tailed test detects deviations in either direction, while a one-tailed test only targets a specified direction.
Pavlic recommends two-tailed tests in most applications because the direction can be assessed after detecting a difference.
The t statistic is a pivotal quantity whose null distribution is determined by its degrees of freedom.

42:22 Effect Size, Sample Count, and the Student's t-Shirt Joke

Power rises when the true difference between means is larger, the number of samples increases, or output variability decreases.
The standardized separation between means scales with √n and inversely with the standard deviation, explaining how additional replications improve discrimination.
A Halloween shirt labeled Student with a heavy-tailed curve is a pun on a Student's t-shirt and the t distribution's fatter tails.
The example reinforces that small samples can yield more extreme t values than a standard normal would predict.

48:05 Checking Normality and Choosing Nonparametric Alternatives

A t test is parametric: it gains sensitivity by assuming a normal model and testing a parameter such as the mean.
Use Shapiro-Wilk tests or Q-Q plots, including those in Input Analyzer, to assess whether normality is plausible.
When normality is untenable, the Mann-Whitney U test is a nonparametric alternative, focused more on medians and generally less powerful.
Other named alternatives include Wilcoxon signed-rank for paired data and Kruskal-Wallis for ANOVA-like comparisons; the key is knowing when to seek a nonparametric method.

53:07 Setting Up Bank Drive-Through Simulation Validation

The bank model includes drive-through arrival inputs, teller service-time inputs, decision variables such as teller count, and performance outputs such as average delay.
Validation compares simulated outputs with real-system observations under similar inputs; input checks also support verification.
For a simple one-sample demonstration, Pavlic treats the real-world average delay of 4.3 minutes as the target, while noting that a two-sample comparison would better represent uncertainty in both data sets.
The example starts with six independent two-hour replications and tests whether simulated average delay matches the 4.3-minute target.

57:00 Rejecting an Incorrect Bank Model and Finding Walk-In Customer Interference

The initial six simulated delays average 2.5 minutes, all below the 4.3-minute target; the t statistic is -5.34, beyond the two-tailed critical value of 2.571 at alpha = 0.05.
The failed validation prompts model diagnosis rather than accepting the simulation as a reliable proxy.
A suspected delay between successive drive-through cars is ruled out, but the model is found to omit walk-in customers competing for teller service.
Adding walk-in demand improves the simulated mean to 4.78 minutes; a t value around 0.71 does not reject equality, but that alone does not prove the model is accurate.

1:03:12 Power Analysis with Effect Size and Operating Characteristic Curves

Failure to reject a difference is meaningful only if the test has enough power to detect a difference stakeholders care about.
Effect size is the minimum meaningful difference divided by the output standard deviation; stakeholders define the acceptable difference, such as a one-minute or five-minute discrepancy.
An operating characteristic curve plots Type II error against standardized effect size for different sample counts; 80% power corresponds to beta = 0.20.
For six replications, the example's 80%-power detectable effect size is about 1.435, equivalent to 2.38 minutes; a requirement to detect a two-minute difference therefore needs more samples.

1:09:10 Paired Difference Tests for Historical Simulation Inputs

The homework uses historical input batches, runs each batch through the simulation, and pairs each simulated output with its real-system output.
For each batch, calculate the simulated-minus-historical difference; the paired-test null hypothesis is that these differences have mean zero.
Compute the mean and standard deviation of the paired differences, then use a one-sample t statistic with the number of paired data sets as the sample count.
For the power question, divide the stakeholder's minimum detectable difference by the sampled standard deviation, then use the 80%-power OC curve; the worked hypothetical effect size of 0.5 requires 34 samples.

Transcript excerpt

0:11 All right. So, um, this week, uh, mainly the big focus your input modeling report with your groups. So, uh, you've got your, uh, groups formed. This is due Sunday. It basically takes the place of a lab but isn't doesn't count as a lab so it can't be dropped. This is a group project. Um only one person needs to update it uh upload it and then the grade will apply to all four. Uh in the next two weeks we've got lab 9 and lab 10. Uh that's on your regular schedule. And then after that there'll be no uh formal labs assigned anymore and you'll be able to use that time just to work on the final project. Um so uh final project again big focus now is this input modeling report. um the uh you know once you

1:00 submit it this weekend then I'll spend um the early days of next week reading over those giving you feedback saying you know what might be too ambitious or what you know so the again the two biggest feedback I give is like well this the what you're planning here like you've picked too many things just pick one of these or it's oh you've just selected something that we kind of already know from queuing theory so why do you need a simulation for this can you pick something that is something that it makes more sense to be uh to be studied with a simulation. So that's the type of feedback that I'll give. Um I just want to see some evidence that you have access to data from your system. You don't have to model all of the data. You don't have to gather all of the data. So if you are gathering data um at

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ted Pavlic.