Stats 21 - Lecture 14 - Pandas Grouping Aggregates

Overview

Miles Chen uses the 891-row, 12-column Titanic DataFrame to teach pandas missing-value checks, category counts, duplicate detection, and grouped aggregation. The lecture emphasizes interpreting results carefully: missing ages reduce the usable sample to 714, `size` and `count` answer different questions, and choices such as dropping rows or filling values can bias conclusions.

Key takeaways

The Titanic dataset has 714 observed ages out of 891 passengers, but only 183 rows complete across every column, so dropping all incomplete rows discards far more data than filtering only missing ages.
Missingness can bias analysis: dropping people without addresses, for example, could disproportionately exclude people in temporary or unhoused situations.
In pandas group summaries, `size` counts rows regardless of missing data, whereas `count` counts only non-missing values in the selected column.
For a 0/1 survival column, the mean is the survival proportion and the sum is the survivor count; first-class passengers include 136 survivors among 216 records, or about 62.96%.
Use `dropna=False` in `value_counts()` or `groupby()` when missing categories need to remain visible, as with the two Titanic passengers whose embarkation port was unrecorded.
For nullable integer totals, `sum(min_count=1)` avoids treating an entirely unobserved group as a measured zero.

Chapters

0:00 Titanic DataFrame Structure and Missing-Value Checks

The Titanic DataFrame contains 891 rows and 12 columns, including passenger class, survival, age, and cabin.
`titanic.info()`, `.isna()`, and `.notna()` help inspect missingness; selecting `titanic['Age']` returns a Series with the row index.
A boolean Series from `isna()` can filter rows with `.loc`, such as locating James Moran, whose age is missing.

8:49 Missingness Rates, Value Counts, and Duplicate Detection

`isna().sum()` counts missing values by column, while `isna().mean() * 100` gives missingness percentages; cabin is missing for about 77% of records and age for nearly 20%.
`dropna()` removes rows missing any column, while `dropna(subset=['Age'])` retains the 714 records with observed ages; only 183 rows are complete across all columns.
`value_counts()` counts categories, such as passenger classes 1, 2, and 3 appearing 216, 184, and 491 times; `normalize=True` returns proportions and `dropna=False` includes missing categories.
`duplicated(keep=False)` marks every row sharing a repeated value, and `drop_duplicates(subset=...)` removes duplicates based on selected columns.

15:38 Group Titanic Passengers by Class and Aggregate Outcomes

Grouping by passenger class and taking the mean of `Survived` calculates each class's survival proportion because survival is encoded as 0 or 1.
`groupby('Pclass', as_index=False).agg(...)` keeps the group label as a regular column and allows different functions for selected columns.
`size` counts all rows in a group, while `count` counts non-missing values; for age, `count` therefore excludes passengers without recorded ages.
Summing `Survived` gives survivor counts, including 136 survivors among 216 first-class passengers, a survival rate of about 0.6296.

20:56 Grouping with Missing Categories and Choosing Denominators

Grouping by embarkation port normally drops missing group labels; setting `dropna=False` retains the two passengers whose port is unknown.
The two passengers with missing embarkation-port values both survived, illustrating why retaining missing categories can matter to a group comparison.
Comparing survival across all passengers with survival among only the 714 passengers with observed ages changes the population being measured.
A known survival outcome should not be discarded solely because age is missing when the question concerns all recorded survival outcomes.

25:59 Clean Category Labels and Convert Numeric Strings

Inconsistent embarkation labels such as lowercase or space-padded `S` and `C` can be standardized with string stripping and uppercase conversion before counting or grouping.
`fillna()` can replace missing labels for clearer reports, but replacement should reflect actual knowledge rather than an unsupported assumption about a passenger's port.
`pandas.to_numeric(errors='coerce')` converts parseable strings to numbers and turns unparseable entries into missing values.
A coerced missing value may reflect a typo or conversion failure, such as OCR reading a digit incorrectly, rather than an originally absent source value.

29:50 Nullable Integer Values and Aggregates with No Observations

`numpy.NaN` can upcast integer data to floating point; pandas' nullable integer dtype and `pandas.NA` preserve integer values alongside missing entries.
Summing a series with no non-missing values can return zero by default, which may falsely suggest a measured total.
Using `sum(min_count=1)` returns missing when there are no observed values, distinguishing an unknown total from a genuine sum of zeros.
A grouped mean for a group with no recorded measurements returns missing, as expected when the data contain no values to average.

Transcript excerpt

0:02 Okay, I apologize for all of the computer struggles but it it happens. Okay. All right, so we've been working with this Titanic data set. We introduced pandas on Monday and today we want to talk about just how do we uh you know, handle some grouping information and information you know, for each record. Okay. So here we set So we think about our our data frame here. It's got 891 rows, 12 columns. Okay. Um

0:49 each row is a a record and we have stuff like passenger class and whether they survived, uh their age at the time and you know, just stuff stuff like that. Okay. We've got you know, some of these variables are grouping variables. Some of them have missing values. Things are coded. Okay, so we can just ask so for some information. We can just do titanic.info and it just gives you kind of like this is how many values we've got that are not missing. Okay, to see if things are missing, okay. Um we have dot isna to see if it's missing and dot notna if it's not missing. Okay.

1:39 Um This is like a new thing. All right. It took you should always have to do like isna and then like false but now you can just go directly not missing and and so anyway, you can just say all right, age. So question is this if I just stop right here so this is a review from Monday, is this a data frame, a series, an array? It is a series, okay? So, we have a square bracket with uh the name in quotes is going to select a series. If we had square bracket with square bracket,

2:25 and that that would return a data frame. Okay, so this is a series. A series is

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Miles Chen.