Lecture 15: Zipf's Law

Overview

The lecture uses a 5,000-word English frequency dataset to introduce Zipf’s law: word frequency generally decreases roughly in inverse proportion to rank, producing an approximately straight line on a log-log plot. Along the way, it demonstrates reading and parsing CSV data in Python, calculating how much of a corpus is accounted for by common words, interpreting dispersion, and visualizing the pattern with Altair; it also covers course grading and lab deadlines.

Key takeaways

In the analyzed English corpus of about 329 million words, the ten most frequent words account for approximately 27% of all counted tokens.
Zipf’s law is visible when word frequency is plotted against rank on logarithmic axes: the relationship becomes approximately linear with a slope near negative one.
A word’s frequency and dispersion describe different properties: frequency counts occurrences, while dispersion indicates how broadly the word appears across texts and genres.
Python’s sum function and list slicing can calculate top- and bottom-ranked word shares without writing explicit accumulation loops.
For computer science, log base 2 can be understood as repeated halving, connecting logarithms to efficient algorithms such as O(n log n) sorting.
The slice expression word[::-1] reverses a string and supports a simple dictionary-membership exercise to find word pairs such as “live” and “evil.”

Chapters

0:00 Runestone Grade Imports and Week Eight Course Logistics

The instructor describes merging Canvas and Runestone CSV gradebooks with a Python script, matching students using names and email addresses.
The script uses dictionaries to associate email addresses with names and student names with Runestone grades; dictionaries are scheduled to begin in the next class.
The instructor says labs 4 and 5 should be completed by Monday and warns that late or missing work can result in zeros pending a demo.

9:20 Academic Progress Reports and Lab Demo Expectations

Academic midterm reports can mark students satisfactory or unsatisfactory, with codes for grade, participation, missing work, or attendance.
Zeros may be entered as a prompt to complete assignment demos; the instructor says students can still get them corrected by demonstrating the work.
The week-eight deadline is presented as a checkpoint to prevent students from falling too far behind.

11:00 SETI, Language Fingerprints, and Zipf’s Law

The project asks how SETI could distinguish communication from random noise without knowing the language being detected.
Zipf’s law offers a possible language fingerprint: a small number of words occur very often, while many words occur rarely.
The lecture notes that similar frequency patterns have been observed in human language and communication systems studied in bottlenose dolphins, humpback whales, and squirrel monkeys.

14:30 Inspecting the 5,000-Word Frequency Dataset

The dataset contains columns for rank, word, part of speech, frequency, and dispersion.
Rank orders words from most to less frequent; frequency counts occurrences, while dispersion indicates how evenly a word appears across texts.
The word “re” illustrates low dispersion despite a notable count because it is concentrated in legal writing rather than broadly distributed across genres.

18:40 Parsing CSV Rows into Python Lists

The example opens the word-frequency CSV, reads its lines, and splits each row on commas.
Five lists—rank, word, part of speech, frequency, and dispersion—store the corresponding fields using list indexing and append.
The header row must be skipped before converting rank and frequency to integers and dispersion to a float.

26:20 Zipf’s Law and the Unsustainable Word-Frequency Joke

An XKCD graph jokes that extrapolating the rising use of “sustainable” would eventually make every sentence consist only of that word.
Zipf’s law predicts a rank-frequency relationship in which higher-ranked words are much more frequent than lower-ranked ones.
The lecture gives the rough pattern that the most common word can occur about twice as often as the next, while emphasizing that actual datasets do not match perfectly at every rank.

29:35 Measuring How Much the Top and Bottom Words Contribute

Python’s sum function totals the frequency list, yielding approximately 329 million counted words in the corpus.
The top-ten calculation sums frequencies at indices 0 through 9 and divides by the total, showing that the ten most common words account for about 27% of the corpus.
The bottom-ten calculation uses negative list indices and shows that the least frequent words contribute only a tiny share.

36:15 Dispersion and Examples of Rare English Words

The least frequent entries include words such as “plaintiff,” “middle class,” “laundry,” “manual,” “pistol,” and “stereotype.”
Dispersion helps explain genre concentration: “plaintiff” is more likely in legal writing and news than in fantasy fiction.
Comparing rare entries with words around rank 100 illustrates how common words are shared across far more contexts.

39:20 Logarithms as Repeated Splitting for Computer Science

The instructor explains log base 2 as the number of repeated halvings: a collection of 64 can be halved six times to reach 1.
Log base 10 can similarly describe repeated division into ten groups; for example, 1,000 takes three such splits to reach 1.
Logarithms matter in computer science because algorithms such as sorting can run in O(n log n), and logarithmic scales help display data spanning very different magnitudes.

47:35 Building a Word-Rank Chart with Altair

The demonstration uses Altair to create a chart from rank and frequency columns, mapping rank to the x-axis and frequency to the y-axis.
The instructor contrasts Altair with Matplotlib, which is the library used for their scientific publications.
On ordinary axes, the frequency curve drops sharply because the most common words have counts in the millions while lower-ranked words occur far less often.

53:15 Log-Log Plots Reveal the Zipf Pattern

Applying math.log to both rank and frequency makes the steep curve appear approximately linear.
Changing the logarithm base from 10 to 2 changes the displayed scale but not the basic pattern.
The plotted line has a slope near negative one, the characteristic rank-frequency behavior expected under Zipf’s law.

57:10 Finding Reversed Words with Python Slicing

The final exercise checks whether reversing each word produces another word in the dataset, using the slice syntax word[::-1].
Examples include “part” and “trap,” “live” and “evil,” and “pot” and “top”; palindromes such as “I” also match themselves.
The lecture ends with reminders to submit and demo labs 4 and 5, and explains that zeros can be corrected by completing the required demos.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Professor Rosen.