Python (Fall 2026) Module 12-7: File Challenge 1
Overview
Bill Moseley outlines Module 12's two optional file challenges, then focuses on Challenge 1: building a Python web crawler that reads HTML, collects links, and visits pages. He recommends starting with his four-page bmoseley.com site and restricting followed links to that domain so the crawler does not expand indefinitely; Challenge 2 instead adds file persistence to an assignment, with pickle optional.
Key takeaways
- A basic crawler repeatedly extracts links from HTML, queues them, and visits newly discovered pages until its links are exhausted.
- Restricting traversal to bmoseley.com keeps the assignment's crawler within Bill Moseley's roughly four-page test site instead of expanding onto large external sites.
- Python string processing can be used to inspect HTML and collect links into a list for subsequent visits.
- Challenge 2 is a general file-persistence task: students can save and reload data from assignments such as an ATM, a game record, or calculator history.
- Pickle is an optional way to save assignment data; students may choose another approach.
Chapters
- Students may choose either of two challenges or attempt both.
- Challenge 1 builds a web crawler that follows links and analyzes page text.
- Challenge 2 adds data persistence to an assignment, such as saving ATM account data, Rock-Paper-Scissors win-loss records, or calculator history.
- Python's pickle module is allowed for Challenge 2 but is not required.
- A browser requests a page from a web server, which returns HTML that the browser renders visually.
- The crawler can inspect the returned HTML as text and use string processing, such as splitting it into lines, to find links.
- A typical crawler starts from one address, adds discovered links to a list, visits each page, and repeats the process.
- Crawler output could be a list of visited pages or, with more work, a visual map.
- Bill Moseley provides a set of about four simple pages on bmoseley.com for the assignment, including links, an email address, and an image.
- Only follow links whose addresses are on bmoseley.com; external links to Apple or Bakersfield College could lead to sites with thousands of links.
- Other discovered links can be cataloged separately or handled another way, at the student's discretion.
- Reading downloaded HTML as text does not execute embedded JavaScript, so this crawler approach does not run webpage scripts.
Transcript excerpt
0:00 So, for this week, in, if you go into Canvas, you will see this. Module 12 has two file challenges. You do not have to do both of them. Although, if you want to challenge yourself, you can try both of them and see how it goes. There are two challenges. Challenge one is to build a web crawler application that will scan for, scan pages for links and follow the links from page to page, analyzing the text on each page. Challenge number two is to, it says enhance the employee database assignment, but in actuality, it's literally any assignment where there's a data result. I, I just want you to save the
0:54 data and reopen it when that, when they, when the app opens up. So, for example, I'm going to show you an example with the ATM assignment, saving the account data. That's one of the ones that you could use for this. Although, I'm sure you can think of other possible, you know, you could save the win-loss record for rock, paper, scissors, or something like that if you wanted to. You could save past calculations for the calculator assignment. There are lots of different ways that you can do this. Now, you don't have to use, you don't have to use pickle, but you can use pickle. Okay? So, you can use pickle or not. So, file challenge number one, let's take a look at that one first. This is the more difficult of the two challenges, but for some of you it's maybe more
1:49 interesting. A web spidering application is, is used for all kinds of things. Google uses those to catalog web pages. Most search engines use them. That's how AI absorbs information online. That's how people who want to send you spam harvest email addresses. Web spidering application
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Bill Moseley, Ph.D. - Computer Science Educator.