1786 lecture5 Oct6

Overview

Jonathan Rose frames the ECE 1786 project as a chance to build an agentic system for a demanding real-world task, targeting gaps that tools such as Claude and ChatGPT cannot handle autonomously. He uses his fantasy baseball team as a case study and outlines how teams should define inputs and outputs, research prior work, gather evaluation data, measure success, manage project risks, and deliver tightly constrained proposals and presentations.

Key takeaways

A strong agentic project begins by testing a current harness such as Claude or ChatGPT, then targeting a specific capability gap rather than assuming the model’s first successful demo is sufficient.
An autonomous fantasy baseball manager must do more than recommend players: it must handle league rules, weekly roster changes, injuries, free agents, trades, and potentially the initial auction.
Every project proposal needs a testable success measure and example data; when an LLM acts as an evaluator, human judgments should validate its decisions.
Teams should define inputs and outputs explicitly and explain domain-specific context for a diverse audience that may not share their background.
Project planning should break work into estimable tasks, identify assumptions and data-access risks, and specify fallback options within the term’s limited schedule.
The proposal’s 200-word cap is enforced with a 1% grade deduction per excess word, plus 5% if the word count and penalty are not reported.

Chapters

0:00 ECE 1786 Project Schedule, Team Commitments, and Peer Reviews

Assignment 2 is due Friday at 9:00 p.m.; its placement earlier in the course is intended to inform project choices.
Project approval in principle is due around October 13, followed by the proposal document and slides on October 19.
Teams will present in one of three sessions on October 20–21, attend a second session for peer review, and submit a progress report November 17.
Final presentations are scheduled for December 8–9, with slides due December 7 and the final report due December 15.

6:00 Find an Ambitious Project Beyond Existing AI Harnesses

Rose asks teams to target tasks that were previously impractical and identify a gap between their goal and what current systems such as Claude or ChatGPT can do.
A promising search area is human work that could be assisted or automated, including domains outside electrical and computer engineering.
Teams should draw on a hobby or personal experience, or learn enough about a domain such as music, medicine, or education to define a useful problem.
Project thinking should iterate among what the system will do, why it matters, and how it can be built.

9:18 Explain the Problem to an Audience with Different Backgrounds

Rose introduces the project goal: autonomously run a fantasy baseball team, from selecting an initial roster to managing it week by week.
Many classmates may not know fantasy sports, so proposals must explain unfamiliar concepts rather than assume shared knowledge.
He compares this communication challenge to past projects about games such as Elden Ring, whose mechanics may be unfamiliar to the audience.
The proposal and presentation should account for what classmates, teaching assistants, and instructors are likely to know.

12:18 Fantasy Baseball as a Multi-Step Agentic Task

In a fantasy league, participants select real professional players; their teams accumulate points based on those players’ real-world statistics.
Rose’s Canadian Baseball League has run for 38 years, and its website provides standings, player information, and league statistics.
Managing a team involves tracking separate batter and pitcher categories, responding to injuries, replacing underperformers from a free-agent pool, and evaluating trades.
Because roster changes and decisions recur weekly, the task combines ongoing monitoring, rule-following, and action selection.

19:17 Test Claude and ChatGPT, Then Define the Remaining Autonomy Gap

Rose tried Claude and ChatGPT 5.6 on his league website; with guidance, the systems found relevant information and suggested player decisions.
He still had to direct the systems to consult the league’s seven-page constitution, showing that an impressive demonstration is not full autonomy.
The initial player auction is a separate challenge involving a limited budget and dynamic bidding, potentially large enough to be its own project.
Teams should test current agentic harnesses first, then define a focused, special-purpose harness that addresses the remaining gap.

22:46 Research Prior Work and Design Measurable Evaluation

Teams should examine two or three related papers, conferences, journals, or operating businesses and identify what remains unsolved.
Candidate systems can be tested with offerings from OpenAI, Anthropic, DeepSeek, Google Gemini, or other providers; local models and open-source harnesses are another option.
Assignment 2 introduces running a local model through LM Studio, which can support experiments with locally hosted models.
Proposal plans must identify the data needed to test the system and a quantitative, or where necessary qualitative, measure of success.

27:33 Write the Approval Form and 200-Word Project Proposal

The approval form should name the project and state its goal in the team’s own words; Rose specifically advises against using generative AI to write that statement.
Teams must be able to explain their system’s inputs and outputs clearly enough that a TA can understand what it is meant to do.
The proposal should cover the problem and rationale, related work, data, system architecture, success metrics, task plan, collaboration, and project risks with mitigation options.
For LLM-as-a-judge evaluation, teams should compare automated judgments with human labels on example data to check whether the judge is reliable.
The proposal has a strict 200-word limit: each extra word costs 1% of the grade, and omitting the word count and penalty calculation costs an additional 5%.

37:26 Prepare the Four-Minute Proposal Talk and Peer Review

Proposal presentations are limited to four minutes and eight slides; slide text must use at least 20-point font, and both teammates must speak roughly equally.
Teams should practice at least two or three times and speak without reading notes or slide text; the instructor will stop talks when the timer ends.
Slides must be submitted as PowerPoint or PDF for presentation from the instructor’s laptop; evening sessions are in Sanford Fleming 1101.
Students will submit short peer-review feedback on another team’s proposal, then use the scheduled time to discuss project ideas with their partner, TAs, and Rose.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.