1786 lecture4 Sep29

Overview

Jonathan Rose connects LLM fundamentals to practical agent engineering: prompts select contextual knowledge alongside trained parameters, while retrieval-augmented generation, context-engineering harnesses, tools, and reasoning help models perform structured work. He applies those ideas to a behavior-change therapy example and lays out a project workflow requiring a distinctive problem, clear success measures, human validation, and documented software development.

Key takeaways

An LLM’s output depends on both parametric knowledge stored in trained weights and contextual knowledge supplied at inference; engineers should test conflicts between the two instead of assuming which one will dominate.
RAG reduces context size by embedding document chunks and a question, retrieving the roughly 10–20 closest chunks by cosine similarity, and supplying those passages to the model.
A harness turns a model call into a system: it assembles context, interprets tool requests, runs conventional software, and feeds results into the next model step.
Tool access can create serious security exposure: a model with permission to read files and make web requests could potentially disguise data exfiltration as a legitimate tool call.
Prompt quality needs empirical validation: iterate on examples, test separate cases, and use human-labeled ground truth to verify both model outputs and LLM-as-judge evaluations.
A strong course project starts with a meaningful what and why, defines measurable success, and adds multiple agents only when testing shows one model is insufficient.

Chapters

0:00 ECE 1786 Assignment 2 and the Project-Oriented Course

Assignment 2 introduces API use, prompt engineering, and running a local model; Rose expects most students to use existing models rather than train them.
Rose asks the six students without partners to meet during the break and says he has shared potential partners’ work habits and interests.
The first half of the lecture covers LLM engineering concepts; the second half explains project scope and requirements.

5:00 OpenAI APIs, Token Costs, and Assignment 2 Preparation

Assignment 2 Part 1 asks students to call OpenAI programmatically, select economical models, and budget under about $2.
Rose directs students to read API concepts, including token counting, tool use, and other software layers that address common LLM application problems.
Tokens—not words—are the units providers charge for, so students need to understand tokenization and its practical cost.

8:00 Prompt Engineering Through a Smoking-Cessation Example

Assignment 2 Part 3 asks students to soften overly direct statements; Rose draws on research into motivational, behavior-change therapy for smoking cessation.
A blunt prompt such as “You’re here to talk about your gambling, right?” can be reframed as a softer, permission-seeking question.
The goal is not to tell clients what to do: Rose says directive advice often fails because people change only when they are already ready.

11:00 Iterating Prompts and Testing Beyond One Example

Rose’s prompt-development method starts by defining explicit criteria for a good output, then iterates until the prompt works on an example.
After the first example, test additional cases to catch regressions; he suggests a small set of roughly 30 examples, such as 10 for iteration and 20 for testing.
Reliable agentic systems need more than a plausible-looking answer: use checks, model-based evaluation where appropriate, and eventual human review.

16:00 Prompts, Tokens, Embeddings, and Trained Parameters

LLM inputs are tokenized and converted to embeddings; the embeddings themselves are trained along with the neural-network parameters.
Prompting does not update model weights, but it selects which trained parameters and learned capabilities participate in a particular computation.
Rose presents this as a useful way to connect prompt engineering with the training concepts students will study later.

19:00 Contextual Knowledge Versus Parametric Knowledge

An LLM’s output depends on its current context and its trained neural-network parameters, which Rose identifies as two distinct sources of knowledge.
Context can include a prompt and, in some models, up to about one million tokens—roughly 750,000 words.
Parametric knowledge is what the model learned during training, including next-token prediction across human-written material.

24:00 When Model Knowledge Conflicts with the Prompt

Rose cautions that there is no dependable universal rule for whether contextual or parametric knowledge wins when they conflict.
His example instructs a model that the sky is normally green: one response initially describes a blue sky, then follows the green-sky instruction when prompted again.
System prompts, post-training, and jailbreak attempts can all affect behavior, so engineers must test conflicts instead of assuming a fixed priority.

31:00 Putting Whole Documents into an LLM Context

Sam Altman’s GPT-4-era demonstration put a long American tax reference into the context so the model could answer questions grounded in the document.
Supplying a complete knowledge text can help when the model lacks reliable information in its parameters or needs to apply detailed rules.
Large contexts increase token charges for hosted models and processing time for models running locally.

34:00 Retrieval-Augmented Generation: Embeddings and Chunk Selection

Retrieval-augmented generation (RAG) divides a large text collection into chunks and computes an embedding for each chunk.
The system embeds the user’s question, ranks document chunks by cosine similarity, and places roughly the top 10–20 in the model’s context.
Chunk size, overlap, embedding model, and retrieval structure affect whether RAG retrieves the information needed to answer.

40:00 Context Engineering for Structured Therapy

Rose distinguishes context engineering—the broader design of what enters each model call—from RAG, one particular retrieval technique.
His smoking-cessation therapy system organizes work into four tasks: engaging, focusing, evoking, and planning.
The system must track stages and strategies and respond when a conversation loses engagement, rather than blindly following a fixed sequence.

47:00 Harnesses: Software That Shapes LLM Context

A harness is the software that assembles engineered context, passes it to an LLM, examines the output, and decides what should happen next.
The loop may incorporate human input, model output, and results from other software before building the next context.
Rose expects course projects to build a meaningful system around an LLM, not simply submit a one-prompt demonstration.

52:00 Tool Calling Connects LLMs to Regular Software

Hosted systems such as ChatGPT, Claude, DeepSeek, Qwen, and Gemini can include harnesses that search the web, handle files, or open spreadsheets.
For a PDF task, a model can request a PDF-to-text tool instead of processing the file’s binary representation itself.
The harness detects a tool-call token or structured request, invokes the named software function, and returns its output to the model’s context.

59:00 Combining LLMs with Algorithms and Conventional Code

Tool calls rely on ordinary software to interpret model output, execute a function, and format its result for another model call.
Rose warns against asking an LLM to perform tasks that established software handles better; a tuned optimizer is more suitable for the traveling-salesman problem.
Effective systems combine language-model flexibility with conventional algorithms and code, and test for malformed tool requests or outputs.

1:03:00 Tool Permissions and Local-Model Security Risks

A model with tool access may gain the ability to search the internet, read files, or perform other actions through the harness.
Rose outlines a potential attack: a malicious model could disguise data exfiltration as a web-search tool call if it has broad permissions.
Local inference alone does not guarantee safety when connected tools can access private files or networks; restrict permissions and be cautious with tool calling.

1:05:00 Chain-of-Thought Prompting and Step-by-Step Reasoning

Rose traces modern reasoning features to chain-of-thought prompting, an idea associated with a seminal research group at the University of Alberta.
A worked example shows a model how to solve a tennis-ball arithmetic problem step by step, improving its answer to a similar apples problem.
Reasoning prompts encourage intermediate steps rather than an immediate guess, and the amount of reasoning can affect model performance.

1:13:00 LM Studio and Running Models on Personal Computers

Assignment 2 asks students to use LM Studio to download and run an open-weight model locally.
Rose says model compression and other methods have made useful models feasible on computers with around 16 GB of memory.
He recommends GPT-OSS 20B as an option that can run on a CPU, and emphasizes checking available system and GPU memory before downloading models.

1:20:00 Why the Project Is Central to Engineering Education

Rose describes the open-ended project as the main opportunity to learn how engineers turn an idea into a working system.
Students must explain their idea, plan, and results clearly; communication is treated as part of engineering rather than an add-on.
A strong project can become a concrete interview example, and students may share recorded final presentations on LinkedIn with permission.

1:24:00 Define the Project’s What and Why Before Its How

Rose asks teams to begin with what problem they want to address and why it matters, postponing technical implementation details.
He encourages students to draw on interests outside electrical and computer engineering, such as music, sports, psychology, or community work.
A proposal that opens by describing an “LLM that…” leads with the how; the project’s goal should be understandable without naming its implementation.

1:28:00 Project Scope, Agent Decomposition, and Evaluation

Rose cites past ideas including IELTS support, literature-survey writing, medical diagnosis, crossword hints, email summarization, and animation.
Before building multiple agents, test whether one model can solve the task; adding agents without need creates complexity rather than better engineering.
Projects must define how success will be measured, use human labeling as a ground-truth check, and validate any LLM judge against human judgments.

1:35:00 Project Approval, GitHub Workflow, and Presentation Milestones

Teams are generally pairs, must develop unique projects, and need Rose’s approval; each team will also receive a TA mentor.
Students must use the course GitHub repository and show commits throughout the term, rather than uploading work only at the end.
The next milestone is approval in principle; proposals and slides are scheduled for October 20–21, with peer reviews, progress reports, final presentations, and a final report to follow.
The approval form asks for a project name and a two-to-three-sentence statement of what the project does and why it is worthwhile, without implementation details.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.