UUtah Data Mining | Fall 2026 | L20 : AI-Enabled Science
Overview
Modern science has shifted from testing mechanistic hypotheses with purpose-built experiments toward predictive science, which learns functions from large datasets and evaluates them on held-out data. The lecture traces how cross-validation, open data and code, Kaggle competitions, ImageNet, and self-supervised learning enabled this shift, then examines how AI can accelerate scientific workflows while humans remain essential for defining questions, judging evidence, and guarding against misalignment.
Key takeaways
- Predictive science can succeed without mechanistic understanding: an MRI classifier used hospital logos as proxies for sickness and still performed well on an IID test split.
- Cross-validation is a powerful common evaluation method for supervised prediction, but its reliability depends on training and test data sharing the same distribution.
- Frictionless reproducibility requires sharing the dataset, executable code, and exact scoring function so researchers can reproduce results and make controlled improvements.
- ImageNet's public image challenge helped establish deep neural networks as effective computer-vision models, while self-supervised masking created training labels from otherwise unlabeled text.
- AI agents can accelerate data preparation, baseline modeling, and iterative experiments, but humans remain essential for formulating meaningful scientific challenges and judging evidence.
- AI-generated code and scientific outputs need independent tests and oversight because optimizing a specified metric does not guarantee scientific validity or ethical alignment.
Chapters
- Project groups received feedback on intermediate reports, and the remaining two homework assignments are planned as shorter tasks.
- Mechanistic science seeks simple rules that explain processes, such as Newton's force equals mass times acceleration.
- The traditional workflow forms a hypothesis first, collects experiment-specific data, then uses statistics to test the hypothesis.
- Predictive science prioritizes forecasting future outcomes over discovering the mechanism that generates them.
- Its central assumption is that future observations resemble past observations, even when measurements are noisy or incomplete.
- Big data changed the workflow: collect and store data first, define a prediction objective, then search for a function that performs well.
- The standard machine-learning setup assumes observations are independent and identically distributed, or IID.
- Supervised learning pairs input features X with outcomes Y, such as recent stock prices and the next day's price.
- Cross-validation compares models by training on one split and evaluating predictions on held-out data, provided both splits represent the same distribution.
- A useful predictive function must be computable, and preferably efficient enough to run on future observations.
- Parameterized functions such as neural networks can be optimized with gradient descent when gradients are available; automatic differentiation can help estimate them.
- Linear regression predicts with a parameter vector dot product and commonly minimizes squared error, optionally with regularization.
- Linear regression and other models can predict accurately without identifying the physical or causal process behind an outcome.
- An MRI model learned hospital logos as a proxy for patient sickness because one hospital's cohort was sicker, illustrating a confounder that survives an IID test split.
- Correlated features, hidden variables, and complex interactions can all support prediction while obscuring the mechanism.
- David Donoho's essay, "Data Science at the Singularity," motivates frictionless reproducibility as a way to let many researchers or AI agents work on the same optimization problem.
- Sharing data lets others verify results and test alternative methods; human-subject research, especially healthcare, faces privacy barriers.
- Sharing executable code on platforms such as GitHub makes experiments easier to rerun and extend than paper descriptions alone.
- A reproducible challenge needs a released dataset, executable methods, and a clearly specified scoring function.
- Kaggle competitions can provide daily feedback on a validation split and use a separate final split to reduce overfitting or attempts to game the leaderboard.
- ImageNet's large, publicly available image dataset and prediction challenge demonstrated that deep neural networks could outperform established computer-vision approaches at scale.
- Self-supervision creates paired examples from unlabeled text by hiding a word and training a model to predict it.
- For example, masking "night" in "It was a dark and stormy night" creates a prediction target without human annotation.
- This approach supplied large language models with vast training corpora and vector representations, reducing reliance on costly hand-labeled datasets.
- Scientific machine learning attempts to constrain a learned function so it follows known physical rules, combining data-driven prediction with scientific structure.
- Protein structure prediction illustrates the approach: models use sequence data while incorporating constraints about how atoms interact.
- The broader challenge is to combine flexible machine learning with mechanistic knowledge without assuming every system has a simple governing model.
- One strategy is to fit a simpler, interpretable model to a complex predictor, but the simpler model often loses predictive accuracy.
- Sparse methods such as LASSO seek simpler explanations when a genuinely sparse underlying mechanism exists.
- Researchers also study activation spaces in large language models, though interpreting their high-dimensional geometry remains difficult and uncertain.
- Cross-validation can mislead when future data no longer comes from the training distribution.
- AI-generated text may increasingly replace human-generated text, while robots change the environments they are meant to model.
- Time series and other sequential settings violate simple IID assumptions because earlier events can affect later outcomes.
- Large language models and agents can search existing data, prepare and clean datasets, and write initial analysis code.
- AI can find baseline models and make rapid variations when a dataset and scoring function are already specified.
- Defining the scientific question, choosing a meaningful evaluation, and applying domain knowledge still require substantial human guidance.
- A closed-loop system can propose hypotheses, rank them, run code or simulations, and use experimental results to suggest the next test.
- Recursion Pharmaceuticals has pursued machine-learning-assisted hypothesis generation and automation in drug discovery for roughly five to ten years.
- Reliable testing and evaluation remain major bottlenecks, and AI-generated research submissions can be poor when human oversight is insufficient.
- An agent optimized for a scoring function may exploit the metric without serving science's broader goals or ethical standards.
- Human researchers can also fabricate data under pressure, but scientific communities have mechanisms for detecting and penalizing misconduct.
- Misinterpreted instructions, underspecified goals, and imperfect safeguards such as Anthropic's model constitution make human review important.
- A recently discussed MIT method used open-weight models to adjust model parameters through a black-box process and reportedly improved open-model performance.
- Open systems such as Linux and BSD illustrate how open implementations can support broad scrutiny and development.
- Open weights do not reveal the training corpus, leaving questions about memorization, data provenance, intellectual property, and evaluation contamination.
- AI-generated code should be tested and evaluated rather than trusted because it appears plausible.
- One proposed teaching approach treats AI coding tools as flaky compilers: provide detailed instructions, inspect outputs, and add extensive checks.
- Evidence such as test suites, scenario coverage, and observed behavior can help establish that generated code works, even when its internal process is difficult to understand.
Transcript excerpt
0:47 Uh scripts, let's get started. Oh um so welcome back everyone. Um so um so almost all the groups had their project feedback on on the intermediate report. I hope you and your group were able to discuss afterwards and kind of take the feedback and chart a um so plan forward. If you have any more follow-up questions on that, feel free to ask me before after class and office hours on email. Um so I'm receptive. Uh so was there a question? Okay, great. And then um so if you are um so then the um other thing was the dimensionality
1:35 reduction homework was due today um and we'll be moving on to kind a different part of class like on noise um and understanding kind of uh kind of things like anomalies and outliers of um so so starting on Thursday. Um there will be two more homework assignments left, but they're both going to be on the shorter side. They're kind of like two half homework assignments. So, um so I I I I hope you can spend most of the rest of the semester working on your projects. Uh so if you got feedback and a lot more stuff to do, hopefully it give you space to explore it. Um so today I'm going to give a little bit different sort of
2:24 lecture than than I usually do. Um so in class I've got slides. Um so this is to uh uh uh um so this lecture will talk about how we should think about modern data analysis and how this is affecting science. And then I'm going to kind of loop into how this kind of advancement that's kind of change of thinking about how we think about science the last 20 years has
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.