Building and evaluating AI for deployment: evidence before and after

MLHC pre-conference workshop 2025

Wouter van Amsterdam, MD, PhD

Department of Data Science Methods, Julius Center, University Medical Center Utrecht

2025-08-14

Welcome to the workshop!

Organizers - Workshop Chairs MLHC 2025

Uri Shalit

Visiting Researcher, Google DeepMind

Associate Professor, Tel Aviv University

Wouter van Amsterdam

Assistant Professor, University Medical Center Utrecht, Netherlands

Anna Goldenberg
U of Toronto

Brett Beaulieu Jones
U of Chicago, Med. School

David Kent
Tufts Medicine

Jean Feng
UC-SF

Jenna Wiens
U Michigan, Ann Harbor

Julia Vogt
ETH Zurich

Mark Sendak
Duke

Michael Oberst
Johns Hopkins

Muhammad Mamdani
U of Toronto

Rajesh Ranganath
New York University

Shalmali Joshi
Columbia University

Shengpu Tang
Emory University

Vincent Jeanselme
Columbia University

Xiaoxi Yao
Mayo Clinic

Why are we here?

• AI is changing healthcare
• pre deployment, need the right evidence to know AI will improve healthcare
• post deployment, ‘things will change’, how do we know the AI is still performing (in terms of impact on healthcare) as expected?
• FDA and EMA (and others) are developing regulations
• not all questions resolved, maybe we can help

Morning schedule

9-12am: focus talks, 30 minutes each

• Wouter van Amsterdam - Aligning development, deployment and monitoring for AI: a causal perspective
• David Kent - Evaluating prediction models in practice
• Anna Goldenberg - to Trial or not to Trial?

10.30-11.00: coffee break

• Mohammad Mamdani - Application and Evaluation of AI in Healthcare
• Jean Feng - Monitoring clinical AI: Where are we now and where do we want to be?

12.00-12.30: Plenary Discussion

12.30 - 13.30: Lunch

Afternoon schedule

• 13:30: group work; work on questions, draft manuscripts
• 16:00-16:45: groups present ideas
• 16:45-17:00: wrap-up

Goal of workshop:

• identify pressing questions that we can answer / work on, potentially start drafting follow-up work

References