Course · Subscriber

Model Risk Management for LLMs

What happens to a 50-year-old bank validation discipline when the model is a black-box LLM nobody trained, and how to govern it without grinding deployment to a halt.

8 modules·About 3.5 hours·Available now

AI in Financial Services covers where AI creates value. This covers the validation-and-governance machine that decides whether any of it reaches production, the most-asked and least-taught topic in regulated AI right now.

We map the revised model risk management guidance onto an LLM you did not build and cannot fully inspect, then build the inventory, tiering, validation, and monitoring that survive an exam without becoming a launch blocker.

Who this is for

Model risk officers and second-line risk and audit staff
AI/ML leads inside banks who keep getting blocked at the validation gate
Product owners shipping GenAI features into a regulated environment

What you will learn

Map the revised interagency model risk management guidance onto an LLM you did not build and cannot reproduce
Build a materiality-based model inventory and risk-tiering scheme that survives an exam
Validate a vendor model through challenge, benchmarks, eval harnesses, and golden datasets instead of retraining
Stand up continuous monitoring for drift, hallucination rate, and stability with thresholds tied to materiality
Document a defensible accountability chain and shift governance left instead of bolting it on at launch

Prerequisites

Familiarity with what a model is and the basics of bank model governance

Syllabus

01From SR 11-7 to SR 26-2: What Changed and What GenAI Is In Scope For7 min
The 2026 revision retired SR 11-7 and replaced annual revalidation with risk-based oversight, while putting generative and agentic AI explicitly out of scope. Here is what that means for anyone running LLMs inside a bank.
02The Inventory Problem: Finding Every LLM Already Running in Your Bank8 min
You cannot govern a model you cannot see. This lesson is a practitioner's playbook for discovering every LLM already in production across your bank, including the ones nobody told you about.
03Risk-Tiering by Materiality: Sizing Oversight to Impact8 min
How to tier LLM-backed models by exposure and purpose, and how to make the tiering decision itself defensible when an examiner asks why a model sits where it does.
04Validating a Model You Cannot Open8 min
How to run real model validation on a vendor LLM when the weights, training data, and methodology are off-limits. Conceptual soundness and vendor challenge become behavioral, evidence-driven exercises rather than code review.
05Evals as Validation: Golden Sets, Benchmarks, and Challenger Models8 min
When you cannot retrain or open an LLM, your validation evidence becomes a curated golden set, targeted benchmarks, and a challenger model. This lesson covers how to build each one so it holds up to effective challenge.
06Continuous Monitoring: Drift, Hallucination Rate, and Stability Thresholds7 min
How to run live monitoring on an LLM you cannot retrain, with drift, hallucination, and stability thresholds tied to the model's materiality tier rather than a single house number.
07Documentation and the Accountability Chain8 min
What regulators expect to find in the model file, and how to build a record for an LLM you cannot open so that a named person owns every decision.
08The Fast Path: Governance That Ships Instead of Blocks7 min
A practical operating model for moving low-risk LLM use cases to production in days, not quarters, by pre-deciding the controls and reserving full validation for the models that can actually hurt you.