Model risk
Banks, insurers and other federally regulated institutions already run model risk management. An AI agent does not need a parallel governance regime beside it. It needs to produce the same class of evidence the existing regime expects, in a form a validator, an internal auditor and a supervisor will accept.
The anchor requirement
Guideline E-23 takes effect 1 May 2027 and applies to all federally regulated financial institutions. It is principles-based and risk-based, and it applies with proportionality to the size and risk profile of the institution and of each model. It explicitly covers artificial intelligence and machine-learning models as well as models supplied by third parties. For an agent workload that means the same discipline as any other model. The institution identifies and inventories the workload and rates its risk. It develops and validates the workload with evidence. Approval comes before deployment. After that the workload gets monitored in production, change-controlled, and tied to a named owner across its lifecycle.
It is not an AI regulation. E-23 governs enterprise-wide model risk. Its expectations were written for capital, pricing, actuarial and credit models long before an agent was on anyone’s roadmap. Institutions already have model risk policies and validation functions, along with inventories and independent challenge. Large consultancies and specialist model-validation firms already serve that market well.
The gap is narrower and more specific. A retrieval agent behaves unlike a scorecard. Its inputs are documents rather than features, and its output is language. Its performance shifts when a provider ships a new model version, and its worst failure mode is a fluent answer the source does not support. Conventional validation techniques do not produce evidence for that on their own. We build the agent workload so it emits validation, monitoring and inventory evidence as a by-product of running. That evidence goes to your model risk function, which keeps the challenge and the decision.
Source
Source: OSFI Guideline E-23, Model Risk Management: effective 1 May 2027 and applicable to all federally regulated financial institutions (FRFIs). It is principles- and risk-based, and explicitly includes AI and machine-learning models as well as third-party models. Read it alongside your own internal model risk policy; the guideline sets expectations, your policy sets the standard you will be held to.
Mapping
This is what a validator asks for, and what each stage of the engagement produces. Nothing here replaces your model risk framework; it feeds it.
| E-23 expectation | What it asks for | Governance Sprint | Six-week Pilot | Managed AgentOps |
|---|---|---|---|---|
| Model definition and inventory | Whether the system meets the institution’s model definition, and a complete inventory entry | Draft entry: purpose, inputs, approved sources, owner, dependencies | Entry completed with model provenance, versions, prompts and retrieval configuration | Entry maintained with version history through every change |
| Risk rating and proportionality | A risk rating that determines the depth of validation and oversight applied | Proposed rating with the factors and reasoning recorded | Rating tested against measured behaviour and revised if warranted | Rating reviewed on schedule and after any material change |
| Development and validation | Documented development, data lineage, and evidence the model performs as intended | Evaluation plan, acceptance thresholds and golden dataset design | Golden dataset built and run: groundedness, citation accuracy, safety, latency, cost | Regression suite re-run monthly, with drift and degradation reported |
| Deployment approval | Approval before production use, by someone other than the builder | Approval path and decision rights mapped to named roles | Acceptance pack assembled for the approver, including what failed | Re-approval triggers defined and enforced for material change |
| Ongoing monitoring | Production performance monitored against expectations, with thresholds and escalation | Metrics, thresholds and escalation routes proposed | Instrumentation built and the operating baseline captured | Monthly governance report, threshold breach escalation, corrective actions tracked |
| Change management | Controlled change to model, data and configuration, with testing and rollback | Change classes and authority levels defined | Change control exercised from request to release, including a tested rollback | Every model, prompt, connector and corpus change logged and tested, and reversible |
| Third-party and vendor models | The same expectations where the model is supplied externally and internals are opaque | Provider dependency map with the transparency gaps written down | Compensating outcome tests where provider internals cannot be inspected | Provider release monitoring, with re-test on any model version change |
| Lifecycle governance and accountability | Named accountability from development through use to retirement | Shared-responsibility matrix and control-by-control ownership | Owners named per control and exercised in operation, not on paper | Named operating owner on both sides; retirement and exit plan maintained |
Evidence in operation
The month after go-live is where agent workloads quietly fail. A provider ships a new model version, someone reorganizes a source repository, someone edits a prompt to fix one complaint, and nobody re-runs anything. Continuous evaluation is the control that catches all three.
Monthly governance reportSample, figures are illustrative
Groundedness / citation pass rate
98.1%Above threshold
Critical safety test pass rate
100%Threshold met
Human escalation rate
6.4%Watch
Unit cost per completed task
$0.11Within budget
Median latency
3.9sWithin service-level objective
Open corrective actions
1Action required
Changes released: 4 · Rollbacks: 0 · Incidents: 1 (P3, resolved) · Access reviews completed: 1 of 1
Because it is the same record every month, produced by the same instrumented pipeline, against thresholds the institution approved before go-live. It is dated, versioned and reproducible, and it reports the failures as prominently as the passes. Nothing in it rests on our characterization of how the system is doing.
The control model behind it is documented under security and governance, and the work delivered against it is on what we have built.
Boundaries
Warning
White Glove Labs does not provide an audit opinion, a certification, or independent assurance under any professional standard. We are an implementation and operations firm. We design controls, build the workload, run the evaluation and produce the evidence. That makes us the first line, not the second or third.
Where independence is a requirement, a separate qualified party provides the challenge. That party is your internal validation function, internal audit or an external firm engaged for that purpose. We support the review with the artifacts, plus the data and the access it needs. We do not object to challenge. A firm that builds the system cannot also provide independent assurance over it.
The same separation applies to legal and jurisdictional questions. Statements about which laws bind a provider, its infrastructure and its support organization are matters for your counsel. We document the architecture, the data flows, the administrator access paths and the exit position. We set out the trade-offs under sovereign AI so that counsel and risk can reach their own conclusion on the record.
Other regulated buyers
Outside the federally regulated financial sector, the pressure to govern an agent comes from somewhere else. The control work is largely the same; the reviewer, the vocabulary and the timeline are not.
Where to start
Both were chosen because a person already reviews the output, the baseline is already measured somewhere in an operations report, and the failure modes are observable rather than theoretical.
Classification and field extraction against a documented taxonomy, with confidence thresholds and an exception queue. Baselines already exist as queue age and rework rate, so the value evidence and the model performance evidence come from the same instrumentation.
Identity-aware retrieval that returns only what the requester is entitled to see, with the agent drafting and a person sending. Containment, handle time and escalation reasons are already reported, and entitlement enforcement is directly testable.
Your model risk policy decides, using its own model definition. Institutions are widening that definition to capture AI systems that inform decisions even when they produce text rather than a number. The practical test is whether output influences a decision the institution is accountable for. Where the answer is yes, treat it as in scope and rate it proportionately. Where it is a productivity tool with no decision influence, record that reasoning rather than leaving the question unanswered.
Not us. We produce the development documentation, the evaluation design, the golden dataset, the results and the inventory entry. Your validation function or an independent third party then reviews that package, tests our claims, and forms its own view. We schedule the engagement so validation is not a surprise at the end. We share the evaluation plan in the sprint. We invite validators to challenge the acceptance thresholds before the pilot begins rather than after the results are in.
More than a model name. Purpose and decision context, the approved source corpus and its refresh arrangements, and the base model and version with provider and hosting location. Retrieval configuration, prompt and tool definitions under version control, the evaluation suite and thresholds, and the human approval point. Known limitations and excluded uses, the named owner, and the change history. That last item is the one a validator needs first, because it shows what changed since the last review.
1 May 2027 is the compliance date, not the start date. Building an agent workload without inventory, ratings, validation evidence and change control just moves the work later. Someone then has to reconstruct it from logs and memory, often under time pressure. Constructing the evidence while you build the system costs little extra. Reconstructing it afterwards is expensive. The parts you cannot reconstruct become findings.
The other clock
Eight weeks is the delivery time we are held to, measured from the day the engagement starts. In a regulated business the contracting step is rarely what delays it. The reviews are: third-party risk, information security, and for a federally regulated financial institution the model risk review that Guideline E-23 requires.
Not the contract. A statement of work under an existing master services agreement or a purchase order is a short step. What takes the time is supplier onboarding and the internal reviews. Those often run in series before anything begins: onboarding, then security review, then third-party risk, then model risk, then the engagement.
Run in series that sequence can outlast the delivery it precedes. It does not need to run that way.
Supplier onboarding, the information security review and third-party risk assessment can all run alongside the ten-business-day sprint, because none of them depend on the sprint’s findings. What they need from us is the supplier evidence pack, which is available within two business days of a request and does not require a call first.
The model risk review is different, and the order matters. Under Guideline E-23 a model risk assessment needs the model inventory entry, the intended use, the data lineage and the evaluation design. Three of those four are sprint deliverables. Starting the model risk review before the sprint means your reviewers assess a description; starting it at the end of the sprint means they assess documented artifacts. The second takes less of their time and produces a defensible file.
Note
Your risk functions own their own timelines and their own conclusions. The sequence is a proposal and commits to no review duration.
Next step
A structured 30-minute working session on a single candidate workload. We identify what your model risk framework would demand of it, what evidence does not exist yet, and whether it is worth a sprint.
Source
Dollar figures on this page are the sources’ own or recorded amounts, not our prices. Our prices are in Canadian dollars (CAD), plus GST/HST.
