What has to be true before OSFI Guideline E-23 takes effect

The guideline defines a model as an application of theoretical, empirical or judgmental assumptions, or of statistical techniques, that processes input data to generate results. Artificial intelligence and machine learning (AI/ML) methods sit inside that definition. Model risk is the risk of adverse financial impact that comes from designing a model and from using it.

Read those two sentences together and the planning question changes. There is no threshold at which an AI system becomes subject to model risk management, and no separate AI guideline arriving later. An institution that has been treating its AI work as a technology initiative rather than as models has a definitional problem before it has a compliance one.

The guideline states three expected outcomes. Model risk is well understood and managed across the enterprise, using a risk-based approach. Model governance covers the entire model lifecycle.

Appendix 1 of the guideline sets out what is tracked. The first group applies to every model in the inventory. The second applies to models rated non-negligible.

Two details decide how much work this is. Vendor and third-party models are in the inventory, and externally developed models are assessed for a risk rating on a standalone basis. And decommissioned models stay in the inventory for a period the institution considers reasonable, so retirement is a recorded event rather than a deletion.

Credit and capital models are the ones a model risk function already tracks. The harder list to produce is one that includes the machine learning model a business unit bought as a feature inside a software product. That gap is the practical starting point.

The guideline asks for clear, measurable criteria on quantitative and qualitative dimensions. The quantitative factors are the importance of the portfolio a model covers, plus its size and growth, plus the potential financial, operational or security impact. The qualitative factors are business use or purpose; model complexity or level of autonomy; the reliability of data inputs; customer impact; and regulatory risk.

There is a relief valve. Institutions may include a rating category implying negligible inherent model risk, which exempts those models from the full lifecycle governance requirements. That category is where the effort of the whole framework is contained, and defining it defensibly is worth more than any other single decision in an E-23 program.

A trigger event prompts a review and update of the rating. The guideline gives a drop in performance and a material change in a model's use, data or infrastructure as examples. Where model risk falls outside the institution's risk appetite, the institution sets remediation actions.

Independence here is functional separation from the people who built the model. It does not require a separate legal entity, and it does not require a dedicated validation department. An internal reviewer who did not develop the model satisfies it. So does an external specialist. What does not satisfy it is a developer signing off on their own work, however senior.

What triggers a model review?

The guideline names five triggers. New model development is the first. Model modifications come second, including changes to algorithms, parameters or supporting operational components. Third are breaches of model performance, such as material errors and drift, or threshold violations. Fourth are significant data changes: a new source, an altered definition or a shift in quality. Fifth are scheduled risk-based periodic reviews.

Three of those five are events a monitoring system detects rather than dates in a calendar. An institution without instrumented monitoring cannot know when three of its five review triggers have fired, which is why monitoring and review are one program rather than two.

For AI and machine learning models specifically, review includes examining novel methodologies, algorithms and tools, and evaluating the level of explainability for the model workings against the intended use.

Evaluation criteria have to be quantitative and qualitative together. Performance metrics sit on one side. On the other is an assessment of whether the model still operates inside the scope that its approval covered. The operational factors go wider than accuracy. They include changes in model performance, in how much the model gets used, in the input data and in external dependencies such as version updates. They also include changes in the characteristics of whatever the model represents.

Two things need writing down in advance: the thresholds that define a breach, and the criteria for what counts as a material model modification. When a threshold is crossed, the team shares the issue with the relevant stakeholders under escalation procedures agreed before the breach. A contingency plan covers model unavailability, deterioration in performance and outright failure, each with its own escalation path.

The guideline says plainly that AI and machine learning models bring their own monitoring challenges. It names autonomous decision making, autonomous re-parametrization and a higher potential for model drift. A monitoring design carried over from a static statistical model will not detect a model that changed itself.

1. Find the models, including the ones nobody calls models: vendor features, machine learning inside a purchased platform, and anything a business unit built. The inventory cannot be risk-rated until it is complete.
2. Define the negligible-risk category and defend it in writing. This decides the size of everything downstream, and it is the judgement OSFI is most likely to test.
3. Rate what is left with criteria you can show, both quantitative and qualitative, per the guideline.
4. Fix the independence problem before the review backlog. Decide who reviews, separate from who develops, and confirm the reviewers have the skills for AI models specifically.
5. Instrument monitoring, because three of the five review triggers are detected rather than scheduled. Thresholds, escalation, and a contingency plan for failure.
6. Then work the backlog by rating, highest first, and record the review dates in the inventory as you go.