Step 3 · Operate
An agent that passed its acceptance tests in June is not the same system in December. The model changed, the prompts changed, the source content changed and the entitlements changed. AgentOps is the discipline that keeps a production decision defensible after it has been made.
Definition
Managed AgentOps is the operating discipline for an AI agent already in production. We run the evaluation suite on a schedule and after every change. We control what anyone may change in the model and its prompts, in the tools the agent calls and in the knowledge base it reads. We monitor quality, safety, latency and cost against agreed thresholds. We review who and what the agent can reach, and we classify and respond to incidents. Each month we report to the people accountable for the system. The output is evidence that the system still does what we accepted it to do.
Note
Silent drift: the agent gets quietly worse and nobody notices, because getting worse looks like a plausible answer with a citation that no longer says what the answer claims. Without a scheduled regression run against known-good cases, the first signal is a complaint. By then the organization cannot say when it started or how many outputs were affected.
The service
We re-run the golden dataset from the pilot on a fixed schedule, after every model or prompt change, and after any material change to the source content. We report results against the thresholds your owner approved at acceptance, and a drop below threshold triggers a defined response. We add new failure patterns found in production to the dataset so the suite gets harder over time. The dataset itself is version-controlled.
Changes fall into four classes: model (version, provider, parameters), prompt (system instructions, templates, routing logic), tool (available actions, scopes, permissions) and knowledge base (sources added or removed, reclassified or re-indexed). Each class has its own approver and rollback path, and its own test requirement. No change reaches users without a passing evaluation run recorded against it.
We instrument quality, safety, latency and cost continuously instead of sampling them at report time. We spot-check live answers against their sources. We also run a standing set of known hard and unsafe questions through the system, to catch failures before a user does. We track response time and cost per completed task for each workflow, and anomalies alert the named owner. Monitoring also covers what the agent refused or escalated, and what it could not answer.
A scheduled review covers every human and machine account the agent uses and every system and tool it may reach. We check each against the rule that it should hold the narrowest access that still works, and against the access design accepted at pilot. The review looks in particular for departures, role changes and reclassified sources. Each review carries a date and a named reviewer, and lists the actions taken.
A defined severity scheme covers permission-boundary breaches and safety failures. It also covers fabricated or mis-attributed citations, as well as availability and cost events. Each severity carries a response time and a notification list, plus a containment action and a corrective-action record. We detect, classify and report. Your institution owns its own regulatory and public notification obligations.
Every change is reversible to a known-good configuration, and we test the rollback. Escalation paths for users are live and monitored. Anything tripping a threshold, a permission rule or a safety rule reaches a named human, and the volume of those escalations is a reported metric.
Reporting
One document, same structure every month, written for the sponsor and the risk committee. It records whether the system still performs as accepted.
| Section | The question it answers |
|---|---|
| User and task volume | Who uses the system, and for what? Is usage growing, flat or quietly abandoned? |
| Groundedness and citation pass rate | Are answers still traceable to approved sources, and do the citations still support the claims? |
| Critical safety test pass rate | Did the zero-tolerance suite pass in full this period, and if not, what was contained and when? |
| Human override and escalation rate | How often do people correct or escalate the agent, and is that trend moving the right way? |
| Latency and cost per completed task | Is the system still within the performance and budget envelope agreed at acceptance? |
| Changes released | What changed in the model, prompts, tools or knowledge base, who approved it, and what testing preceded it? |
| Incidents and corrective actions | What went wrong, how we classified it, what we did, and what remains open with an owner and a date? |
| Business outcome trend | How the operating measure agreed at pilot acceptance has moved, on the same baseline query. |
| Decisions required | What the sponsor must decide this month: threshold changes, scope changes, accepted risks, funding. |
Monthly governance reportSample, figures are illustrative
Groundedness / citation pass rate
98.1%Above threshold
Critical safety test pass rate
100%Threshold met
Human escalation rate
6.4%Watch
Unit cost per completed task
$0.11Within budget
Median latency
3.9sWithin service-level objective
Open corrective actions
1Action required
Changes released: 4 · Rollbacks: 0 · Incidents: 1 (severity 3, resolved) · Access reviews completed: 1 of 1
We measure adoption and value on the same baseline established in the governance sprint and confirmed at pilot acceptance: the same query, the same exclusions, the same caveats. Where a measure cannot be separated from other changes in the business, we say so in the report and claim no credit. A figure is publishable only if a named person can point to a dated artifact behind it.
Regulated industries
Guideline E-23, Model Risk Management, is effective 1 May 2027 and applies to all federally regulated financial institutions (FRFIs). It is principles- and risk-based, and it explicitly covers AI and machine-learning models as well as third-party models. A generative AI agent in a bank or insurer sits in those two categories.
| E-23 expectation | What AgentOps produces |
|---|---|
| Model inventory currency | A maintained inventory entry per agent and per dependent model, covering version, provider, purpose, owner, data classification, tools, downstream use and last validation date. We update it as part of change control, so it is current before any audit. |
| Ongoing monitoring | Continuous monitoring of quality, safety, latency and cost against thresholds, with defined triggers and alerting, and a documented response for each breach. |
| Validation evidence | Versioned evaluation datasets, dated regression results, coverage of the critical safety and permission-boundary suites, characterized failure modes, and residual risks with compensating controls. |
| Change management | Classified changes with named approvers, pre-release evaluation results, release records, tested rollback paths and a full change history per agent. |
| Third-party model risk | Provider, version and hosting recorded per model, with the shared-responsibility matrix and the jurisdictional position kept current as contracts and platforms change. |
Warning
White Glove Labs does not provide an audit opinion, an assurance opinion or an independent validation opinion, and nothing in an AgentOps report should be represented as one. We are the operating party. Independent review, effective challenge and the institution’s compliance position with E-23 remain with your model risk management function and your own independent reviewers. We will state that in writing to any reviewer who asks.
Source
Source: OSFI Guideline E-23, Model Risk Management. Effective 1 May 2027, it applies to all FRFIs, is principles- and risk-based, and explicitly includes AI and machine-learning models and third-party models. Sector context is on OSFI Guideline E-23 and AI agents; federal institutions should read when an AI assistant triggers an Algorithmic Impact Assessment (AIA) instead.
Service levels
We set service levels per engagement against the workflow's real criticality. The table below is the standard shape; we agree the numbers in the operating schedule and attach it to the contract.
| Commitment | Standard shape |
|---|---|
| Operating hours | Canadian business hours by default, with extended or on-call coverage priced as an option |
| Evaluation cadence | Scheduled full regression run each period, plus a mandatory run before any change is released |
| Severity 1 response | Permission-boundary breach, safety failure or unavailability. Containment first, notification to the named owner, corrective action recorded |
| Severity 2 and 3 response | Quality degradation, latency or cost breach. Triaged within the agreed window, remediated on a scheduled release |
| Change lead time | Standard changes on the agreed release cadence; emergency changes permitted with post-hoc review and a recorded rationale |
| Access review | Scheduled per period, plus on any role, entitlement or source classification change |
| Governance report | Delivered on a fixed date each month, with a review session with the sponsor |
| Exit | Handover of datasets, test bench, runbooks, inventory and history on termination, with no exit or handover fee after the minimum term |
A monthly fee against a defined operating schedule, minimum six-month term. We confirm final fees in writing before any engagement, and they vary with scope and access, and with security requirements and procurement route. Platform, model and infrastructure consumption charges are yours; we report them and add no markup.
Next step
One entry point is acceptance of a pilot we delivered. The other is an agent someone else built, running in production, with no golden dataset and no change record.
Source
Dollar figures on this page are the sources’ own or recorded amounts, not our prices. Our prices are in Canadian dollars (CAD), plus GST/HST.
In that second case we run a short baselining engagement first: build the evaluation dataset, establish current performance, document the control and access position, and agree thresholds. Only then does a monthly operating model mean anything. You cannot monitor drift from an unknown starting point.
Related reading: how to evaluate a knowledge agent for the method behind the golden dataset, and the Canadian Sovereign Agent Deployment Playbook for the whole delivery path. The control model is on security and governance. The workflow best suited to AgentOps first is the service-desk knowledge agent, because its volume makes drift visible early. From demonstration to operating service sets out the six lines every agent in service has to be able to fill in.
