01 · Approved sources and classification
A named, versioned corpus with an owner per source.
- Fails when
- An unapproved source enters retrieval and gets cited authoritatively.
- Test
- Retrieval returns only approved source identifiers.
Playbook
A complete, twelve-chapter method for taking one high-value knowledge workflow from candidate to a production decision inside a Canadian public institution or regulated organization. Published in full on this page, with the supporting templates available on request.
Orientation
This is the method we run on engagements. It is written out so you can run it yourself, run it with someone else, or read it before deciding whether to run it with us. Work through it in order: each chapter produces a written artifact that the next chapter consumes. Chapters 1 to 4 decide whether to proceed. Chapters 5 to 8 decide what to build and how you will know it works. Chapters 9 to 12 decide how it runs, what it costs and how it gets bought.
A program owner, chief information officer (CIO), enterprise architect or risk officer. The institution can be Canadian federal or provincial, a Crown corporation, a health or education organization, or a federally regulated financial institution. It assumes an internal knowledge workflow where a human already reviews the output (briefing, triage, service-desk research) rather than an automated decision about a member of the public. That distinction shapes everything that follows.
Chapter 1
Residency is usually the first question asked about a Canadian AI deployment, and it settles a small part of what sovereignty covers.
It means your institution can state, in writing, who can compel, administer, observe, change, interrupt and recover the service, and can act on those answers. Residency tells you where data is stored and processed. Sovereignty covers legal jurisdiction; infrastructure ownership; administrator and support access; encryption key custody; model provenance; telemetry; update control; incident response; and exit. A Canadian region operated by a foreign-controlled provider may satisfy residency while leaving that list unanswered.
A central governed workload surrounded by eight control domains: data location and residency; legal jurisdiction; infrastructure control and administration; identity and support access. The other four are encryption and key custody; model and software control; operations and telemetry; and continuity and incident response. Residency is one control among eight, not a synonym for sovereignty.
Record the eight domains in writing. For each one, name the responsible party, the evidence you have seen, and the date you saw it. Domains where the entry is “not yet established” form the work list. An architecture review that covers one of the eight has not covered the question.
The obvious objection is that this is disproportionate for an internal drafting assistant. Often it is. Proportionality is the point: depth of evidence follows the classification and decision impact settled in chapters 3 and 4. But proportionality is decided after asking, not instead of asking. The long-form argument is in Canadian data residency is not AI sovereignty, and the control-by-control detail is on the sovereign AI page.
Chapter 2
Choose the first workflow for provability rather than for visible impact. You are buying an answer to whether your institution can operate a governed agent at all, and you want that answer cheaply.
Run every candidate through this sequence. Failing a gate does not disqualify a workflow forever; it disqualifies it from being first.
One where a person already reviews the output and a baseline metric already exists, with finite, classifiable source documents and 100 to 300 real historical cases to test against. In practice that means policy and research support; document intake and triage; or internal service-desk knowledge retrieval. Avoid anything that decides an entitlement, a benefit or a sanction for a member of the public as a first workload.
Score the survivors rather than arguing about them, using a weighted matrix like the worked example on the Governance Sprint page. Any consistent method puts disagreement into a cell where evidence can settle it. The three patterns documented on this site: policy and research support, document intake and triage, and the service-desk knowledge agent.
Chapter 3
Classification is a property of every artifact the system touches, including the derived copies nobody thinks of as data.
The characteristic failure is classifying the source corpus and approving the design. Then someone discovers that prompts, retrieved passages, embeddings, evaluation datasets, logs, caches and backups are all derived copies living somewhere else, under a different control regime. An agent architecture produces more copies of your information than a traditional application does, and each copy inherits the classification of what it was derived from.
| Artifact | Question to settle | Common oversight |
|---|---|---|
| Source corpus | Who owns and approves each source, and at what classification? | A single high-classification document in an otherwise low corpus raises the whole boundary. |
| User prompts | Where do they go, how long are they kept, who can read them? | Users paste sensitive context into an interface classified for public information. |
| Retrieved passages | Do they cross a boundary in transit to the model? | Retrieval is treated as internal even when inference is not. |
| Embeddings and indexes | Where do vectors live, and are they treated as derived content? | Vectors get filed as metadata rather than as a representation of the source. |
| Evaluation dataset | Is it real case data, and who may hold it? | Golden datasets built from real files end up on unmanaged laptops. |
| Logs and traces | What is captured, for how long, and who can query it? | Full prompt and response logging quietly creates a new records holding. |
| Caches and backups | What persists after deletion, and where? | Deletion in the application is not deletion in the platform. |
Two rules keep this tractable. Design to the highest classification present in the boundary, or remove the offending sources from it. The second option is often the cheaper one. Write down the personal-information position early: whether it is present, necessary and reducible. Record how it will be handled consistently with the Privacy Act and your institution’s privacy instruments. Involve your privacy office here, not at the acceptance review. Our control model is on the security and governance page.
Chapter 4
The Directive on Automated Decision-Making is narrower in scope than it is often assumed to be, and it reaches systems that often go unchecked. Screen deliberately and record the reasoning.
It applies to production systems that make or support administrative decisions or related assessments about clients. It excludes test and research systems. A retrieval assistant that returns passages and citations to an analyst, who then independently reasons to a recommendation, generally falls outside that scope. But the boundary is a judgement your institution makes and records, and the Algorithmic Impact Assessment is how you document it. Screen early; the answer shapes the architecture.
The screening questions are short. Is the output used in an administrative decision about a client? Does it produce a score, ranking, recommendation or eligibility signal? Would a reviewer in practice adopt it without independent reasoning? Production or research? Does it touch protected characteristics or proxies for them? If the answers point toward the Directive, complete an Algorithmic Impact Assessment and let the impact level drive the peer-review obligation. Levels II to IV require peer review, and qualified reviewers may come from government, academia, civil society, a contracted specialist, or a board specified by the Treasury Board Secretariat.
Warning
A low-impact assistant can become a high-impact system through drift. Someone adds a recommendation field, or an eligibility hint, or an auto-send path, and nobody re-screens. Put re-screening into change control from day one, so the trigger is the change, not the audit.
The full treatment is in when does an AI assistant trigger an Algorithmic Impact Assessment. Federally regulated financial institutions run the parallel exercise against Office of the Superintendent of Financial Institutions (OSFI) Guideline E-23, effective 1 May 2027. It covers AI and machine-learning models, including third-party models. See OSFI Guideline E-23 and AI agents.
Chapter 5
Six stages, each with a named owner, an acceptance test and a documented failure mode. A control with no acceptance test cannot be shown to work at an incident review.
Six connected control stages. One: approved sources and data classification. Two: identity and least-privilege tool access. Three: model and prompt controls. Four: evaluations and acceptance thresholds. Five: human approval and escalation. Six: monitoring, audit, incident and change control. Stage six feeds evidence back into every earlier stage, so a failed check in production reopens the controls that should have caught it.
A named, versioned corpus with an owner per source.
Entitlements enforced at the retrieval layer, not in the prompt.
Pinned versions, versioned prompts, recorded provenance and licence, tested rollback.
A golden dataset with numeric thresholds agreed before build.
An explicit decision point, one-click escalation, a logged outcome.
Traceable requests, classified incidents, change control across model, prompt, connector and corpus.
Stage 06 is the one a budget most easily leaves out, because it is the only one that requires continuing effort after launch. Budget for it in chapter 10.
Chapter 7
Three shapes cover most Canadian knowledge-agent workloads. Each buys certain controls and pays for them somewhere else. The useful work is naming the trade in writing.
| Dimension | Managed application programming interface (API), Canadian region | Private or dedicated deployment | Open-weight model on Canadian-controlled infrastructure |
|---|---|---|---|
| Control | Weakest. Jurisdiction, provider administrator access, telemetry and version changes sit largely outside your authority. | Middle. Tenancy, network path and key custody improve; the operator and its supply chain remain third-party. | Strongest. Model, telemetry, versioning and rollback are yours. No inference leaves the boundary. |
| Cost | Lowest to start. Pay per token with no floor. Cost scales directly with usage. | Higher fixed component, often a committed capacity or platform charge, with lower marginal cost at volume. | Dominated by graphics processing unit (GPU) capacity, which is paid whether or not anyone uses the system. Cheapest only at sustained volume. |
| Effort | Days to a working environment. Operating burden is mostly application-side. | Weeks. Networking, identity integration and provider assurance review dominate. | Months of capability build, or a partner. You own capacity planning, patching, evaluation and an operations rota. |
| Model quality | Current frontier capability, changing under you unless versions are pinned. | Usually the same family as the managed API, with more control over when versions move. | Strong and improving on retrieval-grounded knowledge tasks; still a gap on the hardest reasoning and on long multi-step tool use. |
| Exit | Application is portable; prompts, evaluation suites and embeddings usually need rework. Verify export of logs and indexes. | Contractual exit terms matter more than technical ones. Confirm what leaves with you and in what format. | Best position: weights, indexes and infrastructure are already inside your boundary. Exit is a migration, not a negotiation. |
Choose the cheapest pattern that satisfies the classification and decision impact established in chapters 3 and 4, and no cheaper. For unclassified internal knowledge work where the value case is unproven, a managed API in a Canadian region is usually the rational first move. Where model telemetry is a hard prohibition or the corpus carries higher classification, an open-weight model on Canadian-controlled infrastructure is the stronger position. Decide with a written trade-off, not a preference.
A fourth shape, splitting traffic by classification across two patterns, is legitimate. Treat the routing decision as an evaluated control with its own test suite and a fail-closed default: a misrouted request is a disclosure, not a performance issue.
Chapter 8
Acceptance is a number against a threshold agreed before anyone built anything. Without that, acceptance rests on a demonstration rather than a measurement.
Build the golden dataset from real historical cases: 100 to 300 items, stratified across the question types, source areas, languages and edge cases the workflow contains. The people who do the work label them. A named adjudicator resolves disagreements and the reasoning is recorded. Refresh it on a schedule and whenever the corpus or the workflow changes.
| Dimension | Measured as | Illustrative threshold | Gate type |
|---|---|---|---|
| Groundedness | Share of answers fully supported by the cited sources | Sample: 95% or higher | Scored |
| Citation accuracy | Share of citations that resolve and support the sentence they are attached to | Sample: 98% or higher | Scored |
| Task completeness | Rubric score against the reference answer | Sample: 4 of 5 median | Scored |
| Critical safety suite | Named prohibited behaviours, including disclosure across an entitlement boundary | 100% | Pass or fail |
| Permission boundary | Requests by an unentitled test user for content they may not see | 100% | Pass or fail |
| Abstention | Correct refusal when approved sources do not answer the question | Sample: 90% or higher | Scored |
| Latency | Median and 95th percentile to a usable answer | Sample: 5s median, 12s at the 95th percentile | Scored |
| Unit cost | Total inference and retrieval cost per completed task | Set against the baseline budget | Scored |
Two disciplines make the scorecard credible. Safety and permission suites are pass or fail, never averaged, because a 99% pass rate on disclosure means one in a hundred disclosures. And the institution signs every threshold before the build starts, so nobody can renegotiate acceptance at week six. You will find the full method, including red-teaming, bilingual evaluation and the release gate, in how to evaluate a knowledge agent.
Chapter 9
Six weeks, gated weekly, ending in a production decision with evidence attached. The time box prevents an open-ended demonstration that is neither completed nor stopped.
A gated six-week sequence. Week one: workflow baseline and acceptance criteria. Week two: secure access and identity, then the evaluation dataset. Week three: grounded workflow with citations and human review. Week four: safety and privacy testing, then bilingual, accessibility and performance testing. Week five: limited user rollout and operating feedback. Week six: acceptance report and production decision. Each week ends at a gate, and nothing moves on without its evidence.
Note
Any gate can stop the pilot. A stop at week two costs a fraction of a stop at month nine. It leaves the institution with a documented decision, a measured baseline, a classification screen and a reusable evaluation dataset. Write the right to stop into the engagement terms.
How we run this is on the six-week pilot page.
Chapter 10
Production creates a standing operating obligation. A budget that covers the build and not that obligation runs out at this point.
On the first day of production, five things become somebody’s standing job. Name that somebody before go-live, on both sides.
Two structural changes matter beyond the checklist. Re-screening against the Directive becomes an item in change control rather than an annual event. And for federally regulated financial institutions, the model enters the model inventory with an owner, a validation schedule and documented limitations, as contemplated by OSFI Guideline E-23. See Managed AgentOps.
Chapter 11
We publish no savings figures, payback periods or return-on-investment claims, because we have none we could defend. What we can give you is the cost structure, so you can model it with your own numbers.
The right unit is cost per completed task, meaning a task a human accepted and used. It is not cost per request and certainly not cost per token. The gap between those two is where budgets go wrong: a system that is cheap per request and needs three attempts and an escalation is not cheap.
| Variable | What it is | What moves it |
|---|---|---|
| Retrieved context volume | Tokens of source material sent with each request | Chunk size, number of passages retrieved, whether the corpus is filtered before retrieval. Usually the single largest driver. |
| Output length | Tokens generated per response | Response format, whether the agent drafts or summarizes, prompt instructions about length. |
| Turns per task | Requests needed before the user has something usable | Answer quality, interface design, whether clarifying questions are supported. |
| Tool and retrieval calls | Search, database and connector invocations per task | Agent design, caching, whether the plan is fixed or model-decided. |
| Model tier | Which model handles which step | Routing simple steps to smaller models; the routing itself must be evaluated. |
| Rework and abandonment | Tasks paid for that produced nothing usable | Groundedness, abstention behaviour, user trust. Invisible unless measured. |
| Human review time | Minutes of reviewer effort per task | Citation quality, escalation design, interface friction. Often larger than inference cost. |
| Fixed platform cost | Capacity, environments, monitoring, evaluation runs | Architecture pattern from chapter 7. Dedicated and self-hosted patterns front-load this. |
| Operating cost | Regression runs, change control, access review, incident handling | Change frequency and control depth. Real, recurring, and easy to omit from a business case. |
Model your break-even honestly. The fixed-cost patterns only become cheaper than usage-based pricing above a sustained volume you should calculate, with a dated set of assumptions, before choosing. Then re-run it: inference pricing has moved repeatedly, and a cost model needs a date attached to be usable.
Chapter 12
Settle the procurement route before the design is finished: it constrains scope, timeline and sometimes architecture.
Ask for evidence rather than assurances, and be sceptical of any claim of certification, approval or inspection that is not accompanied by a document you can read. What we can and cannot evidence today is set out in the supplier evidence pack, sent on request, and the buyer-side view is on the government page.
Working artifacts
Eight templates support the twelve chapters, each described here in the open. The files are working documents we send on request; the method is on the page, not inside a file.
Honesty
How we know this
Source
Next step
The cheapest way to apply this is to score one candidate workflow against the chapter 2 criteria and see which gate it fails.
Source
Dollar figures on this page are the sources’ own or recorded amounts, not our prices. Our prices are in Canadian dollars (CAD), plus GST/HST.
Bring the workflow as it runs today, the operating problem you want solved, and the classification if it is known. Bring the sponsor who can fund a decision, and name the systems involved. Thirty minutes, and you keep the notes whether or not we work together.
Related reading: AIA screening for AI assistants, how to evaluate a knowledge agent, residency versus sovereignty, and our control model.