Bilingual evaluation for Canadian government AI workflows

The problem is evaluation design rather than translation, and it has a specific shape.

Under the Official Languages Act, federal institutions must communicate with and provide services to the public in both official languages where the obligation applies. In DesRochers v. Canada (Industry), 2009 SCC 8, decided 5 February 2009, the Supreme Court of Canada read that duty as substantive equality rather than mere availability. It stressed services of equal quality in both official languages. The federal official languages policy suite carries that equal quality formulation into departmental obligations. The standard is not that a French output exists.

An artificial intelligence workflow can satisfy the first and fail the second in at least four ways, and a single overall accuracy score will hide all four.

Retrieval asymmetry. If the corpus the system searches is ninety percent English, a French question will retrieve fewer relevant documents, or it will retrieve English documents and answer from them. The answer may be correct. It is still a different service.

Terminology drift. Government French is not general French. Prestation, allocation and indemnité are not interchangeable. A model that treats them as synonyms produces text a program officer recognizes instantly as wrong and a member of the public does not.

Register collapse. French administrative writing carries formality conventions English does not. A model trained mostly on English patterns often produces a French that reads as translated. Comprehension survives. Trust does not.

Refusal asymmetry. Teams may tune safety behaviours mainly on English. The same question may get an answer in one language and a refusal in the other. That produces a service difference no accuracy metric will show you.

The most consequential decision comes before any model choice. Do you author the evaluation set in both languages, or author it in one and translate?

Translated evaluation sets systematically understate French failure, because a translated question carries English sentence structure that makes retrieval easier than a natively phrased French question would. If your French test cases were written by translating English test cases, your French pass rate is optimistic and you do not know by how much.

The practical rule. Author a meaningful portion of the evaluation set natively in French, drawn from real French language enquiries your institution already receives. Do not back translate them into English for parity. The two sets are allowed to differ, because the population asking in French is asking different things.

Report every one of these separately by language. Never report a blended figure.

Retrieval coverage. For a set of questions in each language, what proportion of the documents a human expert would consider relevant did the system retrieve? This is the first place to look for asymmetry.

Factual accuracy against a French language ground truth. Graded by a bilingual subject matter expert, not against a translation of the English rubric.

Terminological fidelity. Does the output use the institution’s own approved French terminology? Objectively checkable against existing glossaries, which makes it quick to find and cheap to fix.

Register and readability. Assessed by a francophone reviewer against the institution’s own published communications, not against a generic readability score.

Refusal and escalation parity. Run the same sensitive or out of scope prompts in both languages. Any divergence in whether the system answers, declines or escalates is a service inequity and should be treated as a defect, not a curiosity.

Time to correct answer, measured from question to answer and including any human review step. If the French path routes to a smaller reviewer pool, the service is slower even when the model performs identically. The public experiences that number.

The Directive on Automated Decision Making requires an Algorithmic Impact Assessment before a system goes into production. It scales approval authority with impact level. The Secretariat’s assessment tool guidance describes publication in both official languages on the Open Government Portal. Nothing we can find in those instruments asks you to disaggregate your evaluation by official language. Nothing in them stops you either. An assessment that reports a single blended accuracy figure for a bilingual public facing service has not assessed the service.

The Treasury Board of Canada Secretariat last modified its Guide on the Use of Agentic Artificial Intelligence on 22 May 2026. It asks institutions to “keep logs of tools used, actions taken, approvals given and key inputs/outputs.” Language is a logging field. If your logs do not record the language a request arrived in, you cannot produce disaggregated evidence later. You will rebuild it from nothing at the worst possible moment.

The National Institute of Standards and Technology Artificial Intelligence Risk Management Framework organizes this work under govern, map, measure and manage. Bilingual evaluation sits squarely in measure, and the framework’s insistence that measurement be tied to context is precisely the point. The context here is a statutory obligation, not a preference.

Weeks one and two. Collect real enquiries in both languages from existing channels. Have a bilingual expert author the French set natively. Target a minimum of one hundred cases per language, weighted toward the enquiry types that dominate your volume. A single-language workflow is sized at one hundred to three hundred cases on the evaluation method. A bilingual one needs that floor twice over. A French set drawn as a share of an English-sized total is too small to move a threshold.

The six weeks below are the evaluation track, not a separate engagement. Inside the six-week knowledge agent pilot this work runs alongside the build rather than after it, which is why the French cases have to exist in week one.

Week three. Build the ground truth. For each case, record the correct answer, the documents a human would cite, and the approved terminology. This artifact outlives any model you choose, which makes it the most durable thing the pilot produces.

Week four. Run the baseline. Report all six measures by language. Expect a gap. The gap is the finding, not the failure.

Week five. Attack the largest gap only. If it is retrieval coverage, the fix is corpus work rather than model work: locating the French language source documents that exist but were never indexed.

Week six. Re run and write it down. The deliverable is a document, not a score, saying what was measured, in which language, by whom, against what standard, and what remains open.

Say so, in writing, before launch. A documented decision to launch with a known and quantified service gap, with a remediation plan and a date, is a defensible governance position. An undocumented gap discovered by a member of the public is not.

Handling this well depends less on the model than on two decisions. Treat French as a first class test condition from the start, not a localization task. And write down what you found either way.

Previously: a practical Protected B AI architecture checklist. Next in this series: from demonstration to operating service.