The business case for a six-week knowledge agent pilot

An uncomfortable opening for a business case, and the only honest one. It gives a budget holder more to work with than a supplier’s confidence.

Canadian business sector labour productivity fell 0.5 percent in the first quarter of 2026, after a 0.3 percent decline in the fourth quarter of 2025. Those are the figures Statistics Canada published on 3 June 2026. Unit labour costs rose for a fourth consecutive quarter.

The longer picture is worse. Statistics Canada research published 22 December 2025 puts Canadian labour productivity at about 83 percent of the United States level in 2002 and 73 percent in 2019. Annual growth ran at 0.84 percent against 1.56 percent.

Adoption is moving fast against that backdrop. Statistics Canada publishes annual analyses of artificial intelligence use drawn from the Canadian Survey on Business Conditions. The share of businesses that reported using artificial intelligence to produce goods or deliver services in the past twelve months went from 12.2 percent to 19.2 percent. The first figure comes from fieldwork from 1 April to 5 May 2025 (analytical article published 16 June 2025, 9,103 respondents). The second comes from fieldwork from 1 April to 6 May 2026 (analytical article published 11 June 2026, 9,251 respondents). The headline figure first appeared in The Daily on 27 May 2026. Same survey, same question, comparable base: seven percentage points in one year.

Four controlled studies, all published, all with stated samples, and they do not agree.

The National Bureau of Economic Research (NBER) reported a field deployment of a conversational assistant to 5,179 customer support agents in working paper 31161 (April 2023, revised November 2023). It found issues resolved in an hour rose 14 percent on average, and 34 percent for novices. The effect on experienced high performers was minimal.

Organization Science published a preregistered randomized trial with 758 knowledge workers. On tasks inside the model’s capability frontier, participants completed 12.2 percent more tasks, 25.1 percent more quickly, at significantly improved quality. On a task deliberately placed outside that frontier, participants using the tool were 19 percent less likely to produce a correct solution. Disclose this when you cite it: three of the nine authors are affiliated with the consulting firm that supplied every participant, and that firm sells artificial intelligence advisory services.

An incentivized experiment with 453 college-educated professionals was published in Science in July 2023. Average time taken on mid-level professional writing fell 40 percent while assessed output quality rose 18 percent.

And a randomized trial by the research non-profit METR (Model Evaluation and Threat Research) was published 10 July 2025. With 16 experienced open-source developers across 246 completed issues, it found developers using artificial intelligence tools took 19 percent longer. They had expected a 24 percent improvement, and afterwards still believed they had achieved one. Small sample, and the authors say so. The perception gap is the finding.

Then the Canadian firm-level evidence, the most important item in this list. Statistics Canada research published 22 April 2026 used the Survey of Digital Technology and Internet Use, pooled across 2019 and 2021 and linked to administrative business microdata. It found adopters showed a 16.8 percent higher productivity level at baseline, and 10.2 percent controlling for initial productivity. Once complementary capabilities were controlled, the gap fell to 5.1 percent, which is statistically insignificant. The authors’ own conclusion: “there is no statistically significant direct association between AI adoption and productivity,” and the earlier premium “reflects the critical role of complementary investments.”

Note the vintage. That data is from 2019 and 2021, before the current generation of tools, so it does not settle the question for today. What it establishes is that the organizations that got a productivity result had already built the complementary capability, and the ones that bought the technology alone did not.

You cannot read your way to knowing which of those results describes your organization. You can only measure it.

Neither Statistics Canada nor the Treasury Board of Canada Secretariat publishes a figure for typical pilot duration. No named research organization with a stated method gives one either. Anyone quoting a norm is quoting a habit. So six weeks has to be justified on design grounds. It can be.

Six weeks is long enough to establish a baseline, run a real workload and observe a change. It is short enough that the baseline is still true at the end, and that it lands inside one budget cycle and one set of staff assignments.

The failure mode to design against in pilot reporting is the absence of a documented before state. A pilot that cannot say what the work took, cost or got wrong before the tool arrived produces a demonstration people enjoyed and nobody can fund against. A baseline is cheap in week one and impossible to retrofit in week six.

The Treasury Board of Canada Secretariat’s Guide on the Use of Agentic Artificial Intelligence, last modified 22 May 2026, points the same direction. It says to “start in a controlled environment and expand gradually to increase understanding of the agentic process.” It also calls for periodic manual performance by human experts for comparison, regular audits, and spot checks “to catch slippage early.”

The Secretariat’s Business Case Guide, which carries no published date, states its own applicability without a threshold:

This guidance applies “regardless of the complexity and risk of the proposed investment, and whether or not Treasury Board project approval is being sought.” It “should be used to guide the development of the investment’s business case.”

There is no size below which the structure stops applying. Its three phases and five steps are, conveniently, a workable set of terms of reference for a six-week pilot. Step one is strategic context and business need. Step two is preliminary options analysis. Step three covers viable options with costs, a cost-benefit analysis and risk. Step four is justification and recommendation. Step five is managing the investment: governance, outcome management, risk, business change management and performance measurement.

Step three’s cost-benefit analysis and step five’s performance measurement strategy are precisely what the pilot exists to populate. Write the business case skeleton first and let the pilot fill it in, rather than reverse engineering a case from whatever got measured.

One threshold to watch while scoping. The Secretariat’s guidance for drafters of Treasury Board submissions, last modified 1 April 2026, says proposals involving more than 10 new full-time equivalents require an appendix to the submission.

The Government of Canada AI Register, released 28 November 2025 and last updated 28 April 2026, carries 412 records on a count taken directly from the open data service. It describes itself as “a minimum viable product” and notes that “not all entries are complete.” It excludes artificial intelligence embedded in low-risk commercial products. Its builders assembled it from records that already existed, not from a fresh survey.

Western University maintains a separate inventory, the Tracking Automated Government register. As of 1 February 2024 it found that only 4 percent of 303 automated federal tools had publicly available government assessments. That figure predates the AI Register and describes a different inventory. It is still the closest published measure of how much of this activity anyone has documented, and University of Toronto researchers cite it in their 2026 analysis of the register.

That makes three independent reasons to read the register as a floor, and a reasonable prior that your own inventory undercounts what your staff use.

Four artifacts. A pilot producing only the first is a demonstration.

A documented baseline, recorded before anything is deployed: volume, time, cost and error rate on the target workflow.

An evaluation set with ground truth, built by the people who do the work. This outlives every model you will ever choose and is the most durable thing the six weeks produces.

A measured result against that baseline, reported with its limitations, including the cases where the tool made things worse. A pilot reporting no failures did not look.

A decision memo saying go, stop or extend, with the evidence for each, written by someone accountable for the outcome rather than for the pilot.

Value here does not come from the best technology. It comes from deciding in week one what would count as evidence, then honouring that decision in week six, including when the evidence says no.

Previously: open-weight versus proprietary models, a Canadian cost model. This concludes the series. The whole arc is listed on insights.