Evaluating Reasoning Protocols for Dispute Resolution

We design a rubric for assessing long-form legal reasoning in litigation-oriented tasks. It consists of ten metrics: Citation Quality and Relevance, Factual Accuracy and Hallucination Control, Quality of Legal Reasoning, Strategic Relevance, Conciseness, Document Structure, Completeness of Legal Coverage, Internal Consistency, Professional Tone, and Substantive Accuracy to the expected outcome. Please see our repository for full details. 

We choose four of these metrics (Citation Quality, Factual Accuracy, Quality of Legal Reasoning, and Strategic Relevance) as we found they are the most indicative of a high quality answer; for example, Completeness is important, but a model response can mention the correct doctrines without applying them to facts. 

Case Study A: Alignment to Doctrines 

Here, we primarily focus on the core legal issues and how Reasoning Protocols can help align models to court decisions. With the same underlying model, we can improve the model’s performance by guiding it through a long-horizon legal research task.

We first defined the questions of interest, and recruited J.D. candidates to create synthetic evaluation data as hypothetical input-output pairs disputing a legal issue. We base these hypotheticals on real cases from the Ontario Court of Appeal, mainly to ensure there is a judicial standard for the intended resolution that the annotators can reference in their analysis.  

Each annotator is assigned a case where the chosen issue was the main dispute, and creates four artifacts:  

• a folder of synthetic documents that mimic the evidence used in the original case, with names and dates altered;  
• a timeline of the material facts as assessed by the lawyer; 
• a free-form gold answer;  
• and a variation of the documents that changes one or two facts without changing the gold answer. 

The gold answer is drawn from the original case, but the main task is to consolidate the legal arguments over the particular question, so it also reflects the student’s legal understanding. We derive cases from four litigation issues in employment law: Just Cause Dismissal (JCD), Length of Service (LoS), Enforceability of Termination Clauses (Contract), and Constructive Dismissal (CD). 

Figure 1. Just-cause dismissal, aligned comparison.

In Figure 1, we present a scenario with a properly-aligned reasoning protocol across three base models: GPT-5.4, GPT-5.4-nano, and Kimi K2.5. Citation gains are large and consistent across all three models. Legal reasoning improves on all three.  

Additionally, we convert Substantive Accuracy to Expected Outcome into a set of objective pass/fail criteria. Criteria are binary, case-anchored, and specific to the gold answer written by a human. A response passes only by naming the right authority, applying the right test, and reaching a defensible conclusion. Some examples include: 

• Applies Proportionality Principle — PASS if the analysis applies the proportionality principle from McKinley to determine whether dismissal is a proportionate response. FAIL if it relies on a different framework. 
• Identifies Lack of Fresh Consideration — PASS if it concludes the clause is unenforceable for want of fresh consideration. 

Cost Analysis 

We analyze the per-sample cost of the best-performing models in Table 3. 

Table 3. Mean cost per sample on Just Cause Dismissal. 

On the Just Cause Dismissal case study, GPT-5.4-nano with the reasoning protocol scores above plain zero-shot prompting Claude Opus 4.7 on every metric (except Strategic Relevance) and on substantive accuracy, at $0.131 per question against $0.155 — better outputs at roughly 15% lower cost. The protocol’s token overhead is affordable precisely because the model underneath is cheap. GPT-5.4 with the reasoning protocol reaches the highest scores at $2.650 per question, roughly 17x the cost of Opus with zero-shot prompting. That is a premium for mission-critical instances.  

Case Study B: Practice-specific Alignment 

After we anchor in legal cases, Reasoning Protocols can then be used to align recommendations to a firm-specific alignment. This case study focuses on termination notice, a legal issue that does have some court-based precedents, but also firm-specific settlement patterns. We create a reasoning protocol for the litigation issue of termination notice, which is common in employment law disputes. 

Rather than replacing the law, we use firm practices to augment the model’s reasoning trajectory, allowing LLMs to understand the small details that define a firm’s characteristic litigation styles. We specifically emphasize strategies that can be abstracted and generalized across cases — for example, anchoring with ranges instead of a single value during settlement negotiations. 

We extracted the fact pattern of a recent case (Talwar v. Best Buy Canada Ltd., 2025 ONSC 5814) that is not in our database. This is an Ontario wrongful dismissal file with an unusual number of interacting constraints: a 40-year-old repair technician with 15 years’ service and no discipline record, dismissed for alleged intentional access of pornographic files on a customer’s laptop; supervisory duties and direct customer contact; restrictive covenants; immigration status tied to the employer; post-termination statements to prospective employers; a chronic health condition; a contracting local repair market. 

Alignment to firm knowledge 

We also create a synthetic example of a previous client case based on this fact pattern, with a new synthetic name Bergman v. NovaTech Systems. This included the key files of litigation such as demand letters, mediation briefs, and minutes of settlement. These synthetic files serve as a place where the Reasoning Protocol can draw insights. 

We find Claude Opus 4.7 is able to generalize and synthesize high-level strategies from a client file, which can help advise future settlement negotiations on employment law. Below are some examples:

• Anchor notice as a defined range rather than a point figure. NovaTech opened at 10-12 months, not a single number. Mediation Brief of the Defendant at paras. 1-8, 11… 
• Keep ancillary damages heads off the notice line and address them with their own measurement rules. NovaTech separated equity from notice and insisted equity be measured “at scheduled vesting dates rather than speculative later dates.” Mediation Brief of the Defendant at paras. 6-7, 11… 
• Resist aggravated and punitive damages as a separate head requiring independently actionable conduct and proven harm. NovaTech deployed Wallace v. United Grain Growers Ltd. and Whiten v. Pilot Insurance Co. through its brief. Mediation Brief of the Defendant at paras. 8, 11… 

Key Insights 

We prompt Claude Opus 4.7 with the fact pattern (in timeline form) and perform two evaluations: first against the criterion from the original paper with an LLM judge, then again with human annotations by a practicing lawyer. The key LLM judge metrics are summarized in Table 5. 

Table 5. Post-decision assessment, 1–5 scale, with the pre-decision scores for comparison. Memo A is the protocol output; Response B is the raw model. 

In the human evaluations, most of the comments in the Plain LLM response are about unsubstantiated claims or unsupported arguments. The LLM tends to make points that sound plausible but are not supported by evidence. One example is Inducement: — this is a Bardal factor that is not engaged if a person’s tenure is greater than 15 years, but the plain LLM mentions it is applicable while the protocol-guided answer explicitly discards it. 

Final conclusion range. LLMs tend to hedge in their responses, but this does not provide useful information to lawyers when the caselaw is clear. The correct outcome (18 months) was in the predicted range for both responses, but the plain LLM hedges for 18-24 months. In contrast, our RP-calibrated response gives a narrower 18-19 months. 

Plain LLM Favors Breadth over Accuracy. The result reproduces the pattern from the benchmark: citation and legal reasoning rise sharply under the protocol (+1.50 each on a 1–5 scale), and strategic relevance falls (−1.00). The plain response scores well on strategic relevance for surfacing strategies that are not embedded within the test reasoning protocol: allocation between retiring allowance and non-taxable general damages, or early mediation. Our legal expert evaluator validated that these are strategically valuable. We did find the memo drafted under the protocol could mention adjacent legal issues, e.g. Just-cause dismissal, but not to the same degree as the plain LLM call. 

Constraining LLM to Reasoning Protocol does not solve legal reasoning. While the model adheres to the factors more cohesively when calibrated with the reasoning protocol, we note that there are minor issues in style and logical coherence. For example, it cycles between which Bardal factor is the most important to calculating the outcome between paragraphs. It also does not surface some of the strategic recommendations made by the Plain LLM response, likely because the reasoning protocol decreases the diversity in the response.  

Next Steps 

Reasoning protocols work, but today they are expensive to build. Each protocol in this study was written through close collaboration between lawyers and researchers, and that kind of manual encoding does not scale. Most firms do not have their knowledge in an AI-ready form: settlement history sits in closed files, memos and the judgment of senior lawyers. Building protocols firm by firm would require a significant investment in structuring that knowledge. The lab’s next priority is to automate this step, turning a firm’s existing files and expert input into draft reasoning protocols that lawyers review and correct rather than write from scratch. 

The longer-term direction is to move from protocols that steer frontier models to small, specialized models built from the protocols themselves, hosted on sovereign infrastructure so that firm knowledge stays under the firm’s control. Our cost results already point this way: a small model with the right protocol outperformed a frontier model at lower cost. Training and serving such models at scale requires dedicated compute. To that end, the lab is pursuing a supercomputing project through the Canada Foundation for Innovation (CFI), together with Amii, Mila, Vector Institute and NVIDIA.