Reasoning Protocols for Calibrating Frontier Models
Conflict Analytics Team ·
Despite the growing interest in legal AI, there is still much to be understood about how large language models (LLMs) are best used in the legal domain. Legal practice relies on processes and conventions that are often not written down due to its ever-evolving nature. This leads us to question what are the fundamental components of legal reasoning that survive even as the law evolves.
We believe the following:
- There is more to legal reasoning than factual knowledge. In the legal domain, reasoning is not just knowing the principles of the law, but also the ability to apply these principles in a rigorous manner. It is one thing to know the factors that could apply for a certain legal question, and another to apply the factors in a "reasonable" way.
- There is more to legal practice than what happens in court. Much of current legal AI research focuses on upper level court cases or controlled test environments such as legal exams. In both cases, legal reasoning appears less repeatable than what is seen by legal professionals in daily practice. We believe legal reasoning can be extracted and modeled, but court cases are just one data source for capturing the full spectrum of legal reasoning.
- Current AI solutions do not capture the experiential knowledge, such as settlement practice, that decides most outcomes. Recent paradigms in AI model training such as preference reward modeling or outcome reward modeling do not capture the thought processes lawyers engage in their argumentation process. The legal industry is one where the process is as important as the outcome, and the process can span years of dispute resolution. However, current solutions do not model the entirety of this process effectively. They also miss experiential knowledge, such as how disputes actually settle. By most estimates, fewer than 10% of disputes ever produce a written judgment [1,2], so the knowledge that decides most outcomes in areas like employment, torts and IP lives in settlement files and practitioners’ experience, not in the published record.
To all of this, we propose Reasoning Protocols, a new paradigm for structuring domain-specific knowledge that guides AI in long-horizon dispute resolution.
A reasoning protocol captures the fundamental components of a lawyer's thought process and calibrates any frontier model towards domain-specific conventions. The fundamental building blocks include gathering facts from the user, conducting research, and applying research back to the facts.
We extend CoDial [3], originally published in ACL 2026, to become a generalized representation of reasoning. CoDial was originally built around task-oriented dialogue, which modelled interactions between a user and a chatbot in accomplishing a task, such as resetting a password. The core philosophy - that a task can be represented as a repeatable schema - is generalizable to any downstream application, including the legal domain.
Through deep collaboration with lawyers, OpenJustice explicitly writes legal knowledge into reasoning protocols to calibrate language models in domain-specific settings. This has several use cases:
- Firm-Specific Calibration. Frontier models are trained on caselaw, or public court proceedings, but many disputes are resolved through settlement. Once anchored to common law, each individual firm has its own practices and conventions for how to proceed with a settlement. It is impossible to train a frontier model on this knowledge, since it is often proprietary to a firm, and the practices are also subject to change over time, even more than common law.
- Adherence to Legal Doctrines. A reasoning protocol encodes a branching schema of a certain topic or problem, which a language model can use to guide its reasoning trajectory. In the legal domain, this translates to repeatable doctrines or legal tests. For example:
Just Cause Dismissal routes through proportionality (McKinley v. BC Tel 2001 SCC 38), the three-step test (Dowling v Ontario (Workplace Safety & Insurance Board) (2004), 2004 CanLII 43692 (ON CA), 37 CCEL (3d) 182 (Ont CA)), and Wilful Misconduct under O. Reg. 288/01.
- Auditable, repeatable reasoning. We can view navigation through a reasoning protocol as an instantiation of graph traversal. This allows a more structured way to view previous AI interactions, where something went okay vs. where it went wrong. This gives a more rigorous way to identify where a model is well-calibrated or wrongly calibrated, and allows us to further iterate upon and refine the protocols.
More resources
References
[1] Miller, R. E., & Sarat, A. (1981). Grievances, claims, and disputes: Assessing the adversary culture. In Law & Society Review, 15(3–4), 525–566.
[2] Priest, G. L., & Klein, B. (1984). The selection of disputes for litigation. In Journal of Legal Studies, 13(1), 1–55.
[3] Radin Shayanfar, Chu Fei Luo, Rohan V Bhambhoria, Samuel Dahan, and Xiaodan Zhu. 2026. CoDial: Interpretable Task-Oriented Dialogue Systems Through Dialogue Flow Alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 42741–42763, San Diego, California, United States. Association for Computational Linguistics.