Contract quality and AI: garbage in, garbage out?

Artificial Intelligence (AI) is poised to disrupt the way we draft, negotiate, review, and perform under contracts. AI has already significantly impacted the contracting process.¹ Some say the primary obstacle to tremendous advances in AI for contracting is the lack of training data. We argue that the problem is not data quantity, but rather data quality. To make the most of AI for contracting, we must make sure we do not create “garbage in, garbage out” AI systems.

Currently we have a disconnect between people developing AI for contracting and people working to improve contracting through simplification and redesign. By ignoring these new trends, developers of AI are missing out on opportunities to create something better.

A narrow focus on today’s urgent problems and risks is taking us to an unwise destination. We are not on a path toward improving the quality and value of AI contracts and contracting processes for organizations and society.

What to do? In this article we reveal some of the problems that stand in the way of building a better future for AI contracting based on today’s best practices and available data. We also highlight the need for quality and value metrics to get the most out of AI for contracting.

AI is making contracting faster, cheaper, and … better?

Training data drives AI success. But will access to more contracts and widely used clauses lead to success with AI for contracting? Today’s training data comes from current contracts. Research and experience tell us that current contracts are not optimal–in fact, they lack in many respects. AI alone cannot tell good contract-related training data from bad or make recommendations on how to improve contract content.

So, is the tail wagging the dog? AI and data analytics could help improve contract quality and value, but first we need human input to get there.² First, we humans must establish our goals (our mission and vision, if you will) and agree what these concepts of quality and value mean. If we don’t, we cannot rigorously evaluate current contracts and contracting processes, training data for AI systems, or the outputs of AI systems.

Do not develop AI without closely examining your training data

The best AI systems excel at learning from training data. In this sense, an AI system is like a mirror, reflecting an image based on its training data. For example, when we create an AI system to automate a hiring process, it ought to be no surprise when the AI system exhibits biases in current hiring processes and perhaps amplifies them.

Likewise, AI-enabled contracting systems will mirror our current processes and contracts as represented in training data. But do our current contracts and contracting processes reflect the quality and value that we aim to generate in the future?

Numerous studies suggest that current contracts fall short in many respects. Contract authors’ drafting habits have led to documents that are far from optimal–in fact, documents that many users find incomprehensible.³ We must watch out for the risks of accepting, without question, current contract content, language, and style. Although these might be legal-friendly–helping to manage legal risk or win a dispute in court–they are not business-friendly. In fact, in many cases, they work as obstacles rather than business enablers.⁴ Whether produced by humans or machines, such contracts can seriously harm relationship quality and contract value.

Common sources of training data will not take us where we want to go

When companies develop AI-enabled contract generation and review tools, where do they obtain their training data? It’s no secret that many begin with publicly available contracts, such as those filed with the U.S. Securities and Exchange Commission (SEC) in the EDGAR database. An AI tool creates a first draft of a contract or extracts clauses during a contact review based on what it learns from its training data.

We would not be the first to suggest that a publicly available repository like EDGAR may be a suboptimal training dataset in many respects.⁵ Many inhouse templates and clause libraries have the same problems as those found on EDGAR. Lawyers might think they have gotten contracts right, while businesspeople would disagree. If technology projects are led by or worked on with lawyers, they might be happy with the results, but the outcome might be far from ideal for other stakeholders.

IACCM surveys show that year after year, limitation of liability, indemnification, and other risk mitigation clauses top the list of most negotiated terms, even though these are not considered to be the most important terms^.6 Consequently, most negotiations are suboptimal–they focus on the wrong subjects. Risk mitigation terms prevail, seeking to minimize the negative consequences of failure rather than securing success.⁷

Research about the psychological effects of contracts on relationship quality hints that effective contract design is not only a matter of selecting the right clauses, but also of communicating them in the right way to promote trust, creativity, flexibility and collaboration.⁸

Research also tells us that contracts could enhance relational governance mechanisms and inter-organizational relationship performance, yet the careless use of contracts can negatively affect both.⁹ Which of these classes of contracts do we want to add to our training set for AI systems?

Moreover, many analytics and AI failures can be traced to the use of training tools that do not represent the environment where those tools are deployed.

• For example, if a dataset in which most of the photos are men is used to train a facial recognition system, it should be no surprise that the system works better for men than it does for women.
• Likewise, if a contract review tool is trained and tested mostly on contracts in metals and mining deals, it should be no surprise if the tool doesn’t work as well on software deals.
• Similarly, if a tool to automate the negotiation of contracts is trained on data representing the negotiations where the parties are large companies with lawyers on both sides, it should be no surprise if the tool doesn’t work as well when smaller companies or individuals with no legal assistance are involved.
• In addition, jurisdictions vary, and countries have different contracting cultures and practices. For example, German business contracts are known to be much shorter than their American counterparts.¹⁰

A related problem with contract repositories is the need to differentiate between terms found in templates and in clause libraries and terms appearing in negotiated agreements. If we want to create initial drafts of contracts or automate the negotiation of contracts–unless we work within a large organization with access to different versions of its contracts–we may not have the right data. Some have found ways to overcome these obstacles, but it’s not clear whether or not these workarounds will generate satisfactory results.

A public dataset for training and benchmarking

Establishing a public dataset of contracts for training and benchmarking AI tools for contracting is one possible way to help address the lack of metrics for quality and value. A focus on a common dataset has fueled progress in other areas, such as the Enron dataset of emails for E-discovery¹¹ and ImageNet for object classification.¹² This public dataset would provide a common source for training, evaluating, and benchmarking systems. But such a project would not completely address the lack of quality and value metrics.

To illustrate this, if we could create an AI system that generates draft contracts that experts could not distinguish from the actual contracts in a public repository, would we want to use these contracts? Whether our answer is yes or no, our goal must be to move beyond anecdotes and intuitions and instead use empirical methods to demonstrate the quality and value of contracts and contracting processes.

First, we need to agree on the meaning of contract quality and value

The biggest problem, as we see it, is not about access to data or quantity, it is about quality. We have no standardized methods for assessing contract quality and value. Scholars and practitioners have only recently started to discuss the purpose and functions of contracts, and there is no agreement about what “good” or “fit for purpose” means or looks like.

Different disciplines value different things, and not all professionals within one discipline agree either. Lawyers will tell you that contract quality varies widely, but if you ask them to describe this variation, few can move beyond untested lawyer norms and personal anecdotes.

Contracting today is not a data-driven exercise. Likewise, law practice is not an evidence-based endeavor.¹³ As a consequence, we lack the foundational knowledge required to transform contracts and contracting processes to deliver greater value for individuals, organizations, and society.

There are many ways to view contract quality. If we look at contract documents one at a time, we might focus on words and sentences and use readability formulas to measure text quality. We could look for legalese and count the number of complex words and long sentences. Yet research and experience tell us that readability formulas are not optimal methods for assessing quality.¹⁴

Looking at documents through the eyes of a contracts professional or language expert is one thing; but, how intended audiences see them may be entirely different. However short the sentences and easy-to-read the language, people just may not bother to read. If they do, they may still not find what they are looking for, understand what they find, or be able to apply it to the task at hand, even if they are expected to.

While there is nothing wrong with measuring readability (and measuring it is reasonably easy to automate), it is insufficient for our purposes. Usability and functionality matter more, and, like in any other context, user experience must be considered as well.

Joining forces with contract designers

Good contract training data cannot be judged on measuring text qualities alone. When we shift our focus from crafting clear wording or sentences to creating effective communications that produce desired business benefits and good relationships, it becomes clear that there is more to consider.

We must also think of the intended audience and put the users and their needs and expectations at the center. This is not necessarily familiar territory for conventional contract drafters or people developing AI for contracting.

The good news is a growing number of people already work in this space: contract designers. They may call themselves information designers, plain language advocates, innovation evangelists, or transformers–we collectively refer to them as contract designers.

IACCM offers resources on contract design, simplification, and visualization.¹⁵ A growing number of examples exist that lead the way to new contract genres–yet they have not yet impacted the contracts with which AI software algorithms have been trained. We argue that they should.

In our previous work, we have explored the need for measures of quality and value and ways to make contracts more useful and usable by design.¹⁶ In addition to making sure intended users can find, understand and use contract content, contract designers can contribute to the development of tools that make it easier to produce such content. They can bring to the table skills and tools that complement those of contract professionals, lawyers, and technologists. As designers, many of them are experienced in prototyping as well as in choosing and using different evaluation methods. They can help select the most appropriate methods among text-, user-, and outcomes-based criteria.

So, no more garbage in, garbage out. Working together, contract professionals, technologists, designers, and users can co-create criteria and metrics that work, ensure that contracts meet users’ needs and the contracting organizations’ goals, and show how much value is actually produced. This offers everyone an opportunity to measure the success of their work and obtain research-based evidence to show return on investment for clients.

What’s next? Our intended destination

Advances in AI are shaping contracting in unprecedented ways, allowing us to see many things in a new light. As examples:

• We now have access to tools and guidance like never before.
• Automated reviews can flag the presence or absence of contract language in seconds.
• We can create templates and forms quicker than ever based on access to libraries of thousands of clauses.

But, again, quantity does not guarantee quality. We need to stop and think about where we are and where we want to go. We can ask what we value in our contracts and relationships and determine what matters the most.

We can improve AI by incorporating approaches to improve contract quality and value. In addition to contracts’ legal objectives, we need to focus on their business objectives and their role as a communication tool. A huge opportunity is being missed if firms developing AI-enabled contracting solutions and companies implementing them do not consider the world’s next generation contracts and focus on contract quality and value.

Can we afford to continue to muddle along? No. We must establish a vision for the future of contracting; employ scientific rigor and empirical methods; and test our ideas, learn, and continuously improve to reach our intended destination.

END NOTES

1. See, for example, Beverly Rich, How AI Is Changing Contracts, Harvard Business Review (Feb. 12, 2018)
2. For approaching AI as a human-machine partnership, “human led, machine assisted”, see Paul Branch and Peter Wallqvist, Artificial Intelligence partners with humans to create BT success story, Contracting Excellence Journal (Jul. 5, 2019)
3. For the causes and consequences, see Wendy Wagner & Will Walker, Incomprehensible! A Study of How Our Legal System Encourages Incomprehensibility, Why It Matters, and What We Can Do About It (Cambridge University Press 2019)
4. Kate Vitasek, Haapio and Barton: Business-Friendly Contracting, Future of Sourcing (Jun. 23, 2017)
5. See, for example, Ken Adams, EDGAR and Me, Adams on Contract Drafting (Oct. 18, 2015) and Automated Review of Contracts: Some Thoughts on LawGeex’s AI-Versus-Humans Study, Adams on Contract Drafting (Mar. 13, 2018) . For the reasons of poor contract quality, see Ken Adams, Why Are Templates Bad? Adams on Contract Drafting (Mar. 28, 2020)
6. IACCM Most Negotiated Terms Report - 2018 (Top Terms)
7. Kate Vitasek, Libby Weber: The Importance of Framing Your Contracts, Future of Sourcing (Mar. 23, 2020)

8. Stefania Passera, Anssi Smedlund and Marja Liinasuo, Exploring Contract Visualization: Clarification and Framing Strategies to Shape Collaborative Business Relationships, 2(1-2) Journal of Strategic Contracting and Negotiation 69 (2016)
9. Anna Hurmerinta-Haanpää and Sampo Viding, The Functions of Contracts in Interorganizational Relationships: A Contract Experts’ Perspective, Journal of Strategic Contracting and Negotiation (2019)
10. Claire A. Hill & Christopher King, How Do German Contracts Do as Much with Fewer Words?, 79 Chicago-Kent Law Review 889 (2004)
11. See Maura R. Grossman and Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, 17(3) Richmond Journal of Law and Technology, article 11 (2011)
12. Dave Gershgorn, The Data that Transformed AI Research–and Possibly the World, Quartz (Jul. 26, 2017)
13. See Dan Linna, Evaluating Legal Services: The Need for a Quality Movement and Standard Measures of Quality and Value - Draft Chapter for Research Handbook on Big Data Law, LegalTech Lever (Mar. 12, 2020)
14. Karen A. Schriver, Research review finds most readability formulas outdated and overrated. Center for Plain Language (Feb. 7, 2018)
15. IACCM Contract Design and Simplification and the IACCM Contract Design Pattern Library
16. See, for example, Dan Linna (note 13), Helena Haapio, Next Generation Contracts: a Paradigm Shift (Lexpert Ltd 2013), and Helena Haapio & Margaret Hagan, Design Patterns for Contracts: or How You’ve Been Doing Contracts Wrong This Whole Time (and How to Fix It), Juro Blog (Jul. 1, 2019)

About the Authors*

Helena Haapio is a contract lawyer and an advocate of contract simplification and redesign. She is an Associate Professor of Business Law at the University of Vaasa, Finland, and a Contract Strategist at Lexpert Ltd. In addition to her LLM and other degrees, Helena is proud to hold a Master of Quality degree and be a Fellow of the IACCM. She is the author and editor of several books and a co-founder and co-creator of the IACCM Contract Design Pattern Library. She is currently co-editing two books for Edward Elgar, Research Handbook on Contract Design and Legal Design: Integrating Business, Design and Legal Thinking with Technology.

Daniel W. Linna Jr. has a joint appointment at Northwestern Pritzker School of Law and McCormick School of Engineering as the Director of Law and Technology Initiatives and a Senior Lecturer. Dan is also an affiliated faculty member at CodeX – The Stanford Center for Legal Informatics. Before joining academia, Dan was an equity partner at Honigman, an Am Law 200 firm headquartered in Detroit. Before studying law, Dan was an information technology developer, manager, and consultant.

 * Thank you to Northwestern Pritzker School of Law student Research Assistant Katie Nagley for her contributions to this article.

 

Return