The Redline Revolution: Agiloft's Acquisition of Screens & The Future of CLM With Otto Hanson

Made with speech-to-text software, so a few words may be off.

0:00You're listening to the Contract Heroes Podcast, your one-stop shop for all things contract management. And now here are your hosts, Marc and Pepe. Well, Otto, thanks so much for joining us on another episode. I know it's been quite some time since we had you on the, the show, but really excited to learn more about Screens and the new partnership with Agiloft. But I think before we talk about the partnership, we'd love to get a, a little bit more information on kind of the inception of Screens, where the idea came from. So yeah, maybe we'll, we'll use that as a jumping-off point. Great. Well, thank you both for having me. It's been a pleasure following along on your journey and listening to many episodes. You guys have interviewed so many great people, and this has just become one of the great podcasts in our space. So really delighted to be here again. The inception of Screens, as you know, we've been working on the contract review problem and, in particular, the pre-signature contract review problem for many, many years now.

0:56And one of the big challenges with pre-signature contract review is that everybody has different standards. Every company is different. Every l- lawyer is different in what they care about and what they're looking for. And pre-generative AI, it was really hard to build a contract review product that was both accurate and flexible. And at TermScout, pre-Screens, we spent a lot of time trying to nail the accuracy piece, but it was... But the trade-off we kind of had to choose was accuracy over flexibility, and pre-generative AI, that seemed like the right move. But when generative AI came out, it became possible to be both accurate and flexible in, in enabling people to develop contract review s- solutions that work for them. So we saw an opportunity to build something that would enable any expert in any type of contract to program a contract review solution, an AI instance of sorts.

1:54You can think about it like a playbook, but an operationalized AI version of a playbook that can review contracts just like them or, in other words, for the standards that they care about. And this was always, I think, has always been the crux of pre-signature contract review, and it's a crux that generative AI opened the door to in a big way and really changed the dynamic for us. So when ChatGPT-4 came out in particular and everyone saw the benchmarks and how it was performing, we all thought, "Oh my goodness, is there a way for us to build something that's much more flexible, that really lets any company build an AI that really creates contracts for their own specifications?" Thus was born Screens, and basically we were... You know, we split the TermScout team in two back in mid-2023, and half of our team went off and started building Screens. And we built a prototype. We launched it. It was really well-received, but it was just a, um, Word add... Sorry, it was just a web application at the time. But it proved that you could do this, this...

2:58achieve this holy grail of contract review, which is ultra-customizable and also really accurate. And then we went forward and built the Microsoft Word add-in, and once we released the Word add-in, it grew really, really fast. Customers really, really liked it. It got used more than anything we'd ever released. It's like we've released a number of products that sold well, but, like, usage, it's... Y- we all know in this industry, it's like, it's one thing to sell it. It's another thing to get users to actually change, you know, change the way that they work, adopt it, and use it on a regular basis, and the usage metrics were really, really exciting to see as well. So that's a little bit about the why and the how and the history that got us to where we are today. I think that was brilliant, actually, because, you know, as lawyers, Word is our day-to-day bread, and that's the, our main tool to work, either if you're a contract manager of, of you work, working in litigation, whatever you want. So if when you have a plugin that just connects directly to your main tool at work, I think that was a brilliant idea. Well, I saw the demo. I really love it.

4:04I told you, Otto, and one of the things that when I talk to lawyers, and there's a lot of buzz about AI, but there's also a lot of pushbacks because maybe people are n- of course not, not using generative AI the way you should. And some people say like, "Hey, but what about these cases where people were citing cases that are, are nonexistent and the hallucination," you know? Like, and what I always tell them is, like, you just have to find a way on how to, you know, do the correct prompting and find a way on how these tools do not hallucinate, right? So if I know this might be, like, very technical, Otto, but, like, can you tell us, like, a very easy way on how you can train a system like this so it can get accurate, which is, like, I think one of the main concerns, especially for lawyers. Yes.

5:02Before I even go there, let me just flag that to me, the answer I give lawyers when they ask, they say, e- when they, when they phrase a question like that, like, "How do I use AI, and, and how do I do it safely, and how do I keep it from hallucinating?" To me, it really comes down to, like, what are the use cases that you're looking at? And part of what I love about contracts is they're actually a really great low-hanging fruit use case to get started at, and I'll tell you why. Because when you're using AI on contracts, you're not asking it to go out into the universe of case law, for example, and pull in things that are hard to verify. You're actually just saying, "Hey, here's a document, and within the four corners of this document, help me understand if this standard is met," or, "Help me draft this thing." The work product and the source of truth is all right there in front of you.

5:53So for us, it's actually a much easier problem to solve, and we build in, we've always built in a, a design principle or built around a design principle that we call auditability- And what I mean by that is what we tell our product managers is like every feature has to be easily auditable by the user, every result has to be easily auditable. Like, ideally, it just takes one click or even less than a click to immediately see the source language from the contract that helps you prove or disprove whether the AI got it right. So auditability is one answer to your question, and it-- and contracts lend themselves to auditability much better than, for example, case law research. But accuracy is a separate question. How do you get a tool to perform at really high levels of accuracy? We think about this question in, on two fronts. One, what can we as a company do to make our system as accurate as possible? And then two, how can we enable our users? 'Cause to your good point, a lot of it's about prompts. It's about are your prompts performing at high levels of accuracy, which I don't control your prompts.

6:55But what I do control is what can we do from a product perspective to give you t-tools to analyze, to track the performance of your prompts, to upgrade them. So on the internal side that we control, we do a ton of rigorous testing around AI accuracy. And the way we do it is actually pretty simple. We get a dataset, we call it an evaluation set of contracts of many different types. We go create standards that are common standards that most lawyers care about. The limit of liability should not exceed X, or governing law should not, you know, must be one of these things, or there shouldn't be any indemnities for data security breaches. Whatever those standards are, we get a large set of those standards, we get a large set of contracts. We actually have human being lawyers go and find the correct answers, and then we plug in different large language models to that evaluation set and figure out, does GPT-4.0 perform better than, than Claude 3.5, for example, than Google's models, than... And so we can plug in the different state-of-the-art LLMs and measure exact accuracy on those different LLMs, and then we can do that with different processing techniques, like RAG techniques you've heard about, and all the different things that we also control or can do or experiment with to increase accuracy.

8:10Basically, what that does is creates a laboratory for us to just experiment and tweak and optimize all of the technology that we're delivering to our customers and measure those results. And we published a couple of studies in Artificial Lawyer in, in twenty twenty-four. We published two different studies where we actually exposed the results of our internal benchmark analyses with our production LLM RAG processing techniques. Bottom line is we're shaking out ar-around ninety-seven point five percent accuracy on standard identifi-identification and on measuring whether red lines that our platform proposes actually fix the problems that they were supposed to fix. So it's not stylistic whether the red line was, quote, good or bad, but just did it actually solve the problem that it was trying to solve? Also performed right around ninety-seven point five percent, incidentally. So and those studies published in Artificial Lawyer are also auditable, so we actually published, uh, instructions on how someone else can go recreate that study using the platform and see for themselves.

9:09On the second part, what do we do to enable our users to optimize their prompts to make sure their prompts are performing at high levels of accuracy? We have a feature set that we call validation that basically lets a user go create-- I actually just did this for an internal use case. It lets you go create a prompt, analyze a set of contracts, manually check whether the AI got it right or wrong. There's a little thumbs up button. If it gets it right, you hit the thumbs up. If it gets it wrong, you hit the thumbs down, and then we'll track those results for you. So after you check twenty results, we'll show you, "Hey, it's performing at eighty-nine percent accuracy. Maybe we should try improving the prompt and see if we get better results." And so we take you through this feedback loop, basically, where you engineer your prompt with actual evaluation sets that you create until you get that prompt to a place where it's performing to meet the specs that you want. Yeah, that's awesome.

10:00And before we move the next question by Marc about the Agiloft partnership, I just wanna say to everybody that, of course, Otto, you did or with your team, a massive analysis of data with TermScout, and if you guys wanna hear a little bit more of that, you can just tune in our last episode. It-- With Otto, I think it was, like, almost three years ago. Man, it's been a long time. You were at the World Commerce and Contracting event, I think it was in Phoenix or the one in Dallas. But yeah, just we're not going to talk about TermScout because we already did. But thanks a lot, Otto. That was a really great answer, and you solved a lot of my own questions. And so Marc, you can go ahead. Yeah. Yeah, no. Thanks, Pep. And Otto, I, I think when we look at enabling users, right, and, and allowing them hopefully to continue to grow and learn and be able to generate their own prompts, right, that are going to be highly accurate, I think that's important. And I also think it's important to, to get an understanding of how you see screens integrating with Agiloft and, and ultimately, right, working with your joint partners or joint customers to, to really enable them to, to have a better experience when using the applications together.

11:07Yeah. So we see a lot of upside in combining with Agiloft. It's a big part of what motivated us to do the deal and actually team up with Agiloft here. I'm personally very bullish on CLM as a category, as a space. I think personally that contracts are unique enough and complex enough that they just aren't likely to be served by generic solutions. You really need something that understands the nuances of how families of agreements work and how different types of contracts have very different dynamics and aspects and things that matter and the like. And CLM is, as a category generally, is in a really strong spot, I think, to solve those problems and has already made tremendous progress on solving a lot of those problems. But I think it's still early days, and generative AI is transforming so much of what CLM can and will become.

12:01For us, as, you know, pre-acquisition We were a point solution focused really firmly on pre-signature contract review, though we did have a post-signature review solution as well. But our customers frequently said things like, "Gosh, Otto, this is great that we can review our contracts, but after we sign them, w-we have all these obligations that we need to track, and we have all these relationships and renewal dates. And what about approvals?" And there are all these things that CLM does that Screens by itself was not going to go after. We just did not have, frankly, the bandwidth, the resources to go solve all of those problems. But Agiloft in particular, as a CLM, has been on a really incredible growth path. It's a really strong team. There's been a really great cultural fit that we found here. And we think that by combining solutions, we not only make the contract review function a lot stronger at Agiloft, but we also enable our shared customers to do all the different things that they've asked us to do, like build approval processes off of their playbook or screen results, so that if this one standard fails, wouldn't it be nice if we could route that automatically to the VP of Finance because it's a finance-related approval that's needed, right?

13:15Or if the whole thing fails, who does it need to be routed to? How do we expedite that process and the like? The final thing I'll say just about the Agiloft partnership, what we really admire about Agiloft as a company and, and as a product is its data-first approach to contracts. Eric, the CEO of Agiloft, told me in, in our first meeting, he said, "Contracts are merely vehicles for the data that powers businesses. And as a company, we're really in the data business. We're really trying to figure out how does that data actually power better outcomes for businesses." And that, to me, that insight and that approach permeates across everything that Agiloft does, and it's really, while it sits in the contract lifecycle management space, it's really about figuring out how do you deliver maximum value to customers or how you enable customers to derive maximum value from their many, many contractual relationships that sit out there. And I think that's just a really powerful vision for how CLM is and should operate.

14:16And i-in my opinion, I think that was, like, one of the last pieces on the CLM as flexible as Agiloft, because what we always say to our customers is that we can do the best implementations, we can automate any kind of workflow, approvals, templates, whatever you want. But the thing about the negotiation part, that's really hard to be solved by just a CLM, because if you have a shitty template, you know, with terms that are only one way or, you know, unlimited liability, and it will take really long for both parts to agree on something in the middle, that's something that it's gonna be really hard to beat. You know, it's like one of the main challenges.

15:05And one of the things that I saw on the demo and then wh- that I loved w-with these Boost features, it was something like, I think this was like the low-hanging fruit of the low-hanging fruit of the contracts. Like, there are a lot of the times where you see a contract with a con-confidentiality clause that is only one way, and there's, there's a boost, just make it both ways. And it was, like, fixed in just in less than a second. It's, it's like very simple things that can help you during the negotiation process. And if you can use, you know, the community, and this is something that I wanna follow up, Otto. But when you have the technology that can help you to negotiate a contract, right? This was, like, the last piece of the puzzle in order to make, you know, not just legal department, but procurement, sales, or any business department to become m-more efficient. And let's stick on the playbooks, Otto.

16:05And I really love about this community that we were discussing before. Um, how does community and the users play together in order for you to understand what needs to be developed for production? Like, how do you interact with the users and the community? Yeah. Great question, Pepe. So for anyone who isn't familiar with the Screens solution, I'll just share that one of the unique aspects of Screens as a contract review solution is that we've partnered with lots of experts in different contract types and allow them to publish their playbooks. We call them screens on the Screens platform. You use them to screen a contract, right? And the, the big idea here is that I believe AI's higher calling in at least the knowledge workspace, and particularly in the legal space, is not helping us figure out what should or shouldn't be in a contract. It's actually performing what I, I call a much more administrative function of just, like, scaling expertise of, and performing administratively burdensome tasks that it happens to be really good at.

17:10So what we set out to do was say, and what we believe to be true is that experts will always be the first and foremost trusted advisor to businesses who are thinking about whether they should or should not sign a contract. And I'll tell you why. Because people trust experts, and because in contracts and in law generally, rarely do two or three experts agree on the same thing. And if two experts can't agree on whether this clause should or shouldn't be present, or this clause is good or bad in this particular agreement, then how will AI ever get to the quote-unquote, "right answer"? In other words, the subject of contracts and what should or shouldn't be in them is so specific and so nuanced that you really do need experts to drive the analysis or to drive the determination of what matters in the analysis. So the Screens community is a place where experts are sharing their expertise, and then we use AI to scale those experts, to enable those experts to say, "Hey-" I think these 20 things should always be in a data processing agreement, and now any Screens customer can just upload a data processing agreement and see whether those 20 things are met.

18:20The AI is doing the administratively burdensome task of applying those 20 standards to potentially infinite numbers of contracts, right? And that's where I think AI is really powerful. Another administratively burdensome task is the one you mentioned, Pepe, which is one of our customer favorites, which is, gosh, how many times do we encounter a one-sided clause where we just need to make it mutual? And guess what? It turns out AI is really, really good at that, and it's really easy to see if it gets it right because you can do it in Microsoft Word, you can turn track changes on, you can see exactly what it's proposing, and you can decide whether it got it right or not. And, and Otto, I think we'll go ahead and wrap up after this last question, I think, but do you see that in, over the next, you know, couple of months, maybe into the, the next year, do you see that AI is, is going to be driven, you know, to support experts i- in this space, or how do you see the, you know, AI continuing to mature when, when we look at CLM? I do think it will grow and grow and continue to support experts. I don't see a world where AI is replacing lawyers or contract managers or any of the really well-trained people that work in this space.

19:26I'm of the belief that there's so much undone legal work, so much legal work that just isn't getting done today because we don't have time to do it. Every business has a laundry list of wish lists. They all have more legal work than they can actually do. Not just businesses, people too, if you zoom out of just our little, you know, B2B, our little lens in the world, like, there's just so much legal work to be done. And so I'm a big believer that if we can figure out how to enable experts to do more with less, we will actually make a big dent in the world. We'll make this world a better place. We'll enable experts to become much more efficient. Yeah, I love it. I think that's kind of what we're all hoping for and, and working towards. But, uh, Otto, this has been a, a fantastic conversation. We really appreciate you catching up with us. Uh, if folks want to connect with you, learn more about Screens, where's the best place to do that?

20:18So check us out at screens.ai, and you can learn more about Agiloft at Agiloft.com. All right. Well, thank you again, Otto, and thanks everybody for listening to another episode of Contract Heroes. Thanks for having me. Thanks. Appreciate it. This episode of The Contract Heroes is sponsored by Koho Consulting. Koho helps organizations of all shapes and sizes find, implement, and manage their chosen CLM tool. Whether your project is buy or sell side led, Koho can help you navigate the waters of the CLM space to make sure that you're getting a product that's going to fit like a glove for years to come. If you'd like to learn more about Koho, visit their website at K-O-H-O consulting dot com.