AI in Legal Ops: From Manual to Intelligent With Shelle Elzer

Made with speech-to-text software, so a few words may be off.

0:00You're listening to the Contract Heroes Podcast, your one-stop shop for all things contract management. And now here are your hosts, Marc and Pepe. Welcome back to another episode of Contract Heroes. We are still live at CLOC 2026, and during today's conversation, we're joined by Shelly Elzer of NetApp. Now, let's get into the episode. Before we, we dive into the podcast, Pepe's gonna give us a bit of, uh, some information on what we're gonna be talking about. Of course. We're gonna be talking about AI. You know, we're... CLOC, this is the first time that it's outside Vegas. So, since couple of years ago, AI's been everywhere, and I think, like, now people are actually seeing some of, you know, good results. They're actually using it in their work instead of, you know, back in the day when th- there was it just, like, a buzzword. But, so you, you told me about really good results that, that, that you're seeing in your team.

1:01So before we start talking about how you're being held in Astra with your team, can you tell us a little bit on, on your role i- i- in your company? Yeah, sure. So I'm a legal operations manager at NetApp. My primary focus is in the contract space, so contract operations, everything from life cycle management of a contract to the technology of it. Perfect. And about, like, the legal operations, right? You're, you're part o- o- of the legal operations team. A lot of company are still struggling right now, like, how they can start a, a legal operations. So maybe let's start with there, because I'm very curious, like- Yeah... how do you interact with the legal team and with the business departments? Yeah. Well, I think just the scope of legal operations is just going under a huge transformation right now. You know, what it looked like a few years ago is, is quickly morphing and is a lot more technology and AI focused. And, you know, I think a lot of the roles now that we're looking for or looking to add have technology, you know, that technology background, that implementing new, new products.

2:07Just being able to keep up with AI right now is a full-time job. Yeah. So, so I definitely think that legal operations landscape is, is changing rapidly. So for somebody that doesn't have a team, my answer now would be find a technologist, right? Find somebody who, who can dig into your problems and, and solve them, you know, with, with some good process and, and good technology. And to come to CLOC- Yeah... to talk with the people. Right? Right? Definitely, yeah. I think legal operation was built not by, you know, schools, but more like communities. 100%. It's the best community out there. Yeah. Yeah. I think, you know, it's interesting, right, that we, we obviously are talking about AI, and it sounds like the team made a decision to start to leverage, right- Yeah... AgileOps' new AI platform, Astra. And, you know, before we dive into exactly how your team is using it currently, I think we'd love to learn a little bit more about how you decided to, to move forward with Astra, maybe over some of the other applications that the team was evaluating.

3:09Yeah. So we did a, a pretty thorough RFP probably about a year and a half ago, and we had a cross-functional team, right? I knew our primary users would be our legal team, but we added in some folks from procurement as well as, as our operations team, and we just, you know, got our hands on a few demos and really spent the time in other... And I, I think we ended up doing four tools. Wow. And, you know, really getting our hands dirty. How easy is it to build? What are the results? How intuitive is it, right, to, to be able to come in and... Not only, like, building is one aspect, but then there's the everyday usage, right? How intuitive is it for our users? But one thing that was absolutely critical for me for my users was bringing it to where they work, so having that Word add-in in Astra was critical, right? Because if they're in Word negotiating their agreements, that's where the technology needs to be, too. We did a thorough RFP. We, you know, got our hands dirty. And for us, the numbers were clear.

4:13Like, it, it was very... They scored, you know, across the board highest, and so for us it was a no-brainer to, to bring them on. And implementation was so easy in the sense that it's a Word add-in. You know? Like, outside of getting through, you know, security and, you know, new, new vendor onboarding, implementation, you really can be up and running with, you know, a basic screen in hours. Hours. So easy to get up and just start saving time. It... After you made those interviews and demos with more than 20 different vendors, you know, we see this, this problem since we've been involved in the, you know, CLM and these kind of AI tools, the implementations. And a lot of the times, the RFPs are, like, very general. Yeah. They don't, they don't go really deep into the actual problems that they try- Yeah... to solve.

5:07So after, you know, having those over 20 conversations, do you see something that you might have done different during the, while building the RFP, or how would you recommend companies to build their RFP so they can match- Yeah... actually this? You know what? I actually don't. Like, I, I feel really confident in, in the RFP. I think maybe... I don't know. You don't know what you don't know, right? Okay. Now that we're already through it and we have a vendor, it's like, "Well, why can't it do this? And w- w- why..." You know. So I guess maybe upfront I maybe would have scoped, like, what do we really want the endgame of this tool to be? What do we really want it solving for? But, but when we started, first the product that we got, when it was time to roll it out, it, it's rapidly changing and evolving and getting better, right? And so, you know, from an RFP perspective, I definitely wouldn't have done anything. Uh, for us it was like, okay- Everybody's on the table, let's look, let's talk.

6:05But I think the real difference, though, is getting your hands on the tool. Yes. Being able to build a screen yourself, your standards, run it, get those results. There's no better way to test. Because a salesperson can sell you anything, right? Their, you know, three-minute demo is going to be perfect, and they're gonna feed it perfect data to give you the results you wanna see. So being able to validate that yourself is critical. That's what I- Yeah, and so it sounds like when you narrowed it down to those four solutions, was, was there a test period? Did you do POCs with, with all four of those applications? Mm-hmm. We did. So we only ended up doing full POCs with two. Okay. Yeah, yeah. Just sometimes you just know, right? And I think another thing is where do you look at the relationship long-term? Like, for me, partnership is huge. I don't need another vendor. I need a partner, especially in a space that's rapidly changing. They're gonna grow, but we wanna grow with them, right? And our, our demands, you know, are gonna grow as well, and we want somebody that's gonna be able to partner with us.

7:07And so that was a big consideration, too, not just the performance of the tool, the company itself and their ability and desire to partner with us. Yeah, that's great. And have you, have you looked at... So I know that the team is just using Astra right now. Have you started to look at, uh, different use cases for using additional parts of Agiloft? As, as you know, right, Astra's their, their AI application- Mm-hmm. Yeah... but they do have other functionality. Or are, are you using something for workflows, for template generations right now? Yeah. Right now we have a pretty complete tech stack. Okay. But that's today, right? You know, down the road, yeah, I absolutely think we'll, you know, we'll definitely consider other parts. I think for us, there's so much more we still have to do in Astra, right? Like, our, the results that we've been seeing have been so great, that it's almost getting addicting, right? Like, we know what we can save. Let's build more. Let's, let's adjust our use cases and, you know, keep getting it out there into the business. Yeah. And can you give us some ex- examples on how was, like, the before and the after? Yeah.

8:10So we weren't using, for, for redlining, it was just manual review, complete manual review. So when you implement an AI tool, it's kind of fun to start seeing the results, right? Because you can take your manual review time, and then you can take your AI review time. And so far we've seen a pretty significant delta. We have seven live screens, and the average savings time per screen is 150 minutes. Wow. So over two hours per agreement type being ran. I mean, that's, that's huge, right? That's a, a really big deal, and it's, it's giving our attorneys that time to negotiate bigger deals, revenue deals, putting their time where they need to be focused, not on first pass redlining, right? And the more- Yeah... you're using it, are you feeling that it's getting, let's say, smarter? Because, you know, like, there, there's always, I, I would say, like, the rule with AI is, like, you still have to check it. Like, the- Absolutely. Yeah... human- Trust and verify. Exactly. Yeah. To verify, you cannot just, like, put it in the system and just, like, send it back to the customer or to the vendor for the review. Yeah. Yeah.

9:15So I will say that we had a pretty thorough build and test process, right? We have SMEs, so our experts are in there building, testing. Then they have super users who are partnering with them, testing also. So before it is live and sort of out there for the general legal team, it has been thoroughly built, tested. So once it's ready to go, it- it's ready. If... What we have found is when it's not giving us the results we want, there's something in our prompt, right? It's something we need to edit on our end. It's not the tool itself. It's, it's in how we build it. So, you know, we're tracking the quality, right? People can give thumbs down. It didn't... And, and we're just not seeing that. And, but I think part of it is our SMEs and how thorough they are in their, in their testing process. No, I mean, that's, that's extremely exciting, and it, and it's good to see that the team is, is having success. And sounds like you do have those internal super users who are able to really empower the rest of the team and, and roll out a, a solution that's, that's working, that's working for you. Yeah.

10:21You know, I think one thing that's, like, unique with, uh, and not just this, but a lot of the AI tools, right? This is no longer legal op saying, "Oh, here's a tool. Go use it. Have fun," right? These AI tools, you have to really collaborate, be hand-in-hand with your legal team, right? Because you need them, and they need you. Like, to be successful, you really have to partner together. You need their expertise to build these screens, and they're gonna benefit from it, too. So I think it's really cool. We have s- we have a lot of really great attorneys who are AI forward and ready to jump in and get their hands dirty, and that makes, that makes all the difference. Yeah. Yeah. And one of the thing is, like, this is not, like, a plug-and-play solution. You also need to fit it with your internal requirements because you build your playbooks, and then you configure Astra so it can know whenever, if you, if it's a service agreement, if it's a distribution agreement. Like, what are, does that clause library, right? It's, it's not just like, okay, we're just, like, getting our users, and then we can start working it. You need to train it. Yeah. Absolutely. Yeah. And it does.

11:26It takes time, right? Takes time building, takes time putting fallbacks in. All, all of that takes time, but you know, the numbers are showing itself it's worth it. It's definitely proving to be extremely efficient for our team. Yeah. And one last thing, Shelly. Since this is a new piece of software that your organization is using, you know, when there's, like, a new tool, there's always, uh, has an impact with change management. You need to train people Do you feel it's easy to train people to use it? Like, how, how have you seen the, you know, like, newcomers, uh, being, being, being using this tool? Yeah, so actually, I would say from an implementation, like the, the new users, it's extremely intuitive. So we didn't have a problem there with, "I don't know what to do or how to do it." You know, I think the, the only issue we have from an implementation is, you know, you build screen specific. So for us, it's customer paper, first pass, right?

12:24So they may get an agreement in, but it's a red line on our paper. We don't have that built yet, right? So it's, it's not a one-size-fits-all, right? You can only use it for what you have a screen built for. And so that's kind of the hardest part or the slowest part of the rollout, is getting built what you need quickly, right? Because people want to use it, but if you don't negotiate any of the agreement types that we have built, you don't really have a use case quite yet. Right. So it's, you know, the demand is there. We just can't get things built quick enough, right? Yeah, and I think that's... Well, that's the important part of, of making sure that when you are rolling out these screens, right, that it is probably for the, the highest volume that you're seeing first, right, and, and things of that nature. You know, volume's one way to look at it. Another is expertise, right? If we only have one privacy attorney, he can't- True... you know, they, they can't be that expert on all of these deals, right? So getting something like a privacy screen up and running, that empowers attorneys- Yeah... who aren't the privacy experts. That empowers them to have a little bit more stake in the negotiation.

13:29So, so volume's definitely, but definitely specialty is another way to look at it. Where, where are you short on resources? Yeah. No, absolutely. Absolutely, right. Being able to share that person's expertise across the team, right, without, without them even being there. Well, that's, that's fantastic. So Shelley, if you had to give one piece of advice to somebody that's here at Clock evaluating, you know, these, these AI tools, w- uh, what, what would that be? Jump in. Just jump in. Just start, right? I mean, we're waiting for a perfect solution, and it may never exist, but start somewhere, right? If it's not a big investment, but something you can have up and running, even a little bit helps. And, and pick a vendor that is willing to partner with you. Mm-hmm. As you grow, you want them to grow. But just jump in and, and take a chance and do it. Awesome. Well- Yeah... Shelley, thank you so much for sitting down with us. It's, uh, exciting to see how the team is starting to use the application. And yeah, we're, we're excited to see where you go in the future from here. Yeah, thanks.

14:32All right. All right. Well, thanks everybody for listening to another episode of Contract Heroes. This episode of the Contract Heroes is sponsored by Koho Consulting. Koho helps organizations of all shapes and sizes find, implement, and manage their chosen CLM tool. Whether your project is buy or sell side led, Koho can help you navigate the waters of the CLM space to make sure that you're getting a product that's going to fit like a glove for years to come. If you'd like to learn more about Koho, visit their website at C-O-H-O consulting dot com.