Rethinking Legal Work Through Agentic Law With Norm AI

[00:00:00] Your agents aren't producing accurate answers because they don't have a complete semantic understanding of your data. And Denodo is solving this and solving it through semantic consistency. Through semantic consistency, your agents can start making accurate predictions in real time. So see what else Denodo can do by visiting denodo.com to learn more. But now let me introduce you to today's guest.

[00:00:34] What happens when AI becomes the production engine for legal work rather than another assistant waiting beside an attorney's inbox? Well, faster document review is useful. Of course it is. But maybe the bigger opportunity is legal systems that can apply rules, examine evidence and prepare decisions for licensed professionals to supervise.

[00:00:59] Well, today I'm going to speak with the founder and CEO of a company called Norm AI. And my guest will explain how attorneys, local engineers and software engineers are all turning regulatory knowledge into executable workflows. And we'll also discuss where deterministic rules and model-based reasoning begins, why human supervision remains part of every legal opinion,

[00:01:25] and how outcome pricing could encourage firms to examine more evidence instead of just counting hours. So, in short, we've got lots to talk about. So, enough for me. Let me introduce you to my guest now. So, thank you for joining me on the podcast today. Can you tell everyone listening a little about who you are and what you do? Yeah, thanks for having me. I'm John Ney, the founder and CEO of Norm AI.

[00:01:51] My background is in AI research and the application of AI to law and policy in particular. Been working at that intersection for about 14 years now. And this is my second company. I was the founder and CEO of another AI company. And that company ended up building out an AI-powered asset manager and was acquired by TIA Nuveen. And at Norm AI, what we do is we automate the first pass work around legal and compliance work. And a lot of what we do now also is power Norm Law.

[00:02:20] It's the leading AI native law firm in the world. Awesome. Well, thank you for sitting down with me today. And I think over the last, what, three years, there's been a lot of noise surrounding all things AI. And I think every enterprise and inside every industry is trying to understand where it can bring real measurable value and make a difference inside their organization. So I'm curious, from your backstory here, what did you see in traditional legal and compliance work that convinced you that, hey,

[00:02:49] AI could change how the work is produced rather than simply helping lawyers complete existing tasks faster and improve productivity? What did you see there? Yeah. So this started a long time ago. I published something called GovDeVec. So after GovDeVec came out from Google, GovDeVec was the idea of training neural networks on a lot of text in an unsupervised way where you're effectively just predicting the next word across a lot of text from the internet.

[00:03:18] What I did is I took that same basic idea but adapted that methodology to specifically focusing on legal and policy text and then capturing concepts that were latent within that text. So, for example, finding that if we could train on a bunch of texts, the models could automatically learn the distinction between a Republican and a Democrat, the distinction between different branches of government and things like that. So that was why it was called GovDeVec.

[00:03:46] It was about learning these vector representations of government and legal concepts. So this was well before ChatGBT. This was before the transformer architecture. And this was sort of the foray into this world for me and some others. And so then along comes the transformer architecture and the ChatGBT moment in large language models. And so that was the real turning point, though, to make AI really useful from a commercial and really impactful perspective

[00:04:16] around the automation of legal and compliance work. And so at the time, I was still this is still more kind of academic. And then I founded Norm AI to really go hard at that in the commercial applicability of that. And what we found to answer your question is around work where it's it's still routine,

[00:04:37] but it does require what was traditionally human cognitive labor to be able to reason across large amounts of text and precedent and apply that in new situations. And there's still a very important part that sits on top of that. Whenever it's officially providing legal advice or anything very high stakes like that, there's a human overlay.

[00:04:59] But it's about getting that human overlay to something faster and focusing the human attention on the review and supervision of agentic output, where that's actually the highest and best use of expert humans. Incredibly cool what you're doing here. And I love how you were doing AI before it was called, long before the open AI moment.

[00:05:24] And I would imagine that legal reasoning often depends on things like context, interpretation and professional judgment. So I'm curious, what parts can be translated into executable rules? And where must an experienced attorney still remain responsible for that decision? Is it somewhat of a balancing act? Well, yeah, I mean, I would make a distinction between deterministic executable rules

[00:05:49] and non-deterministic things that you don't really know ahead of time how to map it all out. So there's things that are deterministic. And before large language models, you could still apply a lot of automation there. So you could have a lot of forms that could be filled out by just finding the right answer and putting it into a part of a form, for example. And that can still lead to a lot of automation and a lot of efficiency gain.

[00:06:15] But then making the distinction between that and was unlocked by the frontier large language models where that allows you to, in a new situation, apply precedent and guidance, but apply it to a brand new situation. And then the third thing is then the human overlay and the human supervision on top of that. So all three of those things come together. And the way to think about it in a law firm context, for example,

[00:06:42] is everything that goes out the door is supervised by attorneys. But then before it's supervised by attorneys, you can get a lot of the way there with that second thing, with that large language model powered analysis. But the other important thing to point out is that it's not just an off-the-shelf analysis. So this is something where there's already been a lot of judgment embedded in the way in which you would call the large language models.

[00:07:09] So that's done up front before any live matters are flowing through a system. And then that right there already encodes a lot of human judgment in a really specific matter type or specific use case ahead of time. And at Norm AI, you describe AI as the production engine for legal work. And for anybody listening that's maybe inside a law firm right now and maybe bring it to life, everything we're talking about,

[00:07:37] what does an AI-first workflow look like from receiving, I don't know, a regulatory requirement through to producing a determination that can be reviewed and defended? Is there like an AI workflow that you'd be able to share with it just to bring it to life? Yeah, sure. So I'll give you two examples. So one example would be the use of our technology for an in-house team. So you could be inside a large private equity firm, for example, a Blackstone.

[00:08:06] And inside a firm like that, you could have the ability to apply public policy-based AI agents. For example, the review of LP communications, where this is subject to SEC marketing rules and other things of that nature. And then the content can be reviewed automatically by an AI agent in a live workflow to do that first pass review against a rule from the SEC or from FINRA or something like that.

[00:08:35] And then the human can take it from there and finalize that legal and compliance review. So that's one example. The second example is around inside Norm Law. That law firm could be representing a firm like Blackstone in a deal context. So if Blackstone is going to invest in a company, they're going to take a minority stake in that company. They're going to use a law firm to represent them in the deal and negotiate with the other side and finalize the transaction.

[00:09:05] So in this case, the live situation would be documents would come in. They'd be emailed from the counterparty to Norm Law. And then right away, that would go to AI agents that could do the first pass immediately behind the scenes before a human even needs to do anything. But then that is shown to a human. And then that is a licensed attorney that's done, let's say, hundreds of deals in this example before. So they have that judgment implicitly already in their head.

[00:09:34] They can take the output from the agents that did the document analysis and then decide what to do next, whether that's going back to the other counterparty, whether that's going to the client, in this case, Blackstone, to get their take on what they want done or just moving forward with the documents.

[00:09:50] So that way, it allows us to background process, sort of parallel process behind the scenes, the work across AI agents and reserve that human attorney judgment for the edge cases and for the communication with the client and for the supervision of everything. And again, I'm curious for anyone listening outside of the industry, how much time is this saving people in scenarios like that? Well, it's two things.

[00:10:16] So one, it's saving time, but two, it's just seeing more things given kind of the unit of time and pushing out kind of the efficient frontier of what you would even want to look at. So take, for example, due diligence. In due diligence, especially in a minority equity investment like the one I just gave as a case study, you're not going to look at literally every document in a data room usually. You're not going to cross-reference at least every document in a way it could be cross-referenced to other documents into the overall deal.

[00:10:45] But if you have AI that can more scalably do that, you can actually look at more things and turn over more stones and figure out if there's inconsistencies, etc. In a way that wouldn't make sense, especially also we haven't talked about the billing. But if you're charging by hours of human labor, your client's not going to want you to do that. They're not going to say, oh, let's look at everything and charge me hours to look at everything.

[00:11:10] Whereas if you're charging by outcome, if you have a fixed price for a deal and then you want to go above and beyond and do something even better for the client, you're going to use AI to look at even more things and try to figure out how you can help them even more. So that's an example where it's, yeah, it's saving time. But it's more about actually, you know, in the deal context, it's more about being able to see and do more things.

[00:11:36] And before you join me on the podcast today, I was doing a little research on you guys. And one of the things that stood out was that Norm Law, you combine attorneys, engineers and legal engineers. So how do these disciplines work together and complement each other? And what new skills will maybe lawyers need as legal knowledge increasingly becomes embedded in software or AI and other technologies? Yeah. So it's evolving.

[00:12:02] I would say every couple quarters, job descriptions change because of how quickly the technology is moving, how quickly we are reinventing what these disciplines are, and we're doing this interdisciplinary work. So that's the first comment I'll make is that if you asked me that a couple quarters ago, I would tell you something different than what I'd tell you today. And so where it is today, we have different flavors of legal engineers that do different things.

[00:12:27] So legal engineers, they're formerly attorneys, they join and then they focus more of their time on the build out and the testing and validation of AI agents. But even within that scope, that also has some diversity of what they do. So, for example, we have some legal engineers that are working super closely with practicing attorneys at Norm Law. And then they are iteratively building out these agents on behalf of our clients as well.

[00:12:56] And I gave you that example earlier, the deal context. So let's come back to that to make this concrete. So in that situation, we would be onboarding a lot of the specific preferences of the client and the deal context of what they want into an AI agent behind the scenes. And the legal engineer would be really facilitating a lot of that process. So that's one example of what a legal engineer does.

[00:13:19] Another example would be building out some of the policy and the things we do around supervisory AI work. So that's another line of business that we have at Norm AI, where we have AI agents that are checking other AI agents for their legal and compliance with respect to relevant laws and regulations and firm policies. So in that case, a legal engineer is working with the client to tailor an AI agent to the deployment of their AI agents.

[00:13:48] So, for example, if they're going to have a chatbot that talks to tens of millions of clients about investment products, that's a highly regulated activity. And so you're going to have a legal engineer that's going to work with the client to build out an agent to supervise that other agent in that highly regulated activity to unblock that, enable that to happen. So that's an example of what legal engineers do. A couple examples of what they do in different things. Another area is AI engineering.

[00:14:16] So these are software engineers that are focused on AI agents and giving them more capabilities and plugging them into more situations in live deployments. So those are your more traditional software engineers that have more of that training of writing production code. They collaborate really closely with the legal engineers. And then everyone also is collaborating with these attorneys if they're working on norm law related work.

[00:14:41] So at norm law, the attorneys, they're also a unique type of attorney that is really part of this broader mission of how do we reinvent legal services in a way that's more aligned with the client and giving some of the benefits of legal AI to the client through our pricing model and through the quality of our services. And so those attorneys are collaborating with the engineers to build out this technology and further develop it in service of the client.

[00:15:09] So those are the three main kind of stakeholders of what we're talking about today. And a few moments ago, you mentioned outcome based pricing, which sounds incredibly attractive to clients, but legal matters can obviously expand or often change completely unexpectedly. So how do you define an outcome fairly without encouraging shortcuts or transferring hidden risks or turning complex advice into a commodity almost? Is that something you're very aware of too?

[00:15:39] Yeah, for sure. So with outcome based pricing, it does limit the types of work that you can do for now because outcome based pricing, I mean, we're at the frontier of it. We do it better than anyone else in the world, but it's still really hard to do because it hasn't been done for decades. And so there's not the full ability to price something up front that's really complex and to have that evolve over time. So you have to focus on matter types that are still complex.

[00:16:08] So this is still things that, for example, doing deals for top private equity firms, that's still complex work, but that's not quite as unpredictable work as something like a bet the company litigation that you have no idea sort of how it's going to meander and how it's going to turn out and all the different things you'd have to unturn over time. So that's the kind of water column, if you will, of things that are really easy, high volume. We do those two as part of a broader relationship with the client.

[00:16:38] Things in the middle that are relatively high stakes and require a bunch of partners that have decades of experience, which we have to supervise the work. And then at the very highest part of the water column of things that we're not, we don't want to do in the near term and are much harder priced by outcome, like the bet the company litigation. And so we can focus on that first and second thing in a way that still works with outcome based pricing.

[00:17:06] And I was also recently reading about the proposed Delaware AI company framework, which would essentially allow an AI managed legal entity to operate under existing laws and oversights. And for people listening that are maybe a little bit more cautious when it comes to bringing technology into this space, what problem would that solve? And who would be accountable if the agent ever, God forbid, ever enter a harmful agreement or acts beyond its authority?

[00:17:33] Because in some quarters, that's going to be somewhat of a concern, I would imagine. Yeah, for sure. So what this is, this is the development of a regulatory sandbox. So to take a step back, Delaware, it's the top state in the world for corporate value. The more corporate value is incorporated in Delaware than any other jurisdiction in the world. And so they have, they've developed a C Corp. They developed the LLC and the public benefit corporation now.

[00:18:02] So they have this legacy of saying, what is the next gen of a corporate form that the world might need and to innovate on that. When these other ones that are now fully settled effectively, C Corp, et cetera, came out, it was everyone's like, whoa, what is that? Like that's controversial. So we're hitting that again with a new thing. And it's called the AIC, the Artificial Intelligence Company. And it's based off the LLC.

[00:18:30] But in this case, the manager of the LLC is going to be an AI agent rather than a human. But to be clear, there's still humans involved. So there's humans setting it up. There's humans that are ultimately supervising the situation at some level.

[00:18:47] But the point of it is that the coordinating off the liability in a way that enables this to happen, where AI agents can actually power things and have enough of a shield that already does exist, to be clear, with C Corp, that people want to move forward with this. And the point of it, to kind of zoom out, is that this is going to happen no matter what.

[00:19:13] So AI agents are going to be doing high stakes things in the economy, no matter what. And the question is, how are we going to bring that into the fold of the laws and policies that we think are best to govern that activity? So that might happen through Argentina. May all the agents would maybe go there because they're allowing it. It might happen fully on the blockchain outside of the United States laws.

[00:19:40] Or we could figure out a framework to bring it into the fold of the most predictable American legal order for corporate law that exists, which is Delaware law. And that has been refined and iterated for decades now by the courts there as well. And things have been tested. So the idea is let's figure out a way to start to answer some of the questions in a sandbox about what it would mean to have AI agents be taking these consequential economic actions themselves.

[00:20:08] And then let's have the courts get involved, have the courts pontificate on what might be done here, ultimately with the goal of coming out of a sandbox with a situation that unlocks new types of economic activity that could be AI agent driven, but need to be autonomous to make that economically viable. And I can go into some examples of that later. But that's the high level motivation behind all of this. Love it.

[00:20:31] And for this year, I think most companies or many companies have been deploying agents that generate content, advise customers and take commercial actions. And for people listening that may be a little bit earlier on in their journey than where you are right now, any tips or advice on what leaders should be putting in place right now around authority, supervision, evidence and accountability? Before just allowing these systems to operate with greater independence?

[00:20:59] Yeah, no, it's a good question, because you can think about a spectrum here. What we just talked about is the furthest end of the spectrum of a fully autonomous AI agent driven company. On the other end of the spectrum is a lot of things that are out there today where we do have AI agents doing some things inside existing Delaware C Corps. Right. And and then there's somewhere in between where those things can become more agentic and do more inside a existing company.

[00:21:26] And so as people move towards doing more, that's where your question really comes in of how they should do that. And as you move from a pure co-pilot where it's just being, you know, a chatbot for someone and then they're still fully doing everything. As you move from that to more of an agent inside a company that is doing a workflow, a little more end to end of completion of the workflow. That's where this really comes into play.

[00:21:51] And so now to answer your question, I think a lot of it is about having the counterpart of an agent be another agent that is specifically focused on supervising the core economic agent. And that's because humans, they can't keep up. Obviously, the point of deploying the agent is because it's more scalable, at least in some respects, than what a human would do. Otherwise, there'd be no purpose of doing this.

[00:22:15] So if that's the case, then we can't have a human analyze everything that's happening and make sure that it's all happening within the right constructs of what the company wants and what the broader law and policy dictates. And so what I would recommend is focusing on how do you have that countervailing AI agent be imbued with the right understanding of what it needs to check on? And then how do you deploy that at scale to check on those things? And that's a lot of work.

[00:22:44] But we've already found that we do this internally. We dog food this with our own AI agents being deployed in high stakes circumstances. And then we work with large institutions to help them do the same. Incredibly cool. And finally, what's next for Norm AI? What excites you about the future? Where you're heading? Any teasers you can leave us with? Well, we've already unveiled a lot of pretty wild things in this conversation, right? So it's really doubling down on those.

[00:23:12] Doubling down on autonomous AI agent-driven companies. Doubling down on supervisory AI for other AI agents in existing companies. And then number three, the powering of Norm Law, the fully AI native law firm. And for anyone listening wanting to find out more information on anything that we discussed today, and there is a lot to talk about there. Where can they keep up to speed with all things Norm AI? Contact you or your team. Where would you like me to point them? Yeah, norm.ai. They can find it all there.

[00:23:42] Excellent. I'll add links to that, and I'll add a few other things as well to make it easy for people to learn and find out more about what you're doing. We did cover a lot in a 25-minute podcast today, but I urge everyone listening to check you guys out, and let's keep this conversation going. But more than anything, thank you for starting it today. Really appreciate your time. Yeah, thanks for having me. I think John's due diligence example changes the AI productivity story.

[00:24:08] The value extends beyond finishing the same review faster. Yes, an AI agent can examine documents. Maybe the sensible starting point here is to divide the workflow into three layers. As my guest advised there, deterministic rules, model-based analysis, and supervision. Only then can you decide where an agent might act, which outputs require an attorney's approval, and what evidence must be preserved.

[00:24:38] And if operational agents begin making decisions at machine speed, consider whether a supervisory agent is needed to check behavior against company policy and law. Belts and braces stuff. So thank you to my guest for joining me today. You can learn more about Norm AI over on my website, techtalksnetwork.com. You'll find a blog post associated with this episode. I'll put some links in there. But over to you.

[00:25:07] Which legal workflow inside your organization is ready for that three-layer test that we talked about today? Maybe you're already doing it. Maybe you're doing something else. Whatever it is. Again, techtalksnetwork.com. You can leave me an audio message over there, or we can grab a hot coffee or cold beer in a town near you. I have got so many events coming up in the last six months of this year. So please, if you're going to any, hit me up on LinkedIn. It'd be great to meet you. But that's it for today.

[00:25:37] So thank you for listening. Bye for now.