Why Larridin Says Every CIO Needs an Enterprise AI Inventory

[00:00:00] Do you need AI agents that you can trust? Well, with an AI data layer providing real-time connection within your data platforms, you can trust your agents to provide accurate solutions. So, scale your business by trusting your agentic AI accurately getting the work done for you. Trust its capabilities with Denodo. And you can do that by simply visiting denodo.com to learn more.

[00:00:26] Could your CIO produce a complete inventory of every AI system that is operating across your entire business today? And by that, I mean every tool that's brought by departments, features embedded inside existing software and agents created by employees who have since left the company. I think most of us struggle to remember every subscription leaving our bank account each month in our personal lives.

[00:00:56] Now imagine doing the exact same across hundreds if not thousands of AI tools, except some of them are accessing your company data, making decisions and quietly spending thousands of dollars. Well, today I'm joined by not one but two guests and they approach this problem from very different directions.

[00:01:16] But before I introduce you to them, I just want to let you know we're going to be discussing the growing AI visibility gap and explore why one company expecting to find 70 AI tools discovered 220 and how another found an agent built by a former employee costing $1,800 every single month. So if your company is investing in AI faster than it can measure or govern it, today's conversation I think you will find incredibly interesting.

[00:01:46] Might even set off a few light bulb moments and even alarm bells. But enough from me. Let me officially introduce you to today's guests. Well, we have not one but two guests joining us today. Mike, to begin with, can you tell everyone listening a little about who you are and what you do? Yeah, thank you, Neil. My name is Michael Levine. I go by Mike. I am a partner with Hunt and Andrews Kurth, which is an international law firm.

[00:02:12] I am one of the leaders of our insurance coverage and insurance recovery practice where we have 35 lawyers representing policyholders across all commercial lines of insurance.

[00:02:25] One of the big areas that we are working with and watching closely is artificial intelligence and how the use of artificial intelligence will impact all lines of insurance, including cyber technology and more AI-specific lines and risks,

[00:02:44] but also things like traditional environmental and property coverages, excess liability coverages, directors and officers and management liability insurance in particular, all of which are being impacted by AI. Well, thank you for sitting down with me today, Mike. And we also have Russ joining us. Some years have passed since we last spoke, I think, my friend. Can you remind everyone listening a little about who you are and what you do? Sure. Russ Fraden.

[00:03:12] I'm one of the founders of a company called Laridin in the Bay Area, a venture-backed company in the AI measurement and audit space. So helping companies understand you've spent all of this money on AI. Should you be spending more? Should you be spending less? What are you getting from it? What are your employees using it for? How can you best take advantage of these exciting tools that are powering so much innovation around the world? Excellent. Well, welcome back, my friend.

[00:03:37] And, Russ, you've been through several major technology shifts from digital advertising to mobile. Now we find ourselves right in the eye of the AI storm, so to speak. So looking back, what do you think makes this AI era that we're in now different from every tech wave that has come before it? Because we've seen so many, all three of us have. But what makes this one different? Well, look, first of all, it's obviously – it's more similar than different to the prior waves, I will say.

[00:04:05] But I'd say it's different in two ways. And it's interesting having seen all of them from Silicon Valley. For a long time, Silicon Valley sure was the center of innovation. But it would take a long time for these technologies to make their way around the world. It would take a long time before they broke outside of even just the tech industry. So, you know, you'd meet other companies in the Bay Area that weren't heavy users of new technology for a while. It took a long time for, for instance, web advertising to be a major part of budgets.

[00:04:33] It took a long time for mobile to be used from a productivity standpoint. You know, ChatGPT is wildly popular in, you know, villages in India right now. Like these tools have just gone all around the world, every industry, every age group, demographic, everywhere. And so I just think that kind of global – globality is certainly not a word. But you get my point. That's been unique.

[00:04:58] And so the way we've kind of gone from a technology nobody heard of three years ago to it being a major skill that you need to understand how to use, take advantage of, understand the risks for, understand the upsides for in your job, that we've just never seen anything happen this quickly in terms of the workforce. And many business leaders tell me that they've invested heavily in AI, but they struggle to answer one simple question. And that is, where exactly is AI being used across our organization?

[00:05:27] Previously, we had BYOD. Then we had Shadow IT. Now we've got Shadow AI. I mean, why has visibility become such a big challenge now? You know, I think it's just like anything else because of what I said earlier. Because these tools have exploded in popularity, the pull to use them all across the organization has just been much faster. You know, in the old days, what you'd have is a new technology came along. It took a while to sell it into enterprises.

[00:05:55] They took a while to consider the risk. Then they slowly rolled it out across the organization. You know, you had this with BYD, bring your own device, right? You had this where, you know, it used to be that everything was locked down BlackBerry, BlackBerry. And then the iPhone was so great that IT departments just had to figure out how to support it. And that's what you see happening with AI is, like I said, these tools didn't exist three years ago. Now they're driving productivity and marketing, productivity and sales, productivity and engineering.

[00:06:22] And so companies just aren't prepared to, you know, have the infrastructure up front on the security side, on the tracking side, on the measurement side, on the ROI side, on the budgeting side. And so you see them kind of building all of the infrastructure on the fly. So I actually have great sympathy for CIOs because on the one hand, you want to help your company stay very innovative. And the pace of innovation is unlike anything we've ever seen. And on the other hand, you have all the normal security concerns you have. And so you can't really push people to go slow.

[00:06:52] And so just people are on the fly trying to figure out policies and procedures for this. Mike deals with this all the time, which is you can't tell people to stop. But law still applies, right? Budgeting still applies. And so you just have to deal with this on the fly. Yeah, I think just to build on that, you know, AI has evolved so quickly and continues to evolve. It seems almost on a daily basis, right?

[00:07:17] Because there are, look at the scale of use of AI. You have, on the one hand, very obvious, very large, large language models being used. They're very discrete entities. You can run through the whole list of names of different models. On the other hand, you've got more nuanced use of AI within, you know, call it shadow AI, embedded AI, within different tools.

[00:07:45] Still artificial intelligence, perhaps generative, perhaps machine learning. There's a whole spectrum of types of AI. But it's being utilized, you know, in such a wide array of applications and functions in different businesses that, you know, at least from a risk perspective, to try to put your hands around it and understand how a company is using it.

[00:08:11] But every day, it just gets harder and harder and harder because it is so deeply embedded now. And almost everything that we're doing from, you know, again, the large language models all the way down to our smartphones. And I'd love to bring this to life here when we're talking about this AI visibility gap where almost half of AI adoption is happening completely out of IT's line of sight.

[00:08:34] And I'll ask this to both of you, but I'll ask you first of all, Mike, what are some of the most surprising examples of how shadow AI that organizations might have discovered when they start looking? You must have a few stories. I know you deal with this a lot, but anything spring to mind there? So there have been some interesting case examples, and I say case in the legal sense, litigation examples of the use of AI.

[00:09:02] And I suppose this predated the term shadow AI. I think that's a more recent raising.

[00:09:09] But, you know, you had a large lawsuit, class action lawsuit brought against CVS Health Corp in Massachusetts because what it thought was a benign tool to screen prospective employees during the interview process actually was utilizing artificial intelligence to analyze their facial movements and responses during the interview.

[00:09:39] And the problem was that there are certain statutes that govern the use of lie detectors in any context. And they require advanced written disclosure, which was not provided here.

[00:09:52] And this resulted in a very large lawsuit, very costly lawsuit against CVS because they did not realize that the technology they were using was this artificial intelligence enhanced lie detection system within their HR, you know, platform.

[00:10:13] So, you know, I think that's a good illustration of where a company thinking that they're just using a better tool is actually now shifting into artificial intelligence. And look, from our side, you know, we tend to have less stark examples because we're not lawyers. It's more just the amount of times you see, you know, one of our early customers thought they had 70 different AI tools being used across the org and they had 220. Now, by the way, most of them were benign. It was no big threat.

[00:10:41] They had five different AI note-taking tools and they realized, you know, that's kind of dumb. We should standardize on one, you know. But you'll see this now as spend has become a big deal. We had a customer the other day. One of the many things we do is, you know, token tracking, spend tracking across an organization. We had a customer the other day that found an agent that was costing them $1,800 a month, which is not that much money, but it's $1,800 a month, that had been built by an ex-employee.

[00:11:09] No one knew what it did and was just running, spending the company's money. Now, no one had done anything wrong. The employee, I'm sure, had done it on purpose and then had left the company and it was just still running and costing the company $1,800 a month. So, you're just going to have these vectors of, you know, expense that you weren't planning for, one of monitoring, one of measuring. You're going to have these vectors of threat because you have these AI tools being used in your org that you don't know about.

[00:11:36] So, our perspective is, hey, you shouldn't shut this stuff down, but you should know what's happening. You should know how people are using them and what they're using them for. And for years, CIOs have focused on inventories of devices, applications, and indeed infrastructure. So, tell me a bit more about now why now regulators are now demanding inventories of AI systems. And what's this mean for business leaders listening who might be slightly unprepared for this new reality?

[00:12:04] So, let me throw a couple of thoughts out there. I know Russ is going to have some views on this too, but, you know, the regulatory landscape is very problematic in the U.S. Unlike the EU with the EU AI Act, which is a more uniform, albeit very complex, regulatory scheme in the U.S., we don't have any uniformity right now.

[00:12:33] And it is very much the Wild West. So, you look at the Colorado AI Act, TRAGA in Texas, and these are state regulatory instruments that are enforced at the state level. Texas is a great example. The attorney general has the enforcement power. There's no private right of action.

[00:12:58] And it governs not only companies based in Texas, but companies anywhere if they're doing business with people in Texas. Right? So, you've got a Texas statute that pretty much any company with a web-based business has to be now adhered to. And it is going to be the same dilemma times 50. Right? And then you have Europe and the rest of the world to worry about too.

[00:13:27] So, when you think about trying to understand, you know, what systems do I have to worry about? Which ones do I have to disclose? Which ones do I have to ensure that we have appropriate guardrails around? And it becomes very unwieldy very quickly when we start thinking on a state regulatory basis within the U.S.

[00:13:45] And Mike, from a legal and governance perspective, what are the biggest misconceptions that organizations and people listening might have about emerging AI regulations and similar legislations appearing right across the U.S.? Because there is a movement there, isn't there? But it can be confusing.

[00:14:03] I think the misconceptions kind of tie in with shadow AI and the ability of the CIO and the team monitoring AI to understand what the company is using. Because they can only regulate what they know about. And if there are other uses going on that they don't know about, then now they're vulnerable. They're at risk.

[00:14:29] Yeah, and that's always been our perspective with customers is we are not your lawyers. I know for sure you should not have things happening in your organization you don't know about. First of all, like I said, purely there's a security element. There's also a cost element. These products are not free. And so, you know, you've seen this explosion in the press around token cost and AI cost measurement. And our perspective there is like, look, we're not telling you you're spending too much money.

[00:14:58] We're just telling you we think it's crazy. You're spending a lot of money where you have no idea what's being spent on. Such a good point from both of you there. And we've also seen cybersecurity become a board level issue over the last decade and more. So I'm curious, fast forward to present day. Are we now seeing AI governance following the same path? And what questions should boards be asking management for right now? Anything else I hear? A hundred percent.

[00:15:25] Every board that we know about and all the management teams we know about say, look, we want to understand what's being used, what's being used for, how much money are we spending, what groups are being spent on, what models are being spent on? You will eventually, for better or for worse in the U.S., you're eventually going to have a lot of questions asked about use of Chinese open source models. Like I said, whether they should or they shouldn't, those questions are coming, certainly for public companies. And so what models are being used? What tools are being used?

[00:15:53] What are our data agreements in place with those? How much are we spending? What are the groups spending them? On what level of models? We know all of those questions are going to be asked. They're all going to be board level questions, of course. Yeah. Right. And like whether you should care about Chinese open source or not is one of the things it's far above the pay grade of this, this group of people on this podcasters. It's going to be a thing. Yeah.

[00:16:18] And, you know, I would add that, you know, from a board level management liability level perspective, you know, AI has been very problematic.

[00:16:26] And if you look at the litigation, the cases that have been brought involving AI, the vast majority involve directors and officers exposure, management liability exposure because of overstating or understating a company's AI capabilities or the use of AI to enhance what it does as a business. And the term coined there was AI washing, right?

[00:16:55] And companies are just misstating and they're getting called out. It results in a precipitous stock impact, which leads to a derivative shareholder suit square in the wheelhouse of the DNO exposure. The other area that we are seeing become more and more problematic is the use of AI by the boards in their disclosures. Right.

[00:17:19] And this is a problem because the disclosure statements that are filed with the SEC are required to be the actual statements of the person signing the statement. If it's now written by generative AI, it is no longer written by the signatory of the statement and it runs afoul of various SEC regulations.

[00:17:43] So, you know, how much can you use AI to enhance your statement? I don't know. Those are legal issues that are yet to play out. But we know that you can have Copilot write the disclosure statement. That would be bad. But can you have AI review it, revise it, make it more concise? You know, all the way down to, you know, will we use spellcheck and grammar check?

[00:18:13] Again, all AI use. And these are some of the issues that affect the, you know, the definitions of something that is or is not AI for all insurance issues. But just in terms of the board liability, how much can a board member rely on the use of generated AI, you know, to do what they are affirming that they've done themselves? Wow. Certainly food for thought there.

[00:18:40] I can hear a few alarm bells and light bulb moments going on listening to your answer there. And another point that has caught my attention is the growing role of insurers. So I'm curious from what you're seeing here, our cyber insurance providers and tech technology liability insurers, how are they changing the expectations around AI oversight? And looking ahead, what could happen if organizations start failing to meet some of these standards? Yeah.

[00:19:07] Well, it's not just the cyber and tech E&O insurers. It's really an issue now that is across the board. I mentioned directors and officers and management liability. Those policies are being impacted more than any right now because of the AI washing type lawsuits. They trigger the DNO coverage right off the bat.

[00:19:28] The cyber carriers are, at least from what I've seen, marginalizing AI to the extent they can because it is not a cyber liability. Cyber policies typically, traditionally deal with the breach and loss of protected information. And that's not what we're seeing. That's not AI liability. AI liability is something very different. There's no loss of information in most instances. It's not a breach.

[00:19:58] So your breach response coverage under cyber is not going to apply. And carriers are introducing exclusions, not just at the cyber level, but across all lines of insurance to try to limit or outright exclude AI-related risk. And this gets to the definitional issue I mentioned a few moments ago, where if we are going to exclude a risk under any line of insurance, we have to define what that risk is.

[00:20:26] And it's not enough to just say, we're not going to cover AI because AI is now so ubiquitous and it means so many different things to different people that we need a concise working definition. Otherwise, you basically eviscerate all coverage because there isn't much that a business does these days that does not involve the use of AI in one form or fashion. Wow. And to bring us home, a question I'd love to ask both of you here.

[00:20:54] And Mike, I'll ask you this first because it is inspired by your answer a few moments ago. I always try and give people listening a valuable takeaway. So if we have a CEO or a CIO or a board member listening today, realizing that they can't actually produce a complete inventory of their organization's AI systems there and listen to your answer,

[00:21:14] thinking, oh dear, alarm bells are going off, what are the practical first steps that they should be taking over the next, let's say, 90 days to move from just AI sprawl to a more trusted, auditable system of record? And I realize that's almost a podcast episode completely on its own or arguably a series. But any advice there, Mike? Yeah, yeah. And I think you're right. It probably lends itself to a couple hours of discussion.

[00:21:40] But I think, you know, think about how companies would traditionally approach insurance. They would have a risk manager, maybe a small team working under that person to manage the physical assets, the properties, and procure and renew the insurance on an annual basis.

[00:21:59] That same risk manager is not capable of understanding the way AI is used across a company, medium-sized, large-sized company, even a small company. I think AI, more than anything, is going to require stakeholder involvement across the entire business.

[00:22:21] So operations, facilities, HR, finance and accounting, the board level individuals. Everybody is going to need a seat at that table to make sure that they are doing the best that they can to understand how the company is utilizing artificial intelligence. Once they get that inventory, they're going to be in a better position to sit down with the underwriters and scope out what the insurance is going to look like, what any limitations are going to look like.

[00:22:51] But it really takes a village now to understand what that AI usage looks like, what that risk profile looks like. And then it's going to be a conversation with the underwriters. And the one thing that policyholders, companies need to be just incredibly mindful of is the use of AI exclusions with the appearance of these exclusions in their renewal insurance policies. The definitions matter.

[00:23:21] The language matters. And it has to be functional for their business. And look, from our side, like I said, it's less about legal and more about it is absurd to have tools being used in your organization that are costing you money without understanding what's being used, where are they being used, how much you're spending on them. First of all, there's the business side around optimization. But we're just talking, you know, in this discussion, we're really focused on risk management. And, you know, look, it's going to be CFO level.

[00:23:51] It's going to be CIO level. There's going to be some reporting to, you know, audit committee level for public companies, right? I can't imagine any public company audit committee is not already monitoring what tools are being used, where they're being used, what's our data risk, what's our expense risk, you know, what's our budget risk. And so, you know, companies are handling this well. By the way, you know, there's been a huge change first six months ago, nine months ago. Six months and nine months ago, we heard about shadow AI all the time. We don't really hear about it much now.

[00:24:18] You know, people are using tools like ours to track and understand what's happening in organizations. And, you know, people are getting this. As I probably said when we first talked about this, you know, like there's an element where it seems like companies aren't prepared, but it's not because they're dumb. It's because this was a tsunami that hit the entire world, every company all at once. And, like, people scrambled pretty quickly. I've actually been amazed at how quickly people have adopted these tools and figured out how to not block innovation in their company.

[00:24:44] So, you know, I've actually been impressed when I meet, you know, CIOs of very large companies over the last six months. They all know what's happening here. It's just truly a tsunami. I think that is a powerful moment to end on. I cannot thank you both enough for coming on here, talking about the risks and the solutions for the AI visibility gap. And most importantly, the steps to move from AI sprawl to a verified AI inventory that satisfies boards, insurers and follows the law, of course.

[00:25:14] But before I let you go, there's going to be a lot of people wanting to dig a little bit deeper on this. So, Russ, to begin with, where can people find out more information about you and the work that you're doing? Our website's just larradin, L-A-R-R-I-D-I-N.com. And I'm pretty easy to find on that website or social media or whatever. And would love to, you know, happy to chat with people if they want to learn more. Awesome. And, Mike, anyway, you'd like to point everyone? Yeah.

[00:25:39] Our law firm website is hunton.com, H-U-N-T-O-N.com. And, Mike, Russ, I'm easily findable via the website. And listeners can feel free to email me directly at mlevine, L-E-V-I-N-E, at hunton.com. The AI visibility gap is something that a lot of enterprises are struggling with right now.

[00:26:04] And I think we've given people valuable takeaways on how they can make those more informed decisions by solving that problem. If we accomplish anything today and get halfway close to that, then we've done our job. But I hope people listening will reach out to you both. But more than anything, thank you for sharing your insights today. Really appreciate your time. Bye-bye. Thank you. I think the examples from Russ and Mike shared today make that AI visibility gap feel far less theoretical.

[00:26:31] I mean, one company believing it had 70 AI tools and discovered 220. Or the fact an AI agent created by a former employee continued to run at a cost of $1,800 a month. This feels like the enterprise equivalent of finding a forgotten gym membership or streaming platform. Except this one can access systems and spend company money. So the practical lesson here is to begin with visibility.

[00:27:01] Companies need to know which AI tools and models are operating, who is using them, what data they can access, and how much they cost, and who remains accountable for their behavior. And I think Mike's insurance perspective also added another layer to this. AI risk may affect directors' and officers' coverage. Technology policies and other commercial insurance.

[00:27:25] But businesses should be paying close attention to exclusions, definitions, and the evidence that insurers are going to request during those renewal processes. And this means participation from across the company. IT cannot build a complete AI inventory on its own. It's going to need legal, finance, HR, operations, risk teams, board members. They all hold an important part of this picture.

[00:27:53] So a big thank you to both of my guests today. I will have links to everything they mentioned. So I urge you to go check that out. And let me know about your company. Could it produce a complete AI inventory today? If it can, if you've been on this exercise, what did you discover? What are you concerned it would discover? I want to hear your stories. Those stories are what this podcast is all about.

[00:28:18] And then together, we can all learn and share from these business tech perspectives. But that's it for today. Thank you for listening as always. I'll be back again real soon with another episode. Thanks for listening. Bye for now.