How Deloitte Measures the Cost of Enterprise AI

[00:00:00] Once a year, I open up a handful of places for people who are serious about launching their own podcast, but have never actually gotten around to it. And over the last 11 years, I've helped entrepreneurs, business leaders and Fortune 500 companies all launch their own shows. And here is how it works. You record your conversation on Zoom, Teams or whatever works best for you.

[00:00:23] You send me the audio, an episode title and short description, and I will handle everything else from setting up the podcast host and distribution to getting your show on Apple Podcasts, Spotify and the many, many other podcast platforms out there. I will commit to working alongside you to help you launch your first season of 12 episodes in the bag. So you can start 2027 by ticking a season of podcasts off your bucket list.

[00:00:52] Think of me as your not just personal podcast producer, but also your accountability partner to make sure that you actually deliver those 12 episodes into podcast land. Now, I only offer this once a year with limited places available. So if you want 2027 to be the year you finally launch that podcast, head over to techtalksnetwork.com. Send me a message, send me a DM or even an audio message. I'd love to help you make it happen.

[00:01:21] If your AI bill went up last month, could you tell which piece of work made it worth paying? Well, today on Tech Talks Daily, I'm joined by Jim Rowan, who leads AI for Deloitte's US businesses.

[00:01:36] And one of the reasons I'm excited to get him on here today is to talk about Deloitte's State of AI in Enterprise survey that found that 71% of respondents track total AI inference and AI spending, while 44% track cost per unit. Now, this gap got me thinking about how a business decides which AI work is worth expanding.

[00:02:02] And Jim will explain today why the systems around an agent matter every bit as much as the choice of model. So we will talk about what happens when an AI agent needs to read or update an older system, who should be able to approve a payment, and how to test a cheaper model on work that your team is already doing. But enough scene setting for me. Let me introduce you to Jim right away.

[00:02:28] So thank you for joining me on the show today, Jim. Can you tell everyone listening a little about who you are and what you do? Thanks, Neil. Really appreciate you joining the show today. So name is Jim Rowan. I head up AI for Deloitte's US businesses, which for me, that means collaborating with external organizations and helping them figure out how to apply AI successfully.

[00:02:49] It also allows me to chance to work internally on our internal Deloitte AI transformation and really just trying to figure out how to make more with this technology today in the marketplace. So excited to be on and hopefully have a little bit of a chat about AI. Awesome. And it is a pleasure to have you join me today. And we do hear a lot about the capabilities of individual AI models. And we see lots of examples of that on our news feeds.

[00:03:14] But one of the things that made me excited to get you on the podcast today was when I was reading how you argue that the environment around the agent often decides whether it actually creates value or not. So what does an effective agent harness actually do today?

[00:03:30] I think that's one of the big things that we've been seeing this year from an AI perspective is that as you have a model, sort of the brain behind what everything's doing, that model needs sort of the arms and legs, the ability to do tool calling, manage memory, interact, provide security, governance, guardrails, all that kind of good stuff around it. And that's really this harness concept, which we've been talking about for a couple years in the market. So it's not a new thing. But I think what we're seeing is that the model technology continues to advance, right?

[00:03:59] By the time you put the podcast out, right? We'll have new models that have jumped over the last leaderboard and the last benchmark. But inside the enterprise, it's a question of like, well, I don't always need the latest and greatest model. I need to do the thing I want it to do. I need it to complete the use case I've been trying to run. I need to run this process consistently. And so what I find is that the harness kind of element is really critical for organizations. And if they have that established inside their four walls, they can then make the choice around what model makes the most sense for them, for the tasks that they're trying to run.

[00:04:28] And so some might choose to use frontier capabilities because they have frontier level intelligence that's needed. Others might choose to use a different model, a lesser model in the same family or an open weight model that's more tailored to the work that they need to get done. And so that I think is what these harnesses allow organizations to start to do. And if they've got an effective strategy around both the harness, the model management, the selection process, it allows them to, I think, be more thoughtful in their AI implementation.

[00:04:56] So that's why I spend a lot of time with it, I think, with clients and why I think it's a really important topic. And I think so much of what you said there will resonate with leaders listening around the world. And to further bring that to life, I'm curious, do you have any examples of an enterprise AI project where connecting an agent to a business system actually ended up proving harder than building or selecting the model itself? And I do suspect you have a few stories there. Yeah, I've got a lot of those.

[00:05:26] I'm sure a lot of our listeners here today have those stories, too. And maybe I could just sort of, if we think about it in the context of a large enterprise, and a lot of times we see success stories in social media and the hype around a lot of things that are happening in AI. And some of those companies are five people, 10 people. Their oldest system is a year old. Great. That's awesome. Good for them. But a lot of our enterprises are using technology that might be 30 years old, 40 years old. They might have legacy mainframes that need connection into these AI tools.

[00:05:55] And so whether it's simple office applications or legacy applications, it's really important for enterprises to think, but what's that connective strategy between the legacy systems, their existing systems, and their enterprise AI?

[00:06:06] And so I've seen this in a couple of places where organizations have had to stitch together what we call a model context protocol or MCP server around their applications to provide some form of governance and control of AI accessing that system and doing the right tool calls and capabilities against that system that are governed, that are logged, that are tracked, to open up that data and open up the actions that those agents need to take against those systems.

[00:06:34] From complex CRM applications to transactional systems, we're seeing more and more of that. And then I think what we're seeing in the market is enterprise SaaS providers, enterprise ERP providers are opening up more of their systems to allow for that agent-level connectivity, which I think will be a good thing as we think about applying AI across the enterprise into more areas.

[00:06:56] I mean, and I could also think of some of the sillier things that we deal with in this space too is just the duration it takes to get some of the governance processes approved for this. Because the technology is so new, organizations can be very concerned about, should I allow a connector to my email system? Can it read and write? Just read? Can an agent send an email on the user's behalf? If they do, who's controlling that?

[00:07:20] And so while those might seem like super silly and basic questions, they're critical to the governance process. Because if you don't know the answers to those, you don't know how you're going to establish those connectors. And that can take a really long time in enterprises to go through that, especially the first time through. The second, third time through, then you've got kind of policies, procedures, you've thought about it, you've tested what you're comfortable with. And then I've seen enterprises kind of pick up at scale there.

[00:07:43] So I guess where I go with this is that the more time you can kind of spend sort of mapping out where those connectors are, what critical systems you need to get access to, and thinking through that strategy will really start to help accelerate organizations' AI, applied AI journey. And I do suspect we'll have a lot of people listening in organizations that are incredibly excited at the moment. And they might be experimenting with several agents at once, maybe even as much as a dozen or even more.

[00:08:12] So just to, I don't know, maybe manage expectations here and some of the risks that could be involved. What kind of problems appear when those agents have different tools, different permissions, memories, and sources of data? And one of the reasons I want to ask this is because I think it's very easy to end up in a kind of world of technical debt again further on down the line, isn't it? Yeah.

[00:08:36] I mean, I'm glad you're bringing up the technical debt conversation because I've been part of transformations and tech transformations for a number of years. And if I go back in time to the robotic process automation days or RPA days, a lot of the challenges with RPA was sort of applying sort of, you know, automations to processes we didn't want to have execute over a period of time. Like, we really actually should have changed those processes. So it's that, like, pouring cement over things you wanted to change. And I think we run that risk with AI agents as well.

[00:09:03] And we see this, we did a survey, Neil, called our State of AI survey that was released in October, end of October. And it talked about organizations in terms of where they're spending their time. And specifically, we saw in the survey results that 81% of organizations are seeing benefits from AI from improving efficiency and automation. But only 25% are truly transforming what they're doing. And so when you ask this sort of question, like, the risk of the agents and what they're doing, I kind of go back to that survey and say, like, hey, a lot of people are using their agents to drive efficiency.

[00:09:32] So it's short run tasks. It's quick actions. And they're doing that because they can apply either a human in the loop. So they're doing a task and forcing a human to review it before it goes to the next task. And so it's driving some efficiency, but not truly transforming how the work gets done. And the 25%, the ones that are truly transforming, are using agents to do long running and process change.

[00:09:55] And the risk that I think has kept people from moving from that just automation into the full transformation is around the governance and controls of those agents. Like, who's accountable for that process if it doesn't work? What's the backup plan if it doesn't work? Is there a human fallback? Is there a systematic fallback? What's that look like? What, you know, can the agents change data in a system of record? Does it have the full create, read, update, delete, CRUD capabilities? Or is it just a read-only agent against a system of record?

[00:10:24] And so those are sort of the governance challenges that I think have stopped organizations or slowed them down in that transformation journey. And I think we also have the challenge that because we're applying a probabilistic technology to a lot of deterministic steps in a process, you've got to tweak your model choices, your software choices, et cetera, along the way to make sure you're getting the consistent output that you need.

[00:10:47] And I think the other thing that sort of hits you is that people are more comfortable with humans making mistakes than we are with machines making mistakes in the States, too. So even if you've got a process that works better with AI agents, better being the sense that it's more efficient, fewer errors, we're still seeing, you know, organizations say, you know, but there are errors there. So let's have the humans keep running those steps.

[00:11:10] And I think those are the things that are really starting to slow down and slow down sort of that transformation percentage group from getting up into a higher target goal. So those are a lot of things that we're seeing in that governance and control and risk space for folks around agents that I think kind of slow it down a bit. And as the boring ex-IT guy, I'm so glad that you raised the topic of governance there because it is often added after a successful pilot of any tech project, never mind just AI.

[00:11:38] So what kind of controls need to be built into an agent right from the start? And what could go wrong when things like identity, policy enforcement and policy enforcement and audit trails all arrive that little bit too late? I think the organizations that are moving fast with AI have gone slower out of the gate with their governance. And slower by any means the sense that they've taken a beat, paused and said to themselves, okay, like who's on our governance team?

[00:12:08] Who's part of this council that's going to make these decisions? What guardrails do we need to set up? And not just like specific guardrails, but sort of macro conversations about how we think about AI ethics and the use of creative technology, AI and the use of responding on behalf to our customers and handling customer complaints. And so they've set up sort of those guardrails and governance processes up front. And by doing that, they accelerate their ability to deploy pilots later on. Just like you said, right?

[00:12:33] We've seen so many of this go into POC purgatory where you run these POCs and they stop because you haven't run by governance. You didn't think about the implications for all the different use cases and what's going to happen with the data, et cetera. So those are kind of, I would say, the things that accelerate by putting governance first. Now, the questions organizations, I think, need to ask, and this starts top down at the board level, who's accountable for AI? Do you know? Like, do you have a person that's there that's accountable?

[00:13:02] Much like the CISO when it comes to cyber, like we know the accountable party. I think a lot of organizations still don't have that question answered. And so that's one that I'd start just at the basics. And then from there, I think that person, the governance council need to be able to say all those questions around data, customer engagement, internal data, customer data, third-party data that you're using, what those contracts all show, what you're letting your systems do. And then finally, like layering in the regulatory environment on top of that.

[00:13:31] If you're a global multinational, you're dealing with your regular data rules that you have AI rules across the globe. If you're sitting in the U.S., you're dealing with multiple different state interpretations and implementations of said AI rules. And so you're navigating a very complex regulatory environment. And that's what that governance council needs to be on top of, too. Where is AI part of the decision? What's the ruling in the state where that potentially could happen?

[00:13:56] We see this a lot in healthcare, too, is a good example of this, Neil, where it's like, you know, where can AI play a role in health? And so those are things that the governance council really needs to kind of wrestle with and set up and then start to apply that. And then I think this isn't like a one-and-done thing. I think you've got to have this as a recurring process in the organization because back to the first question, the models keep getting better. Their capabilities of what tasks they can take on keep expanding.

[00:14:21] And so that governance team has got to keep looking across new use cases and new applications and new rules and regs as well. So while not the most maybe exciting topic, I feel like it's the one that will slow you down if you don't have your hands all over it and kind of being very thoughtful about it. Yeah, again, such great and incredibly important points that you're making here.

[00:14:45] And for leaders listening who at the end of the day, they do actually want visibility into why an agent acted, which tools it used and what happened next. What should observability look like to them without creating too much data that nobody can make sense of? How are they getting that happy medium right? Yeah, well, it's interesting too. In our survey, we saw that 40% of our respondents actually said they were running autonomous agents.

[00:15:11] And this is a survey about 3,500 global executives in the AI space. So I was kind of shocked by that. I don't know what your take is, but 40% was higher than what I was thinking that number was going to be. So it's great. So then for that organization that are doing those things, they've got to have the logging and tracking. And I think that's critical. The nice thing is that sort of the ability to track all these things isn't expensive. You can write a lot of this data too.

[00:15:36] The key is being able to sift through it and looking for anomalies and being able to fall back steps and processes to be able to catch what happened. And I think early on, you want to have a lot of that data because I think what we're seeing in the market is that you can start to influence the models through reinforcement learning and self-improvement activities.

[00:15:54] And so that full agentic loop, we've kind of talked about it for a while, but we're starting to see the ability to then apply those evals and apply those existing process learnings that the agents are getting from the execution back into how they perform. So I think that's why you want to capture that information and then be able to use it to make your system and processes better. Do you need AI agents that you can trust?

[00:16:17] Well, with an AI data layer providing real-time connection within your data platforms, you can trust your agents to provide accurate solutions. So scale your business by trusting your agentic AI, accurately getting the work done for you. Trust its capabilities with Denodo. And you can do that by simply visiting denodo.com to learn more.

[00:16:41] And I must admit, I've been to more than enough tech conferences and recorded plenty of podcast interviews to have crossed that phrase, human in the loop of every tech conversation bingo card. But where should human approval actually remain part of the process? And how can companies avoid slowing down every single workflow while still giving meaningful control over those high impact decisions?

[00:17:05] Because I think some of the stories behind this and reasoning behind this are so much bigger than that very simple human in the loop phrase that we often hear. That's definitely on the buzzword bingo card for the year, for sure. And hopefully, unfortunately, maybe for your podcast today, I've hit the buzzword bingo a few times. But I think the challenge for organizations is a couple of things in this space. So one, and I don't think you're going to get a consistent answer either. I think that's the other, the number one challenge is in consistency.

[00:17:34] We were doing a panel discussion with a large audience with one at a session. And we sort of did this exercise where we had to answer really quickly about whether we thought an agent could act autonomously or not, or where we would trust it. And the answers varied across our panel. And so if you think about it as like a accounts payable transaction under $1,000 or whatever your threshold is, everyone might be comfortable with an autonomous agent doing that for a large organization.

[00:18:01] And then for a smaller organization, that might be crazy. Like, why would we do that? We're going to review every transaction and everything that goes out the door. Would you allow a customer to purchase entirely through an AI automated system? Or, you know, if the purchase threshold exceeds a certain amount, would you want a human involved to make sure their customer is getting what they expected to get from the transaction? And so I think it, sadly, that the answer is, it depends a lot. And that's why it's so hard. And so I try to encourage organizations to say, well, look, what are your decision rights and authorities today?

[00:18:29] Does every contract go to the CIO that's over a million dollars? Okay, great. If it's under $100,000, who has decision rights over it? And start to really look at those decision rights processes we have for the people that are involved today. And then start using that as a framework to apply it to where you would feel comfortable with the agents doing it. And then there's that group that's like, hey, if they're fully autonomous, like, how are you monitoring them? We're going to see a lot of roles where people are orchestrating series of agents and they need to be able to report on what those agents are up to and take accountability for them.

[00:19:00] So sorry to say it's a bit of a depends, Neil, but I think organizations that are applying their existing governance frameworks and their existing, like, you know, purchasing thresholds and spend thresholds are helpful places to start. Yeah. Another big word of the year that we've got to bring up today, of course, is tokenomics. And I think it often feels like the latest grift in technology where big tech companies are getting everyone hooked on AI.

[00:19:26] Then they start tightening the grip on how many tokens they need and the cost of tokens, etc. And, of course, those costs can rise very quickly when agents call models and tools on a repeated basis. So what have you learned about managing token use, performance and reliability, all without weakening the outcome? What have you seen this year? This is really interesting, too. We did our state of AI report back in January, a report back in January. So January 2026.

[00:19:55] It did not mention the word token in it. And then by our report that we released in October, tokens all over the report. Because it's just, like you said, it's become the topic everyone's interested in discussing. And we hypothesize, and I'll be curious to see where this goes, Neil, that we might not mention the word token cost as much again in the future. Because we sort of peaked in terms of the conversation of, you know, people now aware of what the costs are for the models. But then we're going to kind of go back, I think, to the cloud world of, like, we know what our budget is for cloud.

[00:20:24] Let's run towards our budget and our plan. And so what I'm seeing organizations do to help manage their current costs is a lot of them have dashboards that have visibility into the individual costs. I think in our survey results, we saw about 44% are able to track the unit costs of their AI spent. So not even quite half, but they'll get there, right, in terms of when that survey was in the market, which is over the summer to now, even more people are going to get that visibility.

[00:20:48] So once you have the visibility, then you need to be able to see by persona who's using these tools and how they're being consumed. And some of these are individuals, users, and the IT shop that are cranking away, and, you know, the latest coding harnesses, and great. But you need to see, really, the conversation isn't sort of their spend. And that's what everyone was sort of focused on was this tokes and maxing kind of concept. But what's the output, you know, and how are you doing the measured dollar per output per task that you get for the completion of the work?

[00:21:16] And that's the shift I think we'll see from 26 into 27 is sort of thinking about more cost per outcome. And how do we get to that? And once you start thinking about that, then you're making those choices, like we talked at the beginning about, like a smaller model versus a larger model, open weight versus frontier, et cetera, to get the right set of work done. And I think people will kind of coalesce around multiple different models for different tasks, getting the job done.

[00:21:42] I think the thing we've got to watch out for in this space, too, is that how efficient, and this goes back to the harness a little bit, you know, how efficient are people designing their harnesses to think about kind of the caching of certain prompts that are there, how they're managing the compaction of different things within the queries that are going back into the model and coming back out. So there's a lot of things that organizations can technically do as well to start to manage the cost. But we've seen, you know, the cost continue to go down, right?

[00:22:10] Each sort of time there's a new big announcement, the costs are going down per token. And the tricky part is that organizations still are asking the question, well, where's the ROI? And that I think is really where the focus needs to be and less on the individual token spend. We can kind of get lost in those details. So that's kind of what we're seeing, and I'm optimistic that organizations will get their arms around it and be able to see these dashboards, get visibility into what's going on, and then really focus on the ROI and outcomes.

[00:22:36] And because of the costs that we've highlighted there, and indeed different requirements and different strengths and weaknesses across so many different models, a model agnostic approach seems like the most sensible option here. But where should companies be designing for now if they want flexibility in the future without building another expensive abstraction layer? Any conversations that you're hearing here? Oh, yeah.

[00:23:03] I mean, I think we saw this back in early days of machine learning things and model ops management. And so organizations were building capabilities to swap out ML models over the years. So now we've got our LLMs in here, and we've got to figure out, I think, a couple different things. One, every time a new model release drops doesn't mean that we need to reevaluate every model in our portfolio and see if we need to make a change. I think if we've got tasks that are working effectively with the agents that we have, great.

[00:23:31] Like you stay consistent with that approach and continue to execute against that. Especially if you've spent time, let's say, with an open-weight model providing and doing some form of fine-tuning against it. But if the next iteration of that open-weight model comes out, you've got to really question the value of you kind of applying and wanting to kind of, again, do the fine-tuning process and retesting. So that's sort of the decision framework I'd be looking at is like, hey, what new capabilities are being unlocked by the model? Is there a new price performance tradeoff that we should consider?

[00:24:00] Are the tasks that we have running really well today still performing well? And so we don't need to make a change. And then the other piece I heard on a panel that I thought was really interesting from one of our clients was saying that when these things drop, new models capabilities are coming out. I'm going back and looking at the use cases that we chose not to go after and saying, hey, how can I reapply the latest frontier capability to this use case I wanted to do? And does it unlock something new? So I think IT organizations are going to have to have a model ops capability around this.

[00:24:29] They're going to have to have an eval suite that's specific to them because so many of these models that we're seeing right now are doing so well in the benchmarks. But it's like the tests have been out there. So how do you really grade it? You've got to grade it about your work and your industry and your organization and your results. And so having that eval suite handy, being able to run it through your models and you can use AI to go ahead and automate a bunch of these things, too, will be critical, I think, next steps for enterprises to have.

[00:24:55] And then the other piece that comes into it, too, is just also having a good sense of like if you're using SaaS applications, what models are behind some of those solutions? And do you know what's being used there as well? So you have a full sense of kind of the AI that's being deployed across the enterprise. And you've referenced some great insights from the state of the AI in the enterprise report that we've mentioned a few times today.

[00:25:19] And I'm curious that me looking at the report, some of the things that stand out are how many organizations are just lacking insight into their AI spend and the gap between investment and return, how that is narrowing. And elsewhere, of course, we've mentioned AI governance, how that is becoming more important as autonomy expands. But you're in right in the heart of this space. Did anything surprise you in the insights that came out of this report? Yeah.

[00:25:47] I mean, I think the interesting surprise was maybe two. One was that we've run this report for a number of years. And one of the things that we found year over year over year was that organizations kept saying their transformation was about three years out. And so I think our estimability around AI transformation continues to be off. I think we're going to be in this continual transformation mode. And so that might change our mindset in terms of how we think about running transformations in general. It's not a three years and done.

[00:26:17] It's not a two years and done kind of thing. It's a continual mode of transformation. So maybe that's surprising, but an insight that I think isn't readily available in the report itself that, as you look at it longitudinally, is kind of helpful. And the second, and I mentioned this a little bit earlier, was the number of organizations that are running agents autonomously. The 40% number surprised me a bunch. I was optimistic on that one. And maybe I'll throw in the other one that I'm a little less optimistic on.

[00:26:40] And we saw only an 8% increase in organizations seeing an increase in revenue from AI from 20 to 28%. I would have expected that number to be higher. You know, some upside and some downside in the report there. And hopefully people will find some of the other facts helpful as they think about their business cases and where to apply AI to. And for anybody listening that would like to dig a little bit deeper at some of the insights that came out of the state of the AI survey, see more of your work or connect with you, where should they go?

[00:27:09] Where should listeners visit? Yeah, take a look at our AI Institute page. The Deloitte Institute is where we have a lot of this content. And we put out a bunch of our debriefs and other things. So people can stay in touch with that. Hopefully we'll link it in the show notes. And you can always follow on LinkedIn as well. We put out a lot of our content there too. So yeah, I appreciate the shout out. And hopefully people find it useful. Awesome. Well, I'll add the links to everything that you mentioned there. And I do encourage everyone listening to check that out. Some great information. So please go do that.

[00:27:38] If you are listening. But more than anything, just a big thank you to you, Jim, for joining me today, bringing these insights to life. Really appreciate you, Tom. Yeah, thanks, Neil, so much. Really appreciate it. Have a great rest of your day. The next time someone shows me an AI dashboard, I'll be looking for the cost of getting a piece of work done to an acceptable standard. Because, yes, the token count can help explain the bill.

[00:28:02] But the harder question here is whether the task took less time, needed fewer corrections or produced a better result. And Jim's point today about approval limits will stay with me, too. Because if your finance team already knows who can approve a purchase at different amounts, this already gives you a starting point for deciding what an agent can do and when a person needs to step in. So massive thank you to Jim today.

[00:28:29] And remember, you'll find the state of AI in the Enterprise Report and Deloitte's AI Institute and how you can find that in various links in the show notes over at techtalksnetwork.com. But I'd love to hear how you're measuring this and everything we talked about today. What outcome would persuade you to expand an AI workflow that is already in use? Let me know your stories. Good, bad, indifferent. I'd love to hear them.

[00:28:58] techtalksnetwork.com. Send me a DM. Connect with me on LinkedIn just at Neil C. Hughes. But that is it for today. So thank you as always. And I will return tomorrow with another guest. Bye for now.