KPMG - Why AI ROI Depends More on Workforce Behavior Than Technology

[00:00:00] So a huge thanks to Denodo for supporting the Tech Talks Network, helping us produce more than 60 interviews a month. And when it comes to trusted data products, it all starts with the right foundation. And trusted data products start with Denodo because they can help you create, manage and deliver business-ready data products faster with secure real-time access across all of your data sources.

[00:00:26] And you can learn more by simply visiting Denodo.com Why are so many organisations increasingly spending heavily on AI, but still struggling to prove that it is making people more productive? Well, my guest today is from KPMG and we're going to have a conversation that gets right to the heart of AI at Work.

[00:00:55] Because KPMG, together with Macomb School of Business at the University of Texas in Austin, recently analysed 1.4 million real-workplace AI interactions. And some of the findings there will challenge a lot of assumptions about what really separates AI users from everyone else.

[00:01:17] And spoiler alert, it turns out that the most effective users were not always the people using AI most often. They weren't always the most technical people either. The stronger performers were actually those who had learned how to work with AI in a different way. They framed problems better. They guided the tool. They iterated. They used AI as a reasoning partner and brought it into the broader flow of their work.

[00:01:47] And this matters because many organisations are still treating AI as a rollout. And they give people access, count the logins, track the usage, maybe even the tokens and hope productivity follows. But my guests today will argue that the real challenge is behavioural. Companies need to build capability, confidence, trainings and peer networks all around AI so people know how to use it well.

[00:02:13] And we'll also talk about the AI capability gap. Why adoption metrics only tell part of the story. And also we'll look at what first behaviours look like in practice. Why workforce training may decide which organisations turn AI investment into measurable results. But enough for me. I would now like to introduce you to my guests right away.

[00:02:41] So thank you for joining me on the show today. Can you tell everyone listening a little about who you are and what you do? Rassan Shears, I am a principal at KPMG and I have the privilege of leading our AI programme, which we call AIQ for the US firm. I've been with the firm for a very long time. I will date myself if I say how long.

[00:03:05] But suffice it to say, I've had the opportunity to get married, have two beautiful children, have one off in college, all while being at the firm. So I've definitely been a part of the firm's evolution and journey for a long time. Rassan Absolutely love that. What a great story. And one of the reasons I was excited to get you on the podcast today is your recent research analysed 1.4 million real workplace AI interactions.

[00:03:31] And I think what I love about stories like this is there's so much hype at the moment. But I go to so many tech conferences and I've noticed a real shift this year where everything is now looking at real workplace use cases, real measurable outcomes, business value. It's like we've stopped looking at shiny demos and getting right back down to it. But I've got to ask, what surprised you most when you looked at how people are actually using AI day to day at work versus how executives might be assuming that they use it?

[00:04:00] Rassan Yeah, you know, I think the thing that was most surprising is what surfaced around sophisticated use. You know, we, our program has really grounded in awareness, access and adoption, making sure people understood what was possible, making sure our professionals had access to the latest and best AI capabilities and then adoption, which we started, which like a lot of companies do, around use.

[00:04:26] But what this research allowed us to double click and triple click, in fact, into is what use actually makes an impact. What use allows us to deliver increased value for our clients and therefore advance our business. And so the greatest aha was sophisticated use and how it differs between frequency versus approach.

[00:04:51] And it presented to us the opportunity to expand our thinking about how we should be focused on growing the skills necessary for us to get to outsized impact and value. And one of the real standout findings from, for me, from the report is that the most effective AI users were not necessarily the most technical users in the organization.

[00:05:17] So what was it that separated those sophisticated AI users from everybody else? Yeah, the first thing was really their ambition, how much they were willing to challenge the AI in what the AI could do. So their ambition was one of those things. The other was their ability to think about, to think, right?

[00:05:37] Really thinking before engaging the AI, using it as a thought partner, a reasoning partner, and really leveraging the cognitive capability that is available in our AI tools and the capabilities we put into their hands. And then using it to iterate and delegating work to it, which is very different than just going searching, go find, or do something a little bit faster than I could do myself.

[00:06:06] Think about creating a white paper or an email, but really saying, how can I extend what's possible? How do I take what I know, my knowledge, and then challenge the AI to do something that I thought maybe was out of reach? And so that thinking, that ambition, and really using AI as a thought partner is what separated the pack.

[00:06:30] And those users, we also found, spent more time with the AI, but it was in a very different use pattern. And because we had the prompts across a wide set of our employees and a broad level of ranges, we were able to see these different patterns start to emerge. And I think long before AI became a thing, the mantra inside every IT department around the world, and indeed the business at large, was you can only improve what you measure.

[00:07:00] And when it came to AI, many struggled to understand what to measure. And a lot of organizations still seem focused on AI adoption metrics, such as licenses issued, chatbot usage, token usage, which is a controversial one as well. But do you think businesses are measuring the wrong things when trying to evaluate AI success and that elusive ROI? Yes.

[00:07:23] Well, the truth of the matter is there aren't really frameworks out there, which is why we took on this body of work and we're doing some more as well, to start to put together a framework of how to measure and what to measure. Because telemetry data is not always easy to measure, but is the most accessible thing.

[00:07:44] And because even our identification of sophisticated use, this differentiated ability to understand how to track and measure the accretiveness of the AI in the hands of our people, is a different bringing together of data sets than what we would have thought about before. And so I think companies are struggling because there's this lack of framework.

[00:08:10] And then two, it requires you to think about this analysis differently. Again, we did this research with a data set over a period of eight months because not only were we learning in the data, our people's behavior was also moving because they were learning along that same trajectory.

[00:08:31] And so we were able to see not only the pattern, but their movement across patterns that helped us to establish what we wanted to change in our learning, what we wanted to change related to hiring and what we wanted to change as it relates to performance. I would be remiss if I didn't say the baseline of this was that our adoption rates, just like standard usage rates, were already in the 90%.

[00:08:57] And so we knew it had to be something beyond that because we had achieved that pinnacle. And yet we were looking for more. And when I was doing a little research on you before you joined me on the podcast today, I loved how you suggested that many companies are treating AI as a technology rollout instead of an organizational capability. Absolutely love that line. But what does that distinction look like in practice and why does it matter so much?

[00:09:25] Yeah, it's, it's, listen, there are two things happening at the same time. Two things are both true. One, it's a cultural revolution. The way we approach work, the way we think about work, the way we think about teaming. You did a great podcast about the digital teammates and what that does and how it changes the dynamic. That is a cultural thing. It requires us to change the humans. But at the same time, I have said this, if you check me out, I've said this before.

[00:09:55] AI is like a toddler, right? And so yes, toddlers can do way more than babies, but you still don't leave them alone. You still put up baby gates and you put them in play pens and you protect them and you guide them and you coach them. You do let them try maybe one or two steps, but nobody lets them go down all the steps unassisted, right? And so you have to think about this cultural transformation and the fact that AI is maturing at a very fast pace,

[00:10:23] but it's still not a teenager yet. And so if you think about, yes, there's growth. Yes, you're amazed. Yes, you see the potential that comes in the hype. Everybody thinks that their two-year-old is going to be able to change the world. But you know what? That's not true about everybody, but we all think it at first. And so that's okay. We have this great ambition and belief, but how far we push it, what we enable it with,

[00:10:48] how we set up the guardrails are going to determine kind of how those two things come together. And so that's, to me, what's happening and what's different. And so we have to meet the humans where they are, manage the AI where it is, and then have those two things evolve together. Because what we have found is the power is meeting them both at the right point so that you can get the most out of them together. Love that.

[00:11:16] And another phrase that really stands out is the behavioral capability gap. And again, I'm curious from everything that you're seeing and hearing and the conversations that you're having, what does that gap look like in organizations? And how can leaders listening maybe identify whether it exists in their own workforce after listening to our conversation tonight? Yeah, you know, so I'm going to give you an answer in two different ways. So I'm going to talk about the organization level, then we'll come and talk about the individual human level.

[00:11:46] So at the organization level, we haven't, in many cases, created enough agency, enabled agency in organizations for our employees. We'll talk about their skills and what they need. But the organization needs to enable agency in a different way for people to work and reimagine process and the way work happens. That's very different than writing your email faster, right?

[00:12:15] It's saying, why are all the steps that we did before necessary in this flow? Because a lot of our processes and our structure are there because they were the boundaries of humans. But now you have an agentic teammate that doesn't have to work in that same way. It doesn't sleep. It doesn't eat. It doesn't rest, right? And so it doesn't get tired. It doesn't get bored. It operates differently. And so what you have to do with it is different. Or what you can do with it is different, right?

[00:12:45] Which really unlocks things. So this agency to think differently, really leading from the front where the organization says, we have a stated objective that everyone is galvanized around that provides this agency for people in the organization to do something different, I think is an organization level capability that has to be established, particularly in regulated organizations. That's tricky to how to think about doing that within the confines of a regulated environment, but still very important.

[00:13:15] I've seen clients who have been able to come up with that combination really do some remarkable things that nobody thought was possible. Now, let's talk about the human side of that equation. You know, if you could see me, you would see I have some gray hair, so I'm a little bit older. And we all have our preconceived notions about what must be true. It must be true that those who are digital first,

[00:13:41] earlier career professionals are going to be better at some of this. But what we found is not necessarily. What is necessary though, is that you understand how to get the most of AI, which means that you understand the difference in the capability between models. You understand the difference in the capability between tools and that you can employ them in a way with your knowledge of the business situation

[00:14:09] that allows you to get to outpaced value. And that we found was in our manager and above ranks. In fact, they could bring those two powerful things together, which caused us to say, hmm, just being digital native, just being a high user doesn't move the needle enough. I need to really make sure that our professionals understand both the value of that business connection and understanding of industry function, coming together,

[00:14:40] a process to come together with the tools and how to get the most and the best and highest use of each of them. And so that is a way we pivoted, like how we think about training. It's a critical thinking, yes, but it's also critical thinking and ambition and knowing the right thing. I give an example that likely for you, for me, I know for sure.

[00:15:05] No one has to tell me if I should use an Excel product, a Word product or PowerPoint. I know based on the thing I want to accomplish, which one is the best tool. It's intuitive. If I were not in the workplace and I was in the kitchen, I know what kind of knife I need based on what I want to cook. Like I don't even think about that anymore. Well, soon, soon, with the right level of enablement in our workforce,

[00:15:31] people that are in the workforce are going to understand how to do that innately with the AI capabilities that they have access to. And that's going to make them so dynamic, so impactful, and really unlock the value that is purported in the hype. It will become the reality. Wow. And from your perspective at KPMG, when we're talking about how important the workforce training

[00:15:57] is and the cultural changes, how important is it when compared to maybe the technical elements, the underlying AI models and the tech stack itself? It's up there, isn't it? It is definitely up there. You could have the best AI, have access to the earliest models, have access to the greatest or the most compute. But if you don't have people who are endeavoring with ambition, who understand what to use for

[00:16:22] what situation, who are not willing to iterate, use it as a reasoning partner, really engage with the AI differently, you're not going to get the return. And so it is for sure, for sure true that those who are most sophisticated are going to go seek high-end AI capabilities because they believe and they understand how it will amplify their impact.

[00:16:48] But you cannot only think of this as a tech rollout. It's just not going to deliver the ROI that is hyped but realized in many organizations. It has to be the two of those together. It just doesn't work otherwise. And another area I know you're passionate about is AI-first behaviors and peer-led champion networks.

[00:17:13] And one of the things I try and do on this podcast is inspire those listening to take some of the examples that they learn about and maybe try and bring it to life in their own organizations. And if we can inspire anyone to that, are there any examples of organizations that you're seeing redesigning workflow successfully so AI has become part of how work gets done rather than just another tool that employees occasionally use? You don't have to name any names, but is there anything that come to life there? Yes, I'll give you a couple examples.

[00:17:43] And I think some of the things that are working are tried and true techniques. So first is bringing AI into the flow of work, right? And that isn't just making the tool accessible. It includes adding instruction about when to use the tool, creating nudges and say this is a good opportunity for you to consider, and then referencing the tool so you can start to connect the dots between the work and the tool that makes most sense for the work that's in your environment.

[00:18:12] It's also putting AI, we call it our quad, where we bring a quad residency into individual offices so you can go and get in-person help. It's amazing how with this technology, you really, really seek out help from a human, but putting humans in close proximity to each other so that they can help each other and you can learn and you can learn from your peers. So that's one.

[00:18:39] So in the flow of work, both physically and in technology, in the flow of work. The other is, and you referenced it, our peer network. This is a time where we are learning from each other in new and very impactful ways. And not just learning about the art of the possible, which many talk about, but really very pragmatic things. Hey, guess what? Today I created an agent that allowed me to clear calendar space and optimize how I'm going

[00:19:09] to prepare for this meeting based on shifting things around. That's really awesome. It's a great idea, especially if you're time crunched, right? Sharing that with your peers who are likely having the same kind of challenge, like helping to do that. Also connecting groups that are at different levels of what I would call understanding. And we can see that based on how they're engaging with the AI and giving them a bit of a buddy,

[00:19:38] if you would. And so that they're able to learn together and they feel safe, right? And you want this psychological safety because it's very new. And there's a lot of this, not to quote a TV show, but fear factor in the system about job safety and other things. And so helping people feel safe in their learning. Because at this time, at the speed of change, everyone has to learn to be a learner.

[00:20:04] It is like a critical skill set that you can retool yourself because what was true in January is so far different than what is true today in May that if you weren't like a voracious learner, learning, practicing, listening, engaging, setting aside time just to focus on your own development so that you can be a more impactful employee, it's going to be hard for you.

[00:20:29] And so that kind of safe environment and creating that connectivity has been the things that I've seen clients do and that we've done that have really moved the needle the most. And I think we really have all learned so much over the last three years. And it's very easy to forget just how far we've come. But I think we're now starting to think bigger than just that low-hanging fruit. We're starting to get things out of pilot phase and out there.

[00:20:56] But as we look into the future, maybe the next six months, 12 months, 18 months, what do you think will begin to separate the organizations that are genuinely benefiting from the AI economy, from those that are still spending heavily on AI but struggling to show those measurable business outcomes? Do you think we're going to start seeing this? I do. The leaders are going to break away from the pack. If you were to have asked me this six months ago, maybe the end of 25, I would have said,

[00:21:25] hey, everybody's learning. Like, get in. Start now. Well, that's not true anymore. The leaders are separating. And it's a very uphill because of the pace of change. Getting in the game is really important so that you're not left behind. So, and I don't say that as a scary thing, but I say it as a true thing. I think about myself as a kid. Listen, everybody else got the training wheels off and their bikes are gone and I'm the only

[00:21:53] one left that can't make it to the park because I'm still trying to figure it out. Right? You need to go. You need to get in this game. You need to be able to move because those bikes are going to become electric and you won't even know it because they'll be at the park and off to the beach. And so, you really, really have to get started. And I do think we're going to see that separation in this year even more materially. So, that's one. I think the other thing that I would say about that is the move from pilot to enterprise scale

[00:22:23] is hard. It requires you to challenge the status quo thinking about a lot of your policies, a lot of your procedures, to ask yourself why you don't do things, why you must do things. And that goes back to that agency I talked about. If the professionals in an organization do not believe that they have been enabled with

[00:22:48] a level of agency to try different things, question in new and different ways, you're going to see yourself falling behind because you're beholding yourself to old rules. There are analog rules in an age of AI that are not fit for purpose for future ready organizations. And so, if you're not taking a step back and challenging that, thinking, you know, it's not new for people to say, think outside in, but this really flips it on its head even in that approach.

[00:23:17] And you have to reaffirm, and I'm not saying you throw everything out, but you do have to reaffirm that it's fit for purpose in the world in which you're operating. And maybe going back to your prior question, another thing that I've seen people do is shaking off the hackathon games of the past. You know, there used to be the hackathon around. Everybody did hackathons back in the day. It's really challenging people and giving them agency in a controlled environment just to say, hey, if none of these rules applied, what do you think we could do with it?

[00:23:48] Change the finance organization. If none of these old rules applied, let's just see what you think we could do. And having the opportunity to try that, every single client who's done it, they may not accept something soup to nuts. The technology might not be ready. Their data may not be ready. Their cloud infrastructure may not be ready to do that, right? Their legacy systems may not have this enabled.

[00:24:14] However, they're able to take bright spots and nuggets and create a future, a North Star that they believe is attainable and start moving that direction. So give your team agency, create that safe space and question everything you thought was true. Wow. And I think that is a powerful moment to end on. And I'm sure there's going to be a lot of people listening inspired by your words today.

[00:24:40] So for anyone listening that wants to learn more about how KPMG translated the insights we talked today into a set of AI-first behaviors supported by things like practical playbooks, training, peer-led champion networks, and that behavioral capability gap that's emerging inside organizations that are investing heavily in AI or even connect with you or your team, where would you like me to point everyone listening? Feel free to check out our site. We have a lot of thought leadership that we're making available readily.

[00:25:11] Follow me on LinkedIn. We're pushing out lots of content and doing lots of LinkedIn lives with clients to share learnings. Listen, this is a season where almost everything people are trying is new and we can all learn from each other. And so set that time aside, be a student of what's possible and come and check us out and use us as a vehicle to help you learn more, to make an impact in your organization.

[00:25:40] And I will post links to everything there. And anybody interesting in learning about why workforce training might become the real drivers of artificial intelligence, ROI, or how leading organizations are building the internal capabilities and operating structures needed to turn AI investments into measurable performance. Please go check that out. Meet me on techtalksnetwork.com as well. I'll be interested in hearing what your thoughts and your takeaways and what you're doing.

[00:26:09] But more than anything, thank you for sitting down with me today and starting this conversation. Pure gold. Thank you. Absolutely. Wow. What an incredible conversation. My guest today was someone I could have just spoken with for hours. And I think the conversation moved beyond access, adoption, and tool choice and into something far more practical. People need to learn how to think with AI. They need permission to question old processes.

[00:26:36] They need training that goes beyond prompts and shows them when to use AI, how to guide it, how to connect it to real work of the business. And I love that comparison she shared of AI to a toddler. Because yes, it can do remarkable things. But it still needs guidance, boundaries, and human judgment. And I think it was this framing that cuts through a lot of the hype. Because it reminds us that productivity doesn't just magically appear when a new tool is switched on.

[00:27:05] It actually comes from people learning new behaviors. Leaders creating safe conditions for experimentation. And organizations rethinking how work actually gets done. And the big takeaway for me here is that the AI winners might not be the companies with the biggest budgets, but more likely to be the companies with the strongest learning culture around AI. And that means playbooks, peer-led champions, coaching, safe experimentation,

[00:27:34] and a willingness to revisit processes that were built for a pre-AI world. So I'll include links to KPMG's work, the report we referenced, and my guest profile in the show notes over at techtalksnetwork.com. But I'd also love to hear more about your thoughts. Is your organization measuring AI access?

[00:28:01] Or is it measuring whether AI is genuinely changing how your work gets done? Love to hear from you on this one. So please drop by, let me know. Other than that, I'll be back again very soon with another guest like this. If you'd like to join me on here, please message me and let me know. A big thank you to NordLayer for backing the podcast and supporting the kind of real-world cybersecurity conversations that we need more of. Because as someone that records 65-plus interviews a month,

[00:28:30] I've personally seen a huge increase in browser-based attacks over the past year, whether that be phishing, malicious extensions, account takeovers. Because the list is long. And it's all happening where people spend most of their time, inside the browser. So NordLayer's new business browser, that's built to address exactly that. It blocks malicious sites before they load. It limits risky behaviors like uncontrolled downloads or data sharing.

[00:28:58] And gives you visibility into how your team interacts with web apps. And it also helps you stay compliant by controlling access and enforcing policies without the need to rely on multiple disconnected tools. So for anyone listening that is thinking seriously about reducing risk in SaaS-heavy environments, this feels like a smarter and more focused approach. And you can learn more about it by visiting nordlayer.com slash browser. But that's it for today.

[00:29:28] So thank you for listening as always. And I'll speak to you all again very soon. Bye for now.

[00:34:52] So let me know your thoughts. I'll be back again real soon with another guest. But thank you for listening today. And I'll speak with you again soon. Bye for now. Bye for now.