Testing the Blast Radius of Agentic AI With NTT DATA

[00:00:00] AI agents are only as strong as the data that they're given. When provided with an outdated data set, your agents could end up doing more harm than good. But not with Denodo. With an AI data layer built within your platform, your agents are provided with real-time data changes. So with Denodo, your agents can finally make the right business decisions. Simply visit denodo.com to learn more.

[00:00:27] What happens when an AI agent follows the documented process perfectly, only to discover that the real process lives inside somebody's head? This is where many automation plans often meet on a Monday morning. My guest today is Bill Wilson, Executive Head of Data and AI Solutions at

[00:00:52] NTT DATA UK and Ireland. And their global public sector work gives him a close view of AI moving from pilot projects into decisions. Decisions that affect real people. So today, I want to discuss his wonderfully practical test for Responsible AI. We see it written a lot on our news feeds, but he simply asks, is the system competent? And what is the worst thing that can go

[00:01:19] wrong? Great starting point. And Bill will explain today how to measure an agent's blast radius, preserve tacit knowledge, use AI in complex public services without removing human agency, and stop the human in the loop from just becoming a proverbial rubber stamp. So if your AI roadmap contains agents, this conversation will offer questions definitely

[00:01:44] worth asking before you give them the keys. But enough from me, let me introduce you to him now. So thank you for joining me on the show today. Can you tell everyone listening a little about who you are and what you do? Hi Neil. Yes, great to be here on the show. Bill Wilson. So I look after AI globally for public sector for NTT DATA. NTT DATA is a big global systems integrator,

[00:02:09] and it's my privilege to work with government customers around the world on their use of AI, getting the best out of the technology, managing some of the risks. My background is in data, so I really came from that world in the last kind of 10 years or so. Awesome. Well, thank you so much for joining me today. There's a lot I want to talk about. You first appeared on my radar when I was reading how you've got a wonderfully pragmatic test for all things

[00:02:35] AI, and that is, is the system competent? And what's the worst thing that could actually go wrong? So tell me more about that and how leaders should apply those two questions before allowing AI, or indeed agents that everyone's talking about right now, anywhere near any real business processes. Certainly. I might ask you in a minute why that came to your profile. But yeah, it's not a complete

[00:03:00] test, of course, because there are considerations like regulation and cost. But as a starting point, it's not a bad place to start. And the reason is because a lot of the work I do with governments, there is understandable nervousness about the application of AI technology to lots of different processes, not just the ones that kind of touch the members of the public and so on. And sometimes a simple way to cut through some of that is to say, well, what's the blast radius of this AI, right?

[00:03:27] And a good example of that would be some of the work I've been doing with something called the Civil Service AI and Data Challenge, which is a competition we run in the UK public sector and are now starting to export abroad. And in that company, several servants, they have the opportunity to submit ideas for how to use AI more proactively and effectively in their day jobs. And quite a few of them are submitting ideas like how can we help the members of the public work through a process,

[00:03:54] an application for something in a much more straightforward way. And you can probably imagine yourself how that might go, Neil. You know, the common example is there are dozens of ways to apply for grants for certain things. They're often very complicated. And actually, if you want to put grant money in the hands of people, why don't we make that more simple? Now, in that scenario, you can certainly use AI solutions to help people understand whether they're eligible, help them work through the process. But at the end of the day, eventually, that application will fall into a standard government

[00:04:21] process for approving someone for that grant. And let's say that the AI is 98% successful. So 98% of people going through that process will have a great experience. 2% of those will go through a process where the AI has led them to think it would be successful, and it turns out not. But there is a well understood, robust, and usually human centric process at the end of that. So in that case, the blast

[00:04:47] radius is relatively small. The number of people who are inconvenient is small, no money is going to go to the wrong place. Maybe in the rest of this conversation, we'll look at other scenarios where the AI could potentially do more damage. And if you look at the EU AI Act, which obviously doesn't apply in the UK, but many of the principles are still useful, thinking about how it could impact people's access to work and employment, how it could impact access to justice. These are areas which are obviously more

[00:05:13] sensitive, where the blast radius could be greater. So I guess that's kind of part one, which is, what's the blast radius. And the second part is, is the AI effective? A different way of answering your question is, if either the AI is ineffective, or the consequences are enormous, you probably want to turn it down anyway. But thinking about, is the AI effective? There's often a number of ways we can use to test that. Things like evals. Quite often, we're putting AI and people together to just check

[00:05:40] the AI is making similar decisions to people. And so if you start in that place, it can be a good gateway to understand which processes or which aspects of a workload can be trusted to be done by AI. So far this year, I've been to 15 different tech conferences, and predictably, every single one of them, agentic AI and AI agents has dominated the theme there. And that's one of the things that

[00:06:08] makes me a little nervous when we're talking about teams with hundreds and sometimes thousands of AI agents off out there, doing their thing, talking to other agents. And you were talking about increasing the blast radius a moment ago. This is just incredibly worrying, just how quickly that blast radius could spread. But I mean, you've argued that organizations can even get agentic AI wrong before they've deployed an agent simply by automating maybe the wrong process. So how do you determine

[00:06:35] whether a workflow should be automated or maybe redesigned first or maybe just left largely in human hands? Yeah, that's a classic problem, isn't it? I guess the first thing to say is, if you took a step back and said, let's review all our processes across this entire enterprise and get them right before we introduce AI, then you'd be waiting a long time to get some of the benefits of AI. The other thing to say is, when it comes to deciding what to optimize, sometimes it's good to look at

[00:07:04] processes where there are a number of opportunities for AI to be employed. So you take an end-to-end business process and you realize a chunk of that functionality through AI. And the reason that's useful is because if you just take one small part of it and add AI, then it can be quite hard to measure the benefits of that. Whereas if you can make significant savings in something where a number of adjacent use cases could be used in one end-to-end business process flow, that can be a really good

[00:07:32] example. And I guess another part of the question is, how long is it going to take to redesign a particular process anyway? Because as you start to apply AI and agendics, you will certainly learn a lot more. And maybe the AI will even suggest different ways the process could be optimized in a way that kind of human-based, paper-based process optimization could never have got to. And of course, there is a traditional answer to that as well. Process optimization is not a new

[00:07:59] science that's been invented by AI. We've tackled these problems before. And so there are traditional ways of deciding what a good process is to optimize and not. And maybe later in this conversation, we will talk about which things are good to leave in the hands of people and which things are good to leave in the hands of agents when it comes to human decision-making and the fine-tuning around that. 100% with you. And another problem I think doesn't appear on most process maps is how

[00:08:27] much of the, just how much experience employees have and make decisions and the tacit knowledge that they have that has never been documented. It's just inside their heads. And we all know people in the workplace like that, but God forbid they were ever to be hit by a bus or something because they've got so much knowledge. So how do you capture that human reasoning? So an agent doesn't just blindly follow a process that bears very little resemblance to how work actually does get done.

[00:08:54] I think that's a fascinating question. And it's good to ask at this juncture as we start to use AI for much more broad and wide-reaching kind of capabilities as well, rather than doing very simple point jobs where that importance of tacit knowledge becomes much more relevant. I'd say tacit knowledge kind of falls into a number of categories, especially if you look at some of the academic literature on this. But one of the areas I've been looking at in tacit knowledge is

[00:09:21] tacit knowledge of how IT systems work. Let's just start there, right? And in those scenarios, what you're really talking about is things that haven't been documented. And we can close the documentation gap because there is a specific outcome the IT system delivers. And then there will be a specific answer as to why it does it that way, right? So quite a lot of tacit knowledge I'm looking to gather is about is this kind of closing those kinds of gaps, which can turn tacit knowledge

[00:09:47] into documentation. And the way you do that, of course, there is not you can't say to someone, can you tell me all your tacit knowledge, even if they wanted to give it to you, they can't, right? There's work being done back to the 1960s about this is very hard to express all that information. So the way you have to detect that is ask kind of very specific questions like, what is this process doing? Why did you make this decision here? You know, can you give me some of the rationale for that? So that's, I guess that's, that's part of the answer where you're looking for something that

[00:10:16] can be turned into documentation. The trickier bit is where the tacit knowledge builds up over many examples. And it's not so feasible to say, okay, here's the actual rule that's been followed. It's more like people are realizing something is unusual, because they're doing the job day to day and, you know, realize something's out of process, right? And in those scenarios, you have to train the AI in the same way you would train people. How do you get someone new in an organization to understand

[00:10:44] that tacit knowledge in the first place? They join your team, they've read the process, but they don't know how the thing actually works. So then they make mistakes. And then, you know, a supervisor will say, no, it's like this. And that's how people acquire that tacit knowledge. And in future, AI is going to have to, you know, learn in a similar way. And perhaps ask questions about why did I get that wrong? We're looking at things like behavioral traces and so on. So picking up evidence of how

[00:11:09] the decision making actually happens, and then probably asking people. Now, the more interesting question will be how we make sure that human beings have enough understanding of the process themselves, that eventually you get to a point where they can operate the right kind of supervision over agentic processes. And one of the challenges there is if the human being isn't in the day-to-day operation, then noticing that something has gone amiss is harder to do, right? Especially when the

[00:11:35] human being has information filtered based on what an agent is doing. Anyway, but perhaps that's a different topic, not germane to your question, but that's part of an answer, I think. Yeah. And I think you've also got a unique vantage point here because your work gives you visibility into how governments are approaching AI on a global scale. So in complex areas, let's say tax, benefits and public services, where decisions can ultimately affect people's lives, where are you

[00:12:04] seeing AI deliver real value without trying to automate the entire decision? What are you seeing there? Yeah, that's a great question. And a key part of your question there was delivering real value. So we're obviously seeing a lot of AI in kind of the prototype and proof of concept stage in those more, let's say, sensitive human-based decision making. If we're talking about areas where AI is already live and delivering value, let me break it down into a few different areas, perhaps. The first

[00:12:31] one is where governments kind of need this information, need this technology by necessity. And that's because either the volume of information that human beings are trying to process is unachievable, you can't get through it, or because you need to react to such a fast speed that you, that AI is required for that kind of fast turnaround. So examples of the kind of volume information, let's say you're trying to do some vetting of an individual, what should they, are they allowed to

[00:13:00] take up this post in government? There's a lot of them getting a human being to look through all their kind of social media to see if they have some extremist views that would make them inappropriate in that kind of scenario. That's very hard to do manually, right? So there's a kind of volume type of problem. And increasingly in law enforcement as well, there's a volume problem where law enforcement agencies are trying to protect the public. There's a vast amount of digital information out there and

[00:13:25] chewing through that without AI is practically impossible, right? So there's an element of necessity. And to take an example, which is more in the kind of, let's say, real-time response space. So for example, at work, the NTT data have done with the state of Tennessee. In the US, we've built a emergency response system to help them deal with disasters and sort of natural disasters and so on.

[00:13:51] And of course, there, the scenario is information is moving very quickly, you have to make decisions very quickly. And therefore, AI being able to process that information and make some recommendations that clearly a human being is going to act on is really important. The other thing I'd be to say for your listeners is people start to feel worried about how AI might be encroaching on areas that are closer to human experience, getting back to the point of your question, things like tax and benefits,

[00:14:16] and maybe the justice system as well. There could be an element in which people are not being well served by human-based processes today. There are long backlogs in many types of government. I know things like case management. In the justice system, we have big backlogs. In the benefit system, we have long backlogs and so on, right? And so actually, there could be a trade-off here where people feel nervous that AI is coming a bit closer to those very sensitive human problems.

[00:14:42] But on the positive side, the people languishing in queues waiting for their issues to be resolved is not ideal either. And so without going into specifics because of the public nature of this podcast, we are working with government departments who are looking at things like case management systems and how we can use AI maybe to accelerate the review process in case management systems. We have live processes that do that. Again, coming back to the Civil Service AI and Data Challenge,

[00:15:09] the winner of that competition this year was in the case management space in the benefit system. Now, what's going on there? Of course, we're not taking away human agency, but the AI is particularly valuable at the start of that process where you're looking at a case, something that doesn't look quite right from a kind of benefits compliance perspective. And what's going on? The AI is out there collecting information, maybe asking for follow-up information before a human caseworker

[00:15:35] gets to see the case. And of course, you're not, again, coming back to Blastered, you're not taking away anything from the human agency there, but you're reducing a huge amount of the admin and hopefully shrinking the time for those cases to be resolved, which is beneficial for the people involved and for the taxpayer. And when I was doing a little research on you, I quickly learned that you've worked across government, banking, insurance, automotive and telecom. And you've seen information

[00:16:02] repeatedly break down as it moves between teams and organisations. And I'm sure you've got a lot of war stories there. But a question I've got to ask is, why does this problem still persist? And what should businesses fix before expecting AI to just make sense of all this fragmented and contradictory information? I realise this is an entire podcast episode on its own. Yes. Great question again, Neil. And I suppose quite a lot of my career has really been focused

[00:16:31] on math as much as AI. Again, what I'd say is similar to the process simplification point, don't wait for all your data to be in an ideal state before thinking about how to use AI. Think about how to fix the data you're specifically going to need for a particular AI process. And that might be things like data quality issues or data ownership issues or data sharing and so on. Now, to try and answer your question, though, about why these messes happen in the first place, which

[00:16:57] stop many processes, not just AI, from working effectively, my observation is there's kind of a few things going on. One of which is teams are working in silos, human beings work in silos. And, you know, it's well documented that IT systems tend to develop in silos as well that relate to how organisations are structured. Between those silos, you can get misunderstandings of things like how data is defined even. So I'll be talking about the same thing when we both talk about customer. And those processes get,

[00:17:26] those problems get worse when there are, say, particular IT systems present in different departments of an organisation that use a set of terminology. And that terminology actually isn't consistent with each other, right? So there's a kind of meaning problem between teams that are struggling to share data. They're literally talking at cross purposes. And in my career, you know, I find this all the time. So you're subconsciously translating between, okay, this team talk about it in

[00:17:53] this way, this team talk about it this way, it's actually the same information, right? But so that could be a sort of inherent blocker. There's another problem around, let's say, sharing data between teams as well. So there's a kind of human resistance, even if the data is understood between teams around, well, shall I give you my data? Are you going to give me yours? I worked on a system for the police many years ago. And chief constables were delighted with the idea of information sharing

[00:18:21] across police forces, as long as they didn't have to share their own information, they were absolutely delighted if everybody else could share theirs. And so you see that asymmetry immediately in that very sort of simple anecdote. And the information of the asymmetry and risk asymmetry plays itself out like this. You're interested in my information, perhaps I don't need anything from you right now. But if I give you my information, then I'm taking all the risk. And also, you're going to come back to me and ask me lots of questions. And you're probably going to want that information in a certain

[00:18:51] format and a level of quality that I don't need. Whilst it's very helpful for you, I'm just being a good corporate citizen now. And so the motivations to share information between teams are often not there as well. And there are kind of fundamentally human type of problems, which, as you say, is the subject of another podcast. But that would be a start point of your question anyway. It really is. And the NCSC and UK Financial Authorities, they have also recently raised concerns

[00:19:19] around agentic and frontier AI. And from a delivery standpoint here, what does responsible deployment, what's that actually look like? Where should organisations insist on that human approval that we mentioned earlier and monitoring escalation paths or hard limits on what those agents can do, guardrails, et cetera? What does that actually look like for people listening? Okay. Another very broad question. But there's some kind of quite useful models out there, which

[00:19:47] would be easier if this wasn't a podcast and had some visualisations. But if you take kind of the three things that agents might have, the ability to take external inputs, the ability to access internal data and the ability to make essentially unsupervised changes to your corporate IT systems of various kinds. Sometimes we say all about access to tools. Now, if an agent has all of those things, it's kind of in the danger zone, as you might imagine. But it's quite a nice model to think about the levels of

[00:20:17] risk. And so if you have agents that have kind of two of those capabilities, less risky. So at a very high level, that's one model, which is a helpful way to think about it, right? I think another consideration is what guardrails you put in place. There's a lot of useful work that's already been done, including inside entity data about what those guardrails might look like and what's proportionate in each case. So it's things like making sure your prompts are accurate, the system

[00:20:46] prompts involve you set up evaluations that include other LLMs as judges. It's about transparency and explainability, feedback loops, and so on, right? So these are some of the practical things. How do organizations choose where to implement those things? I think was sort of the core of your question. And I think another thing to say, perhaps before I answer that very specific question is, it's useful to make those decisions once. So I see organizations going through the same

[00:21:16] decision-making process time and again, whereas you can use a kind of case law type of approach. We have in the past, when we come across these kinds of decisions, we said these agents were okay with these kind of guardrails. There are ways of making that more efficient. The reality is it's on a case-by-case basis in terms of which types of agents are going to need which levels of controls, and also which type of industry you're operating in. So you talked about the NCSC,

[00:21:45] the Treasury, and FCA have also issued their own guidance. And it's true that in financial services, for example, there's a lot more reticence. Some of our banking customers are kind of saying, well, nothing will go out without a human being in the loop somewhere, at least for the time being, right? So it varies both by use case and by industry as to where you set that line, I think. And if we have a look on our news feeds or any show floor of any tech conference at the moment,

[00:22:13] there is so much enthusiasm about AI augmenting workers. But I also loved how you've raised the harder question that surrounds de-skilling. So if AI increasingly handles repeatable work, and also taking some of the entry-level roles, how do we ensure tomorrow's employees develop the experience, the critical thinking and judgment that is required to question some of the outputs, supervise systems doing work that they've never actually learned to do themselves? A bit of a

[00:22:42] conundrum almost. Indeed. And we touched on this slightly earlier, one when we were talking about human in the loop and how fantastic knowledge is a topic we were talking about there. So I think it's fair to say I don't have all the answers here as to how, because this is a kind of systemic type of problem, but it's certainly one that worries me. If you're asking the question about kind of how do we make sure the human beings aren't de-skilled to the extent that they can't be effective humans in the loop, right? What is interesting is

[00:23:13] this problem has been come across before in other industries. For example, airline pilots, you know, the level of automation in aircraft is quite high these days. And actually, in the sort of schedules and training for pilots, they deliberately have to include manual controls of the aircraft to make sure that they don't become reliant on those automated systems. And you know, they've got reasonably good, praise God, because otherwise we'd all be in danger at

[00:23:40] working out where that fine balance is about keeping pilots' skills well honed, while at the same time, mostly using automated systems, particularly when things like takeoff and landing and so on. So there are paradigms there. And I think what will be necessary in the future is for some humans to see the day-to-day cases as well. Because if you aren't looking at what looks normal or having eyes on that, then spotting what is abnormal will be much harder as well. But one of the things we do in

[00:24:09] some of our systems, though, Neil, is we introduce, I guess, kind of trick questions. So we present things to a user to say, can you validate this? You know, what do you think of this outcome? Where we already know the answer, and then see if humans are kind of lapsing into perhaps not such good practices and a kind of authority effect of IT systems as well, which is kind of well known. And if we're not careful, the human in the loop just becomes the rubber stamp, right? So I think we're

[00:24:35] now at that stage where, especially that the balance of work is shifting more to agents, where we need to worry about what the human in the loop is capable of, and how they can be an effective check and balance on what the agents are doing. I suspect we're just scratching the surface of what that really means, Neil. Yeah, I completely agree. And I always try and give people listening some actionable takeaway. So

[00:24:58] if we do have a CIO listening who has dozens of AI experiments underway and increasing in pressure to then introduce agents into the mix too, are there any practical checklists that are out there for deciding which projects deserve to reach production, which need redesigning, which simply just need to be stopped because they're going the wrong direction. Any advice there?

[00:25:24] Yeah, that's another good question. I think we, sometimes we struggle to stop pilots that aren't delivering value. So that's the important area to recognise. It's sort of, if there's a CIO listening to this, it sort of depends on your motivation for wanting to get those things into production. Maybe you want to be famous in your own right, in which case you will be perhaps more interested in use cases that are very specific to your industry rather than, you know, use cases that many other people have solved. So that might be coming to part of your decision making. You know,

[00:25:53] I guess it goes without saying, you know, where's the return on investment, but not just that, but which processes, use cases, agents, is the return on investment easily measurable as well, right? So that we can say, why does that our listener want to get some of these agents live? Yes, they're under pressure, but they also want to turn around and say, I've had some success and this is how I can measure it. So that's another consideration. Also goes back to something I said earlier in the conversation about use cases that can cluster

[00:26:21] together so that the net effect of all those use cases and the agents working in one part of your business becomes much more amplified and, as I say, easily measurable. Another consideration would be diversity. So are you learning something from a number of different types of use cases that you're putting live as well? So clearly the POCs are a learning process. I agree with that, but also putting things live and seeing how they're used in production is a learning process as well. So

[00:26:48] diversity is sometimes quite useful as well. I guess one last thing is, are you tackling use case that a vendor or some SaaS tool that you've already bought is just about to introduce anyway? So how specific is it? Is that use case to your business or are you tackling something that will be easily bought off the shelf within a short space of time? So a few pointers for your listeners there. Absolutely love that. And there's so many big takeaways there and we covered a lot of ground in a

[00:27:15] short amount of time and so much of what you said will resonate with people listening. I suspect some will want to continue the conversation. So anyone wanting to find out more information about anything we covered, NT data or connect with you. Where should I point everyone listening? You can find me on LinkedIn. I'm sometimes on the conference circuit. I'm going to the LEAP conference in Saudi Arabia in a couple of weeks time, speaking there about the future of knowledge management. But you'll find me on

[00:27:41] LinkedIn or of course the NTData website has everything you need. Excellent. Well, I'll add links to everything that you mentioned there. And one of the big takeaways for me is what responsible AI actually looks like and how you ask the question, is the system competent and what is the worst thing that could actually go wrong? If anyone takes away just two things today, I would love them to start with that. But I will have links to everything. I'll encourage people listening to connect with you

[00:28:10] and find out more about NT data. But just thank you for sharing your time and your story with me today. Appreciate it. Neil, thank you for the opportunity. It's been a pleasure. I think Bill's point about the human in the loop becoming a rubber stamp is an idea that I keep thinking about. Because oversight only works when people retain enough experience to recognise when an

[00:28:32] AI system has gone off course and question its output. And this means testing people as well as models, recording the reasoning that never made it to the process manual and choosing projects whose value can be measured in production. And maybe asking Bill's two questions before an agent touches a workflow. And as a quick reminder, they are, is it competent? And what is the worst thing that could go wrong?

[00:28:59] And a big thank you to Bill today for a refreshingly practical conversation about responsible AI, public services, and most importantly of all, human judgment. So you can find Bill on LinkedIn and learn more about NTT data through its website. I'll include links to everything over at techtalksnetwork.com. Feel free to send me a message over there.

[00:29:22] But over to you, where might your human approval process already be turning into a polite click on approve? Love to hear from you. Lots to think about there, but time for me to go now. So thank you for listening. Speak to you soon. Bye for now.