Testing AI That Never Stops Changing With UL Solutions

[00:00:00] Your agents aren't producing accurate answers because they don't have a complete semantic understanding of your data. And Denodo is solving this and solving it through semantic consistency. Through semantic consistency, your agents can start making accurate predictions in real time. So see what else Denodo can do by visiting denodo.com to learn more. But now let me introduce you to today's guest.

[00:00:32] How do you certify the safety of an AI product? A product that could behave differently after its next update. Quite excited to get today's guest on. His name's Dr. Robert Sloan. He's the Senior Vice President, Chief Scientist and Innovation Officer at UL Solutions. And today he's going to join me in explaining credible AI product testing.

[00:00:57] And we're going to move from talking teddy bears to healthcare systems, examining why cybersecurity alone cannot tell us whether AI is reliable, fair, private, understandable and properly governed. And Robert will introduce everyone listening today to three areas buyers should examine. And they are technical performance, ethical safeguards and governance across the product lifecycle.

[00:01:24] And they'll also explain why annual reviews might eventually give way to continuous telemetry capable of detecting meaningful changes after deployment.

[00:01:36] So if you wanted to take a peek behind the curtain of that UL label that you've probably seen, or if your company is in the process of buying a new AI tool, this episode will help you ask vendors better questions, recognize weak evidence and understand what independent testing can prove before trust is ever placed in a product. But enough from me.

[00:02:28] I'm going to take a look at the same way.

[00:02:58] Well, it's a pleasure to have you join me today. And before we go all techie and dive straight into all things AI, et cetera, I'd love for you to tell everyone listening a little bit more about the role that UL Solution plays. And the reason I ask that is I think many people listening will recognize the UL mark, but might not realize the science, the testing, the certification, everything that goes on behind the scenes there. So tell them a little bit more about that.

[00:03:23] Yeah, I agree. Happy to do that, Neil. So UL Solutions is a global safety science leader. Our mission, which all of our employees can repeat by heart, is working for a safer world. It's literally over the doorways when we walk into our laboratories and into our workplaces. So we take it quite seriously. We're a very mission-driven organization. We were founded over 130 years ago to do something for humanity. And it was around the time of the World's Fair in Chicago

[00:03:52] when electricity was transforming everyday life. But there were a lot of unfamiliar risks. And actually, our founder's role, believe it or not, was to be invited to Chicago to help not burn the city down for the second time as electricity was featured in the Columbian Exposition. So this challenge of bringing new technologies forward and safeguarding local areas and the world is

[00:04:17] not new. That's literally what we were founded for. We are known for and we do own the UL Mark, which is on billions of products around the world at this point. It is a recognized symbol of trust, if you will. And we're a player and a leader in what's known as the TIC industry. So TIC stands for testing, inspection, and certification. We also offer software and advisory services. But bottom line for

[00:04:47] the TIC area, testing, inspection, certification is we blow things up. We light products on fire within the safe areas of our laboratories, you know, properly set up. But that's part of our role is to find the breaking point for products and to test them to their limits. And then if they pass, to certify them and put the UL Mark on them. So over 130 years, you know, that's put us close to

[00:05:14] technologies that are changing the way people live and work. We're always being asked to take a look at this new technology that's coming to market soon. It's always better if we get called in early rather than later. We'd rather be doing proactive work ahead of any issues that consumers may face rather than batting cleanup, if you will, later or trying to help clean up later on. The types of technologies

[00:05:40] we have worked with and do work with include EVs, electric vehicles, batteries, including battery safety, lithium-ion batteries being one current focus for us and for a lot of folks, 5G technology, communications technology, autonomous systems, and of course, today AI. We also get pulled in before technologies have hit the market. We're being brought into conversations, for example, on

[00:06:08] quantum computing and cryptography because we know and our customers know at some point, those will become meaningful for them and for the consumers as well. So the technology that's around the corner keeps changing, but that role of working for a safer world for us, that's what stays the same. That's kind of our north star, if you will, that guides us and guides our work to help society benefit from the innovation and from safe products making it into the market while managing the risks as best we can

[00:06:38] along the way. Our customers are in industrial markets, so energy transition, energy infrastructure, data centers, those areas, and also consumer markets, so retail, toys, all sorts of manner of devices, if you will. So we have a very, very broad range of products that we work with. If a product does earn certification, it does get the mark, the UL mark, so the U and the L within a circle. It's one of the

[00:07:07] most sort of recognized symbols of trust out in the marketplace. We take it very, very seriously. We also, from time to time, need to police that mark so that there, if there are any counterfeit examples out there, we follow up on those and make sure those get removed from the marketplace. And so the mark, when you see it or when a consumer sees it, it indicates to that consumer what the product has

[00:07:32] been through in terms of its evaluation, what it's been certified against in order to earn that mark. And that's really sort of the gist of what we do, Neil, in our day to day. And although it is an incredibly serious topic, you really do have the coolest job in the world there listening to you. I would absolutely love doing that. And of course, fast forward to present day here and now, AI is moving rapidly from just chatbots and going directly into products and systems

[00:07:59] that can influence everything from healthcare, manufacturing, finance, transportation, critical infrastructure and toys, even toys now. So how should organizations and individuals think differently around AI safety compared with maybe traditional software quality or cyber security? Because it feels like a whole new level of risk here.

[00:08:22] It is different in certain ways, Neil, that's true. So traditional software is something that you can count on being the same for a period of time. AI enabled products, including software can evolve very quickly. They can change over time. They also are a product of how they are

[00:08:46] trained. So that brings in differences in terms of how we need to ask questions of manufacturers and others who are bringing AI enabled products to us. AI does bring different types of risks that we can go into more detail on what those are and how we are accounting for those. But what doesn't change is kind of the rigor and what goes behind the mark. And also our need to work

[00:09:14] very transparently with our customers and be very dispassionate, frankly, about this testing. I mean, the founder's quote around this is, know by test and just state the facts. So we will evaluate every product through our doors in the same way each time. We're duty bound and accreditation bound to provide the same consistency to each customer, each manufacturer. So no one gets a hard time and no one gets easy

[00:09:44] to do. So we're going to be able to do that. And that's the thing that we're going to do. And that's that bit of how does not and will not change whether it's AI or a physical product or anything else out there. AI is a system type of challenge. So we don't just look at one individual part of how it's constructed. So in other words, if we only look at cybersecurity, you know, great, the product can be

[00:10:12] safe from an attack, but it could still be very unfair, opaque in its reasoning, poorly governed, unreliable, all the rest of it. So AI is a bit different in a sense that we need to look broadly across how was it put together? How is it operating? And then going forward, we do need to account for those over the air updates and things. We've had that in our work before. It's just growing more important

[00:10:41] over time. And it's so important because I think if we go back, what, three to four years ago, there was almost an AI gold rush. A lot of people jumping on the bandwagon, not wanting to get left behind. But thankfully here in 2026, we're seeing a lot more thoughtful approaches to the technology and appreciating its power. And one of the reasons I was excited to get you on here today is you've developed this framework for AI product safety that includes robustness, privacy, fairness,

[00:11:08] transparency, human oversight, change management, and life cycle accountability. As an ex-IT guy, this ticked so many boxes for me, but can you walk me through these building blocks, these foundations, and explain why organizations need all of them rather than just focusing on the ones that they would like? Yeah, absolutely. And if you don't mind, I'd like to actually take a little bit of a step

[00:11:31] doc and explain why these. One of our principal engineers, so we have a number of these folks around who are the experts for their areas within your solutions, took a look at what do we have in common around the world in terms of expectations of AI systems? So whether it's AI regs in Asia or GDPR

[00:11:57] in the AI act in Europe or the various executive orders in the US that we've had, they, many of them have common themes or pillars among them. There are differences, but we looked for what's in common across them. And as you rightly point out, we have a lot in common across broad ranges of performance. So I would group these really in three pillars. The first is sort of the technical

[00:12:24] pillar, and that does speak to robustness. So that's asking whether the system performs reliably under different conditions. And then also risk management. How has it been assembled to take care of those aspects of performance? Functional safety. Does the AI-enabled product or the AI software perform for the intended purpose? How do we know that? How does the manufacturer know that? So that's the technical pillar.

[00:12:54] First, one of the three. The second, the ethical pillar, just as important. All three are, and for obvious reasons that you'll hear, equally important. Ethics involves fairness and bias. On what data was this AI system or product trained? How do we ensure, how does the manufacturer assert or attest that they used unbiased data to train this? How have they accounted for data privacy

[00:13:22] and protection? As you mentioned earlier, medical's an area that's, you really want AI assistance and doctors want AI assistance when looking through imagery. It helps for earlier detection and better outcomes. But of course, the patient wants to know that their outcome, I mean, think of an oncology test that's a cancer patient. They want their outcome to be kept private. And that's their life, that's their

[00:13:49] information. So that's part of the ethical pillar. Transparency and explainability. We, at UL Solutions, we don't really love black boxes, ever. You know, unexplainable systems, that's not something that's going to score very well with us in terms of certification. So that's true for AI as well as physical products in the past. So that's this middle, if you will, ethical pillar.

[00:14:14] And then the last one's really important also, I mean, equally important, and that's governance. How is the data collected and analyzed? How is the manufacturer managing the life cycle of this product and of the AI system? If the data goes out of date, then guess what? The performance is probably not going to be better, right? At some point, just like any other product,

[00:14:39] there's like an expiration date that we need to understand together with manufacturers. So what can they tell us about that life cycle management? And the last one which we've added or shrink, it's always been there, but we've strengthened in the most recent version of this UL 3.1.1.5 is accountability. Who's going to stand behind this AI system or this product? Because oftentimes,

[00:15:05] it's not the product manufacturer who's developed the AI system. It's another specific AI company or developer who's pulled that together. But who is it? Can they tell us who can be reached if there's an issue that's specific to AI performance related to this product? And I would draw an analogy, by the way, Neil, to another area which has not had an easy path and

[00:15:32] still is on a difficult path, and that's social media. So if we had to do it all over again, I think even the social media companies would want to have had the opportunity to tell us more about how's it work? How are these algorithms going to work? How have they thought about the public's safety? So with 3.1.1.5 and with this framework, with this outline of investigation, we have a way for manufacturers across all of these, it's actually 200 criteria,

[00:16:02] pretty comprehensive we think, but we have a way for them to explain how they built the AI system, put it into the product, how they've accounted for all of these different key aspects that are common needs, again, across the world. And then sign on the line, really, and put their name to it and say, here's who's accountable for it and here's what we will assert on how this has been built.

[00:16:27] And if it meets the criteria, then we can apply the UL mark and provide certification. And if it doesn't, then we tell them that, I'm sorry, here's where you were deficient. We do not help them. This is important. We do not help them fix those problems because we can't become a co-designer with our customers and then turn around and, I'm sure we think our own work would be wonderful, but that actually

[00:16:54] would undercut, badly undercut impartiality. And we would, we always safeguard impartiality along the way. So we provide a detailed scorecard. If it scores well, it gets certified. If it doesn't, then they know what the gaps are and what they need to do. And I would imagine another big challenge with AI is that the models don't stand still. There is no once and done, give it the label, then move on to the next thing because they are updated,

[00:17:22] retrained and will continuously evolve over time. So when a product keeps changing like this, how can safety evaluations remain meaningful over a period of time? Yeah. So great, great question. And that's true. I agree with that. I'll answer that directly. But first I want to highlight that that's not new. We have products, of course, and many different

[00:17:46] products that get updated on a regular basis. And so at UL Solutions, we've had to account, we have accounted for this and we've had to account for this for quite some time. As an example, for physical products, we go into manufacturing facilities on an annual basis, unannounced, to check that what's being produced and shipped matches what we certified and what earned the UL mark.

[00:18:12] And if it does not match, then we hold the right and we do delist products and remove those certifications. So to your question on AI, it's a similar approach for 3-1-1-5. Once per year, at a minimum, we want to see and understand what's changed, if anything, and how that product matches up against

[00:18:36] the criteria I described earlier. In addition to that, so that's a minimum of once a year. In addition to that, when a customer signs up to be certified, they are committing to us that they will tell us when they push significant updates through so that we then together take a look. It's time to bring this into the pits and take a look at the product and see how it may have changed over time.

[00:19:02] I think that that's a step in the direction of goodness. Is it all the way where we want to be? No. Where we need to be eventually is to have almost like real-time telemetry, really, on these products and understanding today what is the product versus what we certified yesterday. And have that data feed

[00:19:26] coming in so that we and our customers who want to do this can pick up on the signals quickly and and call out anything that needs further attention live time. That's where we eventually need to be with this. And we're not there yet, but we're making significant progress toward that. And outside of that, for business leaders that could be listening to our conversation today who might be buying AI rather than building it themselves, are there any questions that they

[00:19:54] should be asking their vendors before deploying an AI product? And what evidence should they be expecting to say and how can they distinguish a credible independent evaluation from just more marketing claims and and big promises? Great question. I actually think that the same questions that we are asking the manufacturers and others is what I would ask as a buyer of AI. How

[00:20:23] has this been assembled? They can even use the the structure of 3015 to help frame those questions. The three pillars, technical, ethical and governance. All of those, I believe, are common sense questions that regardless of what region you want to then take your product and commercialize and introduce it into the market. Those will all be in the direction of progress and goodness, if you will, safety

[00:20:51] for buyers rather than producers of AI systems. Documenting data sources, model performance, knowing the limitations, privacy and fairness aspects, bias, all of those are going to be important to purchasers of AI, AI as well as producers of AI software products. And you're someone that spent decades helping

[00:21:17] organizations validate the safety of products across so many different industries and the tech may change but the problems are very often the same. And if you look at AI today though, are there any mistakes that you see organizations repeatedly making as they maybe rush a little bit too headstrong in deploying these technologies and how can they possibly avoid creating unnecessary risks but without slowing innovation? And again, that has always been the big challenge. But any advice there?

[00:21:47] I think we can accelerate innovation with a bit of time spent up front in considering what's the problem that this product is going to solve or why do you need AI within this product itself? I think that one of the common mistakes out there, Neil, is treating AI like a

[00:22:08] catch-all or something that can fit every single problem. It's not in our experience and in our customers' experience, it works better if you really clearly frame how you want AI to interact with your physical product and also with the consumer at the end of the day. Because different vendors, even different models of AI within those vendor sets are going to hit that problem better than others.

[00:22:38] This gets at the functional safety testing that we do within that technical pillar. Is this fit for purpose? Does the AI enablement actually produce the result that is both needed and expected? So, you know, demonstrations help here, pilots help here, pushing these products yourselves, you know, in the early stages of design, all of those pay dividends down the road. So,

[00:23:03] more time up front means better spent time down the road and also a faster path to market, frankly. Because in the earlier stages, that's when it's easier to course correct rather than if you spec something in, much less shared it with your customers, and then you have to redesign, that is a common mistake that can cost a lot of time and money at the end of the day.

[00:23:28] And finally, as we look ahead, and AI inevitably becomes embedded in more and more business-critical and safety-critical systems, and we've got hundreds if not thousands of agents inside organizations and teams. Do you think independent AI testing and certification will become as routine and expected as safety testing for physical products today? And what does that future look like,

[00:23:53] do you think, for organizations that want to innovate while also ensuring that they maintain public trust? Yeah, I think it will. I mean, we certainly believe it will become more routine. I think you have to look no further than the direction of travel, even in the political area. Even the governments at this point are starting to build net forward progress toward routine testing and what's expected for products

[00:24:21] to enter their marketplaces. It differs in different regions, whether it's the EU, the UK, the US, or Asia. But what's in common is this agreement that having no safeguards is a really terrible idea. And so it comes back to this question of what are the basic elements that we can all agree on, that we can all drive toward. And I think, do we have everything perfect in UL 3.015? Almost certainly not.

[00:24:49] Right? We're revising along the way as we learn more. But those basics on bias and trust and everyone wanting their AI-enabled products to protect their privacy, to give them a result that's helpful and useful. Validating that with a third party is helpful. It does drive innovation into the

[00:25:12] marketplace in a way that consumers and all of us can help grow trust over time. That's incredibly important because if we don't have these safeguards and we don't pay attention to at least those basics to start, we could really hold back innovation because of some really bad failures along the way. We don't

[00:25:36] want to repeat some of these examples of new technologies or, you know, I mentioned social media earlier where we don't want to repeat that path. That's a painful path and it's not necessary. If we know what the basics are, we know how to evaluate these systems, then what we're finding is our customers trusted brands already in many, many cases, they want to protect their brand and innovate.

[00:26:02] You know, those are one in the same, you know, that's how they want to serve their customers better. So it all sort of leads to working together to build that trust with third party testing and certification as well. So consumers can look for that, that mark. And in our case, the UL mark, and they know what went behind it. Having AI convey that, that will be very, very helpful at the end of the day.

[00:26:30] Well, thank you so much for speaking with me today. And a big thank you for delivering so many actionable takeaways for people listening and what they should be asking their product vendors, what proof they need to demand and what independent evaluations can realistically demonstrate today. It's a big conversation starter. So for anybody listening wanting to find out more about UL solutions and some of the work that you're doing there, where would you like me to point everyone listening?

[00:26:57] David The official channel, if you will, is UL.com. David The official channel, that is a good starting point for folks to go to. My background and my profile are also there at UL.com and I'm not hard to find on LinkedIn as well. So we are here to partner with folks and to help them. We want to make sure we get safe products to market quickly. And we also ask questions along the way for those that have issues and challenges and may not be as safe. So

[00:27:25] those are the best ways to reach out to our company and to me as well. And thank you so much for the opportunity to join you today on the podcast. And we look forward to continuing this journey together with our customers and kind of we're all on this journey together and learning what is safe and how can we do a better and better job over time of driving innovation, but also protecting public safety along the way. David Well, for anyone listening wanting to learn more about that framework of the core building

[00:27:54] blocks of AI product safety, I would urge you to check out the links in the show notes. I'll also include links to the social channels for UL as well, in particular LinkedIn, so listeners can stay up to speed there. But more than anything, just a big thank you today for coming on here, Rob. Really appreciate your time. Rob Thanks very much, Neil. David I think my guest's story is a memorable reminder that AI safety is already a product question,

[00:28:20] not a distant policy debate. The model may sit behind the interface, but its behaviour can affect a child, a patient, an employee, driver or customer. So buyers need evidence about training data, performance limits, privacy, bias, accountability and what happens after the next update. And certification can provide useful independent evidence, although it cannot be treated as a permanent promise when

[00:28:48] the product keeps changing. And I think Robert's direction of travel is towards regular re-evaluation and eventually near real-time monitoring. So a big thank you to him for joining me today and explaining the science without setting anything on fire during our conversation. But remember, you can learn more at UL.com or connect with Robert on LinkedIn. But over to you, what evidence would

[00:29:16] make you trust an AI-enabled product? TechTalksNetwork.com, love to hear from you. And remember, connect with me on LinkedIn, but don't just send a connection request. Send me a message, tell me you listen or what you like, what you don't like, what you'd like to change. I'm here to serve you all. But that's it for today. So thank you for listening. Bye for now.