How Testlio Balances Automation and AI With Human Insight

[00:00:00] So, a special thank you to Denodo for supporting the Tech Talks network and helping us keep these conversations going. Because moving beyond AI pilots all starts with connecting your models to trusted enterprise data. So, if you're ready to move beyond AI pilots, Denodo can help you connect your AI models to trusted enterprise data in real time. So, you can scale faster and reduce risk.

[00:00:26] So, if you're interested in turning AI into business value, simply visit Denodo.com. Welcome back to the Tech Talks Daily Podcast, where today I'm going to be talking about something every business relies on, but very few people think about until it goes wrong. Yeah, I'm talking about software quality.

[00:00:52] Because in a world where AI is accelerating software development and doing it at incredible speed, what happens when companies start releasing products faster than they can properly test them? Well, my guest today is Dean Hickman-Smith, Chief Revenue Officer at Testlio, a company that's helping some of the world's biggest brands avoid release day disasters. And they do that through crowdsourced testing and human insight.

[00:01:19] And today's conversation arrives at a fascinating moment for the industry. Because AI can now generate code, automate workflows, and speed up development cycles in ways that would have seemed impossible just a few years ago. But as my guest will explain today, speed alone does not create trust. Customers still judge brands based on the quality of the experience in front of them.

[00:01:46] And whether that's a checkout flow, a banking app, a streaming platform, or even an AI chatbot, it's all the same. So I want to talk today about why so many organisations are still over-relying on automation, and how cultural nuance and human behaviour, how these things are continuing to expose blind spots in AI systems, and why software quality has quietly become one of the biggest reputational risks that are facing modern businesses today.

[00:02:16] And my guest will also share insights from Testlio's State of Digital Risk report, where they've seen alarming levels of hallucinations, misinformation, and quality failures hidden inside modern software releases. And somewhere along the way, we'll even talk about aerobatic flying, the Lockheed skunkworks, and why innovation ultimately depends on small teams solving big problems.

[00:02:42] So buckle up for a conversation about AI, software quality, digital trust, and the growing role humans will continue to play in keeping technology reliable. And on that note, let me officially introduce you to my guest now. So thank you for joining me on the show today. Can you tell everyone listening a little about who you are and what you do? Yeah, thanks, Neil. Great to be here and happy to jump straight in.

[00:03:12] I'm Dean. I'm the Chief Revenue Officer here at Testlio, and I run our worldwide go-to-market, so sales, marketing, and the customer kind of outreach. Been here for a year. Very excited to be here. I've basically been in America for 20 years doing a series of scale-ups, mostly in the information security space. So my background is with companies like Netscreen, Juniper, Proofpoint, and then some moves into the biometric space.

[00:03:41] Did some biometric software companies and then, yeah, came across Testlio about a year ago and run the worldwide go-to-market operations here. Outside that, I am a very competitive aerobatic pilot. That's my fun fact for a conversation outside the industry. Well, you can't just drop that and expect me to move on to Testlio. Tell me more about this aerobatic pilot. How did that begin? What do you do?

[00:04:10] I've got to find out more about this. It began at a very early age, and I'm sure you're familiar with the Air Cadet Program in the UK. Yeah. So they gave us some flying experience when we were 14. Got through that, got some flying experience with the Air Force, with the Reserve Air Force in the UK, and I've been flying since I was basically – I've been flying for 40 years, so you can do the math, right? And now I run the Northern California Aerobatic Contest,

[00:04:39] which we just concluded last weekend in an oasis called Tracy, California, which is on I-5 just outside the Bay Area on the way down towards LA. It's a pretty rural desert place that's famous for being a logistics hub for Amazon. And we make – we have a very big contest, actually. We've got about 50 pilots from around America and a couple from abroad come over,

[00:05:05] and we have three days of very friendly but very intense aerobatic competition. And it's a blast. And, yeah, I love it. So it's my kind of – it's my golf, my yoga, my – and again, away from everything else. When you're in the zone for flying, it's a nice mental shift from the day-to-day kind of world of what we do here at Teslio. Awesome. And I could spend an entire episode just talking about that.

[00:05:34] But as I said before we started recording, I started this podcast 11 years ago, and I think it was around 10 years ago that I spoke with the founder of Testlio, Crystal, there. And fast forward to present day, I think software teams are shipping updates faster than ever. But the pressure to release quickly can also increase digital risk. So from your perspective at Teslio, how has that relationship between speed and quality,

[00:06:00] how has that changed over the last few years since AI arrived on the scene? AI is just an accelerator for everything, I think. I think in the old days, it was – QA was kind of the gate, right? You could – in the old days, you could ship fast or you could ship securely. So QA would kind of be that final check.

[00:06:25] Now, you can ship fast, you can localize fast, you can build vast amounts of content, you can modify apps and iterate them on the go. The issue is not now the shipping, it's can you trust what you ship in the real world? And that's the big difference, I think, between the old days where software worked or it crashed or it was easier to monitor.

[00:06:54] Now with AI, you don't get the crash, but you get falsehoods, you get hallucinations, you get inaccurate representation, you get business logic that doesn't align with the business. And that's where we fit in with Teslio and providing humans in the loop to validate that you can trust the software that you are releasing to the public.

[00:07:22] And Testlio's fully managed crowdsource testing platform integrates expert on-demand testers into that release process. And you've also worked with global brands like Paramount, BitPay, and so many other household names. And ultimately, you help them prevent those release day failures. And we've all been there. So what are some of the most common mistakes that you see organizations still making when preparing for those major launches or big update rollouts?

[00:07:51] I think that we work with a lot of innovative companies. So I would say let's talk very generically what mistakes companies make, not necessarily those that we work with. But I think a lot of people are still stuck in the old mindset when they launch software. They use old metrics. Did it deploy? Did automated tests pass? And did it crash?

[00:08:18] Can we learn from crash, from crash dumps, from anything like that? And there's a push to automate as much as possible. And I think that's a big mistake. I don't disagree that we should be bringing automation as much as possible to the table. But I think there's an over-reliance on automation in a lot of companies now. And I think there's an over-reliance on synthetic testing.

[00:08:45] Now, of course, I'm going to say that because I provide humans in the loop. But I think if you're only testing on a pristine or a synthetic data set and you're relying heavily on automation, you're not going to capture the real human experience of what you're shipping. So I would say that's the biggest thing that I see in the industry. And when we talk to companies, we are really talking about, OK, are you confident in what the customer experience is going to be?

[00:09:14] Because your brand is so now directly linked to your app, to your software, to whatever faces the customer. And that's where we come in to bring a kind of quality intelligence layer to things, which involves orchestration of automation. Of course, it involves humans in the loop. It involves regionalization. It involves all sorts of localization testing across a very broad gamut of different test requirements.

[00:09:41] So those would be the typical mistakes I'm seeing at the moment. It's that push to automation and it's a reliance on synthetic and pristine environments to be real test kind of environments. So that, again, brings it back to why Testlio? Why now? I think we bring a real world testing environment to companies that do want to understand what's actually the real world customer experience. And if I look back 10 years ago when I had that first conversation with your founder,

[00:10:09] crowdsource testing was back then seen as unconventional compared to traditional QA models. I think that is largely changing now, a decade later. But what advantage does a globally distributed testing community, what does this bring, especially when applications now need to work across countless devices, regions, networks, user behaviors, etc.? What do you think it brings extra to the table here? It brings nuance.

[00:10:39] It brings real world experience. It brings demographic representation. It brings everything that synthetic and automated testing doesn't bring to the table. So it really reflects on the real customer or consumer experience. It could be the checkout experience. It could be the cultural experience.

[00:11:06] It could be the guidance they're getting from an app, which might in one country be considered polite and in another country be considered very offensive. It gives companies the chance to check out various different age groups, various different degrees of ability, certainly various different demographics. So that's what the crowd brings to the table. It gives you a real world experience.

[00:11:34] And it's being used by lots of big brands that are very conscious of that experience. We do a lot of stuff in the premiership in the UK, for example. So we've got some of the bigger teams in the UK. They have massive audiences in Asia. So they want to know, OK, what does my app work like in Indonesia, one of the most populated countries on the planet? What's my fan base in Singapore?

[00:11:59] We work with the premiership actually itself right now to launch on-demand services in various different countries. So the projects we work on are those where the customer really wants to make sure their brand is represented the way they want it to be experienced by their consumers. Such a great example there with the premiership. And I think that you're so on the money. I think nuance is so important. And critically, it's very often underestimated.

[00:12:27] It's why we see so much confusion sometimes and wrong results. And AI is, yes, changing software development at almost every stage from code generation to automated testing. But how do you see AI reshaping quality assurance itself? And where do you see human testers still providing something that machines cannot? I mean, you mentioned nuance there. Anything else? When we talk about AI testing, as in AI testing itself,

[00:12:55] we work with some of the model companies to help them understand what the customer experience is of the output of their product. And clearly, AI is getting pretty good at validating a lot of its own thought processes. But it's not good. It's good at repetitive. It's good at very, very linear thinking. And it's good at analyzing vast amounts of data.

[00:13:25] The human in the loop, by comparison, is very good at the edge case. The nuance, the random thought process, or perhaps the experience that somebody that's older or younger or maybe has some sort of challenges, how they interoperate with a system is not something that is easily replicated by AI. So you've got this interesting world right now where you've got a great balance.

[00:13:54] AI and automation, let's put them in the same bucket, because automation is using AI to learn more and to get its own feedback. But that, alongside human in the loop, where you're getting culture, you're getting the human perceived context, you're getting business relevance, you're getting social context, you're getting all sorts of other context, which feeds back into the software. That's the kind of perfect scenario where you're balancing the two.

[00:14:24] You're leveraging as much as you can, automated testing and AI-driven testing, and also the fundamental models themselves as they improve, but you're reinforcing it with human input. And based on everything that you're seeing and hearing out there at the moment, what trends are emerging around the risks that companies face as digital experiences become more central to customer trust and business reputation? Any big risks you're seeing there?

[00:14:51] We test a lot of offerings, and we've seen test data that shows that 80-plus percent of releases have significant flaws in them. We've seen up to, again, 80-plus percent. I'd say 82% was the exact figure that we had as of a couple of weeks ago.

[00:15:14] So releases have AI hallucination or misinformation inherent in their offering. So we see a lot of accuracy issues, and it's not very evident accuracy. It's a kind of silent erosion of customer confidence in a solution when the advice they're getting from it is not trustworthy.

[00:15:39] So I think you've got a number of different things right now that if I'm the customer I'm worried about, over time, as people start to use my app, do they increase in confidence? Does it stay level, or do they decrease in confidence? Because people move quickly with their feet. When they feel like an app is giving them incorrect advice, so you've got an erosion of your customer base if you don't get it right.

[00:16:07] Now, on the other side, you've got a trend at the moment called personalization, where personalize your experience. And you can overdo that too. So culturally, am I hitting the nail on the head, or am I getting into the zone of creepiness? If I'm suggesting I know, you know, Neil, you're sitting in your beautiful dining room doing this, and you are actually sitting in your dining room, do you feel like you're being spied upon?

[00:16:36] Is the personalization too much? You know, that's something that AI cannot tell you. But a human who's experiencing a product can tell you whether it's hitting the nail on the head culturally, whether it's appropriate in its amount of personification, whether the experience they're getting is right. There's all sorts of demographic stuff, right, as well. We expect an app to work, and it probably will work in certain geographic locations on certain networks, on certain devices.

[00:17:05] But you take that device to a different location with a crappy network, and a lower performance phone, maybe a different lighting situation, background noise. You know, we talk to companies that are launching, now they're moving more and more into voice. So you've got to test for things like accents, and dialect, and idiom, and how people interact. So there's a lot more complexity now as you roll these apps out, and they shift from text input to voice input.

[00:17:35] All the different cultural nuances have to be validated. So there's a lot of risk now, I think, that's just whether customers like what they're getting, or whether they move away from it. And then, of course, there's the regulatory compliance side, and governments are coming down harder and harder and harder on, you know, things like privacy, identity, consumer protection, age.

[00:18:03] There's a whole bunch of new regulatory pressure that's either privacy or AI policy driven coming through that companies have to be in abeyance of. So the regulatory landscape is tougher. Delivering customer satisfaction is tougher. Customer expectations are higher because you see brands with immense investment in their apps that are delivering really, really top quality products. And you've got to be there, right? If you want to compete, you've really got to be there at that level.

[00:18:33] So I'm convinced it's an interesting time to be in this space. I think you and I probably have a similar vintage. We've been through, you know, mobility, then we went through cloud. Now we're in AI world, and everything is accelerating. It feels like we're two or three or many, many times faster than anything that we've seen over the last couple of decades. So exciting, but lots of opportunities.

[00:19:03] Yeah, I completely agree with you. And I do think many organizations focus heavily on cybersecurity, but digital risk can also come from outages, broken experiences, failed updates, or poor performance. And one of the reasons I wanted to bring this up, there was a moment, I think it was about 18 months ago to two years, and in one week, there was in the UK, McDonald's, Sainsbury's, Argos, and Greggs all had massive tech outages,

[00:19:30] which weren't related to cybersecurity, but simply software updates that went wrong. So do you think businesses still underestimate the reputational and financial impact of good old-fashioned software quality failures? Yes. I think the recognition of seeing those incidents with major, major brands is resonating, but I think at the moment it's still

[00:19:59] at the board level, companies are being forced to adopt AI. At the functional level, they're saying, well, I don't even trust this thing myself. How can I roll out software using a product that I honestly do not already trust myself? How do I put guardrails around it? How do I put data leakage prevention rules in place? the technology behind the ring fencing of what AI can and cannot do is still very nascent itself. But from the top down,

[00:20:30] there's this pressure to improve efficiency, to automate. And the reality is now the app is the company. What that app does is the public image of the company, how it, how the chatbot interoperates, that's the face of the company. That's now the new, like, the sales interface is the chatbot. If the chatbot's crappy, and there are some big, big brands with really bad chatbots, and I'm sure, you know, I think you worked in the telco industry

[00:20:59] right back in the day. Just check out some of the chatbots that are powering some of the biggest brands in the industry. They're horrible. The experience is terrible. It's worse than getting a call center employee five years ago, which took a long time to get to, but they would give you sensible advice. Now the chatbot will take you through this dance, and ultimately not give you what you need. So the chatbot

[00:21:29] is the new commercial interface. The recommendation is the merchandising team now, so the recommendation engine behind it has got to be relevant. It's got a good business logic where you can blow everything. And then the checkout experience, the payments, you know, can you accept my local currency? Can you accept crypto? Can you do my click to pay or my buy now, pay later? There's so much complexity in payments now. All these things are absolutely fundamental to the experience the customer is going to have, and all of them therefore bring

[00:21:59] their own risks to the table outside the security risks that we talked about earlier on. But again, it's all opportunity. There's a whole new raft of startups coming that are putting guard rails in place, business logic in place, data leakage prevention in place, and there's companies like us that are testing all of these operating infrastructures to make sure that we're delivering quality software experiences for companies. And with AI comes increasing pressure on engineering teams. They're challenged with moving faster with

[00:22:29] smaller teams and often tighter budget. And again, this question is a big one and maybe even an episode on its own, but how do you balance automation AI driven testing and human insight without creating blind spots that only appear once the customers are affected there? It's a massive challenge and somewhat of a balancing act, right? It is a balancing act. I think the issue is you don't replace humans with automation. You bring

[00:23:00] them together in an optimal way. I talked about my fascination with flying. I think this is a bit like the adoption of the autopilot back in the day. You don't replace the pilot with an autopilot. You have them working in harmony. So you use automation and I'm putting AI and automation into a very similar bucket here again, but you use automation for regression, for repetitive tasks, right? Regression, performance testing, synthetic workflows, test generation,

[00:23:32] continuous monitoring, and you in concert with, you use that with humans that give you the cultural, the personalization, the payment, the voice, verification, the user experience, accessibility, all those things and they work together. So you automate as much as you can safely

[00:24:01] and what you can't, you bring the humans in, the two come together and it's a feedback loop that gives you a constant feedback to improve your software development lifecycle and that's the future and the people that will succeed are the people that can bring these two things together in an intelligently orchestrated manner. I like that phrase, but that's kind of, that's the vision for Teslio is bringing together intelligent orchestration,

[00:24:31] humans in the loop plus automation where it's relevant. And then feeding it back to the customer in a way that they can make sensible business decisions and continually improve the quality of their software output. And finally, as we look to the future there, as AI generated code maybe inevitably becomes more common and release cycles, they will continue accelerating. Do you think software testing becomes even more important or could AI eventually automate most Q&A work end to end? Listening to you today, I suspect

[00:25:01] we're going to come back to human in the loop there and it's great to hear that, but where do you see this all heading? I think it's that balance. I think what we'll see is, I think we'll see more and more specialist areas where the human continues to be relevant and we will see a broadening of where we can safely automate. So I think the domain experience continues to become more and more relevant. We have a big hyperscaler

[00:25:30] customer that has a co-pilot type product that's bringing together a bunch of different apps. We tested the co-pilot, we tested the apps individually. Now they want domain specialists that are, for example, African accountants that have experience of spreadsheets and PowerPoints. So that through the co-pilot you bring together those two separate things and the human is doing the testing of that AI-driven interoperation and I think that's kind of where this

[00:26:00] world goes more and more and more as the space matures over the next couple of years. So it is exciting. I think if we play our cards right, you know, Tesla is ideally positioned to be an intelligence layer that glues it all together. I cannot thank you enough for sharing your insights with me today but before I let you go I'm going to ask you to leave one final gift for everybody listening and that is a book that you would recommend that we can add to our Amazon wishlist.

[00:26:30] What would you like to add and why? Ah, well I am going to bring together technology and aviation because those are my two kind of outside family. Those are the two things that kind of excite me and there's a book called Skunk Works. I don't know if you've read it. No. If I'm right, it's by a guy called Ben Rich and it's all about incredible scientific development in

[00:26:59] the post-World War, well during World War II, post-World War II, during the Cold War. It's all of the high level, high altitude and supersonic stuff that the Lockheed Skunk Works was developing and this is fascinating for me because of the speed of innovation. Over a 10-year period they went from propellers to hypersonic. They went from the P-38, which was designed by the Skunk Works as a medium

[00:27:29] strike fighter back in the World War II days through to the SR-71 Blackbirds cruising along. We don't actually know the actual speed. I think it was probably Mach 3 at 70,000 feet with a bunch of people, a very small team doing amazing things with slide rules in a shed in the desert in Palmdale, which is I've been there quite often actually. It's just windy desert conditions, not great for doing amazing

[00:27:59] scientific work. But yeah, between that and Burbank they were flying bits and pieces back and forth, but they were an amazing team, very innovative, great minds, small teams, no bureaucracy, got shit done. And I think that is analogous to the perfect kind of world we're in right now where you can do a lot. You can do a lot with AI, you can do so much more research with AI faster than you've ever been able to do. And I think, yeah, just reflecting back on what they did at the Skunk Works, they continue to do it today by the way, but

[00:28:29] it's a very different organization. It was just a great book. It's part sci-fi, reality, innovation, humor, just drama, the whole thing is in it. So it's a great book. Skunk Works, Ben Rich, highly recommended. Oh, it's got the YouTube as well. So he did the P38, the F104, which is that first Mach 1.2, 1.3 called the Starfighter all through the 60s and 70s.

[00:28:58] Then he did the U-2, the spy plane, the ultimate high-flying spy plane still in existence, and then the SR-71 Blackbird, which just blows your mind that those minds could come up with all those things in about a 20-year period. Wow, you've certainly got me intrigued. I will be adding that to the Amazon wishlist. I'm going to be checking that out myself too. You can have that on the beach or in the garden in the sun when you're barbecuing, and enjoy

[00:29:28] the British heatwave that you've got at the moment. 100% with you, and I would agree with you about Testlio being in the right place, right time as well. But for everyone listening that want to find out a little bit more information on Teslio, explore anything that we talked about today, and keep up to speed with reports and news coming out, etc. Commit with you or your team. Where would you like me to point everyone listening? For sure, the Teslio website. We've got an interactive website, teslio.com.

[00:29:58] Hit us up there. From that, we've got a very active blogging world. We all put out a lot of content on the normal platforms, but I'm very happy to connect people up if they hit me up on LinkedIn. Dean Hickman Smith. I'm the only one on LinkedIn. We can make the connections work. What we try and put out through socials and everything else is just it is a complicated time, so we're trying to put out content that is going to

[00:30:28] show people what we think the future looks like. We think the future looks rosy. We do a lot of keynotes and a lot of talking, so if you hit us up on the website, we'll make sure that we keep you up to date on all the latest stuff. Awesome. I will add links to everything you mentioned there, the website, the socials, your LinkedIn, and I will encourage anyone listening that's impacted by anything we've talked about today or we've set off a few light bulb moments to reach out there,

[00:30:58] not only learn more information from the website, but maybe connect with you, ask the questions, connect with me over at Tech Talks Network. It'd be great to continue this conversation we started today, but more than to start it today. It's such an important time to share conversations like this, so thanks again for joining me. Neil, it's a pleasure, and I just want to reiterate what you said. I'm happy to engage in conversation with anyone about this new wave

[00:31:27] of what's happening in the world, how AI's can have some positive outcome, how we can make it have a positive outcome. I think it has tremendous opportunity for positive growth in the world. It can be used negatively or positively, and I like to think that if we come together we can take things in a very positive direction. What I loved about chatting with Dean today was that balance between optimism and realism. And there's no doubt AI is transforming software development and accelerating innovation at a

[00:31:57] remarkable pace. But as Dean pointed out today, faster development cycles also increase the chances of releasing flawed experiences into the real world. And when that app effectively becomes the public face of your business, even small failures can quickly damage trust. But one of the biggest takeaways for me was this reminder that automation alone cannot fully understand human behavior. We need to double down on

[00:32:27] understanding cultural nuance, accessibility, regional differences, voice interaction, personalization boundaries, and those simple human expectations that all require people in the loop. And it was fascinating to hear how companies are trying to balance AI-driven efficiency with some of the regulatory pressure around privacy, compliance, and consumer protection. All of which I think highlights that the tech is moving incredibly fast.

[00:32:56] But as it always has done, governance and trust still have a little catching up today. But I'd encourage everyone listening to check out Testlio and keep an eye out for that state of digital risk report. I'll add a link to that. And of course, if you enjoyed today's episode, subscribe to Tech Talks Daily, leave a review, share an episode with someone that is navigating the challenges of AI, software quality, and digital transformation. And remember, if you go to techtalksnetwork.com, you'll find all

[00:33:25] eight podcasts there on the network and how you can contact me or work with me. And if you do find yourself upside down in an aerobatic plane over California with Dean, let me know how that one goes. But that's it for today. Thanks for listening as always. Speak to you tomorrow. Bye for now.