Moving Enterprise AI From Pilot to Production With Stelia

[00:00:00] Your agentic AI might not be secure even with real-time data and proper guardrails. But Denodo makes sure your business has every avenue covered. By placing all your data platforms under one AI data layer, your business can reach semantic consistency safely and securely. So get your agents on the same page by visiting denodo.com.

[00:00:26] And you can learn more about how to start trusting your agents to make business decisions. But now, back to today's guest. Why can an AI prototype impress everyone on Tuesday and collapse under real customer demand by Friday? Well, today I'm going to be joined by Dave Hughes, Chief Technology Officer.

[00:00:51] And together we will examine why enterprise AI systems struggle when they leave pilot stage. And Dave will argue today that the failure could be hiding below the model, inside the storage throughput. The network capacity, the power availability, data movement and architectural choices made long before a first prompt was ever made.

[00:01:16] And they've also recently launched as a full-stack operating system for enterprise AI. But the crux of our conversation today will be about why modern engineering teams need to recover some high-performance computing discipline. And why expensive GPUs can spend alarming amounts of time waiting for data. And why utilisation and energy efficiency, why all these things belong in conversations with the board.

[00:01:45] So we've got a lot to get through today. So enough from me. Let me introduce you to my guest right now. So thank you for joining me on the show today. Can you tell everyone listening a little about who you are and what you do? My name is David Hughes. I go by Dave. I am currently the Chief Technology Officer at Stellia.

[00:02:07] And at Stellia, I am responsible for the full technology stack within the company, whether that is providing network connectivity to data centres in Europe and the US. Cloud infrastructure that we have internally for self-hosting sort of user-facing products. And then the user-facing products piece is also what I'm responsible for. That would be in the current AI world.

[00:02:34] That would be offering sort of managed inference services, taking an off-the-shelf open source model and applying the company's proprietary data sets to it. That's what we sort of prototype to date. So, yeah, that's what I'm doing day to day at the minute. And then got the fun pieces of corporate IP and everything else all factoring into the equation as well.

[00:03:01] Basically, anything with blinking lights that may or may not send and receive data falls into my remit at some point. You can't beat some PBLs or PGLs, pretty blue lights or pretty green lights. Love it. Oh, indeed.

[00:03:15] There's so much I want to talk with you about today because if we look down our news feeds, we keep seeing the same pattern across enterprise AI where something performs brilliantly in pilot and then struggles when it reaches production, assuming it does reach production as well. So, I was reading how you argue that we're often diagnosing the wrong problem. So, when an AI system fails at scale, I've got to ask you here, where should a CTO look first before blaming the model or buying more GPUs?

[00:03:45] I mean, like anything in life, I like to start from what problem are we trying to solve first and then sort of work towards that. I mean, the engineers that I work with will say that I'm famous for saying, are we trying to engineer a bridge that's rated to hold this tonnage or are we building a bridge and then asking if it holds that tonnage? Like I said, I'm an engineer at heart. I like to make sure we actually engineer things from the get-go.

[00:04:13] I understand occasionally we prototype stuff and say, actually, that's quite good. Can we put it into production because there's people willing to pay for it? And then you end up with the battle scars afterwards. But, yeah, that's generally how I approach it. I think some of the most common sort of examples I've seen all fall into that bucket. CTOs and companies will sort of task some of their Tiger Team sort of members of the engineering staff that they trust to deliver something.

[00:04:41] And they'll give them a very sort of wide scope and say, here's one or two problems we've got within the business. Can you guys come up with something that might solve those problems? And while it might solve the first problem, you'll inevitably get asked, like, oh, can we start applying it to all these other problems within the business? Or can we offer it to 500 customers instead of one customer?

[00:05:07] And before you know it, it just falls over because it doesn't work like that. I mean, a famous example I've seen recently was someone prototyping, or not even prototyping, I'd say proof of concepting, a product that was starting to incorporate AI into the workflow. And they deemed it a success because I think one customer externally had tested it.

[00:05:32] And when I looked at the architecture of it, I picked out one component immediately. It's like you've got a single RabbitMQ instance here as a data bus, effectively. And it's going to fall over by the time it gets exhausted. It's going to get exhausted at, like, 50,000 messages. That's not a lot. So, like, by all means, call it a success if it's a managed service for one customer.

[00:05:58] But offering this publicly and expecting 5,000 customers to sign up to it is going to be a very different experience. So, yeah, just understanding what you're trying to solve before you do anything. I think something that got taught to me very early in my career as a young software engineer was, before you write a single line of code, sit down and actually just sketch out on a whiteboard or a piece of paper what it is you're trying to solve,

[00:06:25] what you're proposing will solve that problem, and try and visualize it. And I've always kept that in mind. Don't try and code your way out of a problem. Understand the problem first. Love it. 100% with you. It's so refreshing to hear you talk this way. And as you said, you are an engineer at heart. You've got the battle scars. And I think that stuff never leaves you. When I was doing a little research on you, I was also reading that you made quite a provocative argument

[00:06:54] that a decade of cloud abstraction has actually weakened systems thinking amongst some engineering teams. And cloud was supposed to free developers from worrying about infrastructure. That was the big promise. So at what point does abstraction become almost a liability? And what knowledge have we inadvertently engineered out of modern teams without thinking about that back in the day? There's pros and cons here.

[00:07:20] I'll frequently say in the sort of AI space as we see it today, we've brought in a lot of bias from both cloud computing and high-performance computing. And again, good and bad bias. Like we say, cloud computing, the abstraction of resource, elastic computing, having programmatic control over everything. It's fantastic. You would never go back to doing it the old way.

[00:07:45] And where this worked was when you had people just developing applications that maybe had to read and write to some storage or read or write to a database. And let's be honest, they didn't have to have mission-critical levels of performance on any of that. There was such a thing as good enough. When we start getting into high-performance computing, machine learning, and AI, you basically, again, understand the problem, what's happening here.

[00:08:15] So, you're loading data from disk or a bunch of disks in some kind of RAID array or cluster, which would mean erasier coding over multiple servers, whatever. Many variations of this from different storage vendors. But effectively, you're reading data from disk over a network and onto GPUs.

[00:08:37] So, that's a bit different from going from building applications on, say, AWS EC2, reading and writing some data to an object storage bucket, an S3, and maybe a Postgres database or something. Because now, all of a sudden, you've got very, very, well, you've got the abstraction of resource there.

[00:08:58] You now have the tightly coupled nature of the actual workload in mind and sense that you now have to be able to copy data at the speed and capacity of the network fabric. If we look at sort of current generation, that's 800 gigabits per second. So, it will be very easy to see if your workload is not actually factoring that in or you've provisioned, say, let's say, mechanical hard disk storage.

[00:09:27] And while you've maybe got the full capacity of the network fabric available, the storage just isn't able to output the data at the throughput required to saturate the network fabric. Like, it's high performance computing, AI, machine learning, whatever we want to call it. It's all quite similar in nature, but it's based upon a sort of triangle of storage, network, and compute.

[00:09:53] And being able, I always like to say to people, you think of old steamships. You had to be able to continuously shovel coal into the boilers at such a rate to sustain high speeds. It's no different than a high performance computing or AI environment where you need to be able to make sure that data mobility or data has mobility around that fabric at the full capacity of the fabric.

[00:10:20] And looking at your background, it also includes helping pioneer what we would now call the NeoCloud model. And you've seen high performance computing disciplines applied at scale. And if I was to ask you to look back, I mean, what lessons from that HPC world should enterprise engineering teams be maybe rediscovering as they build that AI infrastructure today? So what I like about it is just simple engineering rigor and discipline.

[00:10:48] I understand, especially if you want to be at the frontier, there's a level of sort of cowboy cavalier attitudes and mentality required. Whereas that's typically been sort of lost from high performance computing because it was, again, sort of deeply rooted in academia. Or if you look over the US national laboratories, so on and so forth. So you got that sort of, here's your funding.

[00:11:16] Here's your, this funding will purchase this amount of assets and fund the X amount of researchers over a period of time in order to produce research. So that, by nature, sort of produces people working in that space who have, again, very, very strict vigor and discipline. Maybe not as open to being a bit of a cowboy and how they want to approach different things.

[00:11:42] I mean, if I go back to when I was building the first, what was at the time, the first NeoCloud. Some people might dispute that, but whatever. Nah, it was a different space entirely. It was like, at that point in time, the company was repurposing GPUs that was left over from the former mining business that had been put into liquidation. At that point, or hadn't been put into liquidation yet, but they were repurposing a number of assets.

[00:12:09] And back then, AI wasn't really a consideration. So the actual forming of this cloud company happened before I joined. I joined later in the journey. But it had basically been started with an idea of bringing accelerated compute to public consumers at as low as possible entry point in terms of consumption price.

[00:12:34] If you were to look at GCP or AWS back in 2018, 2019, they would offer a couple random workstation-grade GPUs in a server chassis. And you'd be looking at double-digit dollar per hour in terms of cost.

[00:12:50] The initial proof of concept at my previous job, Genesis Cloud, ended up getting consumer-grade NVIDIA GPUs to the market for as low as $0.40 an hour and $0.50 an hour. So if you compare that by a factor of 20, 25, even 30 times compared to what the hyperscalers were doing and offering to the public at that point in time,

[00:13:16] it was a successful POC showing that you can bring accelerated compute to the mass market. Doing that obviously required a bit of a cavalier cowboy attitudes operating out of former sort of crypto mining data centers, which was burned upon at the time. Albeit you see with the likes of CoreWeave getting into bed with Core Scientific today or Cypher mining.

[00:13:39] Google took a 6% position in Cypher mining in order for them to unlock debt financing to get the co-location built for FluidStack. So it's becoming more of an accepted practice today. But go back eight years ago and you were thought of as an utter cowboy of wanting to host this in a former crypto mining site. I even look back at how we'd done things back then.

[00:14:04] It was not necessarily even buying off-the-shelf stuff from OEMs like Dell. We worked with ODMs and done some sort of custom design chassis to get 10 GPUs per chassis instead of eight. We'd done our own networking fabric rather than buying off-the-shelf stuff. We took basically like sort of white box Broadcom ASICs and ran Sonic Open networking on those.

[00:14:30] And it was quite, when I compare it to what I see today, to have been having that introduction like six and seven years ago, it was at the frontier of the market absolutely ahead of its time. But the problem then is when you're ahead of the market, who's dying it? I mean, it was an entirely enthusiast B2C-driven market, very little B2B interest in it. There was some B2B interest, but it just wasn't as high.

[00:15:01] And then all of a sudden you get the sort of GPT 3.5 and four releases going through to 2022 and into Q1 2023. And there's just a sort of explosion in demand for compute. So, yeah, it's been absolutely fascinating to see that sort of track records or just how it's evolved over time. Yeah, it's incredibly cool.

[00:15:29] And the journey that you've been on here and watching it all evolve the way it has. And fast forward to present day, we now find ourselves with this extraordinary amount of attention being placed on GPU availability and this assumption that more compute equals better AI performance. Of course, it is much more complicated than that. And you argue that scaling AI is almost never simply just a GPU problem.

[00:15:54] So on that side of things, what are the networking, storage, memory, data movement, and architectural bottlenecks that CTOs frequently discover only after spending heavily on those accelerators? I'd even take a step back and say the biggest constraint I see is access to land and power at the moment. If you don't have a data center to host this in, everything else is futile.

[00:16:20] You can have people who have got money ready to buy GPUs but they've got nowhere to host them, which is a challenge in and of itself. That then sort of brings about the market hysteria of you have to take data center capacity anywhere you can find it, which again brings its own problems. Now, if I was to say we're designing a cloud from scratch,

[00:16:45] you would start off by designing a sort of service footprint. If you're wanting to build a cloud in Europe, you'd say, okay, I want to be able to have these edge locations in London, Frankfurt, Amsterdam, Dublin, whatever, and I might host the GPUs up in Scotland but be able to build a good service footprint. But you're starting with a design in mind and then fitting the actual product into that design

[00:17:15] rather than just saying, oh, I can get capacity to host this in the Arctic Circle and I have to make it work from there. Or I can host it in Texas in the US rather than the Bay Area and I have to make it work from there. So there's an awful lot of just sheer market realities that people are having to sort of make peace with. And if you want to deploy GPUs, you'll take whatever capacity you can get because you still have to move forward at the end of the day.

[00:17:44] Beyond that, I mean, in terms of architecture, we've not seen a lot change in terms of the actual architecture of how we cluster these things together over the past six years. I think in terms of where we've seen the real change is going from sort of off-the-shelf so-called pizza box servers from Dell and whatever else and going back to the days of actually having rack-scale products.

[00:18:13] NVIDIA were very... done a fantastic job in releasing their NVL72 rack, a fully holistic, vertically integrated product between the network fabric, the compute and the power and everything else. You think back to the days of old when you bought your kit from Sun Microsystems and any of the other big IAM vendors, you would got the integrated rack products. So to me, it's brilliant to see us getting back to that

[00:18:41] because it solves a lot of the problems that we've had in the past 15, 20 years of where I'd maybe go out, buy my servers from Dell, buy my switches from Cisco or Juniper, and maybe buy my storage from EMC prior to the Dell acquisition and bits and bobs like that. And you're basically just hoping that they play well together in a rack. So I think that's one of the biggest challenges that the industry had in the past five years and moving to that vertically integrated rack-scale product

[00:19:11] is going to solve a lot of those. And now the biggest challenge is basically fitting ever-increasing power densities into geographic regions. I mean, take Singapore as a good example. While the MVL72 racks are absolutely the single best thing and single most efficient way of generating tokens at the lowest cost of dollars per token or tokens per dollar,

[00:19:39] finding capacity in Singapore to host that kit is a challenge. And that's just something that if we think back to the sort of build-out of the internet and the sort of digital age, connectivity and capacity back in the late 90s and early 2000s wasn't great. You think of how long it took to get broadband into people's homes and everything else. We're just going to have to go through a phase as an industry where there's a lot of growing and teething problems until we get to the point where we've built out a lot of this infrastructure to the point

[00:20:09] where it can seamlessly be deployed. I think Singapore, again, using Singapore as a bit of an example, but it will be possible eventually to deploy this stuff in Singapore. You've not been through the build process of either doing greenfield development of a new data centre or retrofitting an existing one or whatever. And, of course, an enterprise can own incredibly expensive AI infrastructure

[00:20:36] while surprisingly getting very little productive work from it. We've seen a lot of stories around this over the last few years. There was that, I suspect, slightly exaggerated stat from MIT that 95% of AI projects were not seeing ROI. Then there's the old IT mantra that we should throw in here, if you can only improve what you measure. And I'm curious, from what you're seeing here, especially now there is a focus on measurement and ROI,

[00:21:03] what kind of metrics should CTOs be watching to understand whether their AI infrastructure is genuinely performing efficiently? And what would you want reported to the board? I'm curious what you're seeing here. The way we've looked at it is more about, so because of the nature of our business, being able to, for example, for ourselves, if I'm able to self-host an open source model and have my internal teams consume that

[00:21:32] rather than having to buy tokens from Cloud or ChatGPT or any of these things, I can at least deem that a financial success. Yeah. And the idea is, we like to have the idea internally at Stellia that we don't just have external customers. A customer is a customer, regardless of whether it's an internal team or an external company that I'm providing that to. So that's like the most basic level that I can say

[00:22:01] we can measure an ROI. Am I able to reduce our spend on tokens as an organization by self-hosting this stuff and building a product around it? And more importantly, once I've proven that product, I can then take it to other companies who want to do the same. That's like the lowest hanging fruit if we're looking at other things. It could be an ad tech company that maybe wants to do image and video generation around ads.

[00:22:28] Maybe it's more typical machine learning algorithms around that market vertical. Could be companies that need to do modeling, like weather modeling, for example, if they're focusing on doing agriculture and crop yield and things like that. We've been in a position where we moved one customer off a sort of legacy high-performance computing system from a university and moved them on to more modern GPUs.

[00:22:58] And not just by giving them access to that, but helping them sort of parallelize their workload and making them be able to do sort of model training and inference simultaneously, we've been able to really, really help them improve their output. So I'm looking at it more in terms of outcomes for a business rather than just, oh, GPUs produce these tokens. Yeah, and I think there's something else we should probably throw into the mix here

[00:23:27] is energy, which adds another dimension because AI economics increasingly come down to how much useful computation their organization can extract from every single watt and every dollar invested. So do you think metrics such as energy efficiency, GPU utilization, do you think they will become business KPIs and how should leaders connect their infrastructure efficiency with AI return on investment? Do you think that's a natural evolution?

[00:23:56] Well, absolutely. If I look at the typical enterprise, if they go out and buy a GPU cluster, they're maybe only using it Monday to Friday, maybe 9 a.m. to 5 p.m. Now, you've definitely got a case of, sure, if you're a bigger enterprise and you've got different international offices, they'll be utilizing it, things like that. But you're right, it's going to come down to a utilization game of viewing this as an asset that you purchase. If I was going out to raise money

[00:24:25] in order to build a hotel, the first thing is the people, the people I'm pitching to for money for this project are going to ask me is what's the forecasted utilization rate? And if I turn around and tell them, oh, like 62%, they're going to tell me, nah, don't bother building this. So there's going to inevitably come a point where you'll get from CFOs, if we're going to spend all this money on these assets, we need to guarantee higher levels of utilization, which is inevitably going to come back

[00:24:54] to how do we make sure that we've got jobs running over the weekend and things like that. So I think KPIs are absolutely going to be placed on what is the actual utilization of the GPUs, how much tokens are we producing? Again, are we even breaking even? But also you need to start factoring in the typical enterprise who buys those types of assets are going to be in the position that they're probably doing it for some level of sovereignty

[00:25:24] or whether it's data residency. There's some external factor that's going to push them to want to do that rather than just, again, using Claude or ChatGPT or some other model provider. And I think that's a sticking point at the minute where a lot of businesses accept they've got those external factors placed on them or requirements placed on them, but they don't know how to show that in a KPI feat. We've talked a lot today about the journey that you've been on,

[00:25:53] everything that you've seen, how that knowledge that you've gained along the years will help you now and moving forward. And I'm curious, if you were designing an enterprise AI environment from scratch here in 2026, how would you connect compute, networking, storage, orchestration models and applications so that whole system is designed for production rather than just experimentation? I suspect it's a question you might get a lot, but it's such a massive talking point now. We've seen a lot of people

[00:26:23] get this wrong, but I'm curious how you would approach it. Well, I mean, it comes back to what size of company are you? What's your budget? What's your appetite? And what's your end goal? We've already seen a fantastic model set already by the hyperscalers. I mean, you look at who was the first customer of AWS, it was Amazon. Who was the first customer of GPT? It was Google. So on, so forth. They all built cloud

[00:26:52] not only as a sort of developer-focused productivity thing, but it's an economic model. Cloud allows you to get asset utilization as high as, as close to as 100% as reasonably possible. It allows you to set up availability zones that you can move workloads place to place. Again, if I was, let's say, an algorithmic trading company and I was setting up clusters in different areas, I want a way

[00:27:21] of being able to run those clusters as close to 100% as possible. I want the means of being able to replicate data site to site as a sort of archival and backup and redundancy strategy. There's a whole lot to consider, but I do think one of the problems you come back to is the cost of achieving this and doing it properly at the minute. It's a high barrier to entry, being completely candid. And I always try and give people listening

[00:27:51] a valuable takeaway. So if we do have a CTO listening who already has the luxury of AI pilots, cloud commitments, GPU investments and increasing pressure from the board to demonstrate results, of course, are there any practical assessments that you'd recommend that they conduct now and how can they identify whether their biggest constraint sits in the model or the software, the data, the network, storage, compute or just anywhere in the underlying system design

[00:28:19] before they approve the next round any advice you would offer there, people listening? I mean, a lot of it comes back to what you've bought and what you bought it for. Like again, if I was buying a GPU cluster for model training, that factors in that all the individual compute nodes need to be interconnected. That brings in every increasing cost in the network fabric and switches and so on and so forth. If I'm,

[00:28:49] I go back to an early proof of concept that we'd done at Stellia in terms of having a sort of internal model service so that internal teams could consume open source models at Stellia, I took one server we already had, we put some GPUs in it, we set up a piece of software by the name of VLLM, an inference engine and we were serving models internally. That was a couple of consumers

[00:29:18] then we got a request to increase that so we got an additional server, we done the same thing again, we threw a sort of little load balancer in front of it to help load balance the requests and before you know it, we'd done that four or five times and we got to the stage where I was able to sit and say, okay, we now have enough internal buy-in to this type of product, I can actually sit down and do a bit of modelling as to what it would cost to deliver this quote-unquote properly

[00:29:48] I think it's a lot easier to get spend when you've already got a justification and I think it comes back to having a little bit of that sort of cowboy cavalier mentality that I referenced at the start of the show. If I was wearing my strictest most stringent CTO hat I would not have done things like I just mentioned in terms of let's just install some GPUs in this existing chassis and see what we can make work until I've got an existing customer base for example.

[00:30:18] And I think that is a thought-provoking moment to end on and for anybody listening wants to find out more information or continue a conversation with you or find out more information about some of the work you're doing at Stellia there. I was reading before you came on you've launched a new platform AI operating system full stack operating system for enterprise AI. That's incredibly cool. I'd urge people to check that out. But where should people go if they want to find out more on anything we talked about today? Well we've got the Stellia

[00:30:47] newsroom on the website. We frequently put out articles there. We're quite keen on giving engineers sort of marketing time so that they can sit down and put out technical focused content on why we chose a certain technology at Stellia, why we use a certain programming language, things like that. We've got a strong marketing presence on LinkedIn. The marketing lady's doing a wonderful job with that. We've also got a, I believe,

[00:31:17] monthly or some regular cadence of newsletter that goes out on LinkedIn as well. The marketing team do a good job there with that. If you want to follow up with me, feel free to reach out on LinkedIn. I'm not as active on there as I should be, so if I don't respond to you immediately, don't worry. If not, there's an email contact on the website which marketing take care of and feed and water that. There are many ways to reach out. Love it. I've loved chatting with you today about

[00:31:47] this competitive advantage in AI. Now it lies in deployment excellence and how many leaders are getting that wrong because they were optimising the software layer while ignoring compute, storage, networking and system design. It was so refreshing to hear from you today. You've got this engineering mindset. We hear a lot of BS now, a lot of big promises at tech conferences. It was great review on this. I will have links to everything that you mentioned now. I urge people listening to go

[00:32:17] check that out. More than anything, thank you for taking the time to not only explain all this but in a language that everyone can understand. Really appreciate your time. thanks Neil, it's been a pleasure speaking with you today. So many big takeaways from Dave today and I think one of the most useful lessons there is defining the load before building the bridge. A successful pilot proves that an idea can work for one customer under friendly conditions but production asks whether the system can support thousands,

[00:32:47] move data quickly, recover and justify the money and energy that is being consumed. And this requires CTOs to inspect the path from storage and networking to compute to models and applications. Because GPU utilisation also deserves the same commercial attention as occupancy in a hotel almost. And an expensive asset sitting idle for evenings and

[00:33:16] weekends is very difficult to defend however impressive those spec sheets can look. And I also appreciated Dave's willingness today to combine engineering discipline with little experimental courage because businesses need evidence before they approve the design and it was so refreshing to hear this today. So over to you before buying another GPU. Do you know what your current system is sat around waiting for?

[00:33:45] techtalksnetwork.com You know where to find me by now. I'm nice and easy to get a hold of. So let me know there but it's time for me to go already. I'll be back again tomorrow with another guest but thank you for listening. Speak to you soon. Bye for now.