Bringing enterprise AI inference on premises with Scality

[00:00:00] The leading issue of agentic AI in businesses right now is ensuring agents act with compliance guidelines. And Denodo applies guardrails across your entire data estate. By aligning your company's data infrastructure under one system, these guardrails perform consistently across your platform. So start scaling your business and start with Denodo. Simply visit denodo.com to learn more.

[00:00:27] What changes when your AI application becomes useful enough that its growing token bills start shaping the business case? Well, for organisations moving inference into production, the decisions extend far beyond choosing a model.

[00:00:49] They include where sensitive information is processed, who controls model changes and how efficiently GPUs will use the data that they need. Well, today's guest is from a company called Scality. And we're going to discuss those decisions following the launch of its AI inference factory recently. And my guest will explain why storing and reusing contacts can reduce repeated computation.

[00:01:16] And also discuss how storage, networking and memory all affect the performance of an inference service. And hopefully if we've got time, we'll also discuss where human approval belongs in infrastructure automation. And also how their teams can test sovereignty rules and why recovery plans need evidence that data can be restored.

[00:01:39] So for business and infrastructure leaders listening, if you're considering an on-premises AI, the conversation today will offer a few questions around cost, control and the responsibilities that come with operating the service themselves. But make no mistake, there are a lot of opportunities here. A lot of excitement in this space for good reasons. So enough from me. Let me introduce you to my guest now. So a massive warm welcome to the show, Paul.

[00:02:09] Can you tell everyone listening a little about who you are and what you do? Yeah, absolutely. I'm Paul Specioli. I'm the chief marketing officer at Scality. Scality is a vendor of data infrastructure software. We've actually been around a while. The company started in 2009. And what we specialize in is building data infrastructures for AI, for cloud and for cybersecurity for data protection for backup. Awesome. It's a real pleasure to have you on here with me today.

[00:02:38] There's so much going on in this space. There's a certain amount of hype. You may see a few things on your news feed that might frustrate you as well. But when people talk about an AI data layer becoming an infrastructure control plane, what decisions should that layer actually be able to make? Yeah, I think it's pretty simple. In a production inferencing system, a data layer like ourselves has one job. And that job is to make sure the GPU never computes the same thing twice. Right?

[00:03:08] So think about how a session works. You initiate a session. You send a query, a prompt. The session either has to recompute everything from the information you sent. Or if it's seen it before, it can read it back from storage. Okay? Reading it back from storage is so much faster. It's like 10 to 80 times faster than recompute. People think that, you know, this inferencing is a compute problem. It actually turns out to be a memory problem.

[00:03:34] So one of the things we can do is stage all of this existing recomputations in memory. You bring it back. You don't have to do all that recompute overhead. So that's our job. Make sure that the GPU never has to do the same thing twice. And one of the things I always try and do on here is demystify a lot of the complex technologies we're hearing about. And as AI moves from pilots into production inference there, as you mentioned, what is the data layer underneath? What is it actually needing to do? Well, I think it needs to do a few things.

[00:04:04] It needs to be obviously to make sure that the GPUs are fed, as we talked about, but also to keep the costs predictable, to make sure that you maintain control of your data. And that means, for example, the jurisdiction of your data that governs your data. That's really key, right? If you end up looking at the spend that's happening in AI right now, it's absolutely dominated by the compute costs. I'm sure you've heard how expensive memory and GPUs are.

[00:04:32] Storage is a small part of that spend, but it has so much of a bearing on how efficient the compute is, right? If we can make sure that those GPUs stay fed and make sure that the data is trusted and make sure that it's sovereign and governed in the right places, that's really what's doing the job. Those are the three key levers. We're hearing a lot on our newsfeed. It almost feels like we've come full circle from looking back at the cloud and what happened there, everyone going to the cloud, then coming back.

[00:04:59] Why are enterprises bringing AI inference in-house, and where does storage fit into that economics too? Well, I think it's – once again, it's predictable cost. Think about what happens when you have an AI project that succeeds. Users start using more tokens, right? The more tokens you use, the more expensive things get, and you end up having this problem that we all had in the cloud where you have these surprise charges, right? These hidden fees. Well, here it's not even hidden. You're just going to spend more on tokens, right?

[00:05:27] So what you want to do is make sure that that gets minimized. One of the ways you can do that is to control the infrastructure on-prem, on the servers that the customer owns already. Now they're not renting from some AI provider. But moreover, you can make things cheaper by making sure that you're using data efficiently. One thing you'll notice in the world of these public AI providers, they make it cheaper to use a token that's been cached, about one-tenth of the price.

[00:05:56] So even they realize that compute is expensive. Well, on-prem, you can make sure that that happens because you paid for the infrastructure one time. Now you can amortize it the more and more you use it. And I'm curious, what is it that makes data placement so difficult when performance, cost, sovereignty, which is proving to be a big topic over here in Europe and beyond, and cyber resilient, when they're all pointing in different directions? Yeah, that's really a hard one, right?

[00:06:22] And it's one of the things we thought about when we came out with the latest version of our software is that you can't just put everything on Flash anymore, right? It's too simple of an answer, and we all know what's happening with memory prices like NAND Flash. It's gone through the roof. It's hard to get. So there's really no single right place for the data. There's only a right place for each workload, and performance obviously wants things next to the GPUs.

[00:06:47] It wants it fast, but if you want things inexpensive, you want a denser, colder tier, right? The thing that you can't trade off is sovereignty, right? If you need to make sure that something's located in a certain geography, that's something we can control. But the practical answer is this. As long as you have one namespace and that namespace can move data between hot Flash, warm HDD, cold tape, or cold cloud, it's a policy decision where you put things.

[00:07:17] It's no longer a big migration project, which used to be a huge headache. Yeah, 100% with you there. And for business leaders listening, how should their organization decide which infrastructure decisions can be automated and which still need human approval? Because it's a big topic right now, isn't it? It's a huge topic, and we face it ourselves, right?

[00:07:39] Because we agree that as the data grows, as these things get more complex, you're not going to add much more human headcount to manage all this extra stuff, right? So our sense is you can automate the recommendations, but keep a human on anything that you cannot undo, okay? So, for example, very simple to automate health checks, forecasting, recommendations. I would automate that. But deleting data, changing policies, moving data, you need a person to approve that, okay?

[00:08:08] And that's the way we've implemented it in the product. Love it. And vendor neutrality also sounds attractive, maybe even idealistic to a certain extent. But where does it become difficult when storage platforms, cloud services, AI tools all expose different capabilities? Yeah, that's also a tricky one. I think the neutrality argument just means convincing yourself that every platform is not the same.

[00:08:35] You really don't want to let one vendor's differences become your lock-in, right? So don't fall back to the lowest common denominator. S3 is a common API language, but the features underneath it, like data retention, locking of objects, they behave different on all platforms, okay? Our approach is to be open code, okay? Every change we make, every fix we make gets committed upstream. People can view the code. There is no scality-specific fork, okay?

[00:09:04] The other thing we do is we run on standard servers. So we don't sell you our proprietary storage server. You can buy it from HPE, Lenovo, Supermicro, Dell, whoever you like, okay? So being as open as we can, that's really the best way to get to neutrality. And another question I'd love to ask you here, what does a credible immutable recovery plan look like now? Especially when we're living in this time where attackers could have been inside an environment long before the incident is discovered.

[00:09:32] It's a big talking point now with the quantum stuff as well. Now we get now and decrypt later, for example. Again, it could be a time bomb, right? It absolutely could. I think even two years ago I was hearing that most of these intrusions have been going on for months before they're detected. You know, now we're entering an era where AI can actually help power the ransomware. So I think things are going to get more scary. The answer to your question is obvious. You have to assume they're already in, okay? Yeah.

[00:10:00] And then make sure that there's one copy of your data they cannot touch and prove you can restore it fast, okay? What we found in all the studies, and this isn't just from us, but from big vendors like Veeam, attackers go after the backup first. It's very pernicious, right? They realize that if they go after the backups, you can't restore, and now you're really stuck, and you're going to have to pay the ransom or whatever else they're extorting you for, right? So immutable really has to mean something.

[00:10:27] It means that even the administrator can't override things because you get social engineering attacks where people are threatened, right? You have to set these retention periods really longer than an attacker is likely to sit undetected, right? And then I would say restore speed. You have to get a fast restore speed because if you have something that's safe, you want to get it back as soon as possible. And by the way, we're in an era where you're not restoring one file or an hour's worth of data.

[00:10:56] You're probably having to restore a whole data center, right? So performance really matters. In the end, we think that's what's important, and we have actually put our money behind it. We have a cyber guarantee. If you get attacked on our system, we pay you $100,000 USD. Oh, wow. And for people listening, maybe their teams need to be testing that governance and sovereignty rules still hold when data moves across so many different locations and storage tiers. Always try and give people listening a valuable takeaway.

[00:11:24] Any tips, hints, advice, or best practices that you could provide there? Yeah, I think our rule of thumb is that if you have policies for this and you don't test it, then you're really just residing on hope. You know, people have gotten really good. Corporations have gotten really good at testing code. It's continuous. It's after every change. I think what we still see is that governance and sovereignty checking happens kind of once a year at audit time. You need to do these tests, and you need to do negative tests.

[00:11:54] Try to move data somewhere where it isn't allowed and see that the system refuses, okay? You can actually tag data and say its jurisdiction is here. Now you can make sure that every test moves it to somewhere that is enforcing that rule, right? And always just keep an always current record of where all the data is. Number one problem in the organization is lack of visibility. Where are my backups? Where are things stored? Which one's the good one? Make sure that you have a current visible record of that.

[00:12:22] And you've seen so many big changes in this space. I know you've been in this space a long time, seeing so many big changes and all these different trends. But what is it that changes for infrastructure teams listening when their AI inference runs on infrastructure that they actually own? Yeah, I think the parallel here is to the cloud, right? We had public cloud starting in the 2006-2010 era with Amazon and all the like.

[00:12:47] Public cloud sounded great, and then everybody went into it and had this shock bill of all these surprise charges. I really think the same thing can happen here in AI. So the big change is infrastructure teams need to become the owner of the AI bill, okay? They're the ones who can manage it and make it smaller. If they do that, they will start tracking metrics like how much time to the first token, how much reuse is there, right?

[00:13:14] How much of the context becomes data that they need to actually manage. They can then make decisions more consciously about when does a model need to change, right? They'll find out things like the network matters more than people expect it. It's really the limiting factor a lot of times in the performance to keep the GPUs fed. So I think the lesson is the infrastructure team needs to become the owner of the AI bill. Then they'll optimize it.

[00:13:40] And I must admit, when you started talking today, especially around the topic of storage, I immediately thought you'd be a great fit for the IT press tour, which is something I've been on multiple times over the last, what, four or five years. And I noticed before you joined me, you were actually on that tour way back in 2010, 2011, 2012. How much has changed around storage in that time? What excites you? It's been such an inflection point.

[00:14:08] I'll tell you, back in that era, object storage was kind of emerging, right? And I was actually with another one of the object storage vendors that was coming out at the same time. You know what we all were pained by at that time? There was no standard API, no standard interface, no protocol. In fact, I remember being the product manager for such a product and everybody said, the number one feature we need is file, file-based access. Well, that kind of defeats the purpose of an object store.

[00:14:34] So, one big inflection point, everybody embraced Amazon's S3 API. It's a big deal, okay? Number two, a big inflection point was cloud-native, containerization and Kubernetes, right? Because it meant the way that storage was provisioned became more automated. That was a really big deal. We had to react differently. We had to have different APIs. And then third, this idea that storage would be instrumental in cybersecurity. Backups, as I said a few minutes ago, are critical, right?

[00:15:03] They're kind of your last line of defense. That was a big deal. Before that, we were just thinking, oh, backups need to be faster than tape, so let's put them on disks, right? That was the whole deduplication era. Now we realize it's not about the backup, it's about the restore, right? So, you need to be able to have that copy. It needs to be safe. It needs to come back really quick. And look, I can't ignore it, but I got to say, this last era, this current era, AI, is the other inflection point, right? We never really thought that we'd be used for AI inferencing, right?

[00:15:33] That's a big change. And now making sure that storage is fast enough for that is the real key. It really is. And, of course, we're talking today just a couple of weeks after you launched your AI inference factory to bring enterprise AI on-premises. Where do you go from here? What does 2027 look like for you? What excites you about that and where you're going from here? Well, I think now it's a lot of the things we've talked about here, right?

[00:15:56] Making sure that people understand that controlling their own fate in terms of their own infrastructure, we think sovereignty is a huge deal. Especially in Europe, but also broadly speaking around the world. People are going to know what the lineage of their data is, what the lineage of their code is. This whole open code model is something we're pushing. AI inference factory uses open weight models. We think that's a big deal also in terms of making sure you can lock down and trust the models that you're using.

[00:16:25] You know, you've seen this too if you use one of the frontier models. They change it on you almost weekly, and it's very hard to control. So we think controlling, maintaining the cost, maintaining the sovereignty of AI is what's next for enterprises. If you read a lot of the industry literature, you'll see the prediction is that more and more of inferencing becomes on-prem and really to an end state of hybrid. So people are using it partly in the cloud and partly on-prem. I think that's the future. Awesome.

[00:16:52] And for anybody listening interested in exploring that AI inference factory and learning more about how they come up to bring their enterprise AI on-premises, where should they go? Where can they find out more information and maybe connect with you or your team? Yeah. Well, do you mind if I give a couple of plugs? Yeah, exactly. The website, scality.com. We have an ancillary blog on another property called solved.scality.com. You can read all about the thought leadership and these ideas that I've talked about.

[00:17:21] And then selfishly, me and my CTO host a podcast on YouTube, not anywhere to compete with yours, but it's called the Scale Out Podcast. And we talk about topics that are relevant to us, right? Things like KVCache and inferencing and all the tech underneath it. So those are three places people can find out more. Awesome. I will add links to everything. For people listening, if you go to techtalksnetwork.com, there'll be a blog post associated with this episode.

[00:17:46] Not only will I include the links that you've mentioned there, I'll also embed one of your YouTube episodes. So please go check that out. But more than anything, Paul, just thank you for coming on here, talking about all this very complex technology in a language everyone can understand and the value behind it too. Really appreciate your time. Thank you. I value the opportunity, Neil. Thank you. I think Paul's explanation of reusing context gives infrastructure teams a way to think about costs.

[00:18:12] The question is how much the work the system repeats, where the information is held and whether it can be retrieved efficiently. Owning the infrastructure, yes, it brings responsibilities because teams will need to understand utilisation, operating costs, model changes and the rules that are governing their data. So I think ultimately the business case will depend on the workload and the organisation's ability to operate it reliably.

[00:18:40] I also appreciated Paul's advice on testing restrictions because trying an action that should be refused can reveal whether a policy actually holds in practice. And recovery exercises, they show whether protected data can be restored when needed. So a big thank you to my guest. You'll find all the product information, technical blog and scale out podcast over in the episode notes at techtalksnetwork.com.

[00:19:09] But over to you, which of your AI workloads would you consider running on your own infrastructure? And what evidence would you need before deciding to do that? After listening to today's episode, show me your thoughts over at techtalksnetwork.com and I'll get straight back to you. But that is it for today. So thanks for listening as always. Bye for now.