Why AI Agents Fail in Production: TrueFoundry CEO on Building Reliable AI Systems

[00:00:00] - [Speaker 0]
Your agentic AI might not be secure even with real time data and proper guardrails, but Denodo makes sure your business has every avenue covered. By placing all your data platforms under one AI data layer, your business can reach semantic consistency safely and securely. So get your agents on the same page by visiting denodo.com, And you can learn more about how to start trusting your agents to make business decisions. What separates another impressive AI demo from an AI system that can actually run a business? This is one of the questions at the heart of today's conversation because I have the CEO and cofounder of TrueFoundry joining me.

[00:00:54] - [Speaker 0]
And before launching the company, he worked at Meta helping build conversational AI systems that served more than a billion users. And it was this experience that gave him a front row seat as to what it actually takes to operate AI at scale and a massive scale at that. And also why success has far less to do with the model itself than you might be thinking. Today, my guest is helping enterprises move AI from proof of concept into production by tackling those infrastructure, governance, observability, and operational challenges that are often responsible for determining whether an AI initiative will succeed or fail. And it's a platform trusted by organizations including NVIDIA, Siemens Healthineers, and so many others that we're not allowed to mention.

[00:01:45] - [Speaker 0]
But most importantly, I wanna hear about the lessons he's learned and maybe offer a fascinating glimpse into where enterprise AI is heading next. So you can expect us to discuss why AI is fundamentally a systems problem, why so many agents that look brilliant during demos struggle in real world environments, and what reliable autonomy. What does that actually mean when businesses begin trusting AI with increasing complex workflows? So if you're trying to understand what enterprise AI adoption actually looks like beyond the headlines, demos, and keynote presentations. Today, we're gonna take a practical look at the challenge that maybe your organization is facing right now and some of the solutions that are emerging to address them.

[00:02:31] - [Speaker 0]
But enough from me. Let me introduce you to my guest who will talk about all this and join us in exploring the next chapter of AI and why it will be won by the companies that build the strongest systems, not necessarily the biggest So thank you for joining me on the podcast today. Can you tell everyone listening a little about who you are and what you do? Thank you so much for

[00:02:56] - [Speaker 1]
having me over, Neil. I'm Nikunj, cofounder CEO at TrueFoundry. Based in the Bay Area, and I come from a machine learning background. I had the fortune to lead one of the conversational AI teams at Meta, shift one of their first deep learning models on device, and got a chance to contribute to Meta's internal ML platforms as well, and have always worked at the intersection of building ML models and ML infra. At TrueFoundry, we are taking up taking up learnings from Meta and adapting it for the requirements of large enterprises, building a product called AI gateway, which helps enterprises, connect, observe, and govern their AI applications.

[00:03:38] - [Speaker 1]
So think about all agents, all MCP servers, all models connected in one single control plane, where you can observe and govern all your traffic.

[00:03:50] - [Speaker 0]
It's incredibly cool what you're doing here, but one of things I always try and do with my guests is go back in time a little and look into their origin story. And when I was doing a little research on you, I read that before founding TrueFoundry, you worked at Meta on conversational AI systems serving more than a billion users, which is just phenomenal. But what lessons from operating AI at that kind of scale are still missing for many of today's enterprise AI conversations? Because I imagine this gives you somewhat of a unique vantage point. For sure.

[00:04:20] - [Speaker 0]
For sure. Yeah.

[00:04:21] - [Speaker 1]
I think there was there was so many learnings for, for me, like, you know, working at Meta, and using some of Meta's internal ML platform. I think the biggest one of this is when we were building any ML applications, right, or conversational AI applications at Meta, shipping any models to production, as an ML engineer, I could focus on the the business problem that I'm solving, the model that I'm building, basically. Right? And everything else around that was essentially taken care of, taken care of for me. Right?

[00:04:54] - [Speaker 1]
So I could literally go from completely ideating a model to launching it in production on thousands of servers to billions of users, as you mentioned, all within a single day. Whereas oftentimes, in other enterprises, this can this can, like, gonna take up weeks, months, or sometimes even longer than that, to ship anything meaningful, that is production ready. Right? So that that, I think, is the biggest, call it, the impact that I've seen, like, you know, given that the systems that we had at Meta versus what we have outside. The if I have to zoom into this, right, the learnings, what are some of these systems that help me operationalize my machine learning applications of the developer?

[00:05:35] - [Speaker 1]
Well, when I'm building a model, by default, I don't have to worry about logging things. Like, everything by default is observed. Everything by default is logged into an underlying database. There is a clean query layer built out so that if I wanna debug something, I do not have to reinvent the wheel. It's all right there for me.

[00:05:54] - [Speaker 1]
If I have to ship it to production, I don't have to rewrite my code in a certain way. Any kind of governance, like, you know, things that Meta as an enterprise wants to apply to all of its applications, That just happens by default. I don't have to work with five different teams to, like, get approvals and stuff like that. Right? So all of these little details that can result in a lot of delay are taken care of.

[00:06:18] - [Speaker 1]
And this is, like, you know, some version of Meta had built its own internal control plane for building its AI applications, and we are trying to take those learnings and make something similar available for the rest of the enterprises out there.

[00:06:30] - [Speaker 0]
And we do hear a lot around breakthrough through models, but you've argued that AI is actually more of a systems problem. So what is it that business leaders and indeed tech teams often misunderstand when they focus solely on the model rather than the the infrastructure around it?

[00:06:47] - [Speaker 1]
The the proof is in the pudding. Right? Like, if you think about the applications that are being built today, how many times as users of these models or builders of our applications do we run into the limitation of the model itself? Right? Like, in a way, the model is not able to answer this blah question in a nice manner.

[00:07:05] - [Speaker 1]
It's very, very rare. Right? Today, the industry has gotten to a point where the capability of the model is no longer the rate limiting step. In fact, we are not building applications that leverage the models to their full potential. It's about understanding our workflows.

[00:07:23] - [Speaker 1]
It's about codifying those workflows into meaningful model queries. It's about building the systems that capture. Like, when you make a call to a model, are you logging the responses? When something goes wrong, are you able to debug this? When the model itself goes down, are you able to fall back to another model?

[00:07:42] - [Speaker 1]
Right? Like, you know, are you building this resilient systems around this right now? Do you have a kill switch? Fundamentally, these models are stochastic in nature. Agents are stochastic in nature.

[00:07:53] - [Speaker 1]
What if something goes wrong? Do you have a kill switch that you have built into the system? Do you have guardrails that you have baked into the system? Right? And how you manage the life cycle of each of these components is what is basically what sits in the middle of organizations that can ship applications to production versus organizations that that are not shipping applications to production.

[00:08:14] - [Speaker 1]
It's actually not the quality of the models. I genuinely believe it's the systems around it.

[00:08:20] - [Speaker 0]
And over the last two years, we've seen many AI agents perform incredibly impressive in demos, but often struggle when deployed in real business environments. So what are the the most common reasons that AI projects fail once they move from proof of concept and pilot phase and into production? What Yeah. What's missing here?

[00:08:41] - [Speaker 1]
Like, I think in general, right, this entire, notion of proof of concept, like, if you think about it, we build out our demos, proof of concepts in a very controlled environment. Right? In a in a lab type of an environment. So there are no bad actors in a lab environment. Nobody is trying to intentionally break your system or have has an adversarial, like, you know, intent.

[00:09:07] - [Speaker 1]
Similarly, in in your demo environment, you are maybe not testing out the hardest of prompts, you're just not able to simulate things that can go wrong outside of your typical case scenario. You don't have, like, you know, a lot of load being sent to the model. You do not you're not necessarily simulating that workload where maybe other people are sending out a lot of workload to the model, which, by the way, is the real, like, scenario in production because LLMs fundamentally are a shared resource where other people's usage of the model can impact the quality of your own application. Right? During the demo environment, hopefully, the entire model is not going down or your data center is not going down.

[00:09:49] - [Speaker 1]
So a lot of the reasons why your applications break in production, which erodes the trust of the business running these applications, are like, they just don't happen during the POC phase. Right? So everything in the POC looks rosy and, like, in a good to go. But as you start putting things to production and if every component in your system had about 99% perfection rate, if you multiply that 99%, like, 10 times, you suddenly have a system that breaks much more often than you can afford in production. Right?

[00:10:21] - [Speaker 1]
I think that's the that's the core reason why, like, you know, things that look perfect in demos just don't work in production.

[00:10:31] - [Speaker 0]
And further looking at your career, you know, you currently work with organizations far and wide, including NVIDIA and Siemens, Siemens Healthineers. When enterprises began scaling AI across so many multiple departments and use cases, etcetera, I'm curious what kind of new governance, security, and operational challenges suddenly appear? Because, again, at the foundations, this is incredibly important, isn't it?

[00:10:58] - [Speaker 1]
Absolutely. Absolutely. So I think some of the organizations that you mentioned and some that I cannot name right now, these organizations operate in, extremely regulated industries. Right? They, when they are shipping their AI applications to production, they want to make sure that, like, of course, nothing breaks.

[00:11:19] - [Speaker 1]
The the applications run well. But also the fact that when these applications land in in in an external user capacity, they are also exposing themselves to a bunch of regulatory and compliance risk. But sometimes, even simpler reputational risks are actually like, you can't do enough damage to these companies that they need to be extremely wary for any application that they're putting in production. Right? So what we have seen, organizations that have done very well in terms of productionalizing generative AI, agent decay applications are the ones that took the time to build out this internal plumbing layer of how or, like, you know, a governance layer of how they expect their developers to ship these agents into production.

[00:12:04] - [Speaker 1]
This could mean a variety of things. This could mean setting up certain guardrails for the type of datasets that these models, are suddenly privy to, right, versus being able to set out certain kind of, policies around how much cost you are okay to invest into building certain application that prevents, any kind of, cost spillage by, human errors. That also prevents the the swim lanes. That also presents the swim lanes that if I expect the model to operate and answer questions in these three domains, then those are the only three domains our chatbot will answer will present answers in. And if there is a question like, tell me a joke or who's the president of The United States, etcetera, etcetera, those questions will not be answered, not because the model cannot answer them, but because that can derail the application and directions that you just don't have any control over.

[00:13:00] - [Speaker 1]
Right? So organizations that have taken the time to build out that entire layer of control before they ship these agentic applications to productions are the ones that are actually coming out with flying colors, in this world in this agentic world.

[00:13:16] - [Speaker 0]
And everything that I'm seeing at the, what, 15 plus tech tech events I've been to around the world this year suggests that we're moving towards multi agent systems where different agents collaborate to complete complex tasks. But what does reliable autonomy actually look like in practice away from those keynotes and demos on stage? And how far are we from organizations really trusting agents with their mission critical workflows? Right. For sure.

[00:13:43] - [Speaker 0]
Yeah. So I think, all of this starts from,

[00:13:49] - [Speaker 1]
being able to define scope very well. So if you think about these multi agentic systems, right, of course, there's like a this super agent and sub agent architecture that people are building. What needs to be done to get to a reliable architecture with which we are able to ship these applications to production is creating swim lanes for these agents and defining what is expected and what is not expected. Right? The agent knows that these are the seven different types of questions that I'm allowed to answer.

[00:14:21] - [Speaker 1]
If the nature of the question goes beyond a certain degree, then I may need to escalate. Now this escalation can happen to, I don't know, like a like a super agent that that was built to evaluate the responses, or it can also happen to human beings. Right? And as these agent to agent interactions happen, a human being who designed or is overseeing the design of this system has some controls over it. Right?

[00:14:49] - [Speaker 1]
That maybe, like, it does not go into an infinite loop of communication. Right? How does this agent to agent interaction break from that loop? And the fact that anything whatsoever that is happening in these interactions, you have a complete audit trail of that. Once we start building out these multi agentic systems with, this kind of a system laid out, that's when, like, you know, we believe that we are moving towards reliability.

[00:15:15] - [Speaker 1]
All that said, one of the other extremely critical, things to note here is choice of the application that you want to run-in an autumn completely automated fashion is very, very important. Like, you don't want to take something extremely mission critical and directly make it completely automatable. Right? But at the same time, you can actually take something that adds a lot of business value at scale. But, given these are stochastic systems, if one thing were to go slightly wrong, it does not break your entire business.

[00:15:48] - [Speaker 1]
So I think the choice of the applications is the other thing that's that's extremely important.

[00:15:53] - [Speaker 0]
Yeah. I completely agree. And, also, I think compute cost, that remains one of the biggest barriers to AI adoption. More recently, we're adding AI tokenomics into the mix too. So how can organizations balance performance, speed, innovation, and all these things without allowing their infrastructure cost to spiral out of control.

[00:16:13] - [Speaker 0]
Again, big topic right now.

[00:16:15] - [Speaker 1]
Absolutely. Like, this is, pretty much the number one problem that we're hearing from all of our customers that how do they keep their AI cost in check. And, do this end actually, like again, like, you know, I I think of all of these problems as a systems problem. Right? For example, the default behavior of anyone is no matter what question I'm asking, I will go to the most powerful model.

[00:16:39] - [Speaker 1]
Again, all of us end up relying on like, all of us think that our problems are models' problems. Right? But if you send out a simple prompt to an, Claude Opus model or a Claude Sonnet model or a much cheaper Claude HiQ model, right, you will likely get a very similar response, but the cost dynamics of that would look very different. Right? Similarly, there are other levels that control your cost.

[00:17:02] - [Speaker 1]
For example, if you send out your request to what we call as a common pool of LLMs, right, Those are much cheaper, but if you end up buying your own provision throughput units with guaranteed SLAs, those could be a lot more expensive. So similarly, like, you know, there are there are different providers that have different cost dynamics with different types of models. So when you are sending out your queries to different LLMs, you want to figure out that based on the complexity, based on the cost structure that you have in organization with with different model providers, and based on how a certain model is doing in a certain region in terms of uptime and availability and latency, etcetera, you can actually create a meaningful trade off between quality, latency, and cost. And you need to have these queries routed through what we call as an AI gateway layer, where model gateway is a part or a model proxy is a part. And you can either set up rules as an organization that runs the query routing optimized to your needs, or you can let the AI proxy layer also automatically decide which query goes to which models and end up, like, making this three way optimization for you around quality, cost, and latency.

[00:18:12] - [Speaker 1]
So that's our and, like, we have seen organizations that have adopted this kind of an architecture have actually, in some cases, at scale in production, saved even over 75, 80% of their overall model cost. I repeat, 75 to 80% of their overall model cost. And this is by designing the systems right, choosing the right queries going to SLMs, and the right queries going to, like, know, these complex LLMs and stuff like that has resulted in massive, make or break type of situation for these enterprises.

[00:18:44] - [Speaker 0]
And there's also a growing discussion around AI gateways, model governance, MCP servers, and agent orchestration. It's almost an episode on its own now. But for people listening and hearing, these terms for the first time, how do all these pieces fit together, and why are they becoming so important for enterprise AI? Because the whole topic is so much bigger than just AI in inverted commas. Absolutely.

[00:19:10] - [Speaker 1]
So let me explain this, like, you know, just from from first principles. Right? If if we are building an agent, an agent needs to, first of all, talk to a model. Right? An an agent needs to have access to the brain of the system, which is usually a a large language model.

[00:19:27] - [Speaker 1]
But then the agent, as I mentioned in some of the earlier comments here, that the agent may want to decide which model a certain query is getting routed to. Right? So typically, it's working with more than one model providers. The interaction between an agent and a model happens through a layer that we call as model gateway. So that's the first piece in this puzzle, the model gateway layer that helps you route queries to different models.

[00:19:52] - [Speaker 1]
The second thing is once you have the brain of the system, then you need the actuators. Right? Like the the arms of the system in some way. Right? And that's where you interact.

[00:20:02] - [Speaker 1]
The agent interacts with the environment. And that happens through MCP servers. That's the most common protocol with which an agent interacts with other tools and actions that it may wanna take or dataset it may wanna read. So this environment interaction happens through a layer that we call as MCP gateway. This is where you would connect a bunch of off the shelf MCP servers like GitHub, Jira, Slack, Atlassian, whatever it is that you're using, or any internally developed MCP servers which may have access to your internal databases or maybe invoking some internal APIs.

[00:20:35] - [Speaker 1]
Right? So now our agent has access to the brain, is the LLM, has access to the actuators, which is the MCPs. And then the agent may want to interact with other agents, and it's almost like, you know, as a group of agents coming together, they accomplish some of the important tasks. Right? So that agent to agent interaction happens through the agent gateway there.

[00:20:54] - [Speaker 1]
So these are the three most important components, model gateway, MCP gateway, and agent gateway. The three of them combined into what we call as AI gateway. And this AI gateway ends up becoming this pipe through which then all of your token traffic is flowing, and then it implements your observability and governance layer. Because once you ensure the traffic is flowing through one pipe, you actually can easily implement the observability and governance layer on top of that.

[00:21:21] - [Speaker 0]
And if we look ahead further on into the year and even with one eye on 2027 already, which sounds crazy just saying that out loud, but what do you think AI development teams will be doing differently from today? And are there any other assumptions about building, deploying, and managing AI systems that you think could quickly begin to look outdated as we progress?

[00:21:41] - [Speaker 1]
For sure. So, well, I think I don't need to repeat this, that software as we know it today or as like, know, it's already widely different from how we knew it yesterday. And I believe the same thing will happen, like, you know, six months, one year down the line. It might even happen, like, three weeks down the line. Right?

[00:21:58] - [Speaker 1]
Who knows in the in the AI weeks and AI years what happens? But I I I strongly believe that given we are so early in the AI adoption still, some of the problems that are, like, just standard systems problem that we have learned a lot through our software engineering cycles will become commonplace in the AI world as well. And I'll give this and, like, you know, explain this through an analogy. The fact that when we write software, we don't think about the fact that, our software is packaged as a docker container. We check-in our code into GitHub.

[00:22:36] - [Speaker 1]
Everything goes through a CICD pipeline, and everything is by default observed through underlying monitoring system that we have. Right? These are just basic assumptions, that we don't even think about. This is second nature for us when we build out, some of these software engineering applications. We believe that there will be standardization that will come in from a systems perspective into how AI is being developed.

[00:23:01] - [Speaker 1]
Right? The fact that any AI calls are going through any model calls are going through a gateway layer, all MCP authentication authorization is happening through, again, the gateway layer, and agents are able to interact with each other. Every call is logged without the developer needing to monitor this. And all of these things comply with a with an organization's policies, their brand voice, their, like, you know, the representation that they want to carry externally. These things will become second nature in the AI world.

[00:23:32] - [Speaker 1]
That, I think, will be a major shift that will happen, in the very near future. If we think slightly longer than this, I think there's a new system that will emerge which will implement the same things, the same kind of infrastructure principles that apply to human beings building these agents. We will have an agent native infrastructure and control plane layer that will get developed in the next maybe couple of years or so that will come to life. So I think those are two major shifts that will happen in the in the near future.

[00:24:07] - [Speaker 0]
We've covered so much in a short amount of time today. And for anyone listening wanting to learn more about anything we discussed or how your platform helps teams take AI from prototype to production with simpler deployment, governance, compute efficiency, and dig a little bit deeper on anything we talked about. Where would you like me to point everyone listening to that?

[00:24:27] - [Speaker 1]
Well, we actually write, share a lot of our learnings working with our customers, as blogs and content on our own website, truefoundry.com, truefoundry, truefoundry.com. We also share quite a bit on our LinkedIn page as a company, and I personally share, a bunch of these insights from from my own LinkedIn account as well. So you can find find us either on our website at proofhondy.com or on LinkedIn. Like, if you just search my name, Niquinj Bajaj, you should, have it.

[00:24:59] - [Speaker 0]
Awesome. Well, as I said, we covered a lot there from the lessons you learned building AI at Meta at such a massive scale, why most AI agents look good in demos but often break in production, and what reliable autonomy actually means for long running multi agent workflows. So much more for people to dig into. So I urge them to go over to the website. I will include links to everything you mentioned.

[00:25:21] - [Speaker 0]
Go check out some of them blogs and maybe continue this conversation. But more than anything, thank you for starting it and bringing all this to life today in a language everyone can understand. Really appreciate your time.

[00:25:31] - [Speaker 1]
Absolutely appreciate it. Thank you so much, Neil, for having me over.

[00:25:34] - [Speaker 0]
What I loved about today's conversation is it always came back to fundamentals. And at a time when much of the AI conversation focuses on model capabilities, benchmark scores, and the latest announcement, his perspective was a reminder that successful AI deployments are actually coming down to operational discipline and governance, observability, resilience, and infrastructure. And I think it was his experiences at Meta that also provided a useful lens into what mature AI operations look like. That ability to move quickly, monitor everything, establish guardrails, and build reliability into the systems from day one. All this stuff isn't the glamorous work, but it might be the difference between an AI initiative that scales and one that stalls.

[00:26:25] - [Speaker 0]
And perhaps the biggest takeaway of all is that enterprise AI is becoming an infrastructure conversation as much as a model conversation. So as AI agents, models, m m MCP servers, gateways, and governance frameworks become increasingly interconnected, maybe you and your business will need systems that help manage complexity without slowing innovation. But over to you, are your is your organization spending enough time focusing on the systems around AI, or are too many still focusing solely on the models themselves? And what do you see as the biggest challenge when moving from prototype to production at scale? Yep.

[00:27:08] - [Speaker 0]
You know where to find me by now. Techtalksnetwork.com. You find me on all social channels at neil c hues. But that's it for today. So thank you for listening as always, and I'll speak with you all again tomorrow.

[00:27:20] - [Speaker 0]
Bye for now.