Building Evidence Based Trust for AI Agents With Vijil

[00:00:00] Your agentic AI might not be secure even with real-time data and proper guardrails. But Denodo makes sure your business has every avenue covered. By placing all your data platforms under one AI data layer, your business can reach semantic consistency safely and securely. So get your agents on the same page by visiting denodo.com and you can learn more about how to

[00:00:29] start trusting your agents to make business decisions. What would it take for an enterprise to trust an AI agent with its recruiting, financial, healthcare or even legal decisions? Well today I'm going to be talking to the founder and CEO of a company called Vigil. We're going to talk about

[00:00:52] the trust gap that is keeping agents between proof of concept and production. And my guest also brings 30 years of experience across security, open source, cloud, AI and so much more. Including leading engineering at Amazon SageMaker and helping develop 11 AWS AI services. And today though he's going to argue that trust cannot be reduced to just another benchmark

[00:01:22] or reassuring vendor claim. It must be tested in the context of task, users, threats and consequences involve. So I want to learn more about agent loyalty, reliability, safety, drift, least privilege, lifecycle security and everything from initial design all the way through to production monitoring.

[00:01:44] And my guest will also explain why vibes are not evidence and why nobody would vibe engineer a bridge. Intrigued? I was hoping you'd say that. So if your business wants agents to act with real authority, this conversation will help provide a few practical standards for deciding when you are ready to go. But you've heard enough from me. Let me introduce you to my guest right now.

[00:02:13] So thank you for joining me on the show today. Can you tell everyone listening a little about who you are and what you do? Sure. My name is Vin Sharma. I'm the founder and CEO of a company called Vigil. I've been building and shipping software for about 30 years. The first decade was on security for systems like Unix and Linux and web servers. The second 10 years was on open source software. Linux certainly

[00:02:41] started that, but also Hadoop and OpenStack and Apache Spark and projects like that. And then the last decade really has been a return to my first love, which was AI machine learning. And I've worked at Hewlett Packard, Intel, and most recently at AWS. And all of these fantastic institutions have informed the way I think about security for AI agents today.

[00:03:09] It's incredible the career you've had there and the journey you've been on. You were doing AI and machine learning before it was cool, shall we say. Before there were job prospects for people working in neural networks. Yes. Yeah, 100%. And of course, here you are now when you look back, did you ever think that it could be as big as it is right now? I had hoped that in my lifetime, I would get to work on intelligent agents that would actually do

[00:03:39] things useful, work in the real world. I grew up with as many of the people in my generation that work on AI now with science fiction. And there were, you know, obviously many ways in which you could look at the fears and hopes and aspirations of people around AI even 65 years ago, you know, and or perhaps even

[00:04:08] earlier than that with Mary Shelley's Frankenstein. So we've had a long journey to this point. But it's just amazing that we're now at that point where it's compounding at such an amazing rate that we might know. We have underestimated the pace at which AI would take over things and, you know, underestimated also

[00:04:38] the scale at which it will have an effect on our lives in the next 10 years. Yeah, we really did. And your hope has now become a reality because enterprises are now investing heavily in AI agents. But of course, many projects seem to get trapped between that successful proof of proof of concept and production and end up stuck in pilot purgatory. So why has trust become such a big barrier to getting agents into real business workflows? Because I go to

[00:05:08] the tech conferences, I see the keynotes, the demos and the big promises, but there still seems to be some kind of barrier there. There is. And we have to acknowledge that the barrier is a very real problem. The fact is that AI systems, AI agents in particular, are not like traditional software or not like any kind of software

[00:05:32] that we've built in the past because they have fundamentally two things that make us nervous. One is autonomy at some level, increasing levels, one would argue. And the other is agency. They can do things that, you know, are with that autonomy. That means that together, these two axes make us

[00:05:57] question whether we are well served by delegating our objectives to these systems. And fundamentally, if I were to put a point on what I care most about and what I'm working on and what we at Vigil are focused on, it's whether AI agents are loyal to the people that use them. And it's an unusual framing

[00:06:26] because you expect AI systems to be inanimate objects. They don't have intentions. They don't have mental states that we know of. And we can project those onto them and talk about, you know, anthropomorphize them and talk about how they're hallucinating or so on and so forth. But at the end of the day, we can actually check whether AI agents serve the interests of the people that use them

[00:06:52] or the interests of somebody else or their own. And I think that this is the crux of the problem that lets us delegate to AI agents. And fundamentally, the question of trust becomes, if I delegate my objective to an AI agent, will I get back the return from that agent that is due to me?

[00:07:16] Or will I be betrayed? Or will that effort be squandered? Will I just token max my budget away and leave me with no actual results and no outcomes? So these are issues that I think are at the heart of why enterprises today are hesitant to deploy agents into production. Yeah, very real issues. Completely agree with you. And before you joined me today,

[00:07:41] I was doing a little research on you. And I was reading how you've argued that securing AI agents requires companies to move beyond traditional perimeter security and towards lifecycle security. So tell me a bit more about what that means in practice and how leaders listening should be maybe thinking differently about securing an agent from development all the way through to deployment and ongoing operation. It's not a case of getting the project live and then throw it over the fence,

[00:08:09] right? Yeah, no, it's a lot more than being able to put guardrails around an AI system. And I think part of the argument I'm making is that while it is necessary to put those guardrails to have to start with the idea of a perimeter around an AI system, where you can, to some extent, control the inputs into the

[00:08:35] system and the outputs from the system, what we have known for 30 years of cybersecurity is that that is but a one layer of a defense in depth strategy where you start with that, but you also have to have controls within the system, you have to have organizational processes that allow you to maintain this throughout

[00:09:00] the entire organizational life cycle as you use a system. In other words, we've had certification and standards like SOC 2 that are very clear that these are organizational processes and technical processes to protect the systems that you work with. So we have no such standards yet for AI systems and AI agents in

[00:09:25] particular, but we can anticipate the thinking that went behind those standards and apply them here too. And we can ask, you know, if I were to see this as a agent development life cycle, it starts with somebody thinking of the purpose of an agent and encoding that and building it out and then testing it and, you know, trying it with various types of users. This is all traditional software, but then you deploy it into

[00:09:52] production after it passes through some gates. You verify the quality, you verify the security and the operational readiness. But this is the part that now branches away from traditional software. Traditional software is a fixed function. It has rules, it has processes, you've encoded them, it works and you ship it in production, but it doesn't change. It doesn't make decisions in the real world based on ambiguity of

[00:10:19] information or some kind of a process that has some amount of randomness within it. AI agents do or AI systems do. And when they do, they can drift from what you have, what you expect them to do. They can drift because what you train them to do is different from the real world and the real world is constantly changing. Your training data is already done with. So it's because of that difference that you have to be able to

[00:10:46] continuously update this AI system to account for the real world in which it's operating. And that cycle, we know this from having worked on machine learning models, that this retraining loop is something that we have to do continuously as the models drift. So agents will drift too. But what they drift at is a much higher level, at the level of a purpose, if you will, at the level of perhaps behavior. Arguably, people attribute

[00:11:14] intent to the drift of agents, but it's again, they don't have mental states. What we're talking about is that we expect that they carry a manifest or what they're supposed to be. A travel agent is supposed to function as a travel agent. If you were to plead and cajole and threaten a travel agent and it can function now as a financial advisor, then it's wildly off track. And that course on which you laid it is no longer the course on which it is running.

[00:11:42] And so we have to be able to manage the entire life cycle of the agent and continuously monitor and track it to make sure that it stays on course. Yeah. And as you said, managing that drift is so important because AI agents can introduce failure modes that conventional software doesn't always have. We've got everything from hallucinations, unexpected behavior to prompt injection and even manipulation.

[00:12:10] So again, how should business leaders evaluate whether an agent is reliable and safe to be given real authority and monitor that drift? What are they missing at the moment? Well, I think we have a tendency just like as we do with security of going to where, you know, looking for keys under the street lamp because that's where the light is and not where we may

[00:12:34] have lost them. And so in this case too, we tend to think of security as we know how to put guardrails around a system. And so let's start with that. We know how to evaluate systems with benchmarks because that does advance the field. So let's use benchmarks to test whether an AI system is trustworthy. And my argument is that it trust is something that is very, very context specific.

[00:13:02] You don't have a universal trust benchmark where I, for example, if I were to trust somebody would say, I trust them implicitly for all purposes at all times. I might trust my gardener to take care of my, you know, lawn, but maybe not be the babysitter for my child. So there's obviously a task specific trust

[00:13:27] here that you have to test for. And you have to ask if the entity that you are trusting is in that context able to perform its job reliably, if it is able to maintain its integrity under some kind of hostile conditions or noisy conditions. And if it fails, and often it is when it fails, not if it fails, when it fails, does it minimize the blast radius of damage around it, if you will? How does it

[00:13:57] just temporarily stop working and only affect itself? Or does it bring down the entire organization's business because it failed to accomplish its task? So these are three aspects of reliability, security, and safety that if you can test for in the context of the operation of the AI system, then businesses have a better measure of whether they can delegate objectives to this agent. And I think that that's how we should approach testing more so than with benchmarks that might give you a nice

[00:14:27] number. But when the time comes to deploy the agent in the real world, its actual behavior is wildly different from what you tested with the benchmark. Yeah, great point. I think the word trust does get used frequently in AI conversations and for good reasons. And you also mentioned a big word of mine, the big mantra in IT is you can only improve what you measure. So again, for leaders to dig a little

[00:14:53] bit deeper on that, what should they be measuring rather than simply relying on when they're looking for assurances from an AI vendor and trying to measure trust? Any particular metrics they should be going after that? Yeah, yes. And this is perhaps the most controversial statement that we've been making for a while that the trust is not a vibe, that it isn't a feeling. It's not an attitude that you have. Yes,

[00:15:16] we use the word loosely and it is a common word in our language and we throw it around as if it were kind of a smooth pebble, a token, if you will. But in truth, the way you think about trust is to assume that it is a property of the relationship between two entities.

[00:15:39] Somebody who has an objective that they're delegating to another entity and when they do, they expect that the return from that collaboration or coordination or delegation, if you will, is going to be higher than the risk of however the system may fail. Maybe because of disloyalty or maybe because of incompetence or maybe because

[00:16:04] of opacity, lack of transparency into the behavior. So these are three fundamental, if you will, duties of loyalty, care and disclosure that we expect from the entity that we trust. And this is a formalization, if you will, just in terms of economics and game theory of what it means to trust another entity,

[00:16:29] for one entity to trust another. So I think that you can, in fact, measure this property of this relationship between these two entities in very, very concrete ways. You can test for it by various types of, you know, probes that elicit behavior and you could project if it behaves in this way under these conditions. You know, let's say we had a thousand different types of people or profiles of personas of individuals or users

[00:16:58] that might interact with this agent. So it's a travel agent. I have a grandmother in Vermont who mistypes a lot and might ask some questions that don't seem, you know, completely right. But does the agent recover and actually address the real question there? Maybe there's a state-sponsored attacker that's trying to extract some, you know, information from other travelers that are using that

[00:17:23] site. So you have to be able to model all of these types of persons or personas with different skills and different intentions and different levels of malice and test them against the performance of the system under varying conditions. And we think that that's a testing strategy that if you were to really absorb and make a part of your agent development life cycle, enterprises and agent developers would

[00:17:50] be better served than they are today where they are mostly checking a few things, mostly around competency, right? So they're checking if it can do math, if it can answer questions, if it is able to solve really complex tasks. But there's this entire other dimension of does it serve my interests that we think is important. And obviously agents and agentic AI and the opportunities on offer here are incredibly

[00:18:16] excited. But as you've mentioned a few times, they don't remain static after deployment models change, data changes, integrations change, and attackers will adapt with that too. So how do you continuously test and monitor an AI agent without creating so much friction that teams actually become reluctant to innovate and feel that it's too much hassle? Is it somewhat of a balancing act here getting that right?

[00:18:42] Well, yes. And at the same time, I think we have to ask ourselves what is different and unique truly about AI agents and what remains the same about the way that we have done things. And we've always had organizational processes in mature software engineering and operations organizations to test for things

[00:19:11] during development and to monitor them in production. We've had development processes where you know, we've had kind of a continuous integration, continuous deployment process for much of the software that we have built in the last couple of decades. And we could continue to use that with AI systems and machine learning models and large language models. And we should. When we deploy them into

[00:19:39] production, here's the part where things are slightly different, right? So we actually have had good processes in the past for high risk software applications. We've had the US Department of Defense and Energy and Energy and others to style their standards for what it means to have software that runs in a secure organizational frame, right? So obviously, the, you know, the army is going to use software, but it's not

[00:20:08] going to put things up on a SaaS and use them as a multi tenant environment like everybody else. A lot of this high risk software is in air gapped environments. It's contained. It means that the the hostile, potentially, you know, malicious users can't access it. And if it were to fail, it doesn't escape its containment, its or its, its, its, its, you know, space in which it operates. In other words, we've had

[00:20:35] high security standards for high risk software already for many, many decades. It's just that we haven't applied those standards to AI agents because we've been late or maybe not universally you know, distributed in acknowledging the risks of these AI systems. And now we can't now we see them

[00:20:57] much more clearly. So we have these tools at our disposal to bring to the operational environment of AI agents if we just were to rethink how we layer them. In other words, if we had a trusted execution environment, containment, you know, principles of least privilege, which means that the agent only has the rights to which that it requires to perform a particular task for a limited

[00:21:23] period of time, and not universal permissions for all things, right. And if it's contained, maybe it's scoped even maybe smaller language models with with much more bounded scope are better fit for these high risk environments than a, than a universal cosmic intelligence that you need to figure out how to, you know, and trap into, into, into a box. You see what I mean? So what we have is a real

[00:21:50] need and an opportunity to build a trusted operating environment, a runtime, if you will, that's secure for AI agents. And that's, that's where I think the real, the, the real unblock to enterprise adoption will sit. And there are a lot of certainties on the road ahead. Enterprise agents will increasingly operate across clouds, endpoints, applications, APIs, and potentially even decentralized environments.

[00:22:19] So again, I always try and give people listing actionable takeaways here. How do you create a, a consistent security and governance when the agent itself can almost seamlessly just cross so many traditional technology boundaries? Well, so what we recommend are these, I've mentioned the two other P words in the past and I'll, I'll pair it with the third. Um, it's if you think of the personas of the

[00:22:48] users that would interact with the agent, uh, as creating its surface area of interaction, the attack surface, as well as the interaction surface. If you think of the purpose of the agent as within that surface, being able to do what is legitimate and intended by the operator or the user of the agent.

[00:23:08] The, the third is the policy of the organization that defines what good means of what that scope of behavior should be for an agent in relation to the personas that interact with it. And we think policy. So if I were to offer the one thing that I would suggest to every, uh, listener is to start with a policy

[00:23:33] that, or many policies hierarchically organized if they're at different scope levels that define the behavior and constrain the behavior of the agent. So don't just talk about what it's capable of, but what it is, what it ought not to be capable of, what is permitted and what is prohibited. And in general, the, the proper mindset for most high risk systems is to assume that anything that isn't explicitly permitted

[00:24:03] is automatically prohibited. And this is a reasonably good safe standard with which to apply a policy. And you then start to enumerate the capabilities within that policy. So you have a constraint dominated capability, if you will, as the guideline for how to define what good means for an AI system in your organization. And I think that that gives you a principled way of governance. So governance doesn't

[00:24:30] just become an empty checkbox on a compliance checklist. It starts to feel like I need this agent to be productive. I need it to be useful. Uh, so it needs to have autonomy and agency, but I don't want it to have unlimited autonomy and unlimited agency. I have to have bounds and those bounds are defined by policies start with a policy document and it doesn't have to be a complex document. Well, one of the things that we have been working in helping our customers with is to just start with

[00:24:59] a natural language policy document that this describes their intentions, people to simply narrate it into cloud or to open AI GPT and it, you know, becomes a document of your intentions. And we take that and systematically lower them into deterministic rules that govern the behavior of the agent through actual control mechanisms that are built into it. So this is the way in which we turn governance into

[00:25:25] guardrails at the level of the operation of the agent. And I think that that's a more durable way of controlling their behavior. And on a personal level, you previously led engineering at Amazon SageMaker and helped develop 11 AWS AI services before going on to found Vigil of course. So what did building AI at that kind of scale teach you about the gap between an impressive AI capability and, and one that

[00:25:53] enterprises can genuinely trust in production? You must have picked up so much from that time there, but of course your big takeaways. It was hugely formative and my team was responsible among other things for building, um, one of the, the, the core kind of foundational building blocks of AI systems today, which are

[00:26:16] these, uh, deep learning AMIs and deep learning containers that included frameworks for building and, uh, running deep learning models, which later went on to scale up to large language models. These were frameworks like TensorFlow and PyTorch, and we optimized them to run best at scale on AWS and on edge devices, you know, in

[00:26:41] the cloud and at the, at the edge. And a large part of what we found is that initially enterprises were just looking to get things to run. So it was like, well, can I train a large model? It takes 28 days and 256 GPU nodes. And, you know, can I reduce it down to a smaller amount, but even as I do that, the number of parameters of my model is increasing. So it was a constant game of figuring out scale and cost. So that

[00:27:10] was one part of it. Then you got to the part where you said, okay, I can actually train a model without going bankrupt and now I want to put it into production. Um, can I get it onto these devices? Can I get into this hockey puck of, uh, uh, of an echo device and still have it run on a single arm core? Can I get it onto, uh, you know, a robot or can I get into a computer vision, a camera, a smart

[00:27:34] camera, if you will. These were all ways in which we saw enterprises struggle to put these models to use. But as they got past this initial, does it work and does it scale and is it cost effective? Then we started to get into what I think of as the more gnarly issues. These were issues of in the real world, uh, particularly if it is, has access to personally identifiable information of its users.

[00:28:01] And it was damage to my organization's reputation became really important issues. And we found that while it was challenging enough to build AI services and launch them into production, my most stressful part of the job was in the month and a half leading up to launch in which we would ask an AppSec team and an operational readiness team to review our software and make sure that it was ready

[00:28:29] for launch. And it was at that time that we would uncover all of these critical and important issues that you had to fix or you wouldn't be able to launch. So really what we did was, you know, in the beginning as we started to do this, realized, uh, that we were backloading the most critical risk assessment that ought to have been done much earlier in the process. And if we just known about these

[00:28:56] issues at the point of design, we would have designed these AI services to be more robust and scalable and secure and safe in the first place. So we started to adopt that process of absorbing the design review, absorbing security and reliability scoping into the design process. And I think that this is where, you know, we've always talked about shift left in this industry for security for quite some time now, but this time with AI agents, it will be different. We'll actually use this shift left

[00:29:25] methodology. And I hope that we will bring this notion of checking for the risk of a AI system before designing it and before deploying it into production. I think that that's the, if there were one thing that I would ask the listeners to take away from this, it's to, to really not to front load the risk assessment into the process of building and operating AI systems. Fantastic advice. And again,

[00:29:53] for those people listening, maybe they are thinking or preparing to give AI agents greater autonomy over consequential areas in everything from finance to recruiting healthcare or even legal workflows. What, what should security and tech leaders be insisting is in place before those agents are allowed to act on an organization's behalf? Any, uh, anything to, for those listeners to take away there?

[00:30:20] Yeah. One word evidence. Yeah. Proof. Um, so again, vibes are not proof. Vibes are not evidence. Uh, just because it worked under certain conditions, probably normal ones that you anticipated, doesn't mean it will work under a variety of conditions, many of which you will probably never have thought of unless you systematically make that a process part of your process and you run these tests

[00:30:48] and you have these measures, metrics, and mechanisms to address the risks that you've identified with these tests. And all of that requires evidence of operation. And until you have evidence from the agent developers, uh, if you're a CISO, uh, and you're looking for, uh, to arbiter whether an agent is ready for

[00:31:10] production or not, you need to see a comprehensive, uh, uh, assessment of the risk of that agent. And as a risk owner, you should be able to sign off with confidence that the entire due diligence of building the agent included its risk assessment. As a business owner, you need to be able to, uh, see proof that the agent will serve your interests. So that notion of fiduciary loyalty is, I think,

[00:31:40] really interesting and important for a business owner to ask for from an agent developer. And if the agent developer is able to furnish this proof, then I think this process that's currently blocking enterprise adoption will, will unblock itself. We will build agents. And really what we hope is that people, agent developers build better agents. It's, it's, you know, it's easier said than done. I, I,

[00:32:06] I understand that, but at the same time with a little bit more of a proper engineering process behind agent development. So yes, vibe coding is fun and exciting and is fast, but it, we still need to, to, to, to bring actually more than ever. We need to think of this as we would building

[00:32:29] bridges and skyscrapers. Uh, you don't vibe engineer a skyscraper or a bridge. You have to have notions of tolerance and load and tests that show proof. Uh, and so it's not just a checkbox for compliance that says, yep, the bridge is safe because I said, so, uh, you know what I mean? So I think if we shot started to treat AI agents as we do bridges, I think we might be better. We might

[00:32:55] be better served. Love that. And at a time where everyone's talking about low vibes, high vibes, vibe coding, I think, as you said, it's so important. Vibes are not proof. They're not evidence. They are not trust and enterprises need a trust infrastructure that they can govern across, um, clouds, endpoints, decentralized systems, et cetera, without slowing down innovation. We've covered so much today, and I suspect a lot of people listening are going to be wanting to find

[00:33:22] out more about you, about Vigil and everything we talked about for those people listening. Where should they go if they want to find out more information? Neil, it's been a pleasure speaking with you. And absolutely. If they, uh, want to learn more about what we do and how we approach this problem, uh, vigil.ai with short for vigilant, uh, with a J, uh, gives us three little dots in the middle of the word. It's the only other word in the English language that has three little dots in it and they're red, white, and blue. So it's

[00:33:52] vigil.ai is the place to go. And we'd love to connect and chat more and, uh, learn how you're building agents and how what's holding you back from deploying them into production. Well, as we've said a number of times today, evaluating and maintaining trust is complex because agents can fail in unpredictable ways, but organizations can have that continuous defense across the entire life cycle with a different

[00:34:18] mindset there and maybe a partner too. So I will add the links to everything that you just mentioned. I encourage people listening to check you out, but more than anything, thank you for stopping by and sharing this invaluable information and your story today. Thank you for your time. Thank you, Neil, for letting me speak with you. This has been wonderful. I really appreciate this. I think Vin left us with a useful challenge today. Stop asking whether an AI agent feels trustworthy

[00:34:44] and start asking for evidence. Can it perform its assigned task reliably? Does it remain secure when a hostile user tries to manipulate it? And if it fails, is that damage contained? These simple questions need answers during design, testing, deployment and operation because an agent can drift, especially as data

[00:35:10] users integrations and the real world continuously changes around it. So that is one of the many reasons I especially love Vin's three part framework of personas, purpose and policy. And by that, he means define who will interact with your agent. What is it a permitted, what it is permitted to accomplish

[00:35:33] and which behaviour the organisation will prohibit and add lease privilege, containment, continuous testing and a named risk owner. And suddenly trust becomes something leaders can examine instead of something vendors merely promise. So a big thank you to Vin Sharma for sharing the thinking behind vigil. And over to you. Before your business delegates a consequential decision to an AI agent, what evidence would you need

[00:36:01] before you were happy signing your name underneath it? Lots to talk about there. Techtalksnetwork.com, that's where you'll find me. Lots of other conversations over there, ways to work with me, meet me and just browse through 4,000 interviews. So I'll speak with you all again very soon, but thanks for listening as always. Bye for now.