What happens when an AI security agent receives access to eight enterprise systems but cannot understand the relationships between the data inside them?
In this episode of The Business of Cybersecurity, I speak with Guy Flechter, CEO and co-founder of Sola Security. Guy has worked in cybersecurity for 25 years, progressing from operational security roles to the CISO position before moving into entrepreneurship. He previously founded Cider Security, which was acquired by Palo Alto Networks for $300 million.
Our conversation focuses on why adding AI agents to separate security tools may increase speed without improving the quality of the decisions. Cloud, identity, SaaS, code, devices, and networks frequently operate through different consoles and data models. An agent working within one of those systems may answer confidently while missing a relationship that changes the meaning of the risk.
Sola Security’s research examined 50 tasks across eight enterprise platforms. According to Guy, providing structural and relational context improved answer correctness by approximately 34% across the tested models. Under full context, 78% of responses were considered fully correct, around 18% were incomplete, and fewer than 4% were classified as complete failures.
Those results show both the promise and present limitations of AI security agents. Connected context can improve performance significantly, but 78% accuracy does not support fully autonomous action in situations where an incorrect permission change or security response could have serious consequences. Guy believes human involvement will remain necessary until accuracy reaches a far higher level.
We also discuss how companies should vet and onboard AI agents. Guy recommends treating an agent like a new employee by defining its permissions, monitoring its actions, controlling what it can retain, and limiting its authority until trust has been earned.
The episode concludes with a discussion about independent testing. Guy argues that buyers need transparent benchmarks with visible tasks and repeatable methods, rather than vendor accuracy claims based on private evaluations.
Should an AI security agent be allowed to act autonomously if its reasoning cannot be independently tested? Listen to the conversation and share your thoughts with me.
Useful Links
[00:00:00] The leading issue of agentic AI in businesses right now is ensuring agents act with compliance guidelines. And Denodo applies guardrails across your entire data estate. By aligning your company's data infrastructure under one system, these guardrails perform consistently across your platform. So start scaling your business and start with Denodo. Simply visit denodo.com to learn more.
[00:00:32] Every cyber security vendor seems to be AI pal in these days. But what if adding a smarter agent to fragmented security tools simply helps each silo become faster at getting the wrong answer? Welcome back to the Business of Cybersecurity podcast, where I'm joined by the CEO and co-founder of Solar Security.
[00:00:56] And he was previously the founder of Cider Security, which was acquired by Paolo Alto Networks for a cool $300 million. But he's back in the game. And today he's going to argue that the biggest constraint facing AI security agents isn't necessarily the model. It's the context.
[00:01:18] And their research tested AI agents across eight enterprise platforms and found that providing relational context improved answer accuracy by around 34%. In other words, understanding how identities, permissions, assets and systems connect can dramatically change what AI agents see. No big surprise there, I guess.
[00:01:43] But today I want to ask what AI native security actually means beyond the buzzwords. How much autonomy should we be giving agents? And how security buyers can separate those impressive marketing statements from the technology they can actually trust. But enough from me. Let me introduce you to my guest right now. Thank you for joining me on the podcast today. Can you tell everyone listening a little about who you are and what you do?
[00:02:13] Hi, pleasure to be here. So my name is Guy Fletcher. I'm the CEO and co-founder of Solar Security. Married plus three amazing daughters. I'm in the cybersecurity trenches for the last 25 years now. I grew up in the security teams in different positions all the way to the CISO position. My last CISO position was at a company called AppsFlyer.
[00:02:39] Then around 2020, during a lot of time thinking at home while COVID was outside, I decided to jump into the entrepreneurship side. Build a company called CIDO Security in the CICD and ASPM domains. Then after two years, Palo Alto acquired us. I worked for Palo Alto for some time.
[00:03:05] And then after walking, Palo Alto decided to jump back again into that journey and build Solar. Wow. Wow. It feels like the scene from the Godfather 3 where he says, just when I think I'm out, they pull me back. I pull back inside. Yeah, yeah. Definitely. I always laugh that it wasn't with my full blessing, with the full blessing of my wife.
[00:03:30] She expected me to at least rest a little bit longer after that journey. But I always say to people, if you have the builder mentality, building something new is something that, you know, it's a feeling that you cannot compare to anything.
[00:03:55] Again, from a professional perspective, of course, not family and everything like that. Yeah, completely agree. And so much has changed over the last few years. And this year in particular, everyone's talking about agentic AI. We've got individuals, teams, entire organizations launching hundreds, if not thousands of AI agents. And security teams, though, they're already working across separate tools for cloud, identity, SaaS, and code.
[00:04:25] So my question to you is, with all this in mind where we are now, what happens when AI agents are added to those individual consoles without addressing that fragmented data underneath them? It's something I don't see talked about at these tech conferences and shiny demos and keynotes.
[00:04:43] Yeah, I think that definitely starting from the first point, what you mentioned in the last two years, it was, you know, it was crazy to see the evolution of the industry, you know, starting two years ago. So not only the AI industry at all, starting in, you know, in the chat with LLM, OpenAI and so on, and then grow a little bit for building stuff.
[00:05:13] And then at some point in the last few months, the key, everybody's talking about honest and everything. But in the last few months, everybody's speaking about the fact that, like you mentioned, the agentic era, everybody using agent, connecting to stuff. But the missing piece in the last few months, the missing piece that everyone is speaking about is the context layer.
[00:05:40] And exactly what you're saying, just adding an agent to do work without giving a context about your company, about understanding, connecting for the agent, the dots. That for us humans that are working for a company, it's like very trivial to, it's part of our day-to-day, part of our onboarding.
[00:06:05] Think about a new employee that comes to a company on the first day, the company not letting him do whatever he can do or do a task in the company, you know, without getting the proper understanding of the company. So this is the same thing.
[00:06:22] You cannot send an agent to do a task without giving him the proper understanding of the environment, on the relationship between assets, between even understanding the policy of a company and past approval and so on. So this is what we see is the missing piece, especially in security. And we see it, of course, in many other areas.
[00:06:51] And before you join me on the podcast today, I was reading your benchmark that found that structural relational context improved answer correctness by approximately 34% across the tested model. So what information did that rationale map provide that the models could not reliably discover for themselves? Tell me more about what you found there. So the LLMs are going back.
[00:07:20] Satya, the CEO of Microsoft said, LLMs are commodity. Everyone can reach LLM. Everybody can start building with LLM. The issue is that LLMs are good in a certain way to answer stuff. Okay.
[00:07:37] And if you want them to be good in answering the question to your company or it relates to your company, you need to provide them, you can say, certain type of information. Go back to the context layer of your environment and to improve the accuracy of their answers.
[00:08:00] Because they don't know how to do that connection and they don't know how to go into a different connection between assets by themselves. The need of doing some precompute relations between assets and between permission services and so on.
[00:08:22] This is something that Sola is doing and showed in the research that once you're doing that and this is something that you can do on the fly with LLM because it will cost you a lot from time and of course from budget and token perspective in order for the LLM to get in that accuracy.
[00:08:56] And your research also reported a best result of 78% answer correctness under full context, which is a great figure. But if we flip that on its head, the other side of the tail here is in cybersecurity, that remaining 22% could still carry very serious consequences. So I'm curious, which tasks continue to defeat the agents and what level of human review still remains necessary? Anything you can share around that? Yeah.
[00:09:25] So first of all, for the accuracy, it's 78% of full and accurate answer from our perspective. Another around 18%, 18%. We said that the agent didn't give a full and accurate answer. Okay.
[00:09:49] Only less than 4% of the answers were, again, following the benchmark and adding all the layers that we provided, only less than 4% were completely, now perspective, now research, failed. Okay. Okay. So 18%, this is what we're saying. It's still something that need to have a man on the loop.
[00:10:12] This is also part of what we're saying that in order for you to get the fully autonomous, everyone else speaking about fully autonomous defense systems and so on. And what we're saying is that you can't get there until the accuracy will get closer to 97%, 98% of the answer, full accuracy. Only then fully autonomous defense system can be achieved.
[00:10:41] This is why, by the way, we released today what we call open security. We took the benchmark that you just mentioned and exposed it to the world, which means that anyone that is building an agent, whether you are part of security team, whether you are a vendor and so on, you can test your agent, see the accuracy, and then start to improve it over and over again for your needs.
[00:11:11] It's something that we did for the community as we realized that, like you mentioned at the beginning, everyone are building now agents. Everyone are doing a lot of stuff around agentic things. So we said, okay, we need to help everyone in the community to get better, to improve their agent. And we started with one domain, ISPM, connected to the research that you saw.
[00:11:40] And we're getting to release more types of benchmarks based on the research that we are publishing in the near future in many other domains. And the benchmark contained, I think, 50 tasks across eight enterprise platforms. So I'm curious, how did you test whether an agent reached the correct answer through sound evidence and entity matching rather than just arriving there through plausible guesses or finding the pattern?
[00:12:09] So we look, in some cases, we look on the final answer. What is the final answer? And we compare it. And it's a research person that ran the test and saw a comparison. It's a combination of it's AI. We have two teams that are working together. It's the AI researchers that, you know, working with the models and testing and building the benchmark.
[00:12:34] And we also have the security innovation team that are working together. And they are one bring the brain for the AI and how to build models and everything around it and how to test them. And the other bring the security perspective.
[00:12:50] So the security researcher looked on the answers and saw whether how much accurate are there and compare it to how they would respond to this, you can say, same questions if they will have it manually to do it. And a buzzword I keep hearing at tech conferences this year is vendors frequently describing their products as AI native.
[00:13:17] And from an architectural perspective here, what separates a platform designed around AI reasoning from a legacy product with a chatbot or assistant added to its interface and being truly AI native? So the core principle is that it's domain slash vendor agnostic.
[00:13:38] That's the key part, how you build your infrastructure to be agnostic to the domain that you are trying to solve or you can solve. This is how we started solar from day one.
[00:13:53] We built solar from the ground up as an AI company, not a company that doing a specific domain and then add the layer, like you mentioned, on top of that of certain like chat GPT on top of the data that I have.
[00:14:11] So the fact that, by the way, one of the research you can, I can share with you after around how we build that brain or that asset inventory, AI asset inventory. It's also very connected to the previous question around pre-computed items that need to be done before LLM can touch the data.
[00:14:34] So from day one, we started solar in a way that give user the ability to ask any question on any domain without us being tied to specific domain. So in order for you to do that, for following your question, you need to build the entire infrastructure in a way that it's not like identity, cloud security, device management, network and so on. Everything is connected.
[00:15:04] Everything has relationship between that. By the way, because of that, this is what we're saying that LLM has next week. I have a talk in AI for it's a big conference happening in Vegas in parallel to Black Hat around the fact that LLM are now again from our research and everything are not fit to solve security challenges.
[00:15:32] And they have a very, very high, but still glass ceiling that they cannot break at some point. And you will always need to do certain work for them in order for them to be able to answer the questions. And you mentioned security a moment ago and connecting security data can 100% improve reasoning.
[00:15:57] But there will be some techies listening say yes, but it can also increase access to sensitive information and create a more valuable target for attackers. So how should companies and people listening control what an agent can see, what it can infer and what it can retain and what it can act upon? Tell me about some of the guardrails there. So first of all, you need to think about it like a human being joining the team.
[00:16:25] Like any human being that you need to understand that you need to put the guardrails and give them the right permission, but also make sure that you are doing the right vetting on onboarding that person. You will not let any stranger, let's say your company, you want to not pick someone from a random person from the street.
[00:16:51] Go, you can join the company and start work and start accessing. You need to do some vetting process. And then after the vetting, the onboarding, you will have all the guardrails from permission perspective, monitoring and so on. So all this process like you're doing for any human being should be done as well to the agents with the understanding that,
[00:17:16] yes, they eventually, if you want them to do the job, they need to access data. They need to be able to see data. And most likely the majority of the data is sensitive to because any company data, the majority is sensitive data.
[00:17:36] But you need to think of it, how you are letting it do the job, then preventing it from doing the job that you want. Of course, like any technology, like any risk, you need to manage it and you need to accept it. But this is also part of what you can say the last mile order for an agent to be fully autonomous.
[00:18:04] Because if you still need until you will feel very comfortable in getting full access, then agent will have limitation about how much autonomous they can be. And as you said at the very beginning of our conversation, this is not your first rodeo. You previously built Ciner Security before its 300 million acquisition by Palo Alto Networks.
[00:18:30] But I'm curious, now you're here again, when you look back, what did you or what did that experience teach you about the strengths and limitations of specialized security products, especially now in present day where customers are attempting to connect insights across so many different domains? Any big learnings from that time?
[00:18:50] I can say that, first of all, the experience taught me that the teams are evolving and need the ability to do or to create what they want. And it's accelerated. Think about solar on day one, when we started solar, the promise was the equivalent of think about lovable for cybersecurity.
[00:19:19] Because we realized that people need to be able to build what they want, need to be able to create what they want. If they want to solve specific problems, they don't need the full-blown big solution for it. And over time, that's the entire world now. You can see it. Everyone are building for them what they need in real time.
[00:19:43] And so this is something that I believe going to accelerate more and more over time. I think that very soon we will see it will start, of course, encoding and then we spread to other areas that from creating something, from asking questions, getting answers to solve or like encoding deploy,
[00:20:11] we will see that becoming the distance becoming smaller and smaller and security teams will start, you know, in a click of a button. Yes, I need to solve X. Solve it now. Go an agent and solve it. They don't need, they don't. The concept of solution is going to disappear.
[00:20:38] We're speaking about it that solution and tools as we understand them are obsolete. Very soon, it's not about whether I have a dashboard, whether I have a UI or agentic experience. It's I need to solve something. Solve it. I don't care how you're solving it.
[00:21:03] I don't care whether it will be with speaking with someone, changing permission to something whatsoever. Just go and solve it. And it's going to be fleet of agents. For the first time, the beginning will be, okay, this is what I need to solve. So and they will go. And right after will be an agent, a fleet of agents will speak with me and say, okay, we already solved that issue for you.
[00:21:31] We recognize this issue and we solved it. And I believe this is going to be one of the biggest changes in our industry and our ability to secure our companies.
[00:21:46] And finally, for everybody listening, when they are assessing an AI security platform now, what demonstrations, benchmark results, audit evidence and real world tests, what should they demand to determine whether it can reason accurately across the existing tech stack? Because we all know when a company comes in and pitches their product, et cetera, and they've got so many to assess.
[00:22:10] It can be so difficult to spot the difference between them all with so many buzzwords and big promises, et cetera. So what should they be doing? Maybe I'm self-promoting, but the best thing for them is the sun, showing it into the sun. And we're calling all vendors to join the open security and share their agent capabilities.
[00:22:36] It's open, it's free, and it's allowing everyone to see the result by themselves. And by the way, it's based on a very clear understanding what was the task. It's not like hidden around or behind walls about what happened there, how they tested it. No, everybody knows. Everybody knows what tested. This is part of what we believe. Again, very connected to that question.
[00:23:05] If we want to create an industry that value transparency and value that we know exactly what's working and what's not working, it should be a very unbiased process for companies. Because if you will not have something like that, unbiased, everyone will say, I have the best agent ever.
[00:23:34] Here, look, he answered 99.999% accuracy on certain benchmarks that you don't know even how it was tested. So I call everyone to join and to support that.
[00:23:54] That's the only, again, the immediate thing that I can think about that can help promote transparency to the buyers and how they decide whether they want to leverage certain agent or certain vendor or not. And for anybody listening that would like to continue this conversation or find out more about anything we talked about today, and where do you want me to point them listening?
[00:24:23] Tell me more about the website and where they can find everything about you, and keep up to speed with developments too. So they can find us in two main websites. They can find us in solar.security. That's the main website. Or they can look for the open.security for the initiative about the benchmarking with all the research and everything that we publish there for agents. Awesome.
[00:24:49] Well, I've loved chatting with you today about why AI-powered has become a marketing label rather than an architecture decision and what that gap costs buyers, and also why cross-domain context, not model size, is a real constraint on AI security agents right now. I'll have links to everything that you mentioned, and I encourage people to check that out and find out more information. I'd love to stay in touch with you and see how this evolves and see how you're getting on next year.
[00:25:18] But more than anything, just thank you for sitting down with me today. Thank you. Thank you again for inviting me, and thank you for the great discussion. I think there's a useful challenge for every security leader coming out of today's conversation. Stop asking whether a cybersecurity product has AI and start asking whether its AI understands your environment. And the research that Guy referenced today suggests that model intelligence alone is not enough.
[00:25:46] Agents need context across identity, cloud, SaaS, code, permissions, and all the relationships that are connecting them. And even with that context, Solar's benchmark reached 78% of fully correct answers, which tells us why human oversight still critically remains part of that equation.
[00:26:11] And also, I liked Guy's prediction that security tools themselves could eventually disappear into fleets of agents. Instead of opening another dashboard, the experience could hopefully become much simpler. Here's the problem. Go solve it. But of course, it's not that simple.
[00:26:31] Getting there requires accuracy, permissions, guardrails, transparency, and independent ways to test whether agents can actually do exactly what vendors are saying in those keynotes and shiny demos that we're all addicted to. So thank you to Guy for joining me today here on the Business of Cybersecurity podcast. Remember, techtalksnetwork.com if you want to find out more about the work that we're doing here.
[00:27:00] And also, I'll include links to solar security and everything we talked about today. But that's it for now. Thanks for listening. Speak to you soon.