How can marketers make better decisions for individual customers without spending their working lives designing, running, and maintaining separate tests?
Recorded at Braze Forge 2026 at the Fontainebleau in Las Vegas, this episode of Tech Talks Daily features my conversation with George Khachatryan, Head of AI Decisioning at Braze. George describes leading product management for AI Decisioning Studio and shares the story behind OfferFit, the company he cofounded before it became part of Braze.
We begin with the practical limits of segmentation and A/B testing. George explains that smaller customer segments can make it harder to collect enough evidence for a useful result, while each new creative option introduces further work. His explanation of reinforcement learning offers a different approach. The marketer defines the objective and the options available to the system, which then experiments, observes the results, and adjusts its decisions.
We discuss the distinction between Decisioning Studio Pro and the newly announced Decisioning Studio Go. George describes Pro as offering flexibility around success metrics and custom data, with data science support required during implementation. Go is designed as a self-service option using data generated within Braze. At the time of recording, he says Go supports email and optimizes click activity with machine clicks filtered out. Marketers choose the journey, creative options, subject lines, calls to action, available timings, frequencies, and guardrails.
The limits are as useful as the possibilities. George recommends a baseline of at least a few thousand clicks per month for a journey using Go, so the model has enough information to learn. He also acknowledges that maximizing clicks will not always maximize conversions. We discuss why those objectives need to be assessed separately, rather than treating improved interaction metrics as proof of additional sales.
George explains how the system can learn from similarities between creative variants and describes daily model retraining as a way to adapt to changing behavior. We also talk about reporting against a business-as-usual control group. He distinguishes the performance reporting available at the time of recording from deeper explanations of why a model made a particular choice, which he describes as an area of ongoing development.
One of the most memorable parts of the conversation concerns customer trust. George recounts arriving with his family for an apartment viewing arranged by an AI assistant, only to discover that no appointment had been booked. His point is that speed and responsiveness lose their value when a company refuses responsibility for the actions of its AI. Transparency and ownership of the customer experience still require human judgment.
We finish with George's advice on readiness, including experience with manual testing, measurement, and customer data. His broader comments about data infrastructure should be considered separately from his description of Go's use of native Braze data.
Which marketing decision would you automate first, and how would you check that it was improving the outcome you actually care about? I'd love you to share your thoughts.
Useful Links
Connect with George Khachatryan
Learn more about the Forge 26 Event
[00:00:00] Scale your business with agentic AI, with limited risk. With Denodo's AI data layer, your agents are provided with real-time company data and guardrails for company protection. Create the business you always dreamed of with Denodo, and you can do that by simply visiting denodo.com to learn more. But now, back to my guest.
[00:00:23] How do you personalise marketing when every customer behaves differently, and your team can only run on so many tests? Well, I'm at Forge26 in Las Vegas, and my guest today is the head of AI decisioning at Braze. And he's someone that began his career as a mathematician, co-founded OfferFit before the company went on to join Braze.
[00:00:50] But here, today, he's going to explain how reinforcement learning can help marketers choose messaging and timing all through ongoing experimentation. And we'll also discuss what people still control, how much data a system needs, and why an increase in clicks doesn't necessarily mean an increase in sales.
[00:01:13] And there's also a memorable story here about an apartment viewing that certainly gave me the question of AI accountability and a very human dimension. Well worth listening just for that story alone. And if you've ever waited for a marketing test to produce a useful answer, again, this conversation will give you plenty to consider today. But enough from me. Let me beam your ears directly to the show floor at Forge26.
[00:01:41] Well, we're here this week at Forge26 in Las Vegas at the Fontainebleau. Big thank you to you, George, for joining me today. Can you tell everyone listening and watching a little about who you are and what you do? Yeah, thanks, Neil. It's my pleasure to be here. Thanks for having me on your show. So I'm George Chacharian. I lead product management for the AI Decisioning Studio product in the Braze AI Suite. Awesome.
[00:02:04] And before we get into Decisioning Studio, can you tell the listeners a little about your background, the OfferFit story, and how that technology ultimately became a part of Braze? Yeah, for sure, Neil. So I started my career as a mathematician. Yeah. And I met Victor Kastouk when we were both doing our math PhDs together at Cornell. And then many years later, Victor Kastouk, we were friends, we had worked together, and he had an idea to use a type of machine learning called reinforcement learning to build a decisioning engine for marketing.
[00:02:34] And so the idea is that this is a type of AI, we can talk about it later if you'd like, that autonomously experiments and self-improves from trial and error. And so Victor had this idea, and we started a company together called OfferFit and ended up building this decisioning engine. And that ended up kind of growing very quickly and establishing the category that's now known as AI decisioning. Yeah. And then last year, we were acquired by Braze. And what kind of time frame was that? What year was this?
[00:03:01] We started in 2020, so it was just under five years before we joined. So that was kind of like the COVID time, nobody talking about AI. Before it was cool, yeah? That's right. We were AI hipsters. You know, it's true of pretty much all my friends because, you know, for Victor and me being mathematicians, you know, what did all the people that we worked with end up doing, right? They either became professional research mathematicians or one way or the other, they found their way into AI. Love it. And marketers, bring the marketing industry into the conversation.
[00:03:30] They've spent years relying on segmentation and A-B testing. What is fundamentally different about AI decisioning? And why do you believe the old way of testing starts to maybe break down when starting to scale? Yeah. There's a few problems with the old way. You know, in the old paradigm, if you wanted to personalize how you market it to folks, you divide your population up into segments. And then you'd have to run A-B tests for each segment. A-B testing is funny. Anyone who hasn't done an A-B test thinks that it's extremely easy to do A-B tests.
[00:03:59] And anyone who's done A-B tests knows that it's actually very painful and takes forever. You've got to design the test. You run the test. It takes you longer than you think to reach stat sig. And then, you know, maybe you don't. And then you have to go back to the drawing board. So it's very laborious and very messy. And there's this paradox that the smaller you make your segments in order to try to personalize better, the harder it is to reach stat sig on any of your tests. And so it's extremely limiting. And then maintaining this afterwards is also very painful.
[00:04:26] Because let's say if you have a new creative that you want to test, we have to redo the whole thing. And so the premise of AI decisioning is to automate that whole process. So no segments, no manual tests. You just set up an AI decisioning agent. You give it a KPI that it's maximizing. So it could be revenue or conversions or engagements, whatever. You give it the set of things it's allowed to do. And then you leave it on its own. And it makes decisions. And it sees the results.
[00:04:53] So, for example, say, you know, for Neo, I'm going to send this message with a certain offer at 2 p.m. on a Tuesday. For George, it's not going to be an email. It's going to be a push notification. And it's going to be at 9.13 a.m. on a Monday. And then it sees what happens. Maybe you make a purchase and I don't. And from this, it learns that, you know, maybe what it did for you was a good idea. What it did with me was like a bad idea for people like me. And it just self-improves. So as a result, you get a few benefits. First of all, it's like dramatically easier because you're not maintaining this very cumbersome system.
[00:05:23] But then also you get higher quality decisions because you're truly making them at an individual level instead of making them for these broad segments. Well, I shouldn't be saying this in public because marketers will come after me. But 10 p.m. on a Friday night, that's probably the best time for a message from me when I've possibly drank a little too much and I wake up the following morning to these emails of random things that I've purchased. But seriously, I mean, Decisioning Studio Pro already existed for high-value use cases.
[00:05:49] But it required data science and engineering resources before. So what problem were customers bringing to you that would ultimately lead to Decisioning Studio Go? Yeah, so potentially in almost any journey that you're running, you want to do AI decisioning. Because why wouldn't you want to personalize decisions and improve through experimentation? But as you mentioned, the challenge with Decisioning Studio Pro is that because you have so much flexibility in how you set it up, you can pick any success metric to maximize. You can bring in all sorts of custom data.
[00:06:18] This creates data science complexity. And as a result, the company that's implementing this needs to have their own resources committed for the install period. And then on our side, we provide forward-deployed data scientists to help them with that. That's all fine and dandy for your highest-value use cases. But what about use case number 15 or number 57 in your stack ranking? And so that's the premise of one of the major announcements we made yesterday, which is Braze AI Decisioning Studio Go.
[00:06:45] And so this Go tier, the premise is that it's completely self-serve. And what allows it to be self-serve is that it only uses the data that's created by Brace. So things like email clicks, sends, opens, and so forth. And it maximizes engagements. And it personalizes off of that existing kind of native Brace data, which is still a pretty rich data set. And so as a result, what it means is you can just get going in an hour if you want, configure hundreds of thousands of different combinations of creatives and times and days.
[00:07:15] And then it'll move. So it's really democratizing access to AI decisioning. We did speak last week, and you described Go as a way to make AI decisioning self-service. So what does a marketer, what do they actually control? And what decisions does the reinforcement learning system, what does that make once the journey is running? Yeah, so first of all, the marketer just chooses which journey to apply it to, right? That's the very first decision. So, for example, do I do this for my abandoned cart or for my win-back or for my refer-a-friend journey?
[00:07:44] Then the marketer chooses what are the creative options. Do I want to have, you know, for now it's email. So do I want to have these five emails? What are they, you know, how are they laid out? What's the messaging? Then what are the variants? For example, maybe for each of them I want to have 10 different subject lines or five different CTAs. These are all decisions the marketer makes. What are the frequencies that I want to make available? The times of day or the days of the week that I want to make available to the AI. The marketer can put on some guardrails.
[00:08:11] Like, don't repeat the same subject line more than two times in a month. Yeah. And then configures all those things and then launches. Awesome. And you also showed some examples where hundreds of creative combinations can quickly become tens or hundreds of thousands of possible decisions. How does this reinforcement learning make sense of that volume and, dare I say, the traditional A-B testing in ways that cannot? Yeah. Yeah.
[00:08:38] The fact that it's able to, you know, just like it personalizes to people, you know, it sees you and me as maybe we're a 200 dimensional vector. Yeah. Right? So, like, it's a collection of numbers that shows things like how frequently we clicked in the past or, you know, how long we've been a customer. All these things are kind of values of these, of features. Yes. The same thing is true of the creative variants.
[00:09:03] So, you might give it what seems like 100,000 different email combinations, but many of them share the same subject line. Or maybe it's different subject lines, but they have the same tone. And so, those things are also encoded as numbers. And what that means is that it learns more efficiently. It doesn't see this as a grab bag of 100,000 unrelated things. It actually sees the structural relationships between them. And so, for example, if there's an email that it sends that performs well, that makes it think better of all emails that share the same subject line. Gotcha. And then it can even learn from more and more data.
[00:09:32] Is it the subject line that's doing the work or is it some other similarity between them? So, it makes it, like, much more sample efficient than if you're doing a bunch of these discrete A-B tests that don't really communicate with each other. Incredibly cool. And one of the things I found interesting in an earlier conversation, I think it was Friday last week, was the continuous learning loop. So, how does that system adapt when customer behavior changes over time? And how do you think about model drift in an environment like that? Because that's a big topic right now. Yeah, that's a big question, Neil. And maybe to explain to some of your listeners this premise that, Neil, you just referred to as model drift. Yeah.
[00:10:02] When you have a predictive model, like, let's say, propensity to churn, you might take a historical data set of customers, look at who actually churned, who didn't churn, and then train a model that will take their data and predict if they'll churn. And these models are subject to model drift because you've trained it once. It's kind of frozen, static point in time. And then as your customer characteristics of your whole population shift, like maybe all of your customer population ages in terms of how long they've been your customer. Yeah. Suddenly, your model starts losing accuracy.
[00:10:31] And it's sometimes quite catastrophically. And so, this is this notion that you referred to of model drift. One of the beauties of reinforcement learning, this technology that undergirds AI decisioning, is that the models retrain every day anyways. Yeah. And so, as a result, if there's kind of these shifts in your data, the models will actually just learn and adjust and continue being performant. And what does a decisioning studio go need before it actually becomes useful?
[00:10:59] Where is that line between having enough information to personalize well and expecting too much from the model? We do see a lot of hype out there. Yeah. So, one of the things that we certainly recommend is having, for that journey, journeys that you use it with, having at least a few thousand clicks a month in the baseline journey. Because otherwise, you're not going to have enough statistics for the models to actually learn well. So, that's one limitation. The other thing is just to realize, you know, for now, it's email. We do plan to over time add more channels. But for now, it's email. And it's maximizing engagements.
[00:11:27] So, in this case, kind of clicks but filtered to remove things like machine clicks. Yeah. And so, now, maximizing engagements usually translates into maximizing conversions but not always. And that's just one of the trade-offs here. Yeah. And what about that getting personalization right without drifting into creepy territory? Something that you're always wary of as well? Because here, you know, if you look at the decisions you're making, it's like which version of the email to send or which version from a generic subject line.
[00:12:04] Yeah. Because the LLM could, you know, show that it knows an awful lot of value potentially. So, I think, you know, there's no fixed solution apart from just using good judgment. Yeah. And I do think, you know, also being transparent with your customers about if and when you're using AI. Like, there's a story that I like to tell. It's kind of a personal experience that I had as a consumer. I was living in Boston at the time. And we wanted to look at a new apartment to rent. Yeah.
[00:12:32] And so, at the time, we had a one-and-a-half-year-old son. And the only time we could see it because of our job schedules was at the end of the day. So, my wife and I pick up our son from daycare. And we're like, okay. And I had booked an appointment to see this apartment. And the way that happened is I emailed the apartment company. And I said, hey, you know, interested in your apartment. You know, can I arrange a viewing? I got an immediate reply from Sidney, who was, you know, I assumed was their employee.
[00:13:01] And Sidney said, absolutely. We'd love to have you come. Here's some available times. And I said, great. Today will work great. You know, let's do it at this time. Sidney says, wonderful. We look forward to seeing you. So, excellent. So, we grabbed my toddler. My wife and I take the train. We come in. And then we're in the apartment building. And the desk man says, oh, no, I don't have anything booked for you. No, no, no. But we emailed Sidney. She said we had something. Oh, Sidney's just our bot.
[00:13:30] And, you know, this is our AI. So, and all the feelings that we had at that point. Because, you know, my son, it's the end of the day. He's crying. And he's upset. And this is all this big production. And we felt betrayed. Yeah, yeah. And it's interesting. If you think about it, like, you know, why? Because initially, the fact that Sidney was an AI was delightful. Because she was so quick. She was so responsive. That was actually great. But the fact the company had concealed to me that this was an AI.
[00:13:56] They had, you know, Sidney had been presented to me as a person. And then afterwards, the company disclaimed responsibility for Sidney's actions. Yeah. And said, it's not our fault. Sidney's not our agent. You know, it's an AI. There's this real feeling of betrayal. So, I think brands just need to keep in mind. It's about, you know, transparency. And making sure that when they use AI. Because consumers love AI when it's used well. Like, you know, Sidney, if there hadn't been that mess up, I would have loved the speed and the responsiveness of Sidney.
[00:14:27] It's really then making sure that you still see them as your agents. Not in the AI sense, but in the sense of taking responsibility for their actions. And you as the company having responsibility for the actions of your AIs and appropriate transparency. Yeah, I think there's almost an entire podcast episode around AI agents and accountability there. So, maybe we'll have to get you back on for that one. But when we last spoke, you mentioned beta customers that were creating use cases you had not originally anticipated.
[00:14:56] So, what have those customers taught you about where AI decisioning works? And equally important, where it doesn't work. What have you learned from these beta testers? Yeah, it's been exciting because, you know, AI decisioning before being a relatively involved process, it's very consultative setting up an AI decisioning use case. So, this notion that a customer would just go and set it up and we wouldn't even know about it is so alien to us.
[00:15:21] Because before, we like, you know, we like become part of their team and we work together for a few weeks and we set it up and we're friends by the end and all this stuff. And here instead, it's like, wait, like we had nothing to do with that. You just went and did it. It's super cool. It's very exciting. And we've seen customers doing it very well. So, these use cases that they've set up, if you look at the performance, they're performant use cases, which makes me very, very excited because that was the whole premise of Go. It was to make AI decisioning truly, you know, democratized and available to where if customers don't want to talk to us, they don't have to. And it'll still work.
[00:15:51] Incredibly cool. So, when a system is making individual decisions at scale, marketers are going to ask what a or why a particular customer received a particular message at a certain time. So, how much explainability do you really strictly give them without oversimplifying what the mod's doing? And you mentioned that great example about Sydney there as well. Yeah, that's a fantastic question, Neil. So, I need to figure out what I'm able to reveal. Yes.
[00:16:17] We have, you know, what we have today is a very robust performance reporting suite. So, you can see, okay, you know, I've gone live with this decisioning agent. How much performance am I seeing compared to my control group of business as usual? How does that performance decompose in terms of, you know, if you're getting more conversions, it's conversions per send times, you know, sends. And you can kind of drill down. You can slice and dice it by segment. You can see which choices the model is making.
[00:16:47] So, that's what exists today. I can share that a very large R&D priority for us has been developing an insights suite that will allow you to dig deeper and start to understand the answers to some why questions. That's probably going to be an announcement that we'd be making at some point in the coming months. So, I'm not able to go into too much more detail except that there are very exciting things coming soon on that front. Oh, you left us with a teaser there. We will be getting you back on for sure.
[00:17:16] So, if we do have a marketing team listening today, they're still mostly running manual segmentation and that conventional dreaded A-B testing that we mentioned there. What is the first sign that would tell them that they're actually ready to move forward towards AI decisioning? Are there any signs that they pick up on or maybe if they're listening and you're thinking, you are ready? To me, it's actually, it's exactly that. It's the fact that they're already doing some manual testing. Because if you've got a team that's like not doing any testing, it's probably too early.
[00:17:45] It means that probably there isn't enough financial value to the business of optimizing those journeys to begin with. Because if it was important to optimize the performance, they'd probably have started running some tests. And when marketers run some tests, it also starts to suggest, you know, that increases the sophistication of that marketing team. They start to understand some of the nuances of measurement. They start to understand some of the levers that they have at their disposal. And that's all very important organizationally to make sure that the organization is ready. The only thing I'd add on top of that is some level of data readiness. Now, you know, it's funny.
[00:18:16] Data maturity is kind of the opposite of driving. Everybody thinks they're below average. So every single company I talk to is like, oh, our data is in terrible shape. You know, are we ready to do anything with our data? And usually they actually are. But there does need to be some baseline. They need to have a warehouse or a CDP set up, some level of unification and customer 360 in place. As long as they've made some strides in this direction, then they're ready.
[00:18:40] But if they don't even have that, if they don't really have a warehouse, if they haven't unified their data, then, of course, they're too early from a data maturity perspective. I think that's a powerful moment to end on. A real CTA for people listening there. I'd love to hear your experiences and how you get on with this. I'll have links to everything that we talked about today. But more on anything, George, thank you for agreeing to meet me in person. It's my pleasure, Neil. It's my pleasure. Thank you.
[00:19:03] I think my takeaway from today's conversation is that better marketing decisions begin with understanding what you're asking the system to approve, to improve. George was clear that optimizing clicks doesn't always mean optimizing conversations. But marketers still choose the creative options, the available timings and the rules around how customers are contacted.
[00:19:27] And I also appreciate his apartment viewing story there because I think accountability becomes much easier to understand when a family arrives for an appointment that was never actually booked. And a quick reply is useful only if the business follows through on its promises. So a big thank you to George. I'll include links to his LinkedIn profile along with the episode. But share your thoughts with me. Which marketing decision would you trust AI to make today?
[00:19:55] And what evidence would you want before giving it that kind of responsibility? Well, techtalksnetwork.com. I'd love to hear your experiences around this. But it's time for me to go now. I've got another event to go to now. So keep an eye on techtalksnetwork.com and the events page. I could be coming to a city near you. And if I am, please slide into the DMs. Let me know. Let's have a hot coffee or a cold beer. But that's it for now. Thanks for listening, everyone.
[00:20:25] Bye for now.