Observability without Overload: Lessons from Chronosphere’s Field CTO

[00:00:00] - [Speaker 0]
One of the phrases I've heard on this podcast very often is data is the new oil. But let's be honest, it's more like the new oxygen, especially with AI because without data, there is no AI. And your business can't function without it. NordLayer, they ensure that your data moves securely between people, apps, and offices through encrypted tunnels. Encrypted tunnels and strict access controls, and it keeps sensitive information out of reach from prying eyes, but gives you the full visibility over how your network is used.

[00:00:38] - [Speaker 0]
Now I've heard many stories on here where a single exposed endpoint quickly turned into a company wide breach. But NordLayer, they will make sure that that doesn't happen to you. So if this interests you, please go to nordlayer.com/techtalksdaily. And if you enter the special code tech daily dash 28, you'll also get 28% off. So please go check that out.

[00:01:05] - [Speaker 0]
Let me know your thoughts. But right now, let's get today's guest on. What happens when observability stops being a tool conversation and starts becoming a business conversation? Well, today, I'm joined by someone who has lived that shift in the real world. His name's Bill Heinlein, and he is the field CTO at Chronosphere.

[00:01:29] - [Speaker 0]
And before that, he spent years inside highly complex environments at United Airlines where telemetry volume, cardinality, and scale were much more than just buzzwords. And throughout his career, he has seen firsthand how a collect everything mindset turns data lakes into data landfills and digital clutter and digital hoarding, and how alert fatigue can drain great teams, and even a hero culture can quietly burn out your best people. But he does believe observability should be intentional, service focused, and tied to SLOs from the very first sprint. So if your CFO knows observability for the cost line rather than the value that you're delivering, he's got some thoughts that you're gonna wanna hear today. So you can expect practical examples, a few scars and war stories from late night pager duty, and most importantly, clear steps that you can use tomorrow.

[00:02:30] - [Speaker 0]
So if you're ready to swap noise for insights and turn your monitoring into a momentum, you're gonna love this one. But enough from me. Let's get our guest onto the podcast now. So thank you for joining me on the podcast today. Can you tell everyone listening a little about who you are and what you do?

[00:02:50] - [Speaker 1]
Sure. I'm Bill Heinlein. I'm the field CTO at Chronosphere. We're an observability platform. I have been in the tech industry, more time than I care to to say, but a little more than twenty five years.

[00:03:06] - [Speaker 1]
A lot of that time spent in the airline industry. And most recently before joining Chronosphere, I was director of observability at United Airlines. So lot of large company experience, a lot of battle wounds over the years. So I can go on for a long time about, observability and other topics. So happy to be here.

[00:03:32] - [Speaker 0]
And that is exactly why I invited you on the podcast to join me today. Because every day, we take a different subject, demystify it, talk about it in a language everyone can understand, make everyone realize we're all battling with the same problems and how we can overcome them as well. So from all the conversations you've had in your current role, United Airlines, everything throughout your career, what is that number one observability challenge that organizations are grappling with right now? What is it?

[00:04:00] - [Speaker 1]
Well, I mean, cost is the easy answer, but, you know, cost isn't the root cause. Right? Cost is it cost happens because, you know, over the years, we have all subscribed in one way, shape, or form to this collect everything model. Right? And it's filled with a little bit of false hope and really creates what I start calling a data landfill.

[00:04:25] - [Speaker 1]
You know? We wind up instead of a lake, you get a landfill. A lot of data in there, not a lot of value, hard to measure the value. And there's a challenge with measuring the usage and the value of the data that you're collecting. So, you know, it's cost, but it's because there is this this collect everything strategy rather than a more curated telemetry strategy.

[00:04:50] - [Speaker 1]
And, you know, things like logs are some of the biggest offenders, and it's really getting down that that telemetry to things that are collected with intent and really bringing down the noise to signal ratio as a result.

[00:05:08] - [Speaker 0]
And we've all been on somewhat of a journey over the last few years. I I I find myself, the older I get that everything happens in cycles, and we've everyone left on prem data centers. That was old school. We're all going to the cloud now. These rumors are all coming back again.

[00:05:24] - [Speaker 0]
So I'm curious. So how do the traditional rules of observability how do they change in cloud native environments?

[00:05:31] - [Speaker 1]
I would argue that observability has never been traditional. It's always had to evolve. I think I don't I I think technology in general is hard to categorize as traditional. If it is, then it's probably something we talked about, you know, years ago that no longer nobody's doing anymore. But, you know, in short, we've had to move from predictable telemetry volumes.

[00:05:55] - [Speaker 1]
Right? In in the world of monolithic applications and, you know, they lived on a few servers in a data center. They admitted a predictable amount of telemetry. You were monitoring a less complex infrastructure, and you understood when it needed to scale, you built that in early on by predicting what your scale was gonna be, and then you knew what to to predict as far as the volume of telemetry you were dealing with. And telemetry wasn't as sophisticated back then.

[00:06:26] - [Speaker 1]
Right? We were still in the era of, you know, early APM. Some teams were still just managing component monitoring. Right? Is the database up?

[00:06:38] - [Speaker 1]
Is the server running? Am I out of disk space? You know, very, very basic stuff that nerds like me used to write a lot of scripts in Perla for years about. But then it moved to cloud and everything exploded. Right?

[00:06:50] - [Speaker 1]
Because suddenly we have servers that last for seconds while they run and then die after they're done. And we have scale that went from a few servers to, you know, tens of thousands depending on the size of the environment and the the scale at which you're operating at the time. And the telemetry that's all there is deeper and and the cardinality just explodes. And so it's really what's changed is the volume of telemetry. And those those more traditional APM tools, the more traditional and early observability tools really struggle with keeping up with the scale of that and doing it in a way that's cost effective.

[00:07:35] - [Speaker 0]
And I also think the rise in cloud costs has served as somewhat of a virtual intervention of sorts to face up to a data hoarding ways. From what you've seen here, what what issues do the collect everything data collection practices bring when the the first step is is gaining control or attempting to gain control?

[00:07:56] - [Speaker 1]
Well, I think there's certainly an opportunity for us to do a show on, you know, your favorite binge watching network about data hoarding, you know, you know, similar to people hoarding in a in a storage unit somewhere. But the collect everything, we already mentioned cost, obviously. But Yeah. You know, if you collect everything, you're not curating. And the problem is is that I think we all got very excited about AI and thought that AI could answer everything, but it's still a classic garbage in and garbage out.

[00:08:31] - [Speaker 1]
Right? If you've got telemetry types coming from multiple sources, you know, different different observability suppliers, maybe, you know, Lord knows a lot of different infrastructure, whether it be cloud or on prem. It's all very different. It has varying levels of enrichment. You throw in application logs that anybody can write and that have varying levels of enrichment and may be missing key things that don't matter to the developer who wrote it, but may matter to other people who look at it.

[00:09:02] - [Speaker 1]
But so you've got this, like, this blood of of data here that's really hard to correlate. And I I always say that, you know, if you can't look at a set of data on some paper and correlate it together, AI isn't going to be able to do that either. It needs something to chew on. And and that is where the collect everything starts to bite you. Right?

[00:09:27] - [Speaker 1]
Because you've got all this data that you can't correlate easily. And when you're in the midst of that 3AM call that we all love so much that you you wind up digging in and following a metric that spikes suddenly, but, oh, guess what? It doesn't really correlate to the problem you're working on and that, oh, by the way, that looks like that happens every third Thursday for some reason because somebody's running a job over here and it's unrelated, but you wasted twenty minutes. So it's that noise to signal ratio that's killing your MTTR, and you're spending a lot of money. And your CFO probably knows what observability is for the wrong reason.

[00:10:07] - [Speaker 1]
The cost instead of the value.

[00:10:10] - [Speaker 0]
Yeah. 100%. And, of course, we'd also throw in operational burden and the dreaded alert fatigue. Also, many sources of burnout and frustration for IT teams. But what what real impact does this have on businesses, and and what can IT leaders do to address it?

[00:10:27] - [Speaker 0]
Something we've been talking about for years, but any way forward here?

[00:10:32] - [Speaker 1]
Yeah. I I think, and I mean, I look. I I had big teams. I had a a team of more than 500 at United, and so burnout was something that I worried about every day, you know, because you get great people and through no fault of anybody's, those great people often wind up being the people that that you build into heroes.

[00:10:54] - [Speaker 0]
Yeah.

[00:10:54] - [Speaker 1]
And a hero culture kills your best people. They burn out, and then the next thing you know, they have this they've secretly turned on the open to work flag on LinkedIn. Right? And then you've got opportunity cost problems and all sorts of things. But the real impact there is you've hired these these engineers being whether they be traditional engineers in the data center or cloud engineers or software engineers, you've hired these brilliant people to make money, to build products that are going to displace your competitors or your market.

[00:11:29] - [Speaker 1]
Right? And this operational burden that happens because on call exists and you're not doing it efficiently takes away that innovation. And so all the capital investment that that you're you're making, making, that your investors are happy that you're making in all this technology to disrupt the marketplace suddenly starts to look more like sunk cost because you're not getting anywhere with it. But you're at the same time, your operating costs are on the rise because your teams are spending all this time keeping the engine running, so to speak. Sorry.

[00:12:02] - [Speaker 1]
Back to airline metaphors. But, to drive or or to keep things running. Right? And so they're not focused on innovating. And that context switching that they have to do constantly is a real thing too.

[00:12:15] - [Speaker 1]
I wrote an article and quoted an APA study not too long ago that talks about context switching can have as much as a 40% reduction in productivity for an individual because of that. I mean, imagine. Right? You're you're in the zone. You're coding.

[00:12:32] - [Speaker 1]
Hopefully, you're not vibe coding too much. And you're, you know, you're really into it. You've got a great thing going on. All of a sudden, the the, you know, I'll date myself. The pager goes off.

[00:12:42] - [Speaker 1]
And then you've gotta completely stop what you're doing, and then focus over here on code that likely somebody else wrote, but you gotta figure out why it's broken while you listen to a a group of people ask you, is it fixed yet? And then you come down from that, and then you go back and you try to pick up where you left off. So the business impact, I mean, certainly is burnout for that individual. But from a business perspective, you're not innovating. You're getting left in the dark.

[00:13:10] - [Speaker 1]
You're you're losing great people. Right? There's a lot of, like, business impact that leaders really have to consider, and it's a real thing. And that's where, you know, observability can either hurt hurt or help you there. You collect everything.

[00:13:24] - [Speaker 1]
It's probably you're probably on the pathway to hurt.

[00:13:28] - [Speaker 0]
And, you know, you're in good company there. I still get flashbacks from those days of that page are going off, man. Don't get me started there. There's a whole world of pain opened up that I've still not dealt with, I think. But, seriously, moving beyond tools, what does a a strategic shift in today's IT infrastructure management look like at a high level?

[00:13:48] - [Speaker 0]
And and what kind of conversations need to happen to kick start that change? Because I it's very difficult to get all the stakeholders and get everyone in a room and kick start the change as required and get everybody on board. But what needs to happen here?

[00:14:01] - [Speaker 1]
I think it's just fundamental change at the top. Right? And by the top, I don't mean necessarily the CIO or the CDO or the CTO. I mean at the the top level of the executives of the organization, including those people. IT has to deliver strategic value.

[00:14:19] - [Speaker 1]
In this world of cloud costs and the cost of IT and the cost of IT labor and all of that, you have to be able to justify the value of your organization. Because IT organizations that don't provide value, guess what? They get outsourced.

[00:14:34] - [Speaker 0]
Yeah.

[00:14:34] - [Speaker 1]
And those outsourced teams don't provide the the great value that are is usually hoped for either. Now in some cases, they do. But I think what you have to do is really start with it's not so much how you manage just infrastructure. I mean, I know that's probably the primary focus of of your conversations on this podcast, but I think it's at a higher level of of making sure that your IT organization, in specific your your IT leader, has a seat at the table and that they are participating in those strategic discussions and advising rather than order taking. Right?

[00:15:15] - [Speaker 1]
And then then, you know, there's the it's easier to, number one, justify. Okay. We're spending a lot on cloud. Well, we made together this strategic discussion or decision that we're gonna invest in cloud because we need to scale. We need to move to we need to be able to support things in this region, whatever your justifications are.

[00:15:35] - [Speaker 1]
But you've done that as a collective group rather than going off and trying to deliver what you think the business wants and then coming back and then being questioned whether you're raising the value or not.

[00:15:49] - [Speaker 0]
And in a world where all these systems are constantly scaling and shifting, and I'm curious, how can teams ensure ensure that that they're they're collecting collecting the right data? Not collecting everything, collecting the right data, making that shift to to make informed decisions without being overwhelmed by the volume? I mean, I know there's an argument that AI will make that volume a little bit easier to digest, but anything you're seeing here?

[00:16:14] - [Speaker 1]
I I think I joke sometimes that, when people get so excited about AI, I say it's the new gluten free. Right? We put AI on a lot of things. You know. At one point, I think I saw that gluten free was on a laundry detergent and I scratched my head.

[00:16:32] - [Speaker 1]
I'm like, why does that matter? But I I think that we've gotta go back to the reality of collecting data with intent. Right? And it goes back to business value. Right?

[00:16:46] - [Speaker 1]
If I work with a team in my past, they get wrapped around this, and it goes back to they'd wanna measure this component and that component instead of thinking about the service they're offering. And so I'll step back and say, tell me five reasons this feature exists in your product. Why did the product owner come to you and say, I want this to happen. Code this. And then what does good look like?

[00:17:11] - [Speaker 1]
And I promise you, if you focus on traffic, error rate, latency, and saturation just as a starting point, you're leaps and bounds ahead of where you'd be if you started to see, is that server up? Is it up now? Was it up ten minutes ago? And instead looking at, is my service at the level I hoped? And build an SLO, spoiler alert, at the beginning instead of at the end.

[00:17:36] - [Speaker 1]
I I've been just as guilty as anybody else in this space. But five or six years ago, I started asking myself, why do we wait until after the release to think about an SLO? Why aren't we thinking about that in the first sprint? Right? And then that informs metrics that you might need to measure that SLO.

[00:17:58] - [Speaker 1]
It informs how you can build an error budget. And then you get to a point where you can look at things with value and you know what you specifically need to collect rather than just throwing everything in a pot and crossing your fingers. And that and and as as this as things grow and as you're scaling, that becomes more and more. You've gotta it's gotta be repeatable. It's gotta be frictionless for a developer.

[00:18:24] - [Speaker 1]
Observability has to be a utility for people, not a burden. It has to be a first class citizen. And I think observability is dealing with the sentiment in most organizations that that cyber had to deal with, you know, ten years ago when we were really doubling down in organizations on cyber because it was becoming a real significant threat for everybody. Right? Developers had to do it, but it was another thing I gotta do, and I got this giant delivery cycle.

[00:18:54] - [Speaker 1]
Observability is sort of in that spot right now. We gotta make it a first class citizen too. And that includes making it frictionless and making it repeatable and doing that by simplifying the approach of how you start monitoring.

[00:19:07] - [Speaker 0]
And we've both jokingly shared stories of our past on that page are going off when we didn't want it to. But I think for many teams, they've almost been conditioned to operate with a a reactive mindset after years of firefighting, fixing p ones, etcetera. So what's the best way to foster that cultural shift that's required to to move towards a much better and stress free proactive data driven decision making? It almost sounds like a utopia me saying this out loud after years of doing the same, but what needs to happen here?

[00:19:40] - [Speaker 1]
I I joke regularly that that I generally talk in euphoric sense, you know, like or a utopic sense rather than, you know, reality, but it is my opinions aren't based on reality and repeated mistakes on my own part. But I I think for us to to get people to shift, they have to understand the power of data and the data that's available to them. I I look at a histogram. Right? The power of a histogram that shows you things like performance or error rate from one minute to the next can help you really think about ways to be more predictive.

[00:20:18] - [Speaker 1]
If you're using SLOs, if you've decided you're gonna shift your mind and do an SLO first and you do it in the sprint and lo and behold, you go to production, you've got a good SLO, and you've got good error budget. Even if it's really simple, you're a step ahead. And now you're saying, okay. I don't have to react and wait till somebody calls me because something breaks or maybe I'm a step ahead and and my tools tell me something broke before the customer. But so many of the things that I saw that were these big newsworthy events, not just at my former employer, but it in general, right, started out as small paper cuts hours, days before.

[00:21:05] - [Speaker 1]
And if there were little data points to tell you, like, hey. You're you're starting to burn your error budget a lot more than you were. That's that's the easiest story to tell to get people to think about how to go from it and get away from, well, my app's up. Nobody's calling about it. Right?

[00:21:21] - [Speaker 1]
I can log in. It's it's a shift in mentality and shift to get them to realize that, look, you got woken up at three, but this started at, like, ten in the morning while you were having coffee. Like, you know, what if what if it interrupted your coffee and started to sleep?

[00:21:38] - [Speaker 0]
Yeah. Oh, now that's progress a 100%. I think thanks to pesky shiny emerging technologies like, yes, AI, AgenTik AI, or whatever. Yep. The next big thing is we seem to have a lot of stalled projects, things caught in pilot.

[00:21:56] - [Speaker 0]
So as a result, leadership teams now are putting an extra special caution on all tech projects. Everything's gotta be about ROI, measurable difference Mhmm. Quite rightly too. So how can organizations effectively measure the ROI of investing in a new observability strategy, especially when benefits aren't always tangible at first or may appear that way? And I suspect this is an area you're passionate about too.

[00:22:22] - [Speaker 1]
I am. And and it goes back to my line. If your CFO knows what observability is because of the cost rather than the value, you've got a problem. To get that to get you there, to get your organization there to realize that value, observability should underpin your business objectives. Right?

[00:22:42] - [Speaker 1]
You don't just decide. I mean, maybe if you're a nerd like me and you see what the cool things are, then you might think, yeah, Absolutely. We should do observability. And you're maybe not thinking about how it's tying to objectives at a higher level. Right?

[00:22:56] - [Speaker 1]
It's cool. I had a leader years ago tell me cool is not a business case. It's not. But if I can tie observability back to if I'm an ecommerce company, I care about how quickly you can buy widgets. I care about conversion rate.

[00:23:13] - [Speaker 1]
I care about how quickly you go through the funnel. Right? And I I care about things that interrupt that funnel and make you abandon your shopping cart. Observability helps you get there. So if you design observability for that application to say, I care about response time and my services.

[00:23:31] - [Speaker 1]
I care about how long it takes somebody to be be presented with the 40 widgets that I can I sell and pay for it and all of that? Then I'm underpinning a business objective that is likely sell more widgets or increase my conversion rate by x percent. So if observability underpins your business strategy, it's easier for people to swallow. And then instead of measuring with, you know, MTTR, which typically only technology leaders care about, then you're you're selling it with the idea of, hey. Look.

[00:24:03] - [Speaker 1]
We drove better performance, which you, mister and missus product owner, care about because it's driving better conversion rate and therefore better revenue based on what your product is doing.

[00:24:17] - [Speaker 0]
And to bring everything that we've talked about here today back to your work at Cronosphere, can are you able to share maybe a real world example of how a strategic observability plan has has actually helped an organization move from that reactive firefighting model to be more proactive and and innovative one and and get that over the line. Anything you can share on that? You don't have to mention any names. I think it'd be useful.

[00:24:42] - [Speaker 1]
Oh, well, I'll I'll tell you. You know, in my former life, I we were early embracers of APM. Right? And we were in a state where there were still a few folks that were saying, you know, my nobody's calling. You know, nobody's calling the service desk.

[00:24:57] - [Speaker 1]
Nobody is you know, there's no problem. I can log in kind of a thing. They were they were still in that mode with with some teams. Getting people to understand the value of, again, that histogram, getting people to be able to understand that a lot of the problems that that keep us on phone calls for hours in the middle of the night started with a paper cut at seven in the morning. And maybe seven in the morning on Wednesday, and it's Sunday at it's Sunday morning.

[00:25:29] - [Speaker 1]
Right? So changing that perspective and showing that this data is is possible to gather, it's not difficult to gather, and that when it's there, it gives you a lot of great things. And one of the best ways is to help people I I call it mean time to innocence. Right? Hey.

[00:25:52] - [Speaker 1]
Look. I know it's not my problem, and I know it's not because here's data. Right? Here's my service. It's calling this.

[00:26:00] - [Speaker 1]
And, oh, guess what? It's your service because I'm calling you and every other third time I call you, it fails. You're killing me. Right? Yeah.

[00:26:10] - [Speaker 1]
I so I we did that. We did that. And and I was proud to say that some of the earliest adopters of APM in our organization were the NetOps team members because they got tired of being blamed for it being the network because it's always the network. Right? That was always a running joke in every organization, I'm sure.

[00:26:29] - [Speaker 1]
But these guys would and gals would get on, look at the look at the tools and say, it's not the network. And as a matter of fact, it's this service right here that keeps failing and it started right when people started calling. So it's it's there are a lot of I can probably think of hours of examples of things like that, both in my current or or with my former employer and as I talk to customers in my current role.

[00:26:55] - [Speaker 0]
And just listening to that story reminds me people were so fiercely defensive back in the day on things like that with, I almost felt like the Spider Man meme with the network guy pointing at the apps guy and the apps guy pointing at the you know, everyone's pointing at each other. But but if an IT is listening to our conversation today, maybe they wanna take one actionable step tomorrow to begin to improve their observability posture. What would you recommend they do? I I appreciate it's not as simple as that, but if they wanna take that first baby step, what should they do?

[00:27:27] - [Speaker 1]
One of my favorite questions when I walk into a customer as we're talking to them is, why did you buy observability and how is it supporting your business goal?

[00:27:35] - [Speaker 0]
Yeah.

[00:27:37] - [Speaker 1]
I I mean and if the answer is I don't know or they stumble for it, then I they probably just need to take a step back and think about it. So many people jumped into observability because they needed something and it was an evolving technology and evolving tool. APM promised some really amazing things easily. And as cloud changed things, that got more difficult and it and it diminished a little bit. People had to work harder to get it.

[00:28:11] - [Speaker 1]
And so it's even more important to have that strategy. And that strategy starts with, why am I doing this? I'm doing this so I can sell more widgets. I know I'm selling more widgets because I'm driving better conversion. I'm driving better conversion because my performance is better.

[00:28:27] - [Speaker 1]
And I know that if I change my performance by x number of of milliseconds, I drive my conversion up by x percentage. Right? So I I think it's really sitting back and asking yourself, why am I doing this? Because somebody's gonna ask you. Don't wait till your CFO asks you.

[00:28:46] - [Speaker 0]
And I think that is a powerful moment to end on today. But anyone we've inspired today, maybe they wanna take that first baby step into improving their observability posture. Where can they find, you or your team and and more information on anything we talked about today? Where should they go?

[00:29:03] - [Speaker 1]
Certainly, you can find me on on LinkedIn, and I'm always, as you can tell, happy and passionate to talk about observability. And and part of my role as as the field CTO CTO as Chronosphere is not to sell, but to guide and help folks with how to do things like an observability strategy. So you can also find me there. We're at chronosphere.io. And, again, we're a cloud native observability company focused on giving you control of your data rather than drowning in it and not understanding what your telemetry is doing for you.

[00:29:37] - [Speaker 0]
Love that. So I will add links to absolutely everything. Make it easy for people to find.

[00:29:43] - [Speaker 1]
Thank you.

[00:29:43] - [Speaker 0]
We covered so much today from observability challenges that organizations are grappling with, collecting the right data to make informed decisions. And as an exciting eye, measuring the ROI of investing in a new observability strategy, the old belts and braces approach of you can only improve what you measure. So great to hear. Right. Just thank you for shining a light on this today.

[00:30:05] - [Speaker 1]
Thank you. Thanks for the time.

[00:30:08] - [Speaker 0]
Now before we wrap, what will you change this week to collect with intent rather than just collecting everything? I think Bill left us with a simple challenge that almost doubles as a compass. Start with why your product exists. Define what good looks like. Set those SLOs early, and let your choices guide the data that you're gathering.

[00:30:31] - [Speaker 0]
And this is how observability can earn a seat at the strategy table and how teams reclaim time for building, not just wasting their days firefighting. So if this sparked an idea, please connect with Bill on LinkedIn. Check out their website. And if you've got anything you'd like to share with me, techtalksnetwork.com, please let me know. But that's it for today, so thank you for listening as always, and I'll speak with you all again tomorrow.