KASLIN FIELDS: Hello, and welcome to the Kubernetes Podcast from Google I'm your host, Kaslin Fields.

ABDEL SGHIOUAR: And I am Abdel Sghiouar.

[MUSIC PLAYING]

In this episode, we sat down with Niki Manoledaki. Niki is a Staff Platform Engineer at Grafana Labs, a CNCF Ambassador, and Green Software Foundation champion, and the core maintainer of Project Kepler. We spoke about green ops, the environmental impact of cloud native technologies, and how Project Kepler tries to solve sustainability challenges in the age of AI.

KASLIN FIELDS: But first, let's get to the news. The CNCF Technical Oversight Committee, or TOC, officially voted to advance Confidential Containers from sandbox to an incubating project. Confidential Containers leveraged trusted execution environments, or TEEs, to secure data in use within cloud native workloads, protecting sensitive data and AI model parameters from hypervisors or cloud provider admins.

ABDEL SGHIOUAR: We came across a very interesting Kubernetes enhancement proposal. KEP 59-72 was created in March this year and is still in discussion within the community. This enhancement would allow main containers to be added to or removed from running pods. This would challenge Kubernetes' long-standing design assumption that pods are immutable execution envelopes.

The main driver here is high performance batch jobs and low latency AI workloads that need dynamic sidecars or context specific containers without destroying the active execution states. It's currently in design review. So if you have thoughts on pod lifecycle rules, now is a great time to join the signal discussion.

KASLIN FIELDS: Red Hat recently announced an open source project called ASAGO, short for AI Safety and Governance Orchestration. The project bridges the gap between compliance frameworks like the EU AI Act or the NIST AI RMF, which is a lot of letters, and actual Kubernetes operational controls. It automatically translates enterprise policies into real-time cluster guardrails and maintains an audit trail for your deployed models. You can check out the repository over on GitHub.

ABDEL SGHIOUAR: The schedule for Kubecon Cloudnativecon North America 2026 is live. The event is taking place November 9 to 12 in Salt Lake City, Utah.

KASLIN FIELDS: CoHDI, apparently pronounced "cody," formerly Infra DDS, was accepted as a CNCF sandbox project founded by Red Hat, AFSAS, Fujitsu, IBM Research, and NTT, CoHDI enables host-level dynamic attachment and detachment of PCIE devices, like GPUs and accelerators, on Kubernetes nodes via dynamic resource allocation without requiring OS reboots. And that's the news.

ABDEL SGHIOUAR: Niki Manoledaki is a Staff Engineer at Grafana Labs, where she leads cloud cost initiatives across multiple cloud regions and cloud resources, like Kubernetes networking and, more recently, AI. She is a CNCF Ambassador and Green Software Foundation champion. Nikki is a core maintainer of Project Kepler, the open source Prometheus exporter for tracking Kubernetes energy consumption. She is also a leading voice in cloud sustainability and an organizer of the environmental sustainability technical community group. Welcome to the show, Niki.

NIKI MANOLEDAKI: Hey, Abdel. Thank you so much for having me on the show. I'm so excited to be here.

ABDEL SGHIOUAR: Awesome, I know that you had a very brief conversation at Kubecon. And it took some time to make this happen, so I'm very excited that you are here. I believe we never actually touched on this topic on the show. I mean, we have, at this point, 260 plus episodes. So it might have happened before my time, but I think it's always good to revisit the topic.

And with that, I have actually my first question, what is sustainability and what is green ops? Because I hear about this all the time. I have no idea what it means.

NIKI MANOLEDAKI: Yeah, let's talk about it. So sustainability is specifically environmental sustainability. In the cloud, we have this term green ops, which is all about optimizing the efficiency of the cloud resources that we use. It emerged roughly in 2022 along with the term finops. It came after DevOps. All of these terms mean how do we measure, optimize the resources that we use.

So green ops is really about cloud usage efficiency, but it can look different for front-end engineers, back-end engineers, platform engineers. So for my domain, I prefer the term operational efficiency because that really captures what we are trying to do, which is that we're trying to make the most out of the resources that we are already using.

ABDEL SGHIOUAR: OK, and so does that necessarily always have like a direct link to cost? Is that something that we always have to associate to how much things cost? Or is it also maybe by optimizing how-- or by making things more efficient, we get more out of whatever we already paid for? Do you see what I mean?

NIKI MANOLEDAKI: Yeah, cost is a good proxy metric for sustainability. Ultimately, we are talking about in what ways can we reduce the carbon footprint of what we are using. So when cost leads to a reduction in utilization or optimization that ultimately reduces carbon, that aligns with green ops. But not all cost reductions lead to green ops. And not all carbon reductions reduce cost.

ABDEL SGHIOUAR: Yeah, makes sense. And the reason I ask you this question is because I guess that some people that might be listening to this because, IT within organizations can be sometimes perceived or looked at as a cost center. So our IT department costs a certain amount of money per year. And then sometimes, it's an investment, it depends really what kind of organizations we're talking about.

And I remember a conversation we had at Kubecon this year about using ARM as a platform for running workloads, the discussion we were trying to have, we were trying to push people towards, well, cost is a way to measure it. But it doesn't always mean that it's going to cost you less. It might cost you the same, but you might just be able to do more with what you have. You see what I mean?

So that's kind of the point of the question I asked you. And so speaking of that, I mean, same question, OK, organizations looking at sustainability and carbon efficient as somehow related to cost, so why should organizations care? Why should they look at their sustainability?

NIKI MANOLEDAKI: A lot of companies care about sustainability because of cost, as we are saying. But cost doesn't always mean that we're improving our proving our sustainability. What else matters? I like to break it down into cost, values, and regulation. So we see a lot of companies that care about this because of their values.

I know, for example, the SNCF train company, which is a user of cloud native tooling, they are a company that cares about sustainability based on their values. A lot of the folks who are there, they operate trains in France. And it aligns with sustainability. So the values align with sustainability. A lot of engineers care about sustainability, even in their life in general. That is a big part of it.

Regulation is another aspect. Regulation, especially in Europe, means that we have to disclose carbon emissions at certain scales. So for example, companies, especially in Europe, might have sustainability KPIs that they're trying to stay below or to optimize around.

CERN is an example of that. CERN is another end user of cloud native tooling. They did some really interesting talks at Kubecon Europe earlier this year, where they shared how they're using Kepler for energy efficiency in their R&D projects. They're testing Kepler to see how it could help them stay below some of the sustainability KPIs they have, some of the energy usage targets they have for the particle accelerator.

ABDEL SGHIOUAR: Got it. So you talked about like the organization level. But now as a developer, I'm a developer. I have my laptop, I'm writing my microservice app that has like 10 requests per second. Why should I care? That's probably a good point to start for people who are going to be listening, because most people are actually not even decision makers. They might be just like individual contributors.

NIKI MANOLEDAKI: Yeah, we've gamified this in Grafana Labs. Just to answer your question with a picture, we build cost metrics that engineers can use to measure the efficiency of the software they're building. And so we've made it really easy to show these cost savings.

In a way, we've gamified the ability to optimize software. And every engineering team can then use these cost metrics to optimize software in whichever way is best in their domain. I think we can't-- as a platform engineering team, we can't really be prescriptive about what's the best way to optimize your software.

But if you have the metrics to show these optimizations, cost savings, et cetera, I think that that can lead to, for example, showing that release by release, the cost has remained the same, or you've optimized, you've led to cost savings. It looks good. Everyone feels good. Everyone's happy.

I do think, yeah, the business cares about cost. But a lot of engineers do care about optimizing their software. And a lot of engineers also care about doing good deeds for the planet, right? I think that's becoming more and more relevant.

ABDEL SGHIOUAR: I mean, I see the point that everybody could contribute in their own kind of little way, right? Because it adds up at the end, right? That's, I think, what you're trying to say, that it's not a single person responsibility, sort of.

So you mentioned platform engineers not having to be prescriptive. So you don't necessarily have to be-- or you don't always have the possibility to tell people how to build things because you are supposed to provide them with platforms and the tools to do things. So how do we-- what are the techniques and the tools that exist today in the ecosystem that could allow platform engineers to help people build energy efficient systems?

NIKI MANOLEDAKI: So there's different levels, right? What we're saying is can we democratize the metrics and best patterns, so that engineers can optimize in their domains and apply best practices in the ways that they know best?

But also, platform engineers can make large impacts with cloud native tooling, for example, the autoscalers. This is a very obvious example. But in Kubernetes, we have Karpenter. We have the Vertical Pod Autoscaler. We have Cluster Autoscaler. We have so many ways-- we have the Scheduler. We have so many ways to make an impact at a large scale and leave the software level, workload level optimizations to other engineering teams.

So those are some examples I mentioned. Those are my favorite ones. I know at Grafana Labs, we use Karpenter quite a lot. There's a great blog post. I'm happy to link to it in the resources.

ABDEL SGHIOUAR: Yeah, we'll make sure to include it in the notes. I really like your approach, because it sounds to me like you're talking about this as being kind of like a multi-layered approach. You have the platform layer, and then you have the application layer, and then whatever layer on top.

And then you also have the other things like the visibility. What you want is you want to be-- to give people access to information about how their application is performing in production is effectively right. I do have a-- while you were talking, I do have probably a stupid question.

So let's say I'm a developer, right? And I'm writing an app. And I just absolutely want my app to log every single thing. I want to write logs about everything happening in my app. Would that be a good example of how I could think about optimizing my application to make it more cost efficient? Because logs means storage. Storage means cost, right? Or to make it more efficient from energy perspective?

NIKI MANOLEDAKI: Oh, yeah, that's a great point that you're making. I've definitely heard of logging as a good entry point into optimizations for a lot of applications. Actually, the Green Software Foundation has this directory per persona, so that you can look at different like a list of examples that you could implement in your software so that it is more efficient and also potentially greener to run.

So the Green Software Foundation has this directory. And there's examples for platform engineers, for front-end engineers, for database maintainers, SREs. There's all these different examples for ways to tackle this. And another great-- I really wanted to mention the "Building Green Software" book by Ann Currie, Sarah Hsul, and Sara Bergman. This book is such a good primer for different ways that engineers can be involved with sustainability. I contributed to the operational efficiency chapter, so it includes a lot of the things that we're talking about today.

There are some very interesting experimental approaches to cloud sustainability that are about spatial-- so it's about load shifting, so shifting workloads in regions where the cloud intensity is lower. I think that one is a little bit harder to implement, although I like to talk about it. It makes us think a little bit more about the physicality of the data centers that we run our software in.

A lot of folks in this area talk about, for example, like Sweden. Could you just run your software in Sweden? Because the country, the electricity grid is powered by renewables. It's food for thought. We should also talk about sovereignty. We should talk about the cost of this regions. We should talk about availability. We can talk about so many different factors that could affect this decision.

And also, I think a lot of engineers don't have, potentially, the choice of the region available to them. But that's how far we can go with this discussion, basically, if we're maximizing for carbon, that's the most extreme level of carbon optimization, I would say.

ABDEL SGHIOUAR: Of course, I mean, it doesn't necessarily have to be always-- I mean, I'm just assuming here that it doesn't have to be always all or nothing, because what you could do is you could use certain cloud regions for certain workloads that are maybe compute intensive, where it would be more energy or more sustainable to run them there, and especially if you don't care about where the workloads are, the locality of the workloads, how fast those workloads will be running, things like that.

I think part of this conversation also would be, you know, if you don't need to run your pipelines during the day, you run them during the evening because it's either cheaper, or the grid has less pressure, or something like that, right? That also would be probably one of the conversations, right?

NIKI MANOLEDAKI: That is definitely, yeah, scaling down when you-- either scaling down, making it easier to scale down through autoscalers or scaling down more intentionally based on the carbon intensity of the grid, or when, I don't know, on weekends, or scaling according to the carbon intensity. You made a great point on latency before.

One of the things that I'm very interested in when it comes to load shifting based on carbon intensity is the energy cost of networking. That's kind of a blind spot. It's very hard to measure that. And that takes me into the measuring versus estimating aspects that is another interesting trade-off that we can talk about.

We've been talking a lot about bottom-up cost metrics, utilization metrics, bottom-up, I mean things that are important for engineers, maybe metrics that are coming from the infrastructure itself, rather than top-down numbers that are, for example, billing numbers, billing cost data.

And where I'm going with this is that it's very important to find accuracy. And the way to do that is by showing the variance between these bottom-up metrics and these top-down metrics. So also with carbon, for example, carbon is usually a top-down metric. It has to usually be accounting level accurate if it's using for what we talked about earlier, if it's used for regulation. So there's a big emphasis on carbon being a top-down number rather than a bottom-up number.

But as a bottom-up number, it can be achieved, for example, using Kepler, which provides the energy estimates or measurements. And then you can take energy and multiply that by a carbon coefficient, for example, looking at the carbon intensity per watt in a region and multiplying the energy with that carbon coefficient. So it can be achieved in different ways.

ABDEL SGHIOUAR: Got it. I want to go to Project Kepler. But while you were talking, one thing that jumped to mind, I'm coming from the data center world in my background. And I think you said something that resonated with me, which is like measuring versus estimating, because measuring can only go as far in measuring what your infrastructure does.

But the internet is built on top of layers of dependencies. And you don't necessarily know what the underlying layers are doing, right? So your cloud provider might have carbon neutral or carbon friendly zones, but they themselves will depend on dark fiber providers that have equipments that might be powered by humans biking or whatever, stuff like that. So I guess I assume it's very hard actually to-- you can't get like the measurements you want all the time.

NIKI MANOLEDAKI: Yeah, we talked a little bit about bottom-up versus top-down as a trade-off or two general categories. Measurements versus estimates is the other trade-off that I look at a lot. Especially with energy, it's very, very relevant because you usually cannot access the real energy measurements of the hardware you're using if you're a cloud user.

If you're running bare metal, it's a no-brainer. You can get those measurements. If you're a cloud user, it's way more difficult. It's a huge challenge. And that's where Kepler comes in, basically.

ABDEL SGHIOUAR: Yeah, so let's talk about Kepler. What is Project Kepler? And where do they fit in all of this sustainability discussion?

NIKI MANOLEDAKI: Kepler is a Kubernetes pooling. It's actually an acronym. Let's see if I can say it without looking at anything. It's Kubernetes Efficient Power Level Exporter. So it's an acronym for this. He's also a really cool astronomer who wrote a thesis about snowflakes. And I thought that was really fascinating. He was also a sci-fi writer.

ABDEL SGHIOUAR: OK, I didn't know that.

NIKI MANOLEDAKI: But Kepler is a tool. It's used for monitoring energy from power meters. And then there is an attribution model that is it goes from-- once you have the power readings, you can then associate that to a process, like a kernel Linux-level process, map that to the associated container, which is then mapped to a pod, which can then be aggregated to a namespace and other Kubernetes internals. So it's then also a Prometheus exporter. So you can then access those metrics in your Kubernetes environment through Prometheus.

There's two main parts to it. The first one is the power meter. And then the second part is the attribution model. It is a complex tool. But what it allows us to do is to plug in different kinds of power meters at the-- when you're configuring it.

And that can be, for example, RAPL, which is a Intel technology. It's the Running Average Power Limiter that gives estimates for power at the CPU level. And once you have access to that, you can then scale those energy readings in yourd up to the namespace or cluster level. But again, it's an estimate at the RAPL level.

There's other power meter backends, if you want to look at it. I like to think of it as a back end, like what are we plugging into? IPMI is the node-level power meter, so that's more of a measurement. That's not an estimate. That's a measurement.

But that is more for the bare metal use case, right? And what we did recently in the project, we had a big rewrite because we had EBPF as a layer in the tool specifically to read the processes. Yeah, EBPF is great because it gives you a huge amount of granularity, but it was actually causing a lot of limitations.

One of them was that it required elevated privileges. And it also was not giving us the most amount of accuracy for terminated processes or short-lived processes. It was just missing some processes. So the team, and also shout out to all of the Red Hat and IBM folks who have been leading Kepler for so long, the Kepler maintainer team decided to rewrite Kepler and remove EBPF.

And now, we're running some experiments to validate the accuracy of Kepler. And the way we're doing that is by deploying Kepler on both using RAPL and IPMI and comparing the power metrics to-- yeah, the readings to assess whether we're-- the variance. Again, the variance is so important.

And leading me back to bottom-up versus top-down, estimates versus measurements, RAPL is kind of a bottom-up number and estimate. And IPMI provides the top-down measurement that we want to compare to get the variance.

ABDEL SGHIOUAR: Got it. And so when you were talking, because I know what ipmi is, I don't know what RAPL is, but my understanding is IPMI will only be useful if you're running on bare metal, right? That doesn't work if you're running VMs. So is that like a bottleneck or a limitation for Project Kepler itself, or for at least the IPMI implementation?

NIKI MANOLEDAKI: Yeah, so the IPMI implementation is for bare metal. We have a couple of projects that are using Kepler on bare metal. Before I move into the bottleneck and limitation, I wanted to say that at this level, on bare metal, Kepler is doing really well.

For example, Project Silva, it's a Linux Foundation Europe project comprised of many European telco carriers. And they're using Kepler in their architecture, on their bare metal architecture, to optimize some of their network functions. And that's beautiful. It works. I think that's a fascinating example.

And the other one is CERN. Again, in their talks at Kubecon, they presented how they are using Kepler on some of their bare metal machines to reach their sustainability KPIs. Now, the bottleneck really happens when you move into virtual machines in public cloud. That previously, before the rewrite, we had trained-- we had a model that was trained, so that we can deploy Kepler on virtual machines when it has less information available.

By that, I mean it doesn't have access to RAPL. RAPL is blocked in public cloud VMs at the hypervisor level because-- well, if someone really wants to go into the nitty gritty of why that is, there's something called the platypus attack. And it's a security issue where if you have access to power readings at the kernel level in a virtualized environment, you can-- it's something about being able to guess encryption that-- some of the encryption that is happening at that level.

I don't know further than that, but point being, it's not possible to access RAPL in public cloud environments. And so for that reason, we had trained-- we had a model trained. It worked well. I mean, it could be optimized. And we're talking with some AI sustainability folks to improve the model. But after the rewrite, we have to train it again. And this is what we're working on at the moment.

Of course, all of this depends on having accurate energy metrics without the model in the first place. And this leads us back to the rewrite and removing EBPF. So we are at this point where we rewrote the tool, we've improved the accuracy. Now, we need to train the model again so that we can deploy to VMs.

We're so close. I'm so excited for when we're on the other side. And we can get folks using it again. And because it has led to so many great architectures, as I mentioned before, Kepler can be used with carbon coefficients to not only get the energy of your cluster, but also potentially, hopefully in the future, it could be used with carbon coefficients to get the total carbon footprint of a cluster.

ABDEL SGHIOUAR: Got it. So I'm going to play a little bit of devil's advocate here. So RAPL being blocked for security reasons, I understand that. Well, I think the other problem that you probably will have or will face, even if you're trying to measure public cloud virtual machines, is that most public clouds, what they do is that they try to create an abstraction layer for the CPU architecture you get on the VM versus the actual CPU architecture running on their bare metal servers.

Because they don't necessarily want to expose what type of CPU they run. And the reason I'm mentioning this is because we've done some experiments with the projects I was working on, where we are running on top of Kubernetes, on Google Cloud, in multiple regions, and we see very, very widely different performances on the app, just because you're landing on different CPUs. And you have no control over that, right?

So because of this experiment, I'm assuming that even then the-- even if you get the real measurements, you're not going to be able to-- they're not going to be as accurate as you want them to be. I don't know. What do you think about that?

NIKI MANOLEDAKI: I mean, I wish I could know how close we are to the underlying CPU infrastructure, how far off they are. Again, to be able to understand the variance and how far we are. At some points, we do need to accept the variance and that it's an estimate. And that this estimate is not going to be as perfect as we'd like it to be.

But yeah, you definitely are making a great point about the next challenge. There's so much abstraction. It's never going to be a measurement. It's going to be an estimate.

ABDEL SGHIOUAR: Yeah, and also, I mean, if somebody wants to go into more kind of geeky details about this, even I think on bare metal, one thing that I learned from my experience is that if you buy an off-the-shelf server, the power rating for the power supplies of the server are not-- the advertised one are not necessarily what the power supply can do, right?

The hardware manufacturers will always underestimate or underreport the performance of their power supplies. And so if you are wondering why I'm bringing this up, because of course, the rating of the power supply has a direct impact on the server itself, right?

And most power supplies are doing AC to DC conversion. So there is like a little bit of a loss in energy because you're converting energy from one form to another. So I think these multiple layers of abstraction makes it even harder, right? So the reason I'm mentioning this is I'm trying to convince myself that estimation is also actually fine and convincing the audience, obviously.

NIKI MANOLEDAKI: Yeah, we also have a lot of university-based research groups that are researching Kepler and publishing papers in the IEEE, ACM. So it's definitely a research stage right now. I think this is kind of a technology that will continue to evolve in the next decades. It's definitely a long-term thing. And we're trying to stay at the very edge of this evolution of the energy usage of the equipment that we use.

And as soon as we solve one challenge, then we're faced with the next one. I know I was just at the Green IO conference in New York, which is really big for the sustainability and ethics of cloud computing and AI. And some of the questions that were posed were OK, when is an estimate enough so that we can start to optimize? Because we can't be chasing perfect numbers. We need to be optimizing as well.

ABDEL SGHIOUAR: Yeah.

NIKI MANOLEDAKI: I think there is value in improving the accuracy of measurements and estimates and taking this as far as scientifically possible. And that's where I'm happy being in that space. I'm also trying to let folks optimize with the data that is available.

So same as you, I'm trying to convince myself and others that it's OK to use estimates, that we're not going to have all of the numbers. We're not going to have all the accurate numbers for the overhead or for, I don't know, networking, for example.

And I think this is a good segue into AI costs because it's up and coming. I think a lot of folks are wondering where that's going. Where are we? It's a unique, new use case for monitoring, optimization, green ops, cost, everything.

I will mention a few of the aha moments that I've had while I'm working on this. And specifically, I'm working on this from the perspective of building cost monitoring solutions. The first one is it's important to remember that if you're using private, proprietary LLMs, you're likely making API requests. It's not something that is-- you're not accessing the hardware the way you would be accessing a virtual machine.

So we had to shift our mindset because for the cost metrics that we've built, for example, for cloud resources like Kubernetes, we were using pricing metrics times utilization metrics. And for example, in a virtual machine, you can get pretty good utilization metrics.

What does that look like in LLMs that are hosted elsewhere? It's an API request. And your utilization metric becomes tokens. So familiarizing ourselves with input tokens, output tokens, et cetera is-- it's greenfield. It's new. It's hard to measure.

Now that is another set of abstractions because we don't necessarily have access to what's happening on the other side. That was definitely a big mindset shift in terms of cost monitoring, very different from other types of cloud resources, for sure.

ABDEL SGHIOUAR: I mean, you're bringing up a very good point. So let's say that you are in a situation where you're consuming cloud-hosted LLM. So yes, you have to kind of shift your mind around API calls, but the cost of input and output tokens, which most probably, you don't know how they're going to look like upfront, which makes it hard.

And it makes it hard even before we get to the cost estimation, it makes it even harder to write your code around that, right? Like how do you talk to service that you don't know what's going to return? But about that particular thing, so I'm wondering, I'm using a cloud-hosted model, my API request would probably be the same as calling a backend, but the variety in carbon emission is the serving itself.

Because the most expensive part of large language models is serving. It's not training. It's like when we serve the model. So how do we factor that in? How do we-- I'm not sure if my question makes sense. How do we reason about that in general about-- is it more cost efficient to run the large language model yourself, or is it more-- cost and sustainability efficient, let's just put it this way-- or to just use a cloud-hosted one? So what's the-- do you have a mental model, I would say? Because I'm also struggling with that.

NIKI MANOLEDAKI: Gosh, yeah, that's the main question right now that everyone is discussing. I look at experts in the field like Dr. Sasha Luccioni, she was at Hugging Face until recently. Her papers have over 300 citations, which is a huge--

ABDEL SGHIOUAR: That's a lot.

NIKI MANOLEDAKI: Huge amount, just to paint the picture of this researcher's profile. She built AI energy scoring methodology that is super interesting to look into. But of course, we only have those numbers for models that are open source.

And the closed source ones are-- a lot of folks are trying to estimate and compare them to other models. Or they're trying to deduce as much information as they can on the energy and carbon footprints of these models. But it's a black box, just like the API request aspect that we were talking about before. We need more information in order to make accurate decisions.

ABDEL SGHIOUAR: Yeah, I guess part of what makes it a black box is between you and the model. You don't know what's happening, right? There might be caching. There might be more than one model. There might be efficient model and not efficient model, mixture of expert kind of models.

And also, you don't necessarily know which hardware it's running on. You don't know-- you don't know a lot of things. So I like this idea that you guys are looking at the resource side of things. So you can take from that and maybe try to implement it in Project Kepler. So that's pretty cool.

So I do have a related question. I think it's just my initial question rephrased. How should people reason about consume a large language model versus host my own large language model from a sustainability perspective?

NIKI MANOLEDAKI: The thing I'm hearing more and more is that eventually, we will be shifting to a future where we wouldn't need the generalist model that can do it all. But if you're conscious about sustainability, you would be using the right model that is suited for the task.

ABDEL SGHIOUAR: Specialized models, basically, yeah.

NIKI MANOLEDAKI: Specialized models. And that is where the research is going. That is where the conversation is going in terms of sustainability and models. But that's if you're able to self-host models. Otherwise, if you're using cloud-hosted proprietary LLMs, then we're moving into can we multiply pricing metrics times token utilization metrics and try to get the best estimation for usage and go from not having any visibility to having even a little bit of visibility and attribution, which is better than nothing.

So you can start looking at what's taking longer, what's costing more. And I think, again, we have to be OK with some level of variance, and abstraction, and not having the best number available, but just enough so that we can take decisions, and make actions happen, and make some reductions happen. So it's the trade-off again.

ABDEL SGHIOUAR: Got it. And I mean that's a good note to end on. The only thing I want to add here is one possible future that we're heading into is that it's possible that those models are going to be so expensive that no one can afford them, because it is super clear right now that most AI companies are running them as a loss. They cost way more than they are charging users, for sure.

So it's not possible that you're buying a subscription for $80 and you can do like millions of tokens. They're losing money on that, for sure. So I mean, that's just my estimation, again. I'm not an authoritative voice in this space.

But I think going forward, either things are going to be substantially cheaper, then they're going to be sustainable. Or they're going to be so expensive that no one can afford them and no one can use them, therefore.

NIKI MANOLEDAKI: My personal speculation on that is I see the challenges of the pricing. So I'm trying to build pricing metrics. I'm trying to build utilization metrics. These are very obscure. They're very complex. They're changing quickly.

It's hard to stay on top of some of these pricing models and understand what's happening and what we're priced for. So I'm just wondering how these pricing models are going to evolve, like you said. So I do agree that's a challenge. And we're dealing with technology that is evolving extremely fast these days. So let's see where we'll be in a year from now.

ABDEL SGHIOUAR: Yeah, maybe we'll have the same exact conversation, but with different metrics. Niki, thank you so much for coming on the show.

NIKI MANOLEDAKI: Thank you so much for having me. It was such a pleasure.

ABDEL SGHIOUAR: Where can people find you? Where can they find your work?

NIKI MANOLEDAKI: I'm primarily on LinkedIn. I'm also on the CNCF Slack. Join the Environmental Sustainability Technical Community Group, we have a channel, TCG Environmental Sustainability. And join also Project Kepler. We have a channel also on the CNCF Slack, #KeplerProject.

ABDEL SGHIOUAR: Awesome, thank you so much. [NON-ENGLISH]

NIKI MANOLEDAKI: [NON-ENGLISH]

KASLIN FIELDS: Thank you, Abdel and Niki for this one. It's always interesting to me to talk about sustainability. I think that sustainability and environmental impact of compute is important and interesting. And I've heard Project Kepler before, but I didn't even know at the beginning of this that was like a sustainability focused project. So I know a lot more about it now.

ABDEL SGHIOUAR: Yeah, I mean, me neither. And also, I didn't know how big of a thing was this, that green ops, that people cared about. It turns out a lot of companies do. So that's pretty cool, actually.

KASLIN FIELDS: Yeah, and I like that you all talked a lot about observability and the different methodologies and approaches one has to take to observe something as challenging as the sustainability, because you have to go through so many levels.

There's the cost optimization aspect, which is, of course, very interesting to businesses. But also equating your usage and the cost to the actual environmental impact of where are you using your electricity from? And how green is the energy on that grid? And thus, estimating from there what your actual impact on the ecology is. And you talk a lot about the trade-offs between measurement versus estimation. And there's so much estimation that has to go into measuring for sustainability.

ABDEL SGHIOUAR: Yeah I mean, I guess it's because part of it is doing like real-time measurements also consume compute cycles, which is not kind of productive probably. So at some point, you will have to just stop and think, you can't measure everything. You will have to either sample or estimate, or maybe a combination of both, right?

KASLIN FIELDS: And you talked a bit about the difference between measuring in perhaps more traditional compute use cases versus AI, and how the equation changes from measuring the usage of the compute, measuring in CPU units of whatever sort versus measuring tokens, which is what a lot of us are doing when we're consuming AI models is the unit of usage is tokens, and then trying to equate that all the way back to the sustainability impact of that usage.

And she mentioned a specific researcher that has done some really cool research on getting those metrics for open models, open weight models, I guess primarily, but doing that in an environment where you are consuming a proprietary model is so much harder.

ABDEL SGHIOUAR: Yeah, because you don't have access to the actual metrics from the provider of the model, right? So you can't really compare whether running your own model would be more sustainable than using a cloud-hosted model. That's why most companies will default to just measuring it based on cost, how much is my cost per token between consuming actual cloud-hosted model or running my own?

KASLIN FIELDS: You also mention that it feels like there's a trend toward more subject-oriented models, more specialized models. I'm very curious about that trend and to see where that goes. I think from a user perspective, the interfaces are very generalized and probably will continue to be very generalized. But as someone who helps to run the infrastructure behind it, oftentimes, that's just an interface to a bunch of more specialized models or agents. So it's an interesting pattern.

ABDEL SGHIOUAR: Yeah, I mean, there is quite a lot of open source projects already that are trying to solve this. So VLM has this thing called Semantic Router, which allows basically-- it's like a lightweight router that sits in front of the models that can route the requests based on the context of what the user is asking. Deepmind have been doing quite a lot of research in this. There's like a model router, which is like a small model that can route requests.

But definitely, I think part of the future would be having some sort of transparent layer that can translate the user requests into-- or forward user requests to the most cost efficient, maybe even sustainable, efficient from a sustainability point of view model, instead of like what's happening right now, which is people just type "hi" in a chat interface. And then you end up doing a round trip to the cloud to talk to Gemini 3 or ChatGPT 5, which clearly you don't need that.

KASLIN FIELDS: You also talked about how cloud providers tend to insert a layer of abstraction between the user and the actual CPU architecture, which makes measuring all of this stuff even harder. So there's all these layers that you have to think about, especially in these AI use cases that maybe we don't have a lot of data on. We don't have a lot of tools in order to drive all the way from the user to the sustainability impact.

ABDEL SGHIOUAR: Yeah, it reminded me of a talk I had prepared about two years ago. I still have the GitHub repo, which I'll probably link in this episode. So on Kubernetes, typically, on any cloud provider, you are able to choose the exact type of virtual machine you want to have on your cluster. So you know what kind of CPU, you know the CPU-to-memory ratio. You know all this stuff.

But the higher abstraction layers, you go with something like serverless, let's talk about Cloud Run as an example, Cloud Run gives you a CPU. And from the documentation, a CPU is a CPU. It doesn't matter where you are running, but turns out, no, it does.

KASLIN FIELDS: When you're talking sustainability.

ABDEL SGHIOUAR: Exactly, but even from a performance point of view, because when you're running on top of Cloud Run or any serverless platform, depending on which region you land on, you might have a completely different CPU architecture. So the performance could be different depending on which CPU your workload runs on. So it matters for performance. But it also, of course, matters for sustainability, right?

KASLIN FIELDS: Yep. Ultimately, the challenge of measuring sustainability is just a very comprehensive observability problem for your system.

ABDEL SGHIOUAR: I think so. I think that that's a very good way of putting it, yes.

KASLIN FIELDS: Yeah, I was thinking a lot about observability tools in the AI era. And I saw something from-- we've had Charity Majors on before from Honeycomb to talk about observability 2.0. And I've been thinking lately, I wonder what observability 3.0 in the world of AI is looking like to Charity right now.

ABDEL SGHIOUAR: Definitely sounds like. I definitely think part of-- I mean, this is something I have heard a lot of people in this space echo over time, is that it seems like the common wisdom in observability space is like collect all the logs and metrics, regardless of whether you need them or not.

KASLIN FIELDS: Which is even more stuff, which impacts sustainability.

ABDEL SGHIOUAR: Exactly, which I think the part probably of the future is going toward a space where we don't really need that. But how do you solve it is, I think, beyond my understanding of how observability actually works.

KASLIN FIELDS: It feels like we just need more and more data, which requires more and more memory and--

ABDEL SGHIOUAR: More storage.

KASLIN FIELDS: And memory is currently impossible to get.

ABDEL SGHIOUAR: Exactly, exactly.

KASLIN FIELDS: Yeah, so sounds like a hard problem that we learned a little bit about today.

ABDEL SGHIOUAR: Yeah, I mean, it was definitely a cool episode to record. It was really cool.

KASLIN FIELDS: Yeah, so check out Project Kepler and read some research papers from Dr. Sasha Luccioni about measuring metrics in AI systems.

ABDEL SGHIOUAR: Definitely.

KASLIN FIELDS: All right. Thanks, Abdel.

ABDEL SGHIOUAR: Thank you.

That brings us to the end of another episode. If you enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @KubernetesPod or reach us by email <kubernetespodcast@google.com>.

You can also check our website at kubernetespodcast.com, where you will find transcripts, and show notes, and links to subscribe. Please consider rating us in your podcast player, so we can help more people find and enjoy the show. Thanks for listening. And we'll see you next time.

[MUSIC PLAYING]