KASLIN FIELDS: Hello, and welcome to the "Kubernetes Podcast from Google." I'm your host, Kaslin Fields.

ABDEL SGHIOUAR: And I am Abdel Sghiouar.

[MUSIC PLAYING]

Drew is a senior DevOps engineer working on platform and cloud teams in Medtronic's Cardiac Ablation Solutions unit. His involvement with the SIG Release team started with version 1.7 and had led the release docs and release signal teams. Drew has previously worked with roles in US public service, spanning K-12 education and Department of Defense. He likes to spend his spare time unplugging in nature. Welcome to the show, Drew.

DREW HAGEN: Great to be here. Yeah. Thanks for having me.

ABDEL SGHIOUAR: Awesome. So this is part of our grand tradition on the show to always have the release lead of the upcoming version of 1.35, or any upcoming version, as a guest. And yours is very likely going to be the last episode of the year. So I guess let's start with this. How is it going? You are still in the middle of the release.

DREW HAGEN: Yeah, things are going well. It's a very chaotic, busy time of year across the board, releasing one of the biggest open source projects in the world, maybe the second. Don't quote me on what the metric is for that, but that's what I've heard, second to Linux. And then on top of that, being a DevOps engineer, sometimes thrown into SRE major incident stuff, too, I just kind of feel like my neck hurts from all the hats I'm wearing sometimes. And of course, we've got the holidays coming up as well. So I'm really looking forward to having this time off to spend with family and kind of turn my brain off a little bit from all of these things.

ABDEL SGHIOUAR: Awesome. At the moment we are recording, we are probably on the last stretch, actually, of the release. It's expected next week. Of course, things can still slip away. But few more days, I guess, right?

DREW HAGEN: Yeah, we actually released on Wednesday, so now the release is fully available. You can download it on GitHub. We have the announcement blog live now. So yeah, I'm excited to see what folks do with our next version of Kubernetes 1.35.

ABDEL SGHIOUAR: Awesome. So let's talk about it. And I usually like to start this with talking about the theme. So as of the moment of recording, the theme is live. The logo is on the website. Can you tell us a little bit about the theme, the inspiration behind it, like where this idea came from?

DREW HAGEN: Yeah, so we named the release name Timbernetes. It uses the world tree as a metaphor for Kubernetes as a global living system inspired by the tree of life, Yggdrasil, from Norse mythology. It honors the resilience and diversity of contributors around the world who sustain the project alongside their daily jobs and life challenges. The theme reflects where Kubernetes is today, reinforcing the core of the platform, setting deeper roots around security and stability, and the foundation for advanced workloads like AI going into the future.

And we're always expanding the branches to support new and more demanding workloads. It continues the narrative of resilience and magic from the last two Kubernetes releases, version 1.33, which was Octarine, the Color of Magic, and version 1.34, which was Of Wind and Will. So we had fun sort of continuing that narrative. I designed the logo myself. I had a lot of fun with it. We actually have some squirrels that are living amongst the world tree.

And kind of fitting the theme of magic, we gave them RPG themes that-- or RPG classes that reflect different release activities. We have the rogue triage squirrel. We have the tech wizard squirrel with a scroll that says "Looks good to me" on it. And then we have the branch management warrior squirrel. So he's cutting down the next version of Kubernetes.

So I wanted to pick something that was fun but also symbolic because, again, I think a tree is a really good symbol for the new foundation where we're rooting with the project right now and also the resilience of all these maintainers that come together with their day jobs, families that they support. I really wanted to honor the resilience of the community.

ABDEL SGHIOUAR: Awesome. I really, really like the logo. I mean, I like every logo of every release that comes out, but this is so much creativity with the Norse mythology, the magical aspect it has and also the representation of the different personas, I guess, of people involved in every release, right?

DREW HAGEN: Yeah.

ABDEL SGHIOUAR: It's awesome. So I guess my next question is, this is your first time being a release lead, if I'm not mistaken, right?

DREW HAGEN: Yeah. I think, typically, people, they have the opportunity to come through and be the release lead once. We have a subproject owner in SIG Release that oversees every version and supports the release lead. But yeah, this is my first time leading the release. I shadowed the release lead once, so every release lead gets three to five shadows to support them. So I had an opportunity of doing that once.

And then I also led two of the subteams. The release lead oversees multiple subteams. Right now, it's down to four. We consolidated some of them. But we have the Enhancement subteam, Release Communications, Release Documentation, and Release Signal, Release Signal doing the bug triage and the CI signal triage.

ABDEL SGHIOUAR: Mm-hmm. And so how was your experience in general, like how did you find it?

DREW HAGEN: Yeah, so I've been on the release team for three years. There were two cycles I took off. But yeah, it's been a really fulfilling journey, getting to bounce around some of the different subteams. I started with CI Signal at the time, before it was merged with Bug Triage into Release Signal. That's where I started, on version 1.27. And that was definitely a pretty busy subteam to start with because I had to dig into a bunch of CI builds that were set up all around the Kubernetes project.

And the Kubernetes project is organized in special interest groups. So there's a bunch of different groups that own different tests that are testing their features. And throughout the development and implementation, bug fixing, and whatnot, we would have to monitor CI builds to make sure that the project continues to be stable. And once we run into issues, we'd have to correlate with GitHub and Slack and reach out to the owners and make sure things were fixed right away. So that was a lot of work. There was kind of a great, fulfilling learning curve to it.

And from there, I continued to work on release docs and other areas of the release team. So yeah, I've got a lot of experience. I eventually started to lead some of these subteams and, like I said, shadow the release lead. So yeah, it's been really exciting to see this project and all the guardrails and all the people that come together and just this whole process that we have set up on delivering the second biggest open source project in the world. It was a great, great perspective and learning opportunity.

ABDEL SGHIOUAR: Nice. I do have a question, just out of curiosity, and I'm going off script here a little bit. Do you work with Kubernetes day to day as part of your main job?

DREW HAGEN: Yeah, I have. Right now, I've been more focused on AWS without Kubernetes, just acutely what I'm being assigned at work. But I've worked with Kubernetes platforms at my last two companies in the K-12 education and Department of Defense roles that I had. Now I'm in medical devices. And I just started this job earlier this year after a layoff and getting a new job, fortunately found something. But right now, we have a legacy environment in AWS, and I'm looking for opportunities to bring Kubernetes in. But yeah, I do have some exposure to it as a user, yeah.

ABDEL SGHIOUAR: So the follow-up question, then, is, having experience hands-on working with Kubernetes, does that help with the role of a release lead?

DREW HAGEN: Oh yeah, totally, because I think one of the things that the release team is most is most influential in is setting the quality gates for users, end users, making sure that throughout all the contributors in the project, we're hitting our guardrails with making sure we have all of our criteria for the Kubernetes enhancement proposals, making sure that we have stability with that release signal process, and making sure that all the new features are well communicated and the documentation is very comprehensive and easy to follow, and also making sure all the user-facing changes are documented.

So yeah, so I think it has been really helpful to be a user for Kubernetes because then I can come in with that perspective and empathy of, OK, what really matters to me as a user? What am I personally excited about to see, and what is it going to be like for the folks after we release this thing? So that perspective definitely helps.

ABDEL SGHIOUAR: Awesome, awesome. I ask this question mostly because I was wondering how much of your time do you spend really diving in versus how much do you spend just coordinating? My previous interviews, there is a lot of work that goes into coordinating between the different teams as a release lead. And do you find your time kind of geeking out to try to figure out what's wrong or having to remind yourself, maybe that's not worth my time? How was your experience?

DREW HAGEN: Yeah. Yeah, I will say definitely coordination, I think, is the biggest thing that the release lead takes on. I had a wonderful team of people to work with throughout this release, both the shadows that were supporting me, the leads that were supporting me on SIG Release as well, that are consistently there release to release, and then also the subproject or the subteam leads, with, like I was saying, communications, signal enhancements, and docs. They were all really great.

And most of the time, when I did start geeking out and digging deep, I would find that my team was doing excellent work. I really had nothing to worry about, and my trust was in good hands. So a lot of what I did was delegation and kind of wrangling cats around the project, if you will, making sure everything's going on the timeline and is going smooth. And even just having the shadows to back me up, they would also-- and being in different time zones, people would pick up on things even before I did. So a lot of it was being an arbiter, making decisions, and, yeah, coordinating things. But again, as an end user, I also was really excited to see what the project was doing and where it was going.

ABDEL SGHIOUAR: Awesome, awesome. Well, I want to spend a little bit of time diving into this new version. I'm going to start here with some stats, and then we can go a little bit into the details. This is 60 enhancements. So that's 17 features go into stable, 19 go into beta, and 22 go into alpha.

And then there is a bunch of deprecations, roughly around, from what I can see here, four or five, I think. And the major one, obviously, is the NGINX Ingress retirement, which will be coming up. The project will switch to best maintenance efforts in 2026, and then it will be fully gone in the future. So let's talk about the new stuff. What can you highlight? What are the things that excite you?

DREW HAGEN: Yeah, there's a lot of really great features that I'm excited about here. I think I'll start with our blockbuster feature. And I mean, this is just something I'm excited about. It's the first thing that we have mentioned on our announcement blog post, and it enables smarter scaling for Kubernetes clusters. That's the in-place pod resource updates-- that is KEP-1287-- going into stable. And what's really powerful about this is that we can make in-place updates to adjust the CPU, memory requests, and limits on the pod without restarting it. It's all updated on the fly.

And that's really big because, as a DevOps engineer that has worked with Kubernetes clusters, I'd have to go in and modify those things and completely flip the pod, which would cause an interruption for whatever application or service I have running on it. So I think this is really huge, as we're running a consistent workload that needs to dynamically adjust itself and vertically scale up, that we have that capability to update it without any outage.

So I'm really excited about that. A lot of good work was put into that. We also have great new features for the scheduling of workloads. We introduced gang scheduling support. So that introduces a new workload object that will group a bunch of pods together so when they're scheduled, they're either scheduled all at once or not at all, which I think is really powerful for AI workloads that might have a bunch of training jobs that are coordinating with the same stateful data set.

In cases like that, there's definitely a lot of coordination and dependencies there, so it's good to have Kubernetes finally be able to do that natively without having to have third-party tools that have to coordinate all of those pods being up at the same time. We also have extended toleration for threshold-based placement. So we can use numerical comparisons to determine what nodes a pod will get scheduled on.

So that could be really big. Say, for instance, you have on-demand nodes or spot instant nodes. You can maybe give a scoring system to that. And then at scheduling time, the pod could determine, OK, I really need something that's reliable and stable, so I'm going to do on-demand rather than a spot instance. There's node declared features, so nodes can have a lot of different versions or underlying operating systems on their hosts and, depending on that, will determine what sort of Kubernetes features those nodes can support.

So we finally have this capability for nodes to declare the features that they do support at schedule time. And then lastly, we have opportunistic batching, which is really great for making some of the same scheduling decisions for pods and jobs that are very similar to one another, so they can get scheduled much faster when it's time to be scheduled again.

And that's not even getting into the security updates that we have with pod certifications, building a better impersonation model so that machines won't join the cluster and pull sensitive information from pods. Introducing user namespaces, so any pods that require elevation to root access will have their own user ID that they're running on the machine, so they're not elevating to full root on the host. Yeah, just a lot of really great features I can go on and on about.

ABDEL SGHIOUAR: Yeah, it's actually interesting to me that a lot of these features that I've been following very closely, because they all sort of have their own unique characteristic and unique use cases, are falling together in this particular release. It's things that have been in the making for a while. You mentioned the in-place pod update. We've been talking about that for a very long time. And to me, it's not an end goal, but it's only just the beginning of a much, much more interesting set of features in the future, like CPU boosting, like a lot of interesting things that you could do, especially for AI workloads.

DREW HAGEN: Mm-hmm.

ABDEL SGHIOUAR: I didn't know about the node-- what was the feature you talked about? The node--

DREW HAGEN: Node-declared features?

ABDEL SGHIOUAR: Yeah, the node-declared feature. That's very interesting. so the node will be able to say, I support this and this workload. That sounds like taints and tolerations, but done better, in a way.

DREW HAGEN: Yeah. Actually, just to clarify, and folks that want to look into this as well, it's KEP-5328. And so this is the actual feature gates that Kubernetes has. A node could be set up to declare what features it supports. I think this is pretty powerful both for AI and for edge computing.

For AI, maybe there are nodes that have a capable GPU that can support those training instances. Then it can declare that it has that available. For edge, you might have a data center that's on the edge that has a lot of different machines that are being used, and it can be kind of hard to manage those with a limited ingress and egress. So being able to label what features that each of those nodes support, I think, is going to be really powerful for these advanced use cases.

ABDEL SGHIOUAR: Yeah, it's a super interesting feature. I was skimming through the blog, and I think among all of this stuff you talked about, one of my favorite features is the pod certificate for workload identity. So basically, being able to supply an identity to pod in an automated way. That's has been, I think, one of the biggest headaches for doing multi-tenancy in Kubernetes for a while, right?

DREW HAGEN: Oh yeah, totally. And there are a lot of third-party tools that you would bring in, maybe Aspire cert manager or whatever. So that's something I'm just really excited about with 1.35 is to see the simplification that we're doing with cluster architectures, making some of these features native so that we don't have to pull in these complex tools.

ABDEL SGHIOUAR: Yeah. And another one that I was just literally looking at today is being able to use images as a storage source. For AI workloads, this is a huge problem, right? AI--

DREW HAGEN: Yeah.

ABDEL SGHIOUAR: These large language models are huge, and you have to somehow make them available to the pod as a volume. But now being able to base the volume itself on an actual image that you pull from an OCI registry is super, super cool.

DREW HAGEN: Yeah, KEP-4639, OCI artifact as a volume, going into beta.

ABDEL SGHIOUAR: Yeah.

DREW HAGEN: Yeah, this is really exciting for edge computing as well, to be able to attach a data set as a container image. I think that's going to be really big for the packaging of the workloads that you would do, is you would roll it out. Maybe you have some removable media that you're using to install those workloads on the edge. So yeah, I'm really excited about that as well.

ABDEL SGHIOUAR: Yeah, and also being able to package both your application and the data it needs as a single format, so you don't have to package them as two different formats, right? So pretty cool.

DREW HAGEN: Yeah, for sure.

ABDEL SGHIOUAR: You are talking a lot about edge. Are you using, or did you have experience working with edge use cases for Kubernetes?

DREW HAGEN: Yeah, at my prior job in the Department of Defense, I worked for a company called Defense Unicorns. It was a startup working on, basically, the continuous delivery of workloads in Kubernetes in air-gapped environments, so things like a submarine or like a fighter jet. So yeah, the Department of Defense has a lot of great use cases for deploying things in an edge environment. So I learned a lot in that role and learned enough about how they do things while I was there to be really excited about edge computing.

ABDEL SGHIOUAR: Nice. Yeah, working for a cloud provider, I don't really get an opportunity to work on use cases with edge computing, so it sounds like something super interesting, actually.

DREW HAGEN: Yeah, I also earlier in my career worked on a point-of-sale system, but that was before I was getting really deep in Kubernetes. But I've read these articles and heard these stories of places like fast food or retail that are deploying Kubernetes clusters on site and managing everything. You'd go and check out your shopping, and it'd be running through a Kubernetes cluster that's completely on site. So that's another awesome use case for edge.

ABDEL SGHIOUAR: Yeah, to my knowledge, I think the first company to come out with this use cases-- they blogged about it-- was Chick-fil-A in the US.

DREW HAGEN: Yes, yeah, that's what I'm thinking about.

ABDEL SGHIOUAR: Yeah. And then it was followed by McDonald's and then a bunch of other companies, and then it suddenly opened up this thing. Oh yeah, we could actually deploy Kubernetes on the edge or in a submarine or in a fighter jet or whatever, right?

DREW HAGEN: Yeah, I mean, I think even Target, some retail companies, are running their point of sale through Kubernetes. And yeah, it's very exciting stuff.

ABDEL SGHIOUAR: Awesome. We're going to make sure to leave some links for people who want to read about this stuff if they want. I mean, this is awesome. I really love how you dived into some of these features. Anything else you want to talk about? Deprecation of NGINX Ingress in 2026, wink, wink?

DREW HAGEN: Yeah. I know that's a really big topic. It's not strictly a part of version 1.35. I'm happy to comment on it, though, because as someone who has been a Kubernetes user for a long time, that's something I've used for a while. I think it served us all well. I think it really brings up a good point about the sustainability of open source that we need the active maintainers available to support these things.

And I think with NGINX Ingress, we saw that it's just hard to continue safely without enough of that maintainer community. And I'm really excited, though, because the community is working towards long-term sustainability and maintainability around other features like the Gateway API, which will be really powerful for tackling some of the same use cases. So anybody that is on NGINX Ingress, I recommend checking out the Gateway API.

ABDEL SGHIOUAR: Yeah. It also brings up a very interesting point about the fact that the maturity of the community means that people care about simplicity and that, at some point, it's just not worth it to continue spending time on something that is unmaintainable. And it's just better to just stop working on it and move on, right?

DREW HAGEN: Yeah, it's definitely a tricky thing, but that's the beast of open source. And Kubernetes just being such a big project and there being the CNCF landscape largely revolving around it and everything is that there's so many solutions that are out there. And we wanted to give time for folks to investigate those and transition off.

ABDEL SGHIOUAR: Awesome. Well, Drew, thank you so much. This was really insightful. I hope you will have some time to relax over the holidays.

DREW HAGEN: Me too, yeah.

ABDEL SGHIOUAR: Anything else you want to add? Anything you want to tell our listeners?

DREW HAGEN: Yeah, I just wanted to give a heads-up. If you're an operator, I wanted to call out some of the deprecations that we have. We have removed support for cgroups v1. So your host OS on your nodes will have to support cgroups v2. We also are deprecating ipvs. That's another thing that's been a burden to maintain, so I suggest switching to nftables.

And then this release is the last call for containerd 1.x. We've transitioned to 2.x a long time ago. I think it was version 1.25. But now we're completely removing it in 1.36. So I thought it would be prudent for me to give a shout out to those things so that we can all check to make sure that we're ready to run this next great version of Kubernetes and unlock these good capabilities.

ABDEL SGHIOUAR: I mean, definitely, these are very good things to shout out, because we all know that not everybody is at the edge of upgrading to the latest and greatest version. So the more you delay--

DREW HAGEN: I've been there.

ABDEL SGHIOUAR: [LAUGHS] I mean, the more you delay it, the more it's going to come to haunt you at some point.

DREW HAGEN: Yeah, I've been in roles before where it's like, wait a minute, we're end of support? Oh, we're end of life? Oh, great. So yeah, no, I feel it. I wanted to call out these things just to make sure it's on your radar.

ABDEL SGHIOUAR: Awesome. Well, thank you so much, Drew.

DREW HAGEN: Yeah, thanks for having me. It's been a blast to geek out about this stuff.

ABDEL SGHIOUAR: Awesome. Thank you for coming on the show. I hope you will take some time to relax and chill over the holidays. I hope that our listeners will have time to do the same. And this is, as I said, going to be the last episode of this year. So we're going to also take some time off. The episode will come out next week. And then next week, as in, we're Friday, so next week somewhere. And yeah, we'll see you all in 2026.

DREW HAGEN: All right, sounds good. Happy holidays, everyone.

ABDEL SGHIOUAR: Happy holidays.

[MUSIC PLAYING]

KASLIN FIELDS: That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @KubernetesPod or reach us by email at kubernetespodcast@google.com. You can also check out the website at kubernetespodcast.com, where you'll find transcripts, show notes, and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.