KASLIN FIELDS: Hello. And welcome to the Kubernetes podcast from Google. I'm your host, Kaslin Fields.

ABDEL SGHIOUAR: And I'm Abdel Sghiouar.

[MUSIC PLAYING]

KASLIN FIELDS: Today we talk with Kat Cosgrove, the release lead for the Kubernetes 1.30 release. We'll go over what's new and what's exciting in this release, including some very old issues that are finally being fixed. And I'm really excited about. So stay tuned for more on 1.30. And make sure you check out the show notes for the links to all of the blog posts with more information. But before we get into 1.30, let's get to the news.

[MUSIC PLAYING]

ABDEL SGHIOUAR: Google Cloud hosted their flagship Google Cloud Next event April 9 to 11 in Las Vegas, Nevada. The event was packed with announcements, like the new Google axiom processor, a type of ARM-based CPU, Google vids, and AI-powered video creation app for Google Workspace. Check out the recap links in the show notes for more details.

KASLIN FIELDS: Amazon EKS announces general availability of extended support for Kubernetes versions. You can run EKS clusters using any Kubernetes version for up to 26 months from the time the version becomes available on EKS. Extended support for Kubernetes versions is available for Kubernetes versions 1.21 and higher. The pricing for the extended support window is $0.60 per cluster per hour.

ABDEL SGHIOUAR: The Kubernetes project introduced a new Windows operational readiness specification. The goal is to have better support for Windows workloads by adding testing to the project.

KASLIN FIELDS: And that's the news.

[MUSIC PLAYING]

I'm Katherine Fields from the Kubernetes podcast from Google. And I'm here at KubeCon CloudNativeCon Europe 2024 in Paris. And we're going to do our release interview for 1.30. I am here with Kat Cosgrove. Welcome to the show, Kat.

KAT COSGROVE: Thank you so much. I love your NPR voice.

KASLIN FIELDS: Thank you.

KAT COSGROVE: You absolutely sound like you should be speaking on NPR. It's very good.

KASLIN FIELDS: It is my absolute favorite compliment. Thank you so much. [GIGGLES] So you're the release lead for 1.30. How is that going?

KAT COSGROVE: I am indeed. It's a lot, actually. I feel like you always get warned that doing the release lead job is a lot of work. The handbook certainly tells you it's a lot of work. It requires buy in from your employer. This is all I'm doing right now. Dell basically cleared my schedule for the quarter to run this release. But no matter how much people tell you that you're going to be busy and that it's stressful, I don't think anybody ever really internalizes how much.

KASLIN FIELDS: Yeah.

KAT COSGROVE: Just talking to previous release leads, it's kind of like a consistent theme, that it's like, wow, yeah, that actually is a ton of work. It's a lot of cat herding. If you think of it like managing actual employees, I have like nine direct reports right now and something like 30 indirect reports. So it's quite a large team to manage. There's a lot going on. It's busy. I'm surviving. But it's busy.

KASLIN FIELDS: One thing I like to say about open-source work is the deeper I get into my open-source work, the more I realize that Kubernetes or any large open-source project is just a company without the company--

KAT COSGROVE: Pretty much.

KASLIN FIELDS: --business part.

KAT COSGROVE: I mean, Kubernetes is more businessy than a lot of open-source projects, just by virtue of being so large. It's the largest Golang open-source project in the world and one of the largest open-source projects in general. So we have a lot more going on than other projects do. But, yeah, it is quite a lot like an org chart.

KASLIN FIELDS: Yep. And speaking of which, I was going to ask you later on, but I think it works well here. So I always like to mention at the end of the release interview that the next release is going to be coming up. And that means that the application for release shadows is going to be open because the release team does use a shadowing application process in order to help train new folks to be able to do the release. And there have been some changes to that this cycle.

KAT COSGROVE: There have, a couple of relevant changes. We got rid of one of the release sub teams. So there used to be six release sub teams, enhancements, docs, comms, release notes, bug triage, and CI signal. And when we switched over to using GitHub project boards, it meant that bug triage doesn't really have a lot of work to do. So we merged the bug triage and CI signal teams into one team, called release signal.

So from 1.30 onwards, there are roughly five fewer spots on the release team for shadows. And we get so many applicants, more than we can handle. It is highly competitive. In a way, it does kind of suck that we have slightly fewer spots. But ultimately it's a good thing when we automate ourselves out of like paperwork and minutia. So sorry about that.

KASLIN FIELDS: But it's a good thing.

KAT COSGROVE: It's a good thing. It's a good thing. It's better for the health of the project. We have also formalized a way to determine what is considered inactivity on the part of a shadow or a sub team lead and a way to remove them. We don't like doing this. I don't relish the opportunity to do it. It is unpleasant.

But the reality is that it is not uncommon for shadows to disappear, go AWOL, just stop responding, stop doing the work. And when that happens, it creates a lot of extra work for the other shadows and for the sub team lead, who's usually the one responsible for picking up that shadow slack.

So then the team lead is doing their own job and the shadows job at the same time. It's not fair. So we do now have a way to remove an inactive shadow or sub team lead and replace them with somebody else, usually with a release lead shadow-- that's one of the things release lead shadows are for-- or with a shadow from a previous Kubernetes release.

Again, I don't want to have to do this. I really don't want to have to do this. But for the health of the release, for the health of the project, we have to have a way to like fully staff the team at all times. And that's not to say that you're always at risk of getting fired, that we can't work with you, it's just that we need you to communicate it.

So like if you ever, for any reason, need to step back from the release, even if it's temporarily or if it is for the entirety of the release, you can always do that at will by just talking to your sub team lead or talking to the emeritus cAdvisor for the release. And it does not negatively impact your future on the release team.

A previous release lead, James Laverick, he ran 1.27 or 1.26, I think, but when he was the release notes lead in 1.20, he stepped back. He was leading a sub team, and he needed to step back. And he came back a few releases later to lead the whole thing. It is not a bad thing. We consider it a good thing and a sign of emotional maturity and responsibility, leadership quality, that you're willing to admit that you can't handle it and take a little break for a while.

So you can always step back willingly. If we have to remove you against your will, we can and will do it now. And that will affect your prospects on the release team. So please don't do that to us.

KASLIN FIELDS: I know when I was a shadow for the comms team, I definitely saw that happen with some of the shadows, that just like, you got accepted to the shadow program, which is really difficult to do because, like you said, they get a lot of applicants. So it's really an honor to be selected for the team.

KAT COSGROVE: 100s.

KASLIN FIELDS: And then some of them just don't show up. One way that it's not like a business is you're not getting paid by the project for this work.

KAT COSGROVE: It is a fully volunteer team. And we have to be cognizant of that. And the fact that it's a fully volunteer team does introduce some additional struggles.

KASLIN FIELDS: So updates to the process there. Just be aware of it. If you do want to apply to be a release shadow, look out for the shadow application for 1.31--

KAT COSGROVE: Please do.

KASLIN FIELDS: --which will be opening soon, I'm sure. But right now we're talking about 1.30. So what all is coming up in release 1.30?

KAT COSGROVE: Oh, my god, so much. We have 11 alpha, 18 beta, and 17 stable KEPs tracked after a code freeze. So it's not a massive release, but it is a little bit beefy. The one after the holidays does tend to be a little bit beefier.

KASLIN FIELDS: Yeah, it's a good spread. I feel like usually one of those is higher than the others. But those are pretty even.

KAT COSGROVE: We've technically got one removal. It's very small. But we do technically have one removal. It's removing the security context deny admission plugin, which was deprecated in 1.27. So you should be using the pod security admission plugin available since 1.25. That's the only removal we have. And it's like so minor, it wasn't mentioned in the mid-cycle blog.

KASLIN FIELDS: Makes sense, because this is a continuation of the [? poded ?] mission controller. Is that the right word?

KAT COSGROVE: Pod security policy.

KASLIN FIELDS: Pod security policy, there we go, PSPs. We did an episode, I think, on PSPs a while ago. It's been in progress for like years at this point, right--

KAT COSGROVE: Yeah.

KASLIN FIELDS: --the removal of pod security policies.

KAT COSGROVE: It has been. We like chronically have some KEPs that are just enormous, that span like multiple releases. And that is absolutely-- that is one of them. But we have a lot going on this release, enough interesting things that we felt like we needed to release like a mid-cycle, take-a-peak type of blog, which I hope to see us continue to do because it feels like a good taste of things to come.

KASLIN FIELDS: It does. I feel like usually--

KAT COSGROVE: It's good knowing

KASLIN FIELDS: --you only get a sense for what's going to be in that release if you're not following KEPs, which is hard to do.

KAT COSGROVE: It is it is hard to do. And no matter how much time you spend in the project, or at least in SIG Release, looking at these KEPs, it changes so fast from release to release. You don't get a good feel for like what all of this stuff is. And the expectation is never that you should have a handle on every single KEP in the release. Nobody in the release team, at any given time, is aware of what every single one of these KEPs, which is why we rely so heavily on the GitHub project board to keep track of it for us because we don't. It's too much.

KASLIN FIELDS: And multiple people, whole teams of people just keeping track of both the work going on the KEPs and the communications that will be going out about the KEPs.

KAT COSGROVE: It changes a lot through the release too. When we started, we started with 92 enhancements.

KASLIN FIELDS: That's a lot more than you ended up with. So that had to have been quite the experience.

KAT COSGROVE: It's pretty typical to start with a lot. And then you lose a lot at code freeze. So we started with 92. We ended with 46. So 46 after code freeze.

KASLIN FIELDS: It's a lot.

KAT COSGROVE: Yeah, it's still a lot. That's only on the slightly large side of average. But it's manageable.

KASLIN FIELDS: So what are some of the, I guess, most interesting bits that are coming up in 1.3 that will probably be the most impactful to users?

KAT COSGROVE: I would say probably take a look at the common expression language for admission control. That is KEP 3488 if you're the type of person that likes to go read KEPs. But it's a more expressive way of evaluating admission requests. So it's a little bit more fine-grained control over what you're doing for admission control, which is I think always pretty cool. What else do we have? Container-based pod autoscaling, so horizontal pod autoscaling based on container resource metrics is graduating to stable finally.

KASLIN FIELDS: Nice.

KAT COSGROVE: So that's exciting. We have a lot of cool stuff coming stable, actually. There's just generally a lot going on in this release that's like. Node memory swap support KEP 2,400 if you like reading KEPs, once again. I know some people do. Stephen Augustus committed hard on inventing KEPs, so that we had more documentation to read, I guess.

KASLIN FIELDS: It's good to have docs.

KAT COSGROVE: It is good to have docs. I am aggressive and passionate about it. Can I talk about another new thing I introduced for 1.30 that's docs related?

KASLIN FIELDS: Please, yeah.

KAT COSGROVE: So if you are not aware of how the release phases work, there is an enhancements freeze and a code freeze. And both of those are hard deadlines that KEP owners have to comply with in order to have their enhancement included in the release. If you are going to miss that deadline, you must submit an exception request, where myself and the rest of the release team and SIG leadership have a discussion and determine whether or not the number of days you've asked extra for is reasonable and how risky it is to allow it in late.

Docs has not historically had that deadline. Even though if your KEP includes user facing changes, documentation is actually a requirement for your KEP to be considered complete. We can't have that KEP in there without docs if docs are required.

But the docs deadline has always been squishy. It's soft. And it's very close to the end of the release. And so it makes it very stressful for the doc sub team and for SIG docs on reviews and chasing people down because it's so close to release day. We really have no wiggle room.

And people are chronically late because they don't take the documentation seriously. So instead, for this release, we introduced a docs freeze deadline. It's March 26 for version 1.30, which is earlier in the cycle than the PRs ready to merge deadline was. And we're requiring an exception request process for this too. So now the docs deadline is no longer soft. It is very firm. And if you miss it and don't file an exception request, we will yank your KEP from the release.

KASLIN FIELDS: There's no point to having a feature if nobody can--

KAT COSGROVE: Use it.

KASLIN FIELDS: --nobody knows what it is.

KAT COSGROVE: Nobody knows what it is. Like, what's the point? We got to have it. So hopefully that improves things for everybody. I know that it's a lot of deadlines back to back, but such is the nature of the beast. And a lot of this documentation is literally just documenting the feature gate. So it's a two-line PR. Come on. Open the thing, right?

KASLIN FIELDS: I remember the comms about that. We tried to make sure that the community knew about that change to the release process because it's very important to have docs for all of the new features.

KAT COSGROVE: And it is a big change. It's a new deadline that people aren't used to.

KASLIN FIELDS: Yeah, understandable. Back to the features in 1.30, we had just mentioned the node memory swap support. Do you want to say anything more about that one?

KAT COSGROVE: So in previous versions of Kubernetes, earlier than 1.30, the node swap feature gate was disabled by default. And now it is being--

KASLIN FIELDS: Enabled.

KAT COSGROVE: Yeah. Yeah, it's pretty old. So we're removing it in 1.30, the unlimited swap behavior.

KASLIN FIELDS: OK. Cool. And there's a lot more detail about this, of course, in the blog post.

KAT COSGROVE: It's really long. There's a lot going on in here. It is still beta, but the new feature is on by default now.

KASLIN FIELDS: OK. Cool. Any other features in 1.30 that you want to call out? Did you mention the user namespaces in pods?

KAT COSGROVE: I don't think I did. But yes, you're right. This is a Linux-only feature. This allows you to better isolate pods to prevent or mitigate several CVEs rated high or critical, including CVE-2024201626. God, we need a better naming convention for these things. So in Kubernetes 1.30 support for usernames spaces is migrating to beta-- woo-- and now supports pods with and without volumes, custom UID/GID ranges and more.

KASLIN FIELDS: I'm kind of interested in that because of-- I feel like namespaces is kind of an overloaded term.

KAT COSGROVE: It is.

KASLIN FIELDS: So understanding what namespace means in the context of a pod versus in the Linux kernel because they're used to create containers in the Linux kernel and then in Kubernetes itself as well. So it's a very overloaded term.

KAT COSGROVE: It feels like we should have named that better so there's a little bit less confusion. But naming things is just one of those chronically difficult things in always programming. We could have been more specific with that one, so it's not such a load-bearing phrase.

KASLIN FIELDS: And one of the other features that you mentioned earlier was about the new dynamic resource, the new auto-scaling thing.

KAT COSGROVE: Yes, the resource based pod autoscaling. So horizontal pod autoscaling based on container resource metrics will graduate to stable in 1.30, which is very exciting. This lets you configure automatic scaling based on the resource usage for individual containers rather than aggregate resource usage over a pod. You can be a little bit more specific there.

KASLIN FIELDS: Interesting because like I always tell folks who are new to Kubernetes, a pod is not necessarily just one container. And honestly, usually it's not these days because of all of the service meshes and things that do sidecar.

KAT COSGROVE: It was when I learned Kubernetes, like seven years ago, now not so much. Things did get way more complicated. That is actually like a fairly complex change. So there's quite a lot of additional documentation linked from the blog, a much more detailed article on that change specifically and some pretty thorough documentation on container resource metrics. It's worth reading through thoroughly if you want to actually understand the impact of that change.

KASLIN FIELDS: Very cool. Because I think a lot of folks, too, who work with Kubernetes and run applications on Kubernetes, you think about it being just one application, one container. But then it's got all of this extra stuff.

KAT COSGROVE: There's so much stuff--

KASLIN FIELDS: And you don't want to be-- you might not want to be autoscaling based on whatever the other stuff is doing. That's pretty cool.

KAT COSGROVE: Unless you just love spending money, I guess. If you just really love running up your cloud bill, I guess, you can do that.

KASLIN FIELDS: Yeah.

KAT COSGROVE: I think most of us don't.

KASLIN FIELDS: Generally. So we've got the one very, very minor removal, which is a remnant of the PSP deprecation. There's some updates happening to autoscaling. And I think that kind of covers it.

KAT COSGROVE: The actual release blog is going to be considerably more thorough than this midcycle blog was. But yeah, there is quite a lot going on.

KASLIN FIELDS: Always make sure that you check out the documentation--

KAT COSGROVE: Please do.

KASLIN FIELDS: --because each change in a Kubernetes release has history behind it.

KAT COSGROVE: It does. It does. I think we've got one very, very old KEP in this cycle. It's KEP 24.

KASLIN FIELDS: 24. Whoa. Which one is that?

KAT COSGROVE: That's pretty fun AppArmor support.

KASLIN FIELDS: OK.

KAT COSGROVE: It's just adding AppArmor support to Kubernetes. But this is one of the oldest KEPs in the project. This KEP was opened July 14, 2016.

KASLIN FIELDS: Wow.

KAT COSGROVE: And it will be in for 1.30.

KASLIN FIELDS: So if you've been looking forward to AppArmor support in Kubernetes, 1.30 is your time.

KAT COSGROVE: It's here for you.

KASLIN FIELDS: I also just noticed that the user namespaces in pods one that I mentioned is 127, KEP-127. So that has to be quite old as well.

KAT COSGROVE: Well, let's look it up and see how old it actually is. This was opened October 10, 2016.

KASLIN FIELDS: Wow. So we have at least two KEPs in 1.30 that have been open since 2016.

KAT COSGROVE: Yeah.

KASLIN FIELDS: The project is making progress.

KAT COSGROVE: It is. It is. With things like this, often the barrier is reviewers. Getting reviewers is always hard. Especially, getting like tech reviews is hard. There's going to be a bottleneck. And the bottleneck is how much resource does a given SIG that owns one of these KEPs have to actually get a job done. So if you're interested in contributing to Kubernetes in ways other than the release team, may I suggest getting involved with a SIG that interests you--

KASLIN FIELDS: Yes, always.

KAT COSGROVE: --and helping because we always need more reviewers, always, always, always.

KASLIN FIELDS: And in order to become a reviewer, you're going to need context. So one thing that I'm going to be talking about later at KubeCon is what I like to call the good first issue trap. If you want to become a contributor in Kubernetes, like join some meetings, talk to people, just be a fly on the wall in the [? flack ?] for a while, get a sense for the context rather than just picking up a good first issue and running with it.

KAT COSGROVE: For sure.

KASLIN FIELDS: It'll take you much farther.

KAT COSGROVE: It will. There's a lot you can't do in the Kubernetes project without like far-reaching contexts. Like, you can't lead a Kubernetes release without having the context gained from having served in other roles on the release team. And you're much better at it if you spent that time talking to other SIG leads. If you have relationships with people in other SIGs, that makes you a better release lead, a better release team member.

For some sub team roles it's a requirement. The comms and docs teams both work with other SIGs like pretty extensively. And that's super valuable experience for excelling through the release team, but also just existing in the project in general.

KASLIN FIELDS: So we've talked a bit about the 1.30 features. What are you most excited about in the release?

KAT COSGROVE: I'm excited about taking a nap.

KASLIN FIELDS: For it to be over?

KAT COSGROVE: I'm excited for it to be over, yeah. Not that I'm not having a fun time. But I am excited to be able to relax because it is quite stressful. I think I am most excited for that super old KEP making it in. I'm cheering it on.

KASLIN FIELDS: It's good to see those, isn't it?

KAT COSGROVE: It is so good to see those. It almost missed it too. They had to file an exception request at code freeze and I think also it enhancements freeze. But they made it. They made it. And I'm cheering for it. He's a cute, little guy. And I want him to make it in.

KASLIN FIELDS: I'm excited about it, both that and the user namespaces one. 2016, blows my mind.

KAT COSGROVE: Ancient.

KASLIN FIELDS: All right. So the toughest question for any release--

KAT COSGROVE: OK.

KASLIN FIELDS: 1.30, brand new, exciting, shiny, new Kubernetes that many, many people don't actually want to upgrade to because upgrading is going to be really painful. Why should they upgrade to it?

KAT COSGROVE: Because we are eventually going to end of life whatever version of Kubernetes you are currently running on. And we're pretty inflexible on that, actually. This isn't Microsoft. We don't do extensions to end of life on enterprise products on Kubernetes. So when it's done, it's done. And it's not going to get any more updates, including security updates. You're not going to get security patches for a version of Kubernetes that's four years old. We're just not going to do that.

So if you care about keeping your cluster as secure as it can be-- I'm not going to say secure because that doesn't exist, but as secure as it can be-- and interoperable with the rest of the internet and updates that other people are making to applications that rely on or interact with Kubernetes, then you should update.

I know that it's stressful. We try to minimize the amount of weird stuff that can happen. But you do have to do it. We can't be sitting on years-old software, especially not when it's something as complex as Kubernetes. The attack surface is potentially massive.

KASLIN FIELDS: And a lot of the changes in this one, like there's only that one very small removal. So there's not a whole lot that's likely to be a breaking change in terms of removing things. And a lot of the features are moving to stable.

KAT COSGROVE: This is not a risky release at all. We're not introducing any breaking changes. We're not removing or deprecating anything sketchy that you're relying on. I mean, unless you are still relying on that like one tiny remnant of PSP. And then I don't know what to tell you. You should probably fix that too. But this is not a risky release at all.

KASLIN FIELDS: So come on over to 1.30.

KAT COSGROVE: Come on over.

KASLIN FIELDS: The water's fine.

KAT COSGROVE: Come on over. We're having a good time. We're trying to build the best release we can for you.

KASLIN FIELDS: And that brings me to my last question, the most fun question of every release. What is the release theme?

KAT COSGROVE: So I made a thread on Twitter three or four years ago, that if I was ever allowed to lead a release I was going to name it Uwubernetes.

KASLIN FIELDS: Uwubernetes.

KAT COSGROVE: And I didn't think I would ever actually be a release lead. I don't think I was actually on the release team at all at the time.

KASLIN FIELDS: And yet, what fate had in store for you.

KAT COSGROVE: Here we are years later. And I don't want it to ever be said that I don't commit to the bit. So I made a threat on Twitter several years ago. And now I'm making good on it. And it is Kubernetes version 1.30 Uwubernetes.

KASLIN FIELDS: Uwubernetes. And for those in the audience who may not know the context of this, would you like to explain the concept of uwu?

KAT COSGROVE: Yeah. So uwu is cute. Uwu is also terminally online, but cute, right?

KASLIN FIELDS: It comes from the face, right? You do a W. A U, a W, and a U.

KAT COSGROVE: It's a little like emoticon from before we had emojis. It's an emoticon. But also sometimes it's like the pleading-face emoji. But that is uwu. But uwu is about being cutesy. And I think that Kubernetes can also be cute. It doesn't have to be like, we're doing serious engineering here. Everything has to be like hard and manly. It can be soft and cute too, you know?

And we're a community of people doing this like mostly for free. There are a few people, like me, who are in like the unique and blessed position of being literally paid to do this. My employer is supporting me while I do it. But most Kubernetes contributors are doing this in their spare time. It is not their job. They do it because they love it.

KASLIN FIELDS: And so you got to have fun.

KAT COSGROVE: Yeah, you got to have fun with it. And I think that's very uwu of us.

KASLIN FIELDS: It is very uwu of us. So come on over. Come on over to 1.30 Uwubernetes.

KAT COSGROVE: Uwubernetes.

KASLIN FIELDS: The water's fine.

[GIGGLING]

All right? Thank you very much, Kat. And I look forward to trying out 1.30.

KAT COSGROVE: Thanks.

[MUSIC PLAYING]

ABDEL SGHIOUAR: That brings us to the end of another episode. If you enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at KubernetesPod or reach us by email <kubernetespodcast@google.com>. You can also check out our website kubernetespodcast.com, where you will find transcripts and show notes as well as links to subscribe.

Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thank you for listening. And we'll see you next time.

[MUSIC PLAYING]