KASLIN FIELDS: Hello, and welcome to the Kubernetes podcast from Google. I'm your host Kaslin Fields.

ABDEL SGHIOUAR: And I am Abdel Sghiouar.

[MUSIC PLAYING]

KASLIN FIELDS: In this episode, we interviewed Priyanka Saggu, Kubernetes version 1.29 release lead and SIG ContribEx tech lead. We spoke about the release, the new features and enhancements, and more. But first, let's get to the news.

[MUSIC PLAYING]

ABDEL SGHIOUAR: The Kyverno project has completed a third-party security audit conducted by Ada Logics, in collaboration with the Kyverno maintainers. In total, six CVEs were assigned during the audit. Four of these had their root cause in the notary verifier, which had not been released prior to the audit. One of the findings was in a third-party dependency to Kyverno and was fixed by the cosine project maintainers. The Kyverno maintainers have quickly addressed all issues found during the audit, with fixes incorporated in Kyverno 1.10.6 and 1.11.1.

KASLIN FIELDS: Google Cloud announced their new Gemini AI model. On top of being Google's largest and most advanced AI model to date, Gemini is built with multi modality. It can reason across text, images, video, audio, and code.

Gemini comes in three versions-- Nano for on-device tasks, Pro for general tasks, and Ultra, the most capable of the three. Google ran 32 well-established benchmarks, comparing Gemini to GPT4 and described it as being substantially ahead on 30 out of 32. Check out the model yourself today through Google Bard.

ABDEL SGHIOUAR: The Linux Foundation has announced its intent to form the High Performance Software Foundation, SPSF, to advance high performance computing. SPSF aims to develop a portable software stack for HPC, making it more accessible and fostering collaboration in developments.

KASLIN FIELDS: The Call For Papers, or CFP, for the open source summit North America is open until January 14, 2024. The event will take place in Seattle, Washington, from April 16 to 18, 2024.

ABDEL SGHIOUAR: The App Defense Alliance initially, established by Google for Android app security, has now joined the Joint Development Foundation, under the Linux Foundation. The new alliance brings together technology leaders, like Google, Meta, and Microsoft, as the steering committee. The alliance aims to strengthen app security across various platforms and foster collaborative implementations of industry-wide standards.

KASLIN FIELDS: And that's the news.

[MUSIC PLAYING]

All right. Welcome to the show, Priyanka Saggu. I'm very excited to be talking about the 1.29 release this time, but first, could you introduce yourself a little bit?

PRIYANKA SAGGU: Hey, everyone. Hello, Kaslin. My name is Priyanka Saggu. I work at SUSE as a Kubernetes integration engineer, and I do a lot of stuff in upstream as well. Currently, like Kaslin already introduced, I am the release lead for Kubernetes 1.29 release cycle. I'm also a technical lead for Kubernetes special-- interest group contributor experience, alongside Kaslin.

KASLIN FIELDS: Where we work together closely, because I'm a co-chair. So co-chair and tech lead, it's like same level.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: For folks not familiar.

PRIYANKA SAGGU: Yeah, and I'm also one of the GitHub admins for Kubernetes and subproject orgs.

KASLIN FIELDS: Yeah. You do so many things.

PRIYANKA SAGGU: I'm learning. Yeah.

KASLIN FIELDS: Our tech leads do such cool stuff in SIG ContribEx because SIG contributor experience is all about making sure that folks contributing to Kubernetes have the processes and tools that they need to contribute. So being a GitHub admin is like a core part of the technical side of SIG contributor experience, because it's a lot of stuff around like how the community uses GitHub. Right? And I'm always amazed at the stuff you all do.

PRIYANKA SAGGU: Yeah. I recently was nominated as one of the GitHub admins. So even I am surprised like how many things goes into being a GitHub admin. There is a lot of stuff.

KASLIN FIELDS: Also, it's open source. So I feel like a lot of open source is just kind of, congratulations, you have a job now. Figure it out.

PRIYANKA SAGGU: Oh, yeah.

KASLIN FIELDS: That's how I feel about being co-chair at least.

ABDEL SGHIOUAR: All of these feels like a full-time job, most times.

KASLIN FIELDS: Yeah. I remember when we were doing the lead cohort, when we were working with the previous leads of SIG ContribEx to get familiar with the role and everything. It was like, great, so what am I in charge of now, I asked one day. And the previous leads were like, well, there's this, and there's that and the other thing. And I was like it's just everything, isn't it? And you don't know exactly where it begins and ends. Do you?

PRIYANKA SAGGU: I still ask the same questions I think a few weeks back to I think Bob, who was the previous lead chair. And I was like, oh, I'm a lead now. What all I'm supposed to take care of?

KASLIN FIELDS: So what do I do?

PRIYANKA SAGGU: Yeah, and I think like he said everyone is feeling the same way, and that was good to hear.

KASLIN FIELDS: So welcome to open source, where we learn in public, which is what we're going to do today on the release 1.29.

PRIYANKA SAGGU: Hey.

KASLIN FIELDS: At least I'm going to be learning a lot today. I'm very excited to talk about-- we talked a little bit about it, and I did a little bit of research leading up to this. So I know there are some very exciting things in this release, which I'm excited to dive into. But before we dive into the changes in the release, let's go over the release theme. What is the theme for this release?

PRIYANKA SAGGU: We are calling Kubernetes 1.29 as Mandala, and it's an Indian art form. Mandala is actually, in English, we can call it universe. So actually, I'm calling the release theme as Mandala [? Brackets, ?] the Universe. So everyone understand it.

KASLIN FIELDS: Oh, I love that.

PRIYANKA SAGGU: It's art form from India, and our logo is actually depicting that art form itself. What I'm trying to maybe show with this logo is our little universe of Kubernetes itself. We have, starting from the release name, 40 team members who are working on making this release happen, backed by hundreds and hundreds of community contributors. And everyone, not just contributors, users, supporters who have worked tirelessly to bring this new release to life, so this logo is for all of them.

KASLIN FIELDS: I feel like it fits perfectly. I'm surprised, honestly, that this has never happened before, because it just feels so perfect to describe the Kubernetes release as its own universe. And mandalas are gorgeous.

PRIYANKA SAGGU: Yeah. I will also talk about who created the logo. So this is also created by one of our contributors in the project, Mario Jason Braganza, and we used-- the base mandala art is actually we took it from someone, Fibrel Ojala. It's an eight-year-old art.

We found it somewhere on internet, and I'm so happy that this logo is bringing it back to life. So thank you, Fibrel Ojala, if you are listening to this podcast, and thank you, Mario Jason Braganza, to create this logo for us. I love it.

Actually there is a little bit of a secret inside, which is no longer a secret. We already discussed it on Kubernetes Slack. So if you read the numbers on these for-- what should I say-- there are squares or I think parallelograms on each side of the circle. And if you start reading it from left to right and then bottom, it reads 1.29.

KASLIN FIELDS: Ooh.

PRIYANKA SAGGU: Script called [? Annapurna ?] SIL, which is on Devanagari script Hindi. So that is where we have written down our release numbers there, 1.29. We try to make it part of the mandala art itself. Yeah.

KASLIN FIELDS: That's so cool, and we'll put a link in the show notes for folks who want to go and see the art. I'm very excited to look at that later. I haven't actually seen it myself yet.

PRIYANKA SAGGU: Oh.

KASLIN FIELDS: And you were saying that this is art that existed previously, but it's been reworked for the 1.29 release. Is that a good way to describe it?

PRIYANKA SAGGU: Yeah. Perfect.

KASLIN FIELDS: Awesome. I'm so excited to look at that art later and such a great theme.

PRIYANKA SAGGU: Thank you.

KASLIN FIELDS: Kind of a universe unto itself. Let's talk a bit about what is in that universe. There's quite a few changes in this one. Right? How many KEPs are there, or how many changes?

PRIYANKA SAGGU: At the beginning of this cycle we started with 72, which is--

KASLIN FIELDS: I feel like that's a lot. Is that a lot?

PRIYANKA SAGGU: Yeah. Yeah.

KASLIN FIELDS: OK.

PRIYANKA SAGGU: It is actually a lot, because the cycle was actually a bit on the shorter side because of a lot of things that happened. So I was not expecting like this release to be this big in terms of the KEPs we are tracking, but we got like a huge number. 72 is a big number, and after code freeze now, we are almost a week or two to the final release date, which is December 13 now. And we are tracking 49 KEPs, so almost 50.

KASLIN FIELDS: That is also a lot.

PRIYANKA SAGGU: Yeah, that is also a lot.

KASLIN FIELDS: The release goes in two stages, as far as these go. Where it's like the KEPs opt in like, hey, we're totally going to get this into the release this time. And then once you hit code freeze, it's some of them didn't quite make it. So it reduces a bit. Right?

PRIYANKA SAGGU: Oh, yeah. Even before code freeze, we have something called enhancements freeze, where Yeah all the people who are actually opting in for their features or any enhancements, they have to have their enhancements or what do we call a KEP, KEP when it is enhancements proposal.

They need to get it merged into a repo. We have Kubernetes enhancements. So we call that deadline enhancements freeze. And the number goes at that milestone, as well. And then we have code freeze, which is where if your code is implemented, you are tracked. Otherwise, we have to leave it for other release.

KASLIN FIELDS: Yeah, so to have 49 make it through both enhancement freeze and code freezes, a lot, I feel like. So definitely.

PRIYANKA SAGGU: Oh, yeah, it is. And out of those 49, we have 11 stable features or 11 stable enhancements coming in. We have 20 new alpha features coming in this cycle, and 18 beta. So yeah, we have a good spread.

KASLIN FIELDS: And the deprecations and removals blog-- or like breaking changes blog is already out, right?

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: Were there any deprecations in this one?

PRIYANKA SAGGU: So we did not have any deprecations, per se. We had two enhancements that were tracked as [? of ?] type deprecation removal, I would say. So I can talk about one of them, which is actually a breaking change, and I think the first thing I would love to talk about so that people are aware.

KASLIN FIELDS: Yeah, definitely.

PRIYANKA SAGGU: So we are talking about this KEP23 called removing entry cloud providers. And for anyone listening it's, KEP number 2395.

KASLIN FIELDS: We'll put a link in the show notes.

PRIYANKA SAGGU: Thank you. So this KEP is going to affect people, some of them. And I want to give a TLDR who is going to be affected. So anybody who is on Azure, GCE, or vSphere.

So if anybody has a Kubernetes environment running on Azure, GCE, or vSphere cloud provider and they are looking to upgrade their clusters from a version older than 1.28, I would say highly recommend to read the blog that's going to come right after our release, the first feature blog. And that's going to tell you what you need to do, because this KEP--

KASLIN FIELDS: Well, before you dive into what the KEP does, to clarify, we were talking about it earlier. And if you're running on Azure, GCE, or vSphere, if you're running your own cluster-- so GCE being Google Compute Engine. So if you're running on AKS or GKE, the control plane is managed by the cloud provider. And so they're probably going to take care of this for you.

PRIYANKA SAGGU: Yeah, not the managed ones, yeah.

KASLIN FIELDS: Yeah, but if you're doing your own-- rolling your own Kubernetes from scratch, you need to upgrade the cluster directly on virtual machines in Azure, GCE, or vSphere. Well, I guess even if you're not using the cloud provider plugin features, this might cause you trouble. So let's talk about why that is.

PRIYANKA SAGGU: Yeah, I would actually start with what this KEP is actually about and what is happening in 1.29. So this KEP started with a discussion in 2018 where Kubernetes community actually created a SIG called Cloud Provider SIG. And the only mission of that SIG was to externalize the cloud provider integrations we have in-tree inside our Kubernetes source code, and to remove all those cloud providers, and make them available as out-of-tree implementations.

So in January 2019, the SIG came up with the draft KEP. And that was approved by the community. And that's where all started.

We are talking about 2019, and we are in 2023 now. After so many years of development and collaboration, this is the first time in 1.29 when this change will start affecting people, because on a basic level, what this change is coming out in 1.29, is we are flipping the values of two flags that are actually used at Kube API server or Kube controller manager and kubelet level.

One of those flags is disable cloud providers, which actually disables any functionality in Kube API server or Kube controller manager and kubelet related to this flag called cloud provider. So before 1.29, if you are to use any of these cloud providers, you would use this cloud provider flag and pass, for example, AWS, Azure, or GCE. But that's no longer going to work once you are on 1.29.

And the other flag is disable kubelet cloud credentials provider, which disables the entry functionality in kubelet to authenticate to a cloud provider container registry for image pull requests. So collectively, these two flags, now they will be defaulting to true values. And what will happen is this cloud provider command line flag will only understand one value called external.

So all the implementations we had for entry cloud provider integrations, they no longer will work after 1.29. So if anybody who is on Azure, GCE, or vSphere, they are going to be impacted. And they will need to take some remedy steps. And the blog is going to talk about what needs to be done with multiple options for whatever is going to be favorable in your situation.

KASLIN FIELDS: And we've talked about this in-tree versus out-of-tree concept before, especially with releases. There's been a lot of work going on in the community for years working on taking things that were originally developed in the Kubernetes code base that aren't really part of Kubernetes like these cloud provider plugins or the storage provider plugins, which we've talked about on previous episodes.

Even the gateway API is kind of an extension of this concept of as much stuff as possible, if it doesn't need to be in Kubernetes, then it should be outside of Kubernetes. Because if it's inside of Kubernetes, then it has to deal with the Kubernetes release process. And a lot more folks have to work together. And it just causes a lot of trouble for everyone involved, both on the Kubernetes side and on the side of whoever is developing that tool.

So the whole community has been going through this process of taking things that were originally developed as part of Kubernetes like the storage provider plug-ins, the cloud provider plug-ins, and making them true plug-ins. Taking them out of Kubernetes itself, and making them something that you have to pull in from outside. So that's what's happening here.

The cloud provider plug-ins still exist. But they aren't delivered as part of Kubernetes, itself. So Kubernetes has to work with a separate tool, the separate plug-in in order to get that functionality.

So what the functionality did, basically, is it's possible to run your own cluster in any cloud provider pretty much. As long as you have hardware, as long as you have machines that have a compatible kernel, you can run a Kubernetes cluster across them. And Kubernetes will do its Kubernetes things of distributing your workloads across them.

But the cloud providers created these convenient plug-ins where you can add the plug-in to your cluster. Now, of course, it'll be outside, so you actually have to add it-- before, it was just there-- where the cluster will know a little bit about the environment that it's working in, the cloud, and be able to have some nice, convenient integrations, even if you're running the cluster yourself. So that's how this is affecting, is if you're running clusters yourself from scratch on these cloud providers, Azure, GCE, and vSphere, and you use this cloud provider plug-in.

Actually, even if you're not using it, it sounds like the control plane and the API server and one other piece of the control plane of the cluster. A couple of parts of the control plane might have trouble because they'll be like, oh, I'm supposed to have this cloud provider plug-in, but I don't now because it's now out-of-tree. So make sure that you follow these remediation steps if you are running your own clusters in these environments and might want to use the cloud provider functionality. Does that sound like a fairly good description, Priyanka?

PRIYANKA SAGGU: Yes.

KASLIN FIELDS: Cool.

PRIYANKA SAGGU: A last-minute note, I would love to also talk about who is not going to be impacted so that people don't worry about this. So anybody who is not on Azure, GCE, or vSphere, they are not impacted. And anybody who is on AWS and OpenStack.

Just a note, those two cloud provider integrations are also there at some point, historically. But they were removed in 126 and 127 from the Kubernetes source code. So if anybody is on those two cloud providers, they are also not impacted.

And outside of these five Azure, GCE, vSphere, AWS, and OpenStack, anybody who is on any other cloud provider, they don't have to worry about this because other providers beyond these five listed were never included in Kubernetes code. And as such, they will be in their life as external cloud controller managers. So that's it.

KASLIN FIELDS: Again, if you're using a managed service like AKS or GKE, then, also, you shouldn't have to worry about this.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: Cool, so that's the one breaking change. Everyone who runs their own clusters in a cloud provider, specifically Azure, GCE, or vSphere, make sure that you go check out that blog post so that you know exactly how to mitigate that issue. Let's talk about some of the other big changes, though, that are happening in this release, because there are some other very exciting things, especially some very old KEPs, even, that are finally getting included in 1.29. So what have you got?

PRIYANKA SAGGU: Yeah, I think there is one KEP which is much anticipated from the community. I think the discussion started in 2015. And while we were discussing version 1.17, and finally they are landing into 1.29 as alpha, I'm talking about KEP [? 1287. ?] It's for in-place update of pod resources. And it's graduating as alpha this cycle.

But on this KEP, what this does, is this will actually make our pod APS spec mutable. So historical context, resources which we allocate to pod containers can require a change for many reasons. Some examples, for example, if the load handled by the pod has increased significantly and the current resources, for example CPU or memory we have allocated to that pod container, they are no longer sufficient and we need to provide more. So in that case, the container require a change.

Or the load has decreased significantly, and our resources are getting wasted, they are unused, to save those resources, we need to change. Our resources have simply been set improperly. We need to set them correctly.

So in all these scenarios, there has always been a requirement to change these resources on our pod containers, but without restarting them. So right now, we can do that. We can change resources like CPU and memory consumption allocation on pod containers. But we need to restart our pod.

This KEP actually brings that very much-wanted functionality which basically, in short, make our pod spec resources, they're mutable. So now, we do not have to no longer restart our pods to actually change our CPU or memory resources we have allocated to our containers. So in 1.29, this KEP is coming. It's graduating as alpha.

There are a lot of other details also related to this KEP on how they are making this happen, making this in-place update of pod resources happen. For example, they are adding something called resize policy for containers so that users can control how their containers are resized. Or they are also adding another field called allocated resources to the containers status in a pod status, which will describe how much node resources are allocated to a pod. So overall, this KEP, the functionality will help us to better utilize our resources.

KASLIN FIELDS: I'm very excited about this one. As I harp on many episodes, always set your pod resource requests and limits. [LAUGHING] And this is such an exciting update for that. And hopefully, it'll make that a little bit easier for folks because now when those resource requests and limits change, maybe if you're using a vertical pod autoscaler or something to automatically change resources, even the pod itself doesn't have to restart in order to change those resources. So it can do it much less disruptively, which is exciting.

And it sounds like as part of this KEP, which, as you said, is a very old KEP that folks have wanted for a very long time. As part of this, it sounds like there's also some additional fields that are going to make it a little bit easier to control the resources of the pod, the allocated and--

PRIYANKA SAGGU: Yeah, that's correct.

KASLIN FIELDS: What was the other one you mentioned?

PRIYANKA SAGGU: Allocated resources is one that will give us some more information in the status section. So status is the section of any Kubernetes resource object, for example, that gives us information of the applied resource. So whatever is actually running inside the Kubernetes cluster, what's the status of that.

We have added a new field called allocated resources there to get some more information. Yeah, in general, you also touched on vertical autoscaling and horizontal autoscaling. There is a bit of implementation details coming out on those sides, as well, as part of this KEP. So look for documentation.

KASLIN FIELDS: Awesome, so I'm very excited about this change. I mean, anything that makes pod operations less disruptive is going to be a good thing. And I think folks will make good use of this. And I hope to see stories about use cases, because use cases are my favorite, in the future.

But there's also another major change in this release. And it's also one that I've heard a lot about. 753, the sidecar containers one, which we've talked about on the podcast before as well, right?

PRIYANKA SAGGU: Yeah, so 753, this is a major KEP that was very much talked about during 1.28, as well, when it was alpha. In 1.29, it is graduating to beta. And I'll just give a brief overview of what sidecar containers are, and maybe touch a bit on what is coming, what is changing in 1.29.

KASLIN FIELDS: I've had to explain sidecar container basics to a lot of folks recently because of this KEP, which is great. I love explaining sidecars. So, yeah, please go ahead.

PRIYANKA SAGGU: Yeah, so for everyone who is listening, I had to go do my homework to actually learn about all the enhancements that we are discussing here. So for me, also, I have my notes. And I'm just going through them.

So for me, if I have to explain sidecar containers to somebody, this is from the docs. They explain it as any traditional init containers that we have in our pods with the added always restart policy. So if you have to explain sidecar containers, it's traditional init containers with an always restart policy. That's what sidecar containers are.

KASLIN FIELDS: Sometimes the perspective of Kubernetes on these concepts is slightly different from the concept of other projects and users. The way that I often explain sidecar containers is just the general concept of like you have a pod, and you have your main container that has the app running in it that you actually care about. And then you have something else, some other container right there within the pod that's sharing network.

It's sharing storage. It's sharing all of the resources. So that allows it to do really cool things with the application like collecting logs or intercepting traffic or all kinds of things. So it's used very commonly in service meshes.

But the perspective of Kubernetes is really cool because it looks at it as a specific type of container within the pod itself. It's very cool. There's a lot of trickiness to explaining exactly how this KEP affects sidecar. So if you want to talk more about the changes to init containers that Kubernetes is doing with this KEP.

PRIYANKA SAGGU: Yeah, actually, you explained it very well. Whatever you discussed about sidecar containers right now are actually the major uses. People know sidecars from those use cases, yeah. So what's coming in 1.29 for this particular KEP, is this KEP is actually graduated to beta now, or will be graduating to beta once we release 1.29.

KASLIN FIELDS: Which is the main thing, right? In 1.28, we also talked about this KEP a lot. So definitely check out the 1.28 episode if you want more.

PRIYANKA SAGGU: Yes, so what changes between 1.28 and 1.29 now is this is going to be they're enabled by default in 1.29. Starting in 1.29, let's say. That was not the case in 1.28 when it was alpha. People had to use the feature gates, enable the feature gates. And then only then they were able to use the sidecar containers.

But now, starting from 1.29 since it's graduating to beta, it will be enabled by default. And what change would happen, is anytime, beginning with Kubernetes 1.29, if your pod includes one or more of these sidecar containers, which is init containers with an always restart policy, kubelet will delay sending the terminate signals to these sidecar containers so that until our last main container has fully terminated, these will stay there. So that's the whole idea of sidecar containers and this KEP that sidecar container stays alive till our main containers are alive.

KASLIN FIELDS: Yeah, the cool thing about Kubernetes perspective on this, I think, is that we would think of sidecars as it has access to all of this stuff, and so you can use it to do all of these cool things. But from Kubernetes perspective, it's like, all right, you put a different container into the pod. And the container needs to live as long as the other container in the pod lives. All right, we'll do that.

PRIYANKA SAGGU: Yeah, that's exactly it is, yeah. And I was just saying, the sidecar containers, if you have multiple of them, they will be terminated in reverse order. So whichever started first will be the one terminated last.

Yeah, however you are defining them in your pod spec in the reverse order of that they will be terminated. That's all about this KEP. This is going to be enabled by default.

KASLIN FIELDS: Yeah, the Kubernetes perspective of things is all about what Kubernetes has to do with it; not what you are going to do with it, which makes it a little hard to understand things sometimes.

PRIYANKA SAGGU: Yeah, exactly.

KASLIN FIELDS: Perfect, so that's in beta. So it's enabled by default now. So this shouldn't break anything. It's not considered a breaking change. But it is a cool new feature for folks who want to use the sidecar pattern for their applications running in Kubernetes.

PRIYANKA SAGGU: Yes.

KASLIN FIELDS: So beyond those two major changes-- there's a lot of other changes in 1.29, right?

PRIYANKA SAGGU: Yeah, I think there are a few stable changes. There are few features that are going G8 this cycle. I would love to talk about some of them.

The first one that I want to talk about is from SIG Auth. And we are talking about KEP 3299. It's about KMS version 2 improvements, or KMS v2.

I'll start with a bit of explanation on a backstory about this. And then we'll come to that, because this was one of the KEPs that took me the most time to understand what this is doing. But it's brilliant. I learned a lot.

So backstory is if, for all the APIs in Kubernetes that allows us or let us write persistent API data. In etcd, they support something called addressed encryption. So for example, we take an example of secret objects. If we want to store secrets inside etcd using this addressed encryption, what we can do is we can ask Kube API server, please encrypt our data first before you store them in etcd, and encrypted with some specified encryption algorithm. So I will provide you what encryption algorithm you have to use. Use that, encrypt it, and then put it in etcd.

And how that happens, is Kube API server process accepts some argument called encryption provider config. This is like a flag where we pass a file in front of it, and that file actually contains an API object. And in that API object, we define what kind of encryption algorithms we have to use to encrypt our data, or more precisely, at-risk data, which is going to be there in etcd.

So in that file, we have this field called [? resources.providers. ?] And in front of that particular field is what take the name of our encryption algorithms. And that's where KMS v2 come in. So we have these many different encryption algorithms we can use called something like AS-GCM or AS-CBC.

Those, I'm reading from the documentation or referring from whatever is specified in the documentation. AS-GCM or AS-CBC, they are good, but they are weak in terms of security and not so fast. And that's where the KMS version 2 came in. What this KMS version 2 encryption provider or encryption scheme does, is it uses an envelope encryption scheme.

So to define it in very simple words, if I, as a user, am asking my Kube API server, let's create or update a resource, and first encrypt it, so what I'm going to do, is I'm going to send this query as an API request to Kube API server. And Kube server, in turn, before adding that data or that resource, YAML for example, and before storing that in etcd, what it will do, is it will do some process to encrypt that.

In case of KMS version 2, what it does is Kube API server first creates something called DEKs, or Data Encryption Keys. And then it sends that DEK to KMS plugin, key management server plugin, and ask KMS plugin to actually encrypt that DEK.

But KMS plugin does not encrypt that on its own. We are talking about Kube API server and KMS plugin inside the cluster. KMS plugin actually asks another entity which resides outside the cluster called external KMS to give it a key called KEK, or Key Encryption Key. And that is what this plugin uses to encrypt our DEKs.

And then Kube API server uses that encrypted DEK to encrypt our data. And that encrypted data is then stored in etcd. So this whole process is actually implemented and improved by many folds as part of this KEP, this whole envelope encryption scheme of Kube API server asking KMS plugin to encrypt something. And KMS plugin, in turn, asking the external KMS to encrypt the DEKs, and then giving all those stuff back to Cube API server.

So all that process is improved as part of this KEP. And what has happened as a result of that, is if you talk about the version one of KMS, there is this test that was conducted which created 12,000 secret objects. And they measured the time taken for this API server to encrypt the resources.

In case of KMS version 1, the time taken was around 160 milliseconds. But in case of KMS version 2, the time was actually decreased by 2000 times. So to encrypt like the same 12,000 secret objects, it just took 80 microseconds.

KASLIN FIELDS: Wow, and with a slightly more secure version of encryption.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: Interesting.

PRIYANKA SAGGU: So it gave us a 2000 times speed boost. So that is amazing. And this change is coming as a stable change in 1.29. This was beta in 127.

And from let's say beta to stable now, what is this functionality which I just tried to describe? I think it was kind of implemented partly in alpha, beta, as well. They added more features to it in 1.29 now.

One of the things they did is they added tracing to this entire process of Kube API server talking to KMS plugin, and then KMS plugin, in turn, talking to the external KMS. They added some functionality to trace this, encrypt operations. They also added a reference implementation this cycle. So they added a reference implementation using PKCS 11, Public Key Cryptographic Standard 11.

They use something called SoftHSM. So using this mock-- sorry, using this reference implementation, somebody can create a mock KMS plugin for testing purpose. And one thing to note here is this reference implementation is actually just for testing purposes only. It's not for production use.

So only if you just wanted to perform tests and want to understand more about this particular feature, you can use that reference implementation. But for everything else right now, I think production use, it's better to use the cloud provider KMS plugins. For example, we have one from GCP. There is something from AWS, as well. So yeah, they have their plugins.

KASLIN FIELDS: I feel like diving deep into this encryption-- I didn't expect to go to this level of security today. But I'm excited that we're here. I feel like when you dive into this level of encryption protocols and security, one common trend is that it distributes.

There's more and more pieces to it, which require authentication and all of this work. And this sounds like a really thorough. And even including tracing in this whole process of including these extra steps into the encryption to make it better is very cool.

PRIYANKA SAGGU: Yeah, I mean, it took me the most time to understand this KEP because--

KASLIN FIELDS: It's very deep.

PRIYANKA SAGGU: There is a lot of stuff happening, yeah, here. And to understand what are DEKs, and what are KEKs, and who is using them, who is creating them, all those pieces. But I would say the KEP authors, they have done a brilliant job with the docs. They have these brilliant diagrams there to explain what is happening, where, the entire flow. And so if anybody who wants to use it, they have made it pretty easy for them to understand. If I can understand it, I think somebody who is going to actually start using it, they can definitely understand that.

KASLIN FIELDS: Which is a good sign. We talked to the SIG etcd folks a couple of episodes ago. So I feel like talking to them about this would also be really cool.

PRIYANKA SAGGU: Yep.

KASLIN FIELDS: But there's also a few more stable-- well, you said there were 11 total. But are there any other stable KEPs that you want to highlight?

PRIYANKA SAGGU: Yeah, so there is one from SIG Storage that's also interesting. So we have three access modes right now before 1.29, access modes for PVs and PVCs, Persistent Volumes and Persistent Volume Claims, which currently are ReadWriteOnce, ReadOnlyMany, and ReadWriteMany.

We have this new KEP coming in from SIG Storage. The KEP number is 2485. It's called ReadWriteOncePod persistent volume excess mode.

So it's like from the name itself, it's very obvious they are trying to add a new access mode. And that's very interesting to what this access mode does. So ReadWriteOncePod is an access mode for PVs and PVCs, which was actually introduced in 122. And that's going stable this cycle.

What this access mode does, is it enables you to restrict your volume excess so you have a persistent volume in your cluster. And you can restrict the access of that particular PV to just single pod in your entire cluster. So this particular new access mode, which is going GA in 1.29 allows you to do that.

This is very much important in situations like when we are talking about stateful workloads, for example, that require single-writer access to storage. So in those kind of cases, this new access mode is-- it's not new, actually. It was there from 122. But it's going to be there by default now, starting from 1.29. And this particular access mode allows us to do that. Allows us to have single-writer access to storage.

One thing like I would want to highlight here, is it's very easy-- just from the names of these different access modes we have, it's very easy to confuse this one, this new one, ReadWriteOncePod with one we have from past ReadWrite ones.

KASLIN FIELDS: Mm-hmm, pod versus not pod.

PRIYANKA SAGGU: Yeah, exactly, the one with the pod and the one without a pod. So the difference between these two is ReadWriteOnce, which is the one without pod. That's also access mode. But that restricts the volume excess to a single node, not a single pod.

So that access mode, if you are using-- if you're creating a persistent volume with that access mode, which means anything on that volume could be accessed by multiple pods on the same node. But this ReadWriteOncePod does that. It's just restricted to one pod in the entire whole cluster. So that is where this new access mode is important.

KASLIN FIELDS: One thing I'm always explaining to folks who are new to Kubernetes is you might not know where your workload gets deployed to. It might be on the same virtual machine as another one, or it might not be. So this helps to manage that, where you can restrict access to storage just to a specific pod.

So you might have several copies of a pod as part of a deployment all doing the same thing. And if they both get deployed to the same node, then they try to write to the same storage. You might have trouble. So this helps to address that, right?

PRIYANKA SAGGU: Yeah, exactly. And this mode is actually available by default from 127. But 1.29 and later, it's actually GA and stable. The Kubernetes API recognizes this, yeah.

KASLIN FIELDS: Awesome, good stuff.

PRIYANKA SAGGU: And one more I have, I think that's a quick one that's from SIG instrumentation. It's KEP 727, kubelet resource metrics endpoint. So a bit of backstory on this one. We have something called kubelet summary API.

That API actually is a source of two things to us. Number one, it provides us some data on some meta on resource metrics. And the other thing it does is provides some information on monitoring metrics. But since it is doing two things, from the documentation-- it's not what I'm saying. It says, because it's a dual purpose, it does a poor job of both.

KASLIN FIELDS: Yeah, I love this. And one thing we were talking about earlier, is I tried to do a little bit of research for this, of course. And I was going through the KEPs. And I realized that there's actually two different places where you can find KEP information.

Traditionally, when I've tried to look up a KEP, I've looked up the issue that is the KEP, which honestly does not tell you much. If you actually want to learn about the KEP, there's not a lot of information there. The issues are mainly for tracking the progress of getting the KEP into Kubernetes and releasing it. But there's also a readme for the KEPs, which is really where you want to go if you want to learn about KEPs. And that's where this great line comes from of, because it's dual purpose, it does a poor job of both.

PRIYANKA SAGGU: Yeah, and just adding to that readme, right next to that readme there is another file KEP.YAML that can give you a historical context of who owns this particular feature, who are the supporting SIGs, or when this cap was alpha, beta, or when it is scheduled, for example, if it's going to be, let's say, planned to be GA in some other future release cycle. You can get those kind of information from that KEP.YAML file, as well. So yeah, both KEP readme, we call it, and KEP YAML file, those are the places where you would want to look at to understand about any KEP.

And now back to this KEP, KEP 727, we were talking about kubelet summary API, doing two things and not doing good at any of them. The 727 KEP kubelet resource metrics endpoint, this basically deals with the first use case of summary API, which is getting resource metrics, or in simple terms, getting information on CPU and memory metrics for pods and nodes. So this KEP actually proposes that we should have a purpose-built API for supplying these resource metrics.

KASLIN FIELDS: Instead of doing a poor job of both. [LAUGHING]

PRIYANKA SAGGU: Yeah, yeah, and so in 1.29, it's actually going to-- the resource metrics API will be there. And I think it was there from previous releases, as well. Specifically in 1.29, they are graduating few metrics data, metrics endpoints to general availability.

And I'll not read through that list of what all is coming as part of this KEP. I think docs will do a better job of that. But some points, like it's going to provide us information like container CPU usage, seconds total container memory, working set bytes, and all these kind of new fields are coming in to provide us better data on our resource metrics.

KASLIN FIELDS: Always important to have better metrics, especially on resources. Set your resource requests and limits.

PRIYANKA SAGGU: Yeah, this one reminds me of the one KEP we discussed early on, in-place update of pod resources, so [? 1287, ?] 77. Yeah, both of these are important if you are looking for optimizing your resources.

KASLIN FIELDS: Which please do. The more efficient we are, the better.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: And then there were a couple more KEPs that we wanted to highlight from beta and alpha, right?

PRIYANKA SAGGU: Yeah, I think beta, I have one very specifically. We talked about one during major changes, which were sidecars. So sidecar is actually graduating to beta, as well.

Another side I want to talk about is KEP 2799. The title is Reduction of Secret-Based Security Account Tokens. This KEP will also require some sort of backstory to understand what this is doing. We know about this concept called service account in Kubernetes. So a service account is something that provides an identity for processes that runs inside a pod container.

KASLIN FIELDS: It's Kind of an account for a service, right? An account for a thing, a workload.

PRIYANKA SAGGU: That's exactly, it is, yeah, an account that provides identity. And here in this case, we are talking about identity for the processes that are running inside our pod containers. So any process inside a pod can use this identity of its associated service account to authenticate actually to the cluster API server. And we do not want this service accounts to be always present there. So that is what is the concern of this particular KEP, that before 122, we used to have these legacy service account tokens.

In simple terms, we used to have these service account objects inside our Kubernetes cluster that used to add a volume based on a secret, where secret represented a service account for the pod. And that secret had some value that just would never expire. So when a pod needed that service account and the service account is automatically created, but now the pod no longer uses it, there is no way-- since we did not have any expiry for those kind of service account tokens, that was a problem, because now these service account tokens are lying there.

And anybody can use them, like any other entity. Any other unwanted entity in the Kubernetes cluster can use them. So this KEP actually proposes some actions to reduce the surface area of those legacy service account tokens.

And what they have done, starting from 1.29, so there is this feature gate they have added. I think it was added before an alpha. But that was not enabled by default. In 1.29, that will be enabled by default.

And we are talking about this feature gate called legacy service account token cleanup. So by the name itself, its work is to actually clean those legacy service account tokens, which are no longer used by any pod. And the implementation detail is very interesting here.

So what it does, is if this feature gate is enabled, which it will be by default in 1.29, so any legacy auto-generated service account token secrets, they will be auto-labeled with a label called kubernetes.io/legacytokeninvalidsince. So if a pod is not using like a service account token which is lying there in a cluster let's say for a year, now Kubernetes will itself add like a label to it, legacy token invalid since. It will add a label to it with that date.

And now it will wait for another year. If for that Another Year or-- we can specify that time period, as well. If no other pod still uses that lying-around legacy service account token, that will be deleted. So that is the intention of this KEP.

KASLIN FIELDS: So it's like very slow garbage collection of secrets that are sitting around and being a security risk.

PRIYANKA SAGGU: Yeah, we want to make sure-- we do not want to delete tokens--

KASLIN FIELDS: That are being used.

PRIYANKA SAGGU: --by any slight chance they might be used. So we are giving enough time. One year is default. But we can specify what kind of cleanup period we want to have, yeah.

KASLIN FIELDS: OK, I feel like when I get a chance to dive into IM any kind of Identity Management stuff, at first, it seems terrifyingly obscure and very hard to understand, which it is. But any chance I get to actually dive into it, I feel like the implementations of those types of features and tools are very interesting. And this is the same.

PRIYANKA SAGGU: Yeah, they mostly work on these labels and annotations, yeah.

KASLIN FIELDS: Yeah, very interesting. And the tie-in with secrets and the classic, don't use secrets for things that you actually want to keep secret because the level of security that they actually provide within Kubernetes is somewhat minimal. So avoiding secrets wherever possible is also a common trend in Kubernetes.

PRIYANKA SAGGU: Yeah, correct. I agree with you. If you just start looking at one component of Kubernetes, it's such a whole learning journey. You will start from somewhere, and you will end up somewhere. And there will still be like a lot of things.

KASLIN FIELDS: And so, so many things you don't know.

PRIYANKA SAGGU: To explore, yeah.

KASLIN FIELDS: Like Joe Beda has said in the past, Kubernetes is too big for any one person to understand at this point.

PRIYANKA SAGGU: Yeah, very sure.

KASLIN FIELDS: Last one.

PRIYANKA SAGGU: Coming to the last one, yeah. Yeah, so we have one alpha, which is interesting. This one is from SIG Network. It's KEP 3866.

Simple, this is for nftables Kube proxy backend. So what this KEP is doing, it's still in alpha. So if you want to use it, you have to enable it through feature gates. But it is actually making an nftables as an available mode where you can start using nftables for configuring your packet forwarding rules.

This proxy mode is actually just available for Linux nodes right now. It's not available for any other nodes. And also, this nftables API is actually kind of a successor to the IP tables API, which is, let's say, the mode right now.

KASLIN FIELDS: Yeah, and there was actually a talk on this at the Contributor Summit. I think it was recorded. I'll try to have a link to that talk in the show notes.

PRIYANKA SAGGU: Thank you, yeah. So this nftables is actually meant to be a successor to iptables. But right now, it's in alpha. And even the people who have actually created it right now, they don't really expect it to outperform other Linux mode at this point of time. But let's say in further iteration when it graduates to beta and GA, we can see that coming.

KASLIN FIELDS: Yeah, the talk featured some very interesting discussion around the trade-offs of whether we should use iptables nftables as the Kube proxy back end. And the main point from that that is sticking in my brain is that they were saying a lot of the Linux community is moving away from iptables, in general, and moving toward nftables.

PRIYANKA SAGGU: That's correct, yeah.

KASLIN FIELDS: Yeah, so it's kind of the successor to iptables, even beyond just this use case within Kubernetes with Kube proxy, kind of in the Linux ecosystem, in general.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: Yeah, cool, so we made it through all of the KEPs. Thank you to everyone who's stuck with us this far. I have just a couple more questions that I want to ask you before we wrap up. Number one, what were you most excited about for this release?

PRIYANKA SAGGU: So for me, I think this opportunity to be the release lead was the most exciting part, because this is my first time stepping into a role where I'm working as a people manager or a project manager, and actually a mixture of both. And I'll be honest, I had like a lot of learning to do during the entire time.

And I learned. And I grew a lot. So that was the most exciting part for me, honestly.

KASLIN FIELDS: Yeah, and being a release lead is a huge job.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: There's a lot of leadership skills that you can gain from it.

PRIYANKA SAGGU: Yeah, correct. And I was a lead shadow twice before. So I've been part of release team from 1.23 till 1.27 I was not on 1.28. And then I now came back on 1.29. So I was a release lead shadow for 1.26 and 1.27 I think at that point if I imagine, I can't imagine what are the-- I'm trying to recall if, when I was in 1.26 and 1.27 as a lead shadow, do I had any idea of what really it means to be a release lead. I think no.

KASLIN FIELDS: It's hard to understand, right?

PRIYANKA SAGGU: Yeah, you get to know when you actually step into the role and you have to take care of everything.

KASLIN FIELDS: Such as open source.

PRIYANKA SAGGU: Yeah, that's correct. Yeah, we started from that, and ended up a full circle.

KASLIN FIELDS: Coming back to it.

PRIYANKA SAGGU: Yeah, yeah. So yeah, just getting into that role and having this whole journey of learning so many things, not just the release processes, but everything outside that, it was amazing.

KASLIN FIELDS: Around leadership and management of all of the different interests involved, and those kinds of things?

PRIYANKA SAGGU: Yeah, and making sure people are happy community-releasing members. And while making sure all that is there to still put the release out, yeah.

KASLIN FIELDS: Yeah, release lead is a service role, service and leadership, making sure that the whole project is working together and releasing something awesome.

PRIYANKA SAGGU: Yeah, I think one thing I loved about this release, when we started, I think I was expecting that it's going to be a bit hard this time because we had a very short code freeze time this time because of the KubeCon. KubeCon actually was scheduled for week 10. And if we look at all the release cycles that has happened in the past, code freeze lands in week 10 or week 11. But we were not able to do either of those because--

KASLIN FIELDS: Yeah, that's a tough time to have a freeze.

PRIYANKA SAGGU: Yeah, we want people-- before the code freeze week, we want the approvers and reviewers from the community to be available. But that was not happening. So we had to really prepone the code freeze to week 9, which actually gave people very less time to implement the features.

So that is where we got 72 in. And it's still like, I can't believe how we managed. It's not our work, actually. It's like Kubernetes work.

KASLIN FIELDS: So many people.

PRIYANKA SAGGU: How people managed to have the code implemented, as well. And we got some very long-standing KEPs, as well, in.

KASLIN FIELDS: Amazing.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: Awesome, I'm really excited for this. And for our last thing, why should listeners upgrade to 1.29? Releases are really hard. Upgrades are really hard to justify. They are a lot of pain. Why do you think our listeners should upgrade to 1.29?

PRIYANKA SAGGU: I think I also share the same feeling that Kubernetes upgrades are hard. But we just discussed about eight enhancements right now. There are 49 enhancements coming as part of 1.29. And 20 of them are alphas.

So there are New. 20 alpha features coming as part of this cycle. And 11 of them are stable. So if you want to try any of them, test any of them, I think you would want to try using Kubernetes 1.29 for that

KASLIN FIELDS: Yeah, especially that in-place update of pod resources.

PRIYANKA SAGGU: Yeah.

KASLIN FIELDS: I think a lot of people will be interested in that.

PRIYANKA SAGGU: I think people would be happy that that's landing now finally After I believe-- how many years? I can't count.

KASLIN FIELDS: You said it started in 2015, I think.

PRIYANKA SAGGU: 12, yeah. So 2015, I saw an issue that was closed recently. And that issue was opened in 2015. And if we work up the KEP issue [? 1287, ?] the KEP issue was actually started sometime in 2019 when we were discussing about version 1.17. So it's like 12, 13 cycles now.

KASLIN FIELDS: I think I was talking with Tim Hockin about this at the Contributors Summit or something. And he was saying there was just so much discussion on this about how they were going to do it. It took a very long time.

PRIYANKA SAGGU: Oh, yeah, every cycle-- if I remember, in past cycles, we have always had exception requests coming from this particular KEP. And this is making a lot of people happy that this is finally in.

KASLIN FIELDS: Yeah, and one more thing I want to mention before we wrap up here that we briefly mentioned at the beginning but I wanted to come back to, is the release did get delayed by about a week. And I was just talking with another community member who's been part of the community for a very long time, Josh Berkus. And he was saying that the reason for this delay is probably the most common reason that Kubernetes ends up delaying releases, is that there was a CVE found in Go.

PRIYANKA SAGGU: Oh, yeah.

KASLIN FIELDS: Apparently security holes in Go are the most common cause of release delays.

PRIYANKA SAGGU: Yeah, we got a report for a pre-- it was a pre-announcement for a security fixed releases coming for 120-point-something and 121-point-something Go versions. And those are actually landing on our original intended release date. So we decided to wait for those CVE fixes to come, and then decided to move ahead with them.

And before we actually wrap up, I have one more announcement or a reminder for everyone in the community. So legacy Linux packages repositories will be going away sometime early 2024. So please, please migrate to the Kubernetes community-owned packages repositories.

And by legacy package repositories, I mean the apt.kubernetes.io and yum.kubernetes.io, which are endpoints to packages.cloud.google.com. So those will be going away. There are two blogs for them if you have to migrate.

KASLIN FIELDS: And we've done a couple episodes talking about it, as well.

PRIYANKA SAGGU: Oh, yeah, so there are a lot of resources out there. But make the move now. It's finally happening. Those repositories will be gone sometime early 2024.

KASLIN FIELDS: Migrate to the Kubernetes-owned repositories, please. It's very important.

PRIYANKA SAGGU: Yeah, pkgs.k8s.io

KASLIN FIELDS: Yeah, and with that, this is a very big episode for a very big release. Thank you so much, Priyanka, for giving us such great insights into all of these many, many changes that are going into 1.29. A lot of really important ones. I'm excited for folks to try out the release. Thank you very much for being on, Priyanka.

PRIYANKA SAGGU: Thank you for hosting me. Thank you thank you everyone for listening.

[MUSIC PLAYING]

ABDEL SGHIOUAR: Well, Kaslin, thank you for that interview. That was really cool. A lot of details.

KASLIN FIELDS: Always a pleasure. Also super exciting to get to interview my co-lead in SIG ContribEx.

ABDEL SGHIOUAR: Yeah, you already work together.

KASLIN FIELDS: Yeah, always good to get to chat more. We're in very different time zones, so sometimes it's difficult.

ABDEL SGHIOUAR: Oh, where is Priyanka based?

KASLIN FIELDS: India, I don't remember where exactly.

ABDEL SGHIOUAR: India, OK. It makes sense, OK. It was actually pretty cool that they went to quite a lot of details in the conversation about the most important KEP, which there are two of them I am very excited about. And we're going to get to talk about them. But before we do that, let's talk about the theme.

KASLIN FIELDS: Yes, the logo is so gorgeous. I finally did get to look at it, and it's so pretty.

ABDEL SGHIOUAR: Yeah, I have it opened in front of me. I also found the text or the chat that Priyanka mentioned in Slack, where somebody noticed that they integrated the numbers. So 1.29 in Devanagari-- I hope I'm not saying that wrong, Devanagari numbers. If you look at the logo-- or you look at the logo for the theme, and you read it from left to right, it's one on the left, dot at the top, two on the right side, and nine in the bottom.

KASLIN FIELDS: They look gorgeous. I can't read them, but--

ABDEL SGHIOUAR: It looks amazing. I'm looking forward to have this as a sticker on my laptop, actually.

KASLIN FIELDS: Yes, that would be so cool.

ABDEL SGHIOUAR: Yeah, so we were chatting before we started recording, and I told you that once I heard in the interview the integrated number. I stopped, and I went to look at the logo. And I was able, actually, to read the numbers.

KASLIN FIELDS: That's so cool.

ABDEL SGHIOUAR: I've never even heard about this script, which apparently is popular in North India, the way they write numbers. And then we were chatting about a fun fact, which is the way we write numbers in the West, people think those are Arab numbers. But they are not. They are Hindu-Arabic numbers. Arab numbers are completely different.

KASLIN FIELDS: And then these ones are, yet, a different kind of Indian-- it must have some connection to Arabic, as well, right?

ABDEL SGHIOUAR: Yes, they look similar to Arabic. So when I was studying in Morocco, obviously, we speak Arabic and French. So we were writing, for example, dates. When you write it, and you have a paper, and you write a date, you write it in both.

So then in the Arabic, you just use Arabic numbers, or the actual Arabic numbers; not what the rest of the world thinks the Arabic numbers are. So if you're listening to this episode, go to Wikipedia and look Arabic numbers. And you will see similarities between the Arabic numbers and the Devanagari numbers.

KASLIN FIELDS: I love language fun facts.

ABDEL SGHIOUAR: Yeah, it's quite interesting.

KASLIN FIELDS: Especially natural ones. But it's so cool how they integrated so much meaning into the logo. There are releases where it's just a fun logo, and it's just cool to look at and enjoy. This one, they were like, no, it has all of the meaning.

Talking about Kubernetes as the universe, it's got the numbers. It's fantastic. I really enjoy it.

ABDEL SGHIOUAR: Yeah, so for reference, the theme is mandala.

KASLIN FIELDS: Yeah, which we talked about the interview. Yeah, I had no idea that meant universe until this interview.

ABDEL SGHIOUAR: Oh, yeah, I was actually looking at the meaning. I'm looking at the release page for the release. We'll leave a link in the show notes. But yeah, let's go into details, 72 KEPs.

KASLIN FIELDS: It's ridiculous. [LAUGHING]

ABDEL SGHIOUAR: It's a lot.

KASLIN FIELDS: I think that was what they started out with. But they ended up with like 40-some, which is still ridiculous.

ABDEL SGHIOUAR: Which is still quite a lot, yeah. So I think that probably a couple of the ones that were really the most interesting is the removing the in-tree code, like cloud providers in-tree code.

KASLIN FIELDS: Oh yes, yeah, the cloud provider removal bit yeah.

ABDEL SGHIOUAR: Correct, so that's basically removing everything that is cloud-specific from in-tree. And by in-tree, I mean in a tree.

KASLIN FIELDS: Yeah, within the tree of Kubernetes, within Kubernetes's own code. Yeah, this is a-- I think I mentioned in the episode, as well, that this in-tree/out-of-tree thing is such a fascinating trend throughout the project to me, because we've talked about before on the 1.28 release interview, a lot of the major changes in that one were also removing storage providers from in-tree to out-of-tree.

So I'm noticing a theme that a lot of the things that we're moving out of the tree of Kubernetes are vendor-specific things, which makes a lot of sense to move out of the open source project itself. They're more like plugins that you can use if you need them. But if they're integrated into the project, then they have to follow the project's release cycle, and there's all kinds of problems.

ABDEL SGHIOUAR: Yeah, I think that that also keeps Kubernetes-- makes the projects more maintainable in a way.

KASLIN FIELDS: Yeah, that's the main goal of actually doing this.

ABDEL SGHIOUAR: And also give people options because then you don't have-- you can choose whatever plugin you want.

KASLIN FIELDS: Yeah, and it minimizes the complexity and the size of Kubernetes itself as much as we can, so that you can just use what you need. And then also, you get better pieces for those pieces because the individual vendors can manage the release cycles of those plugins, themselves, rather than being tied to Kubernetes, and then everybody has to work together. And we know how that works. [LAUGHING]

ABDEL SGHIOUAR: Exactly, yeah, there was quite a lot of-- I'm not going to go into all the details. But the page that describes the deprecations and enhancements or the changes in general in 1.29, we'll have a link for it in the show notes. But the parts you discussed, which was interesting, is this is going to impact people who are doing self-hosted Kubernetes on top of things like GCE, Google Compute Engine, Azure, or what was the third one? vSphere, right?

KASLIN FIELDS: Yeah, and I think this is something that trips folks up a lot who are not very familiar with the Kubernetes space. It's like, Kubernetes is an open source project. And you can run it on computers to tie them together as a distributed system. You can do that yourself, or you can have someone else do it. The people this will affect are the people who are doing that themselves.

ABDEL SGHIOUAR: Yeah, to be precise, self-manage, which means you deploy Kubernetes yourself using something like Kubeadm, or Kubespray, or Cluster API, or something like that.

KASLIN FIELDS: Or pain and suffering of understanding the deep-level components of Kubernetes itself.

ABDEL SGHIOUAR: Yes, that's what Kelsey Hightower--

KASLIN FIELDS: Doing it the hard way.

ABDEL SGHIOUAR: --usually called Kubernetes the hard way, exactly. [LAUGHING]

KASLIN FIELDS: Precisely.

ABDEL SGHIOUAR: Cool, cool.

KASLIN FIELDS: Also, by the way, I wrote a blog post about the in-tree/out-of-tree thing. So I'll put a link for that in the show notes in case people wanted to--

ABDEL SGHIOUAR: Yeah, I saw that one. Yeah, I haven't read it yet, but I saw it.

KASLIN FIELDS: I had a lot of fun doing that.

ABDEL SGHIOUAR: Nice, so there was another one which, actually, you started talking about. And it's so funny because I am literally writing an article with a colleague of mine in Europe about this specific feature, which is the in-place resource changes, essentially being able to add memory or CPU to pods without restarting them.

KASLIN FIELDS: Yeah, that's such an interesting one.

ABDEL SGHIOUAR: Yeah, so what we're writing a blog about, it's a use case. I think you didn't discuss this. But here is the use case. If you're running Java apps on Kubernetes, Java apps typically need a lot of CPU at the beginning of the startup of the application to be able to load all the classes. So all the auto-scaling using Kubernetes doesn't work because you only need that CPU memory during the startup phase. And then the app might not need these resources while it's running.

So Kubernetes has something called the CPU boost feature which was released in, I think, 1.28. And paired with this in-place resource updates, you can essentially create a CPU boost resource that will give more CPU to the pod during a specific period of its startup phase or when the app is starting. And then it will just remove that CPU with the in-place resource changes.

KASLIN FIELDS: Cool, I had no idea about this use case, which makes the whole change a lot more interesting.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: It was already interesting, but--

ABDEL SGHIOUAR: Yeah, I don't think that they introduced it as a way for this specific use case, but it's just one common use case, right?

KASLIN FIELDS: Yeah, but it's one where this is particularly relevant. And I think that's great. A great way to understand how it works.

ABDEL SGHIOUAR: Exactly, we're actually working on an article that will be hopefully coming out soon. And it has even a demo that shows a Spring Boot app, and it compares performance of startup of the app when you use the CPU boost with in-place resource changes compared to not without CPU boost.

KASLIN FIELDS: Interesting, I look forward to that. More reading on my list.

ABDEL SGHIOUAR: Yeah, and then there was the sidecar feature going to beta.

KASLIN FIELDS: Yes, I've been explaining that to folks a lot. I think I mentioned that in the interview, as well. I've encountered quite a few folks recently, especially students and folks newer to the community and newer to the industry who still need-- well, I mean, there's a lot of folks who need an overview of what a sidecar is, because it's a weird concept.

If you've been following the service mesh space which we have, sidecars are like, oh, yeah, we talked about that like two years ago. Or more than that, maybe was sidecars were the big thing. So it's like, oh, yeah, they're doing the sidecar thing. We get that. But a lot of folks, the concept of a sidecar is still very new. If you weren't following the service mesh space or if you don't work with service meshes, I feel like it's not something that you would have thought about.

ABDEL SGHIOUAR: Yeah, but it's not only relevant for service mesh. It's only relevant for things like logging and monitoring, like if you run a logger next to your app. So I mean, we have discussed this feature many, many times before of what this means.

But what I still find very interesting is it took Kubernetes close to 10 years-- it's 10 years in September next year-- for the community to come around and decide-- just for the reference, in Kubernetes, the concept of sidecar is not an actual concept. It's a convention. It's just a common thing that people agreed that we're going to call this a sidecar.

KASLIN FIELDS: Yeah, it was just a naturally-developed way of doing things.

ABDEL SGHIOUAR: Exactly, so it's just very fascinating that it took almost 10 years for people to go like, OK, well, now we need an actual fix.

KASLIN FIELDS: Well, granted, how much of an actual fix is it? It's like taking a thing that already existed within Kubernetes. It's very much like the concept of a sidecar itself. It kind of develops naturally from the existing capabilities of Kubernetes.

It's not like, there's a brand-new thing that is sidecars, and we've been talking about these forever, and now Kubernetes has them. No, it's just like Kubernetes was like, oh, yeah, this pattern is a thing. Here's a better way to do that with the existing functionality.

ABDEL SGHIOUAR: Yes, and for the reference, what Kaslin is talking about, the fix is taking init containers which already existed, and just adding a field to them. That's it. That's the fix.

KASLIN FIELDS: Yeah, that's all it is. And it was discussed. Folks discussed the possibility of creating a full, new thing that's like a sidecar container object.

ABDEL SGHIOUAR: Like an actual resource.

KASLIN FIELDS: Yeah, a resource within Kubernetes. And it was just going to be way too disruptive to the whole project. So this was the easiest way to get it done without breaking anything.

ABDEL SGHIOUAR: Yeah, I know, it's quite interesting. There was quite a lot of KEPs that were discussed. I don't want to go into all the details.

KASLIN FIELDS: There were a lot. It's a long episode. We thank you for getting to this part. [LAUGHING]

ABDEL SGHIOUAR: Yes, thank you for sticking with us. I think there was some skirmish stuff. And there was some like new ReadWrite mode thingy with storage that was introduced. But I think the most important probably, if you are still here, is remember the migration from the repositories, moving from the Google-owned repositories or Google-hosted repositories to the community-hosted repositories.

KASLIN FIELDS: Yes, I need to write several blog posts about that. Everybody, move your stuff. People are not going to move it. I know this.

And it's going to be a whole thing. They're going to turn it off. And there's going to be a firestorm on Twitter somewhere. Somewhere within Twitter, people will be upset.

ABDEL SGHIOUAR: Yeah, while I was preparing for this episode, I saw on, I think, "Hacker News," somebody wrote a blog that says Kubernetes needs an LTS. So it sounds like some of these long-term instability topics are going to stick around for a while.

KASLIN FIELDS: They're working on that. There is a long-term support working group within the open source project right now. Microsoft and a bunch of other vendors are on board trying to figure out what long-term support looks like in Kubernetes.

It's very interesting when you start diving into it how complicated it gets to be like, well, we need people to upgrade. But it's really hard for them to upgrade. And we know that. And we need to support them.

But also, we still need to support them upgrading eventually. So there's a lot of considerations that go into how you make it possible to stay on the same version longer while still keeping you from getting yourself into a bad state, as much as possible, which we know some people want the ability to shoot themselves in the foot. [LAUGHING] So that's another consideration that goes into it.

ABDEL SGHIOUAR: Yeah, I mean, jokes aside-- with all the jokes made about this topic, I think it's a very important topic. Maintainability and sustainability when you run Kubernetes is important, right?

KASLIN FIELDS: Very, very important.

ABDEL SGHIOUAR: Cool, well, thank you very much, Kaslin, for the interview. That was pretty cool.

KASLIN FIELDS: Of course. It was my pleasure to get to do that. And I learned so much. It was so awesome that Priyanka went through and grabbed a whole bunch of KEPs to talk about. And she did a bunch of reading, because that's what we've talked about on previous release episodes.

The job of the release lead is really about coordinating all of the different interest groups involved in a Kubernetes release more than it is about understanding the technical details of the release itself. But Priyanka is a tech lead for a reason. She went in and just learned all the things. And so we were able to talk about it. And that was really awesome.

ABDEL SGHIOUAR: Yeah, thank you very much, Priyanka, as well. And with that, this is our last episode of 2023.

KASLIN FIELDS: Yeah! We've made it as the hosts of this podcast for a whole year now, folks. Can you believe it? [LAUGHING] How we've grown.

ABDEL SGHIOUAR: Yes, we learned a lot. Definitely learned a lot.

KASLIN FIELDS: Thank you for sticking with us or for joining us. We appreciate having you.

ABDEL SGHIOUAR: Yeah, thank you for everybody who listened, who left nice feedback. We'll try to do more cool, interesting stuff next year. And what are your plans for the holidays, Kaslin?

KASLIN FIELDS: Oh, I'm going to visit family. And then my partner and I, we always host a New Year's party where we watch Asian New Year's programs because the Korean program and the Japanese New Year's programs, they go all out. The Korean program is a review of all of the pop music for the year. And the Japanese program has all of these fun little activities that they do with stars and stuff. So they're fantastic to watch. And that's what we do for New Year's.

ABDEL SGHIOUAR: Awesome, nice. I'll be spending my Christmas and New Year's back home in Morocco. I'm trying to run away from the cold in Sweden.

KASLIN FIELDS: Understandable. [LAUGHING]

ABDEL SGHIOUAR: So I'll just be with my family eating good food and being take care of.

KASLIN FIELDS: So we hope you all have a wonderful and pleasant holiday season, whatever that may mean for you. And we'll see you next year in 2024.

ABDEL SGHIOUAR: Yes, we'll see you next year. Goodbye.

KASLIN FIELDS: Bye.

[AUDIO LOGO]

That brings us to the end of another episode. If you enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media at Kubernetes Pod, or reach us by email at <kubernetespodcast@google.com>.

You can also check out the website at kubernetespodcast.com where you'll find transcripts and show nodes and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]