#146 April 15, 2021

Kubernetes 1.21, with Nabarun Pal

Hosts: Craig Box, Daniel Smith

Celebrate the release of Kubernetes 1.21 with release team lead Nabarun Pal from VMware. Nabarun talks about choosing between “hardware” and software, additions and removals from Kubernetes 1.21, and how the Kubernetes project has become more welcoming to people outside the USA.

Do you have something cool to share? Some questions? Let us know:

Chatter of the week

News of the week

CRAIG BOX: Hi, and welcome to the Kubernetes podcast from Google. I'm Craig Box with my very special guest host Daniel Smith.


CRAIG BOX: Welcome back to the show, Daniel. We last had you on episode number 73, which was in October 2019.

DANIEL SMITH: Only a few things have changed about the world since then.

CRAIG BOX: I hear that you got your COVID injection the other day.

DANIEL SMITH: Yeah, I did, and this is very important, but it's actually an anti-COVID injection. They're not giving you COVID.


DANIEL SMITH: They're helping you not get COVID.

CRAIG BOX: A very important distinction.

DANIEL SMITH: Yeah, I want to make that clear.

CRAIG BOX: I find it very interesting, the distinctions between the way that the vaccine rollout is happening in the UK and the US, and first of all, I should say we are two countries that are blessed to be able to have early access to the vaccine and a relatively successful rollout. I realize not everyone is in that situation. But given that the UK has a National Health Service that knows who everybody is, they're making their way down-- they started with people over 80, and they're working their way down. They've just recently opened up to people over 45 in the last couple of days.

But what I'm hearing about the US is they very quickly got to the point where they have an open call, and anyone over 18 can turn up now in most states. And that's just making me feel a little bit jealous because lots of people I know in the US who definitely aren't over 50, or who are in the priority groups that have only just been done here in the UK, they're all going and getting their shot, and I haven't had mine yet.

DANIEL SMITH: Yeah, it just opened up for everybody, so it's been great. I mean, it's been three or four months of seeing other people get vaccines, so I finally was eligible. So I scheduled it as soon as I could.

CRAIG BOX: And did the operation run with military precision?

DANIEL SMITH: Yeah, the one I happened to get did. In fact, I think it may have been run by the military or something. I'm not sure. The guy that poked me was in military fatigues, so I don't know. I don't think it was just a fashion statement.

CRAIG BOX: I've heard story of the Moscone Center in San Francisco, which is where many Google Cloud Next events have happened, as well as many other tech company events. It's now a giant COVID vaccination center which is run by the military, and apparently you basically go in on a conveyor belt, and you're given everything you need, and eight minutes in you're jabbed, and you sit down and wait, and then you're out again.

DANIEL SMITH: Yeah, that's about how it happened. I went to a much smaller one, but there wasn't a conveyor belt. So now, I feel left out.

CRAIG BOX: Well, the UK has been under lockdown for almost five months now, and restrictions are gradually lifting. Some restrictions that lifted this week-- people are now able to go to the pub, but only to drink and dine outside. And of course, when this restriction changed on Monday, it was snowing, so everyone was thinking, should I, should I not? A lot of people who are quite desperate. It did actually warm up by lunchtime.

DANIEL SMITH: So you're saying the UK has developed weather control technology.

CRAIG BOX: It might seem that way, just the general breathing out of everybody thinking, hey, it's been so long, just please warm up just a little. But people here are used to it being cold. They're willing to sit outside. More interesting for me, of course, was the fact that the hairdresser was able to reopen, and I was finally able to take the four or five months of growth off the top of my head.

DANIEL SMITH: I'm sorry to hear that. As someone who, I dislike haircuts, so if you've seen pictures of me, and I have long hair, that's why.

CRAIG BOX: Did a hairdresser wrong you in your youth?

DANIEL SMITH: Believe it or not, no. I mean, not unless you count being forced to have your hair cut.

CRAIG BOX: I actually have a very good barber, and it turns out my barber has the world record for the most number of haircuts in 24 hours.

DANIEL SMITH: I didn't know that was a thing that you could have a world record in.

CRAIG BOX: It is. He set this record in June 2011, and apparently, he had 526 haircuts in a 24 hour period.

DANIEL SMITH: This is counting individual people, not individual hairs?

CRAIG BOX: No, that's the number of people that went through. I suggested to him that maybe he could have broken that record on Monday with the amount of pent up demand and people who were willing to go in and get their haircut, but I imagine he needs to have a pipeline of people waiting to be able to work at that speed.

DANIEL SMITH: I would imagine so, yeah. I've never had my hair cut by a world record holder, so maybe that's why I don't like haircuts.

CRAIG BOX: Well, next time you're in town, I'll hook you up with my barber, and we'll meet up outside the pub on a cold, cold day.

DANIEL SMITH: Having my hair cut in the cold sounds great.

CRAIG BOX: It's better than it seems.

DANIEL SMITH: I'll take your word for it until I visit.

CRAIG BOX: Until then, let's get to the news.


CRAIG BOX: Last week, we brought you the news that Kubernetes 1.21 had been released, and this week, we'll dive into the release with team lead Nabarun Pal. Also diving into 1.21 is the Kubernetes blog, which has posts up about CronJob going GA, three new storage features now in beta, and the ability to suspend jobs introduced in alpha.

DANIEL SMITH: Also on the Kubernetes blog this week, the Kube-State-Metrics project has released version 2.0. Kube-State-Metrics generates Prometheus format metrics for Kubernetes resources like Deployments and Pods by listening to the Kubernetes API. V2 adds performance and scale improvements and support for multiple architectures

CRAIG BOX: A new Ingress project has joined the CNCF. emissary-Ingress, formerly known as Ambassador, was built by Ambassador Labs, formerly known as DataWire. emissary-Ingress is an open source Kubernetes native API gateway built on the Envoy proxy, and joins the CNCF as an incubation level project.

DANIEL SMITH: Lots of people have production infrastructure held together with bash scripts, but dream of the modern world of Kubernetes operators. Dream no more because Flant's shell-operator has hit 1.0 this week. shell-operator subscribes to events from Kubernetes objects and executes an external program when an event occurs, providing it with information about the event.

CRAIG BOX: You're one of the authors of the Kubernetes API server. Is this how you imagined it being used?

DANIEL SMITH: I mean, I like to say that Kubernetes was written in Bash with some go-helper scripts, so why not have the worst of both worlds?

CRAIG BOX: Security vendor Cyberark has released Kubesploit, a new open source tool for testing Kubernetes environments. The framework assists penetration testers and red teams mimic real world attack scenarios. They have also helpfully published a set of rules to detect potential use of Kubesploit in a real attack situation.

DANIEL SMITH: A security issue was discovered in kube-api-server that could allow node updates to bypass validating admission webhook. Unless your plugin cares about the old values for particular fields of node objects on updates, you probably don't have a problem. The bug, which has a CVE rating of medium, is fixed in New Kubernetes releases from 1.18 to 1.21.

CRAIG BOX: In a crowded marketplace of Postgres operatives for Kubernetes, from Crunchy Data and Zalando, comes a new entrant simply called Kubegres. Built by London company Reactive Tech, Kubegres promises simple YAML, a history of production use by their customers, and over 55 test cases.

DANIEL SMITH: I'm shocked that no one has yet released one called Posternetes.

Also on the operator train this week is S3-like storage vendor MinIO with a new operator and a new operator console. The operator includes support for operations like non-disruptive upgrades and cluster expansion that were previously handled in a Helm chart. The console allows anyone in an organization to manage object storage through a web interface with a Prometheus driven dashboard.

CRAIG BOX: Back in 2019, Pinterest pulled back the curtain on how they operate Kubernetes. With us all having spent the last year at home and living life vicariously through the internet, Pinterest has scaled their environments to match. Theirs is a story of overzealous auto scaling and software bugs causing API server overload, and they work through how these problems were identified and corrected.

DANIEL SMITH: SUSE has introduced a cloud native architecture Nanodegree on the Udacity platform. They're sponsoring 300 scholarships to the program, which will be merit based and allocated across a diverse range of course participants. SUSE is actively encouraging women and people from underrepresented backgrounds to participate in the program, which launches in August.

CRAIG BOX: Finally, last week, we brought you the news of the imminent archival of the Apache Mesos project. At the 11th hour, the Apache community found enough interest to keep the attic door locked, and thus, Mesos will continue as was, with a new project chair being elected to kick start this revitalization.

DANIEL SMITH: And that's the news.


CRAIG BOX: Nabarun Pal is the release team lead for Kubernetes 1.21 and a member of technical staff at VMware. Welcome to the show, Nabarun.

NABARUN PAL: Hi, everyone. It's really great to be here.

CRAIG BOX: You have a Bachelor of Technology in metallurgic and materials engineering. How are we doing at turning lead into gold?

NABARUN PAL: Well, last I checked, we have yet to find the philosopher's stone.

CRAIG BOX: One of the more important parts to the process?

NABARUN PAL: Yeah, we're not doing that good in terms of getting the alchemists running, but there is some improvement in nuclear technology where you can turn lead into gold, but I would guess buying gold would be much more efficient.

CRAIG BOX: Or Bitcoin-- it depends what you want to do with the gold.

NABARUN PAL: Yeah, seeing the increasing prices of Bitcoin, you'd probably want to bet on Bitcoin. But again, don't take this as a suggestion. I'm not a registered investment advisor, and I don't give investment advice.

CRAIG BOX: But you are, of course, a trained materials engineer. How did you get into that line of education?

NABARUN PAL: That was a bit interesting prospect. So I studied at Indian Institute of Technology, Roorkee. And we have a graded and equated exam structure, where you give a single exam, and then based on your performance in that exam, you can try any of the universities which take those scores into account. So I went into my university, which is IIT Roorkee.

Materials engineering interested me a lot. I had a passion in computer science since childhood, but I also like material science, so I wanted to explore that field. And that's how I landed in material science. Did a lot of exploration around material science and metallurgy in my freshman year and sophomore year, but then computing, since it was a passion, creeped into the picture.

CRAIG BOX: Let's dig in there a little bit. What did computing look like during your childhood?

NABARUN PAL: It is a very interesting journey. So I started exploring computers back when I was seven or eight. The first programming language, if you call it a programming language, I explored Logo.

You have a turtle on the screen, and you issue commands to it, like move forward or rotate or pen up or pen down. You can basically draw geometric figures. So I guess that was one of the really interesting things where I can visually see how I can draw a square and visually see how I can draw a triangle. It was an interesting journey after that. Learned BASIC, then went to some amount of HTML, JavaScript.

CRAIG BOX: It's interesting because Logo and BASIC were probably my first two programming languages, but I think there was probably quite a gap in terms of when HTML became a thing after those two. [CHUCKLES] Was it something that your love of computing always led you down the road of programming, or were you interested as a child in using computers for games or educational– application software? What led you specifically into programming?

NABARUN PAL: Programming came in late. One of the things that has really intrigued, and this is not just in computing, but in life, I'm curious with things. When my parents got me my first computer, I was a bit curious. I was like, how does this operating system work? What even is running in it?

Because using a television and using a computer is a different experience, but usability is kind of the same thing. You use– the HCI device is a remote, whereas in computer, you have a keyboard and a mouse. I used to tinker in the box and reinstall operating systems.

We used to get magazines back then. They used to bundle OpenSuse or they used to bundle Debian and I used to install them. And that was an interesting experience, 15 years back, how Linux used to be. So I have been a tinkerer all around, and that's what led me to programming eventually.

CRAIG BOX: So with an interest in both the physical and ethereal aspects of technology, you did a lot of robotics challenges during university. That's something that I actually associate with someone who has a background in Logo to be honest. There's a lot of Mindstorms and technology that is based around robotics that a lot of Logo people got into. How was that something that came about for you?

NABARUN PAL: When I joined my university, apart from studying materials, one of the things that my university used to really encourage was to get involved in a lot of extracurricular activities. One of them which interested me was robotics. I joined my college robotics team and participated in a lot of challenges.

Predominantly, we used to participate in this competition called ABU Robocon, which is Asia-Pacific Broadcasting Union conducted event. What they used to do is, every year, each of the participating countries in a contest, one of those countries used to provide a problem statement. For example, one of the years, they asked us to build a badminton-playing robot.

One of the years, they asked us to build a rugby playing robot or a Frisbee thrower, and there are some interesting problem statements around it. You can't do this. You can't do that. Weight has to be like this. Dimensions have to be like that.

I got involved in that, and most of the time at university, I used to spend there, material science became kind of a backburner for me, and my hobby became a full time thing.

CRAIG BOX: And you were not only involved there in terms of the project and contributions to it, but you got involved as a secretary of the team, effectively, doing a lot of the organization, which is a thread that will come up as we speak about Kubernetes.

NABARUN PAL: Over the course of time, when I gained more knowledge into how the team works, it became very natural that you graduated up the ladder and then manage juniors. So I became the joint secretary of the robotics club in our college, which is the Models and Robotics section. This was more of a broader engaging role into evangelizing robotics in the university to promote more events, to promote most students to see the value in learning robotics. What do you gain out of that mechanically or electronically, or how do you develop your logic by programming robots?

CRAIG BOX: Your first job after graduation was working at a company called Algoshelf, but you were also an intern there while you were at school.

NABARUN PAL: Yes, my journey with Algoshelf-- previously, it was known as Rorodata when I joined them as an intern. This was also an interesting opportunity back then for me in the sense that I was always interested in writing programs which people would use, and one of the things that I did there was to build an open source Function as a Service framework, if I may call it, but it was mostly turning Python functions into web servers without even writing any code. The interesting bit there was that it was targeted toward data scientists and not towards programmers. We had to understand the pain of data scientists, that they had to learn a lot of programming in order to even deploy their machine learning models, so we wanted to solve that problem.

After that, they offered me to work for them after my internship, and I kept on working for them after I graduated from university. There, I got introduced to Kubernetes, so we pivoted into a product structure where the very same thing I told you, the Functions as a Service thing, can be deployed in Kubernetes. And I was exploring Kubernetes as to use a scalable platform, and the interesting bit was that, instead of managing pets, we wanted to manage cattle, as in, we wanted to have a very highly distributed architecture.

CRAIG BOX: Not actual cattle. I've been to India. There are a lot of cows around.

NABARUN PAL: Yeah, not actual cattle. That is a bit tough.

CRAIG BOX: While Algoshelf we're looking at picking up Kubernetes, what was the evaluation process like? Were you looking at other tools at the time? Or had enough time passed that Kubernetes was clearly the platform that everyone was going to use?

NABARUN PAL: Algoshelf was a natural evolution. So before Kubernetes-- we used to deploy everything on a single big AWS server whereas we used to deploy everything using systemd. Everything was a systemd service and everything was deployed using Fabric. So Fabric is a Python package which essentially is like Ansible, but much leaner as it does not have all the shims and things that Ansible has.

And then we figured out that "what if we need to scale out to different machines". Kubernetes was in the hype. And then we hopped onto the hype train to see whether Kubernetes is worth it for us. And that's where my journey started as in exploring the ecosystem, exploring the community. How can we improve the community in essence?

CRAIG BOX: A couple of times now you've mentioned as you've grown in a role becoming part of the organization and the range of it. You've talked about working in Python. You had submitted some talks to Pycon India. And I understand you're now a tech lead for that conference. What does the tech community look like in India and how do you describe your involvement in it?

NABARUN PAL: So my involvement with the community began when I was in my University. So when I was working as an intern in Algoshelf, I was introduced to this-- I never knew about PyCon India or conferences, tech conferences in general. But then I was introduced to PyCon India.

And then the person that I was working with just asked me, like hey, did you submit your talk to PyCon India? It's very useful, the library that we were making. So I submitted a talk at PyCon India in 2017. Eventually the talk got selected and although that it was not my first speaking opportunity, that was my second. I also spoke at PyData Delhi on a similar thing that I worked on in my internship.

It has been a journey since then. I talked about the same thing at FOSSASIA Summit in Singapore, and got really involved with the Python community because I used to work on Python back then. So after giving all those talks at conferences, I got also introduced to this amazing group called DGPLUG, which is an acronym for the Durgapur Linux Users Group. It is a group started in-- I don't remember the exact year, but it was around 12 to 13 years back, by someone called Kushal Das, with the ideology of training students into being better open source contributors.

And eventually I liked the idea and got involved with in also teaching last year in 2020. It was also not limited to students. Professionals can also join in. It's about making anyone better at upstream contributions, making things sustainable. I started training people on Vim how to use text editors so they are more efficient and productive. In general life, text editors are a really good tool.

The other thing was the shell. How do you navigate around the Linux shell and command line? So that has been a fun experience.

CRAIG BOX: It's very interesting to think about that, because my own involvement with a Linux User Group was probably around the year 2000. And back then we were teaching people how to install things and Linux on CD was kinda new at that point in time. And there was a lot more of, how do we get involved and what is this new thing?

And then when the internet took off around that time, all of that stuff moved online so that you no longer needed to go meet a group of people in a room to talk about this kind of thing. And I haven't really given much thought to the concept of a Linux Users Group since then, but it's great to see that having turned into something that's now about contributing rather than just how do I get things going for myself.

NABARUN PAL: Exactly. So as I mentioned earlier, my journey into Linux was like installing SUSE from DVDs that came bundled with magazines. Back then it was a pain installing things because you did not get any instructions to do this, to do that. So there has certainly been a paradigm shift now. People are more [open] to reading instructions online, downloading ISOs, and then just installing them.

So we really don't need to do that as part of LUGs. We have shifted more towards enabling people to contribute to whichever project that they use. For example, if you're using Fedora, contribute to Fedora; make things better. It's just about giving back to the community in any way possible.

CRAIG BOX: You're also involved in the Kubernetes Bangalore meetup group. Does that group have a similar mentality?

NABARUN PAL: Kubernetes Bangalore meetup group is essentially focused towards spreading the knowledge of Kubernetes and the aligned products in the ecosystem, whatever there is in the Cloud Native Landscape, in various ways. For example, to evangelize about using them in your company or how people use them in existing ways.

So a few months back in February, we did something like a Kubernetes contributor workshop. It was one of its kind in India. It was the first one if I recall correctly. We got a lot of traction and community members interested in contributing to Kubernetes and a lot of other projects. And this is becoming a really valuable thing.

I'm not much involved in the organization of the group. They are really great people already organizing it. I keep on being around and attending the meetups and trying to answer any questions if people have any.

CRAIG BOX: One way that it is possible to contribute to the Kubernetes ecosystem is through the release process. You've written a blog which talks about your journey through that. It started in Kubernetes 1.17 where you took a shadow role for that release. Tell me about what it was like to first take that plunge.

NABARUN PAL: Taking the plunge was a big step, I would say. It should not have been that way. We really-- after getting into the team, I saw that it is really encouraged that you should just apply to the team, but then write truthfully about yourself. What do you want? Write your passionate goal of why you want to be in the team.

So even right now the shadow applications are open for Kubernetes 1.22. I wanted to give a small shoutout to that. If you want to contribute to the Kubernetes release team, please do apply. The form is pretty simple. You just need to say why do you want to contribute to the release team.

CRAIG BOX: What was your answer to that question?

NABARUN PAL: It was a bit tricky. So I have this philosophy of contributing to projects that I use in my day to day life. I use a lot of open source projects daily. And I started contributing to Kubernetes primarily because I was using the Kubernetes Python client, and that was one of my first contributions.

Now when I was contributing to that, I explored the release team and it interested me a lot, particularly how interesting the mechanics are. How varied the mechanics are of releasing Kubernetes. For most software projects, it's usually whenever you decide that we have meaningful progress in terms of features, you just release it. But Kubernetes is not like that. We follow a regular release cadence. And all those aspects really interested me. I actually applied first time in Kubernetes 1.16, but got rejected.

CRAIG BOX: Mmm-hmm.

NABARUN PAL: But I still applied in Kubernetes 1.17. I got into the enhancements team. And the enhancements team was led by Mr. Bobby Tables, Bob Killen, back then. And Jeremy [Rickard] was one of my co-shadows in the team. And it has been an amazing journey until then. I shadowed enhancements again. Then I lead enhancements in 1.19. Then shadowed the lead in 1.20 and eventually led the 1.21 team..

That's how my journey has been. And my suggestion to people, don't be afraid of failure. Even if you don't get selected it's perfectly fine. You can still contribute to the release team. Just hop on to the release calls and raise your hand and introduce yourself.

CRAIG BOX: Between the 1.20 and 1.21 releases, you've now moved to work on the upstream contribution team at VMware. I've noticed that VMware is hiring a lot of great upstream contributors at the moment. Is this something that Stephen Augustus has his fingerprints all over? Is there something in the water?

NABARUN PAL: A lot of people have fingerprints on this process. And Stephen certainly has his fingerprints on it. I would say that. We are expanding the team of upstream contributors primarily because of this reason; that the product that we are working for is based on Kubernetes. And it helps us a lot in driving processes upstream and helping out the community as a whole. Because everyone then gets enabled and benefits from what we contribute to the community.

CRAIG BOX: I understand that the Tanzu team is being built out in India at the moment, but I guess you probably haven't been able to meet them in person yet?

NABARUN PAL: Yes and no. I did not meet any of them after joining VMware. But I spent a lot of my teammates before I joined VMware at KubeCons. For example, I met Nikita. I met Dims. I met Stephen at KubeCon. I have yet to meet other members of the team and I'm really excited to catch up with them once everything comes out of lockdown and we go back to our normal lives.

CRAIG BOX: Yes, everyone that I speak to who has changed jobs in the pandemic says it's a very odd experience, just nothing really being different. And the same perhaps for people who are working on open source moving companies as well. They're doing the same thing, perhaps just for a different employer.

NABARUN PAL: As we say in the community, see you in another Slack in some time.

CRAIG BOX: We now turn to the recent release of Kubernetes 1.21. So first of all, congratulations on that.

NABARUN PAL: Thank you.

CRAIG BOX: The announcement says the release consists of 51 enhancements, 13 graduating to stable, 16 moving to beta, 20 entering alpha, and then two features that have been deprecated. How would you summarize this release?

NABARUN PAL: One of the big points for this release is, this has been the largest release of all time.

CRAIG BOX: Really?

NABARUN PAL: Yep. 1.20 was the largest release back then. But 1.21 got more enhancements, primarily due to a lot of changes that we did to the process.

In the 1.21 release cycle, we did a few things differently compared to other release cycles. For example, in the enhancement process. An enhancement in the Kubernetes context is basically a feature proposal. We have something called Kubernetes Enhancement Proposals, it's called KEPs. You will hear that terminology a lot in the Kubernetes community. An enhancement is just a broad thing encapsulating that specific document. It's the notion.

CRAIG BOX: I like to think of it as a thing that's worth having a heading in the release notes.

NABARUN PAL: Indeed. [CHUCKLES] So until the 1.20 release cycle, what we used to do was-- the release team has a vertical called enhancements. The enhancements team members used to ping each of the enhancement issues and ask whether they want to be part of the release cycle or not. And then the authors would decide, or talk to their SIG, and then come back with the answer whether they want to be part of the cycle.

In this release, what we did was we eliminated that process and asked the SIGs proactively to discuss amongst themselves what they wanted to pitch in for this release cycle. As in, what set of features do they want to graduate this release? They may graduate things for alpha. They may graduate things to beta. They make graduate things to stable, or they may also deprecate features.

What this did was promote a lot of async processes and at the same time give power back to the community. So the community decides what they want in the release and then come back collectively. And it also reduces a lot of stress on the release team as to ask people consistently what they want to pitch in for the release. You have a deadline. You discuss amongst your SIG what your roadmap is and how does it look like for the near future. Maybe this release, the next release, the next to next release. And you put all of those answers into a Google spreadsheet. Spreadsheets are still a thing.

CRAIG BOX: The Kubernetes ecosystem runs entirely on Google Spreadsheets.

NABARUN PAL: It does and a lot of Google Docs, meeting notes. So we did a lot of process improvements, which essentially led to a better release. This release cycle we came up to 13 enhancements graduating to stable, 16 which moved to beta, and 20 enhancements, which were net new features into the ecosystem, came in as alpha.

Along with the two features also, there’s deprecation. One of them was PodSecurityPolicy. That has been a point of discussion amongst the Kubernetes user base and we also published a blog post about it. All thanks to Kubernetes SIG Security who have been on top of things as to find a replacement for PodSecurityPolicy even before this release cycle ended, so that they could at least have a proposal of what will happen.

For anyone listening to this podcast, I really encourage them to go ahead and read the KEP and proudly put in your reviews about what you feel of the replacement. That would really help us into gauging what the community wants. And it will eventually lead to a better Kubernetes and better replacement for PodSecurityPolicy.

CRAIG BOX: Let's talk about some old things and some new things. You mentioned PodSecurityPolicy there. That's the thing that's been around a long time and is being deprecated. Two things that have been around a long time and that are now being promoted to stable are CronJobs and PodDisruptionBudgets, both of which were introduced in Kubernetes 1.4, which came out in 2016. Why do you think it took so long for them both to go stable?

NABARUN PAL: I might not have a definitive answer to your question. But then one of the things that I feel is they might be already so good that nobody saw that they were beta features but eventually kept on using it. One of the things that I noticed when reading for the CronJobs graduation from beta to stable was the new controller.

So in hindsight, users might not see this, but there has been a drastic change in the CronJob controller. There has been a new controller called CronJob Controller v2. What it essentially does is goes from a polll-based method of checking what users have defined as CronJobs to a controller pull queue architecture, which is the modern architecture of defining controllers. So that has been one of the really good improvements in the case of CronJobs. So instead of the controller working in O(N) time, you now have constant time complexity of how the controller works.

CRAIG BOX: A lot of these features that have been in beta for a long time, like you say, people have an expectation that they are complete. With PodSecurityPolicy, it's being deprecated, which is allowed because it's a feature that never made it out of beta. But how do you think people will react to it going away? And does that say something about the need for the process to make sure that features don't just languish in beta forever, which has been introduced recently?

NABARUN PAL: That's true. So one of the driving factors, when contributors are thinking of graduating beta features has been the prevention of perma-beta KEP. So back, I think, in 1.19, we introduced this feature where each of the beta resources were marked for deprecation and removal at a certain time frame. I think it's two releases for deprecation and another two releases for removal. That's also a motivating factor for eventually rethinking as to how beta resources work for us in the community. That is also very effective, I would say.

CRAIG BOX: Do remember that Gmail was in beta for eight years.

NABARUN PAL: Oh, I did not know this.

CRAIG BOX: Nothing in Kubernetes is quite there yet, but we'll get there.

Of the 20 new enhancements, do you have a favorite or any that you'd like to call out?

NABARUN PAL: There are two specific features that I'm really interested about. In 1.21 that are coming asnet w features. One of them is the persistent volume health monitor, which gives the users the capability to actually see whether the backing volumes, which power persistent volumes in Kubernetes, are deleted or not. For example, the volumes may get deleted due to some inadvertent events, or they may get corrupted. That very information is basically surfaced out as a field so that the user can leverage it in any way.

One other feature that I'm really interested about is the proposal for adding headers with the commandname to kubectl requests. So what this basically means is, we have always been setting the user-agent information when doing those kind of requests, but the proposal is to add what command the user put in so that we can enable more telemetry and the cluster administrators can determine the usage patterns of how people are using the cluster. So I'm really excited about these kind of features coming into play.

CRAIG BOX: You're the first release lead from the Asia-Pacific region, or more accurately, outside of the US and Europe. Most meetings in the Kubernetes ecosystem are traditionally in the window of overlap between the US and Europe, in the morning in California and the evening here in the UK. What's it been like to work outside of the time zones that the community had previously been operating in?

NABARUN PAL: So it's been a fun and a challenging proposition, I would say. In the last two-ish years that I have been contributing to Kubernetes, the community has also transformed from a lot of early morning Pacific calls to more towards async processes. For example, we in the release team have transformed our processes and we don't do updates in the calls anymore. What we do is call out for updates ahead in time of the call, and then in the call, we just discuss things which need to be discussed synchronously in the team.

So what we leverage the meetings right now is for more of discussions. But we also don't come to decisions about those discussions, because if any stakeholder is not present on the call, it gives a disadvantageous situation. So we are trying to move to more of a process where we talk more on Slack, publicly, or talk on mailing lists. And that's where most of the discussion should happen. And also to give a lazy consensus. So what I mean by lazy consensus is come up with a pre-decision kind of thing, but then also invite feedback from the broader community as to how would people like them to see about that specific thing being discussed. This is where we as a community are also transforming a lot, and there is a lot more headroom to go.

In the release team also started to have something like an EU/APAC burndown meetings. So in addition to having one meeting focused towards the US and European time zones, we also do a meeting which is more overlapping towards European and Asia-Pacific time zones. One of the driving factors for those decisions was that the release team is seeing a lot of participation from a variety of time zones. To give you one metric, we had release team members this cycle from UTC+8 all through UTC-8. 16 hours of span. And it's really difficult to accommodate all of those zones in a single meeting. And it's not just 16 hours of span, it's like, what about the other eight hours of span?

CRAIG BOX: Yeah, you're missing New Zealand. You could add another 5 hours of span right there.

NABARUN PAL: Exactly. So we will always miss people in meetings, and that's why we should also innovate more as in, have different kinds of meetings. But that also may not be very sustainable in the future. Will people attend duplicate meetings? Will people follow both of the meetings? That is one of the solutions.

The other solution is you have threaded discussions on some medium, be it Slack or be it mailing list. And then people can just pitch in whenever it is work time for them. And then at the end of the day, give a 24-hour rolling period, and then you basically digest it, and then push it out as meeting notes. Which is what the Contributor Experience Special Interest Group is doing. Shout out to them for moving to that process. I may be wrong here, but I think once every two weeks, they do async updates on Slack. And that is a really nice thing to have, improving variety of geographies that people can contribute from.

CRAIG BOX: Once you've put everything together that you hope to be in your release, you release a release candidate build. How do you motivate people to test those?

NABARUN PAL: That's a very interesting question. It is also difficult for us to motivate people into trying out these candidates. But it's mostly people who are passionate about Kubernetes try out the release candidates and see for themselves what those bugs are. I remember Dims tweeting out a call that if somebody tries out the release candidate and finds a good bug or caveat, then there will be a callout in the KubeCon keynote. One of the incentives, if you want to be called out in a KubeCon keynote, please try our release candidates.

CRAIG BOX: Or get a new pair of Kubernetes socks.

NABARUN PAL: We would love to give out goodies to people who try out our release candidates and find bugs. For example, if you want the brand new release team logo as a sticker, just hit me up. If you find a bug in probably 1.22 release candidate, I would love to send you some coupon codes for the store. Don't quote me on this, but do reach out.

CRAIG BOX: Now the release is out, is it time for you to put your feet up? What more things do you have to do, and how do you feel about the path ahead for yourself?

NABARUN PAL: I was discussing this with the team yesterday. So even after the release, we had kind of a water cooler conversation. I just pasted in a Zoom link to all the release team members and said, hey, do you want to chat? One of the things that I realized that I'm really missing is the daily burndowns right now. I will be around in the release team, in the SIG release team meetings, helping out the new lead in transitioning. And even my job, right now, is not over. I'm working with Taylor, who is the emeritus advisor for 1.21, into figuring out some of the mechanics for the next release cycle. Also, documenting what all we did as part of the process and as part of the process changes, and making sure the next release cycle is up and running.

CRAIG BOX: We've done a lot of these release lead interviews now, and there's a question which we always like to ask, which is, what will you write down in the transition envelope? Savitha Ranganathan is the release lead for 1.21. What is the advice that you will pass on to her?

NABARUN PAL: Three words-- do, delegate, and defer. Categorize things into those three buckets as to what you should do right away, what you need to defer, and things that you can delegate to your shadows or other release team members. That's one of the mantras that work really well when leading a team. It is not just in the context of the release team, but it's in the context of managing any team.

The other bit is over-communicate. No amount of communication is enough. What I've realized is the community is always willing to help you. One of the big examples that I can give is the day before release was supposed to happen, we were seeing a lot of test failures, and then one of the community members had an idea-- why don't you just send an email? I was like, that sounds good. We can send an email mentioning all the flakes and call out for help to the broader Kubernetes developer community. And eventually, what happened was once we sent out the email, lots of people came in to help us in de-flaking the tests and trying to find out the root cause as to why those tests were failing so much. Big shout out to Antonio, all the SIG Network folks who came to pitch in.

And no matter how many names I mention, it would always be less. A lot of people, even outside the release team, have helped us a lot into this release. And that's where the release team comes in. Power to the community. I'm really stoked by how this community behaves and how people are willing to help you all the time. It's not about what they're telling you to do, but it's what they're also interested in, they're passionate about.

CRAIG BOX: One of the things you're passionate about is Formula One. Do you think Lewis Hamilton is going to take it away this year?

NABARUN PAL: It's a fair probability that Lewis is going to win the title this year as well.

CRAIG BOX: Which would take him to eight all time career wins. And thus-- he's currently tied with Michael Schumacher-- would pull him ahead.

NABARUN PAL: Yes. So Michael Schumacher was my first favorite F1 driver, I would say. And then, it feels a bit heartbreaking to see someone break Michael's record.

CRAIG BOX: How do you feel about Michael Schumacher's son joining the contest?

NABARUN PAL: I feel good. Mick Schumacher is also in the fray right now. And I wish we would see him, literally in a few years, in a Ferrari. And the Schumacher family would be back to Ferrari, and that would be really great to see. But then, my fan favorite has always been McLaren, partly because I like the chemistry of Lando and Carlos, the last two years. But it was heartbreaking to see Carlos go to Ferrari. But then we have Lando and Daniel Ricciardo in the team. They're also fun people.

CRAIG BOX: Well, I wish McLaren all the best on your behalf. And I want to say thank you very much for joining us today, Nabarun.

NABARUN PAL: Thanks, Craig, for inviting me to this podcast. And it was really nice talking to you, talking to you about the release, talking to you about life in general. Thanks again.

CRAIG BOX: You can find Nabarun on Twitter @theonlynabarun or on the web at naba.run.


CRAIG BOX: Thank you very much, Daniel, for helping out with the show today.

DANIEL SMITH: Thanks, Craig, for having me. It's good to be on again.

CRAIG BOX: If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter @kubernetespod or reach us by email at kubernetespodcast@google.com.

DANIEL SMITH: You can also check out the website at kubernetespodcast.com, where you will find transcripts and show notes as well as links to subscribe.

CRAIG BOX: I'll be back next week. So until then, thanks for listening.