CRAIG BOX: Hi, and welcome to The Kubernetes Podcast from Google. I'm your host, Craig Box.

[MUSIC PLAYING]

CRAIG BOX: Oh, hello, you. Not going to lie — I've missed you. I know it's been a while. I hope you didn't mind that I needed a little break in these uncertain times. I trust you looked after yourself while I was gone. Are you staying dry?

Hard to imagine, but both sides of my planet were underwater this week. Earlier this year, I brought you a few episodes from Nelson, in New Zealand. Had I been there, I would have been evacuated this week, as the Maitai River burst its banks in a once-in-a-100-year event, which I imagine we're going to see many more of over the next decade.

It's winter over there. But weren't you having a summer holiday, you ask. Well, the way climate change has chosen to assert itself over the UK recently was three days of heat wave, followed by three days of solid rain. Yes, we desperately needed the rain. But we didn't really need it to be pooled up on the A40 so you almost went aquaplaning through it.

Anyway, I'm going to ask you a favor now. I don't ask much, so hopefully it's not presumptuous of me. If you've missed hearing from me these last couple of weeks, would you like to follow me on Twitter? I don't say a lot, almost never about Kubernetes. I posted a picture of a sheep at the start of my break. I probably took hundreds of photos, but I chose to only share that one. I know that's award-worthy restraint for a New Zealander, right?

Anyway, I'm craigbox on Twitter. And if enough of you drop by, I might just post a picture of a horse, too.

Let's get to the news.

[MUSIC PLAYING]

Congratulations to all involved in the release of Kubernetes 1.25. Forty enhancements were tracked, including 13 features moving to Stable, and 15 new features entering in Alpha. Listen on to today's interview to learn more.

The founding team from Rancher Labs are back with a new startup called Acorn Labs. One year after Rancher's acquisition by SUSE, all four founders — including Episode 57 guest Darren Shepherd — had left the company, with former Rancher CEO Sheng Liang remaining on SUSE's board. All have reassembled to create Acorn, an application packaging and deployment framework for Kubernetes. Shepherd's introductory blog talks about the challenge of packaging and deploying apps on top of Kubernetes.

It goes on to introduce their solution — an OCI image containing multiple containers and metadata, and the Acornfile, a docker-compose-inspired method for description of an application without having to talk about infrastructure. Acorn is open source and available to kick the tires in an early alpha/beta version.

While it's a quiet news time through most of the tech world, the GKE team has remained hard at work with a number of new releases. The GKE Cluster List page now includes a new Observability tab. This tab shows infrastructure health metrics, such as CPU, memory, container restarts, and Control Plane metrics. It also provides visibility into ingestion into the Google Cloud Managed Service for Prometheus and the Cloud Logging service.

And while GKE Autopilot has recently become the recommended way to deploy clusters, you can still get power user features in the Standard mode. The maximum number of Pods that can run on a node has increased to 256, limited only by IP address allocation. Given that Compute Engine now has full IPv6 support, we'll keep an eye on that limit being raised further.

The schedule for KubeCon North America '22 was announced earlier this month. In case you were on the fence about attending, you can see the almost 200 sessions that were chosen by the 120-plus members of the program committee. There are also over 80 maintainer sessions, not to mention the two days of co-located events that happen ahead of the schedule. If that isn't enough, or if you're salty that your talk wasn't selected, the Cloud Native Rejekts Conference will be glad to have you the weekend before.

Finally, deep dive blog post of the week award goes to Upbound, creators of Crossplane, for their look at a large-scale deployment of custom resource definitions on Kubernetes.

Nic Cope summarizes the team's work, solving the basic problem of Crossplane needing one CRD for every API type a cloud has. Before Crossplane, you might expect to have five to six CRDs installed in your cluster, maybe breaking into double digits if you had the service mesh installed. To get to the scale of thousands, caches, rate limits, and etcd clients had to be tweaked, and the post explains the process, as well as what work remains to be done.

And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Cici Huang is a software engineer on the Google Kubernetes Engine Kernel Team, and the Release Lead for today's Kubernetes 1.25 release. Welcome to the show, Cici.

CICI HUANG: Thank you for having me.

CRAIG BOX: Let's start by looking at your introduction to computing. I understand you first got a computer when you were a teenager.

CICI HUANG: Yes.

CRAIG BOX: What were you using it for?

CICI HUANG: Back at that time, I was mainly using it for chatting with friends or enjoyed talking with random persons online on some BBS forums, and also maybe some game playing stuff.

CRAIG BOX: Did that go down well with your parents?

CICI HUANG: Not really. While in high school, my parents tried to make me focus more on study, obviously. Because they wanted me to get into a good university. So they used to try to lock the computer so I cannot access. And that's how the journey begins, I guess. I tried to hack the computer. Yeah, initially, by just guessing the password, but later trying to maybe sneaking another guest account or something like that.

CRAIG BOX: They never physically picked it up and put it somewhere else or locked it with a key?

CICI HUANG: Oh, that's a good idea. [LAUGHS] I guess we don't have that many rooms in the house.

CRAIG BOX: Well, you did go into good schools, but you went to study computer science. Was that something that was a common path available to people at the time?

CICI HUANG: Yes. I guess majoring in computer is quite popular because people started to feel the power of computer, I would say.

CRAIG BOX: And you went on to do not one but two masters, one in China, one in the US. Was one not enough?

CICI HUANG: I guess that decision was affected by the fact we are moving to Bay Area. So when we decided to move to the United States, it's a rush decision, I'll say. So I didn't really get a lot of time to apply to a job. And plus, I feel maybe pure developing work is not what I wanted. So I started applying for other majors related to computer, but not like engineering side.

CRAIG BOX: You moved to the US when your husband was finishing his PhD. Do you have a competition between the two of you? Is two master's better than one PhD?

CICI HUANG: [LAUGHS] I don't really think I'm PhD material at that time. Because I always feel like working in the industry is more exciting for me.

CRAIG BOX: I definitely agree. I think that the idea of working for a very long time on putting a tiny, tiny expansion in the bubble of human knowledge. I appreciate people who do it, but it takes a certain special type of person.

CICI HUANG: I agree.

CRAIG BOX: Your first job in the US was working for IBM. But I understand you also interned for IBM while you were in China.

CICI HUANG: Yes. It was like a long time back.

CRAIG BOX: Did the company feel like it was the same place?

CICI HUANG: Not really. Back in China, when I was doing my internship at IBM in college, I was mainly working on the old Rational product, if you still remember.

CRAIG BOX: Mm-hmm.

CICI HUANG: Yeah. But when I got the job in the United States, IBM put a lot of effort into AI and cloud stuff. So I got the chance to work in IBM Watson, which also, not only in the AI side, but also running stuff on cloud.

CRAIG BOX: And winning game shows.

CICI HUANG: Oh, yes. Yes. That's a popular topic at that point.

CRAIG BOX: You bring then a background in both machine learning and cloud to the work that you do. Was the work you were doing in machine learning along with a research team, or was that something that you were interested perhaps in pursuing yourself?

CICI HUANG: Back then, the team was partnering with a research team. So apparently, a research team were taking care of all the AI model stuff, and we as an engineering team were taking care of making the idea true and maybe maintaining and development of them. So I didn't really get a lot of chance to exploring deeply in AI model stuff, I'll say. Yeah.

CRAIG BOX: Perhaps, again, that's the difference between the PhDs, who do all the thinking, and the master's people, who actually have the job of making it happen.

CICI HUANG: Oh, certainly, yes.

CRAIG BOX: One thing you did get experience with at that point was Kubernetes. How did that come into your world?

CICI HUANG: Oh, sure. But at that time, I didn't really — in Kubernetes cluster or the meaning of something, but we do running our service on cloud using Kubernetes, for sure. And we're really impressed by the power of Kubernetes.

CRAIG BOX: That then took you to interviewing at Google?

CICI HUANG: Yes, basically.

CRAIG BOX: Tell me about the process of being matched to a team at Google.

CICI HUANG: I guess the process changed slightly now, but back then you have to pass the interview and pass the committee, and then do team match before you really got the offer. So back then, I was doing the team match with multiple teams. And then, of course, my manager from Kubernetes Kernel Team asked if I would be interested in working on Kubernetes open source stuff. I just immediately said yes.

CRAIG BOX: Is that someone we will know?

CICI HUANG: Yeah. It's Fede. I'm not sure if you know, it's the currently SIG API Machinery Chair.

CRAIG BOX: Mm-hmm.

CICI HUANG: He hired me.

CRAIG BOX: A good choice. Well done, Fede. What was the first thing you worked on when you joined that team?

CICI HUANG: I guess the first big thing I was working on is the effort the community working towards, which is to try to move the entry cloud providers out. So back then, I was working in SIG Cloud Provider, and trying to make this happen. It's still an ongoing effort, as we could see, through a couple of releases. But back then, I was mainly working on separate, the entry cloud provider from the k/k base code.

CRAIG BOX: You were the winner of a Kubernetes Development Award in 2020. Was that for that work?

CICI HUANG: Yes. It's my first time working in the open source community. And that effort is obviously a joint effort, not only inside the Kubernetes open source community, but also has to be partnered with all the cloud providers. So that's an interesting experience.

CRAIG BOX: You are on the Kernel Team for Google Kubernetes Engine. That's not the operating system kernel, though. What exactly does the Kernel Team look after?

CICI HUANG: Oh, yes, I know the name sounds confusing. But we initially named our name as API Machinery, which obviously the majority of the team are working in SIG API Machinery inside the Kubernetes community. And then we feel like the API Machinery as a name also sounds confusing for some of the people inside Google specifically. So we kind of updated the name to Kubernetes Kernel Team.

CRAIG BOX: API Machinery makes up about 40% of the Kubernetes code base. I imagine your team does a lot of work on the upstream Kubernetes project in general?

CICI HUANG: So as Kubernetes Kernel Team, we are not only working on SIG API Machinery, we have the most experienced folks who have been working in the upstream Kubernetes community for many, many years. And we have people across multiple SIGs, as SIG leads or working group leads, who are leading the effort. The members in the team are not only focusing on API Machinery area, but there are leads in other SIGs as well.

CRAIG BOX: How then did you get involved with the release team process?

CICI HUANG: Back then, I was actually a maintainer of the Kubernetes Subrepository, and I was trying to cut releases for that. And then, I think maybe I could pay more attention on Kubernetes Release Team, to see how I could learn from there.

CRAIG BOX: I think that's the thing is you show the ability to do something in a small part of the code base, and people say, oh, she's interested in that, and bring you on to the larger part.

CICI HUANG: Yeah, that's normally how it would work in the Kubernetes community, I'll say, yeah. But then I saw the Release Team shadow application from the Devs channel, obviously. So I immediately submitted my application. But I didn't get picked at the first time.

CRAIG BOX: Right. So you came in with the release after that?

CICI HUANG: Yes. That was 1.22.

CRAIG BOX: Having now been with the Release Team since 1.22, what is the sequence of roles that you went through?

CICI HUANG: I was a release node shadow at 1.22, and then became release node lead in 1.23. I then became the lead shadow in 1.24. And here I come, the 1.25 release lead.

CRAIG BOX: And today is the day of the 1.25 release. So congratulations, first of all, as always, on the release.

CICI HUANG: Thank you.

CRAIG BOX: This release includes a total of 40 enhancements, 15 new features entering in alpha, 10 features graduating to beta, and 13 have moved to stable, general availability. Two features are being deprecated or removed. Let's start with the removal. We've been going a long time on the process of getting rid of PodSecurityPolicy, and I understand it's now completely gone.

CICI HUANG: Yes. There was a discussion going on for quite some time. And the PodSecurityPolicy was initially deprecated in 1.21. And now, with the release of 1.25, it has been removed formally.

CRAIG BOX: The replacement is Pod Security Admission, which graduates to stable in this release. Are you able to track how quickly people stop using deprecated features and start using the new ones as you bring them out?

CICI HUANG: I don't have data handy, but I will encourage people to do it as early as possible. Don't wait until the last minute. And also, like we have announced the PodSecurityPolicy deprecation four releases back. And due to the community deprecation policy, we keep the support for more than a year. So I hope people would get good use of this by a year.

CRAIG BOX: Yes. If it breaks by this point, it's your own fault. You should have been paying attention. We do talk about it on the podcast every time there's a release.

CICI HUANG: Yes. Unfortunately, sometimes people don't pay enough attention until it's out of support or until they have to. That's why we keep shouting.

CRAIG BOX: Indeed. When something breaks, that's generally the first time people will go to try and fix it.

CICI HUANG: Yeah, that is more current.

CRAIG BOX: Now, I know we said before that the GKE Kernel Team is not the Linux Kernel Team. But I do understand that support for C Group's V2 in the Linux Kernel has graduated to stable in this release. What does that new kernel feature mean to someone who's actually just deploying applications on top of Kubernetes?

CICI HUANG: So a little bit of background. C Group is a low-level Linux kernel capability, just to do resource management functionality, like limiting CPU usage or set memory limits. And currently, with some distributions now defaulting to this API, Kubernetes must support it to continue operating on those distributions. So for all the existing users or the newcomer users, there should not be any notable difference in the user experience when switching to C Group's V2.

CRAIG BOX: I don't have to rebuild my containers anywhere?

CICI HUANG: No, not at all.

CRAIG BOX: Another new feature being brought in in this release is support for KMS version 2 API for Key Management. What does that entail?

CICI HUANG: Oh, yes, the Key Management Service within this release, we introduced the KMS V2 for one API to add performance, rotation, and observability improvement. And no user action is required due to this change. We were just trying to address the shortcomings with the previous API.

CRAIG BOX: There are people from many different roles inside the Kubernetes community who come to the Release Team, and specifically to the release lead process. You are coming in from an engineering perspective. And something that I find interesting is that a feature that you are involved in building is actually part of this release.

So first of all, tell me what is new with CRD expression validation.

CICI HUANG: Thanks for asking that. Yeah, I was leading the effort of promoting the CRD validation expansion language to beta in this release. Previously, CRD only supported two major building validations — CRD structure schemas and OpenAPI v3 validation rules. And for use cases not covered by the previous building validation, can only be covered by admission webhooks.

But getting validation admission webhooks does not only involve effort on the development side, but also a lot of effort operational-wise. So that's how the CRD validation expansion language came in place, which offers the power to declare how customer resources are validated using the Common Expression Language. And it was in alpha since two releases back, yeah, and now it's mature enough to graduate into beta.

CRAIG BOX: Tell me, then, about the process not from the release team, but from a person who is trying to move a feature through. What do you have to do to prove to the team or to the SIG in charge that this thing is ready to make that kind of jump?

CICI HUANG: First of all, of course, the enhancement update. So we raised the enhancement at the beginning of the release cycle, and we have to get it reviewed by the SIG leads and all the important stakeholders. So after we got an agreement inside of the SIG, which this feature is ready for the next level, and also there are certain requirements from release team side as well, including the enhancement outline requirement, the review process, the deadline for enhancement phase. And after we got an agreement on the enhancement, then that moves to the implementation phase. Which we just need to get code in before the code phase.

CRAIG BOX: Do you feel differently about your role on both sides of that? You obviously, as a release team lead, want to see a lot of process there and want to make sure that things are ready to be released. But is it hard for you to say, all right, well, hey, there's all this extra processing work for a thing where I know the code is good and I want to get it out there?

CICI HUANG: I think being a release team lead definitely helped me there because I'm more familiar with the process than others are. So it definitely helped me there in preparing my enhancement update and keeping the milestone deadline in mind.

CRAIG BOX: Did you have to open one window, write a document, and then open a different window and tick a big box that says, yes, I accept this?

CICI HUANG: [LAUGHS] I guess, basically, what I said to myself is, don't raise the exception to myself. And luckily I did.

CRAIG BOX: No, let someone else do that.

CICI HUANG: Yeah. So I kept my deadline. Yeah.

CRAIG BOX: Aside from your own new enhancement, is there any other personal highlights of this release for you?

CICI HUANG: I would say we got a lot of feature updating in this release, even though we have a shorter release in the current release cycle. And we have way more alpha features opening in this release. We have, I believe, 15 alpha features in 1.25, which I personally view it as a great sign of the growing power of Kubernetes.

CRAIG BOX: You mentioned a shorter release cycle there. What was the reason for that?

CICI HUANG: Yeah. As you know, we normally have three releases per year for Kubernetes. And as a second release, we have always taken the next release schedule into mind because the third release normally will involve things like US holidays or KubeCon happening. And also, as people aware of the previous release got delayed for a couple of weeks because of an expected change from Golang side, that's also add up on the delay decision.

CRAIG BOX: So they stole two weeks from you, those horrible Golang people.

CICI HUANG: [LAUGHS] Oh, by the way, we upgraded the Golang dependency again in this release.

CRAIG BOX: And why not?

CICI HUANG: Yes.

CRAIG BOX: One thing that has changed with this release is where Kubernetes itself is hosted from. It used to be hosted on a GCR.io domain. It has now moved to registry.k8s.io. What was the reason for that change? And what will it mean for both the Kubernetes team and people who are downloading and installing Kubernetes themselves?

CICI HUANG: Oh, yes. For many years, we have been using kubernetes.grc.io in all our repositories as the default registry for downloading images from, to now, with this current release, we are switched to use the image promoter process to promote images to the official Kubernetes Container Registry and to using the infrastructure provided by SIG Community Infra. And for the users who have the older registry in their configurations, need to make the necessary switch.

CRAIG BOX: Is that another thing that will change over time, there's no immediate deprecation for?

CICI HUANG: We will encourage users to do the switch as soon as possible. And within this effort, we try to host a corpus of images and binaries nearer to where they're used. So the main switch is already done in this release. So we will encourage users just to make the necessary switch as soon as possible.

CRAIG BOX: So people who are running Kubernetes on places other than Google Cloud — shock, horror, how could you — they're actually going to get a better experience with installing Kubernetes?

CICI HUANG: [LAUGHS] I would say maybe not in the current switch. We are currently working on adding the AWS support only as of now. But I'll say, in the future, it will spread across all the possible cloud providers.

CRAIG BOX: With all of these features, how do you make sure that the community learn about them in the depth that they deserve?

CICI HUANG: We will be releasing multiple release blogs together with the main release. So specifically in this release, 1.25, we have 14 feature blogs up and ready to be published within next one or two months.

CRAIG BOX: That's a lot. So you could do one per week and basically you'd be up to 1.26.

CICI HUANG: Yeah. That's a new record we have.

CRAIG BOX: The theme for this release is Combiner. I understand your son came up with this theme.

CICI HUANG: Yes. Apparently, the theme comes from Transformers, which is my son's favorite.

CRAIG BOX: I was going to say, what kind of conversations are you having with him that he's able to have input on this process at five years old?

CICI HUANG: Actually, I was struggling with the release theme idea till halfway through the release. And I was talking about this with my husband at that point. And my son, playing with his Transformers aside. And then he was like, what is a logo? What is a theme?

I was like, theme is a symbol or anything I try to find to represent the project I'm working on. And my son was like, so it could be anything? I was like, pretty much. And he was like, then you can borrow my Transformers. Just make sure to return them back. I was like, yeah, that's a brilliant idea.

CRAIG BOX: It's a big day for you being the Kubernetes release day, but I understand it's a big day for you tomorrow as well.

CICI HUANG: Oh, yes. Tomorrow will be the first day of kindergarten for my son.

CRAIG BOX: That's amazing.

CICI HUANG: Big step.

CRAIG BOX: Every release, we talk to the lead about the advice that they're going to give to their successor. We spoke with James Laverack for 1.24, and he said that the advice that he would give to you is that open communication is really important. He was talking about how he was summarizing release teams to Slack and possibly spamming the channel, and encouraging you to do the same thing. Was that something you were able to take on board?

CICI HUANG: Oh, yeah, that's one of the important things I keep in mind. And I always view the release lead as more like a coordinator role instead of a leading role. So that involves a lot of discussion with others.

CRAIG BOX: Was there any other advice that you picked up through the process that was particularly important to you?

CICI HUANG: Yeah. I remember one advice I got from Jim is, don't underestimate the time. [LAUGHS] Because, yeah, I remember he said, like, the recommended time spent in the release work — maybe, I don't know, no more than 20 hours or 25 hours per week — but always expect more.

CRAIG BOX: Between the release and the enhancement you were pushing into the release, did you have time for anything else?

CICI HUANG: I'll say I'm pretty lucky this release went very smoothly. And I got a lot of time working on my own features and stuff. But maybe not recently, well, two weeks, but earlier, yeah. I will say the release work is not as much as compared to towards the end of the release.

CRAIG BOX: For 1.26, the release lead is going to be Leo Pahlke. What advice are you putting in the envelope for him?

CICI HUANG: Obviously, the open spirit and always communicate things through as much as possible. Yeah, that will be the first advice I will give.

CRAIG BOX: What are you looking forward to doing now that this role will be over for you soon?

CICI HUANG: I'm going to still keep working on the SIG API Machinery, for sure. And I'll stay in the community around. And maybe I will participate in other areas of the release.

CRAIG BOX: As we mentioned before, your son has played a part in scoping this release out and helping you with its theme. Does he enjoy playing on a computer, or is that something that you need to lock up in a different room?

CICI HUANG: Obviously, he enjoyed a lot his computers. But currently, maybe more of the game side or like video side.

CRAIG BOX: There's still time. Do you think you'll get him involved in the Kubernetes project one day?

CICI HUANG: I actually kind of hope like one day, like he's just search his name online and found his name appearing in the Kubernetes release blog. And I hope that will be a surprise for him.

CRAIG BOX: Excellent. That's good parenting right there. Thank you very much for joining us today, Cici.

CICI HUANG: Thank you so much for having me.

CRAIG BOX: You can find Cici on GitHub or the Kubernetes Slack, as cici37.

[MUSIC PLAYING]

Thank you, as always, for listening. If you've enjoyed the show, please help us spread the word, and tell a friend. If you have any feedback for us, you can find us on Twitter @kubernetespod, or reach us by email at kubernetespodcast@google.com.

You can check out the website, at kubernetespodcast.com, where you will find transcripts and show notes, as well as links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]