KASLIN FIELDS: Hello. And welcome to the "Kubernetes Podcast" from Google. I'm your host, Kaslin Fields. And--

ABDEL SGHIOUAR: --I'm Abdel Sghiouar.

[OPTIMISTIC MUSIC]

KASLIN FIELDS: I hope you all had a happy new year, everyone, and that you rested up, and your 2023 started well. We're excited to be back here on the "Kubernetes Podcast." And it's good to see you again, Abdel-- well, hear you again.

ABDEL SGHIOUAR: Good to see you or hear you, as you said. Yeah, 2023's starting to be interesting. So we'll see how this year goes.

KASLIN FIELDS: [LAUGHS] Isn't that always the case?

ABDEL SGHIOUAR: Yeah. This year is particularly interesting. So--

[LAUGHTER]

I hope you had a good rest yourself.

KASLIN FIELDS: Well, I got sick for most of the holidays. So it wasn't that restful. But-- [LAUGHS]

ABDEL SGHIOUAR: Yeah, you're not the only one. A lot of people got sick over the holidays.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: I'm recording from Berlin as we're speaking. I'm attending my first event, actually, for the year.

KASLIN FIELDS: Nice-- getting the year started off right.

ABDEL SGHIOUAR: Yeah, I love Berlin. It's an awesome city.

KASLIN FIELDS: What are you seeing at the event? Are you going to tell us all about it?

ABDEL SGHIOUAR: So it's an internal event. It's the GDE summit for the Google developer experts in Europe. And they are organizing two days where they are gathering 250 people from all over Europe.

And what's interesting is, as of we're recording right now, it's Tuesday. The event is starting on Thursday. Everybody's supposed to fly in on Wednesday. But there is a strike at the airport.

KASLIN FIELDS: Oh, dear. That'll be a problem.

ABDEL SGHIOUAR: Yeah, so the team spent the day just rebooking everything. So hopefully, we will have a little less than 250 people attending. We'll see.

So let's go back to our topics, Kaslin. I'm going to start 2023 with a very interesting question. What do you think are going to be the three major topics in the cloud native space this year?

KASLIN FIELDS: Why would you ask me this?

ABDEL SGHIOUAR: I love to do that, predictions.

KASLIN FIELDS: [LAUGHS] So, so difficult. OK, top three cloud native things in 2023.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: That's the question?

ABDEL SGHIOUAR: Yeah, the major ones.

KASLIN FIELDS: Well, I think the most obvious one is ChatGPT and AIML. Clearly going to be huge this year. Everyone's talking about ChatGPT, it feels like. When it first started coming up, I just saw it everywhere. And I was like, what is this?

ABDEL SGHIOUAR: And I can tell you that no one actually understands what is that.

KASLIN FIELDS: Fair enough.

ABDEL SGHIOUAR: But yeah, that's actually a very good one. Because I think that AI and ML in the cloud native space and in Kubernetes specifically is going to be big this year. We're seeing a lot of traction at Google.

And we're seeing projects pop up left and right. And the community seems to be responding to that. So it's going to be interesting to see how that progresses.

KASLIN FIELDS: And I don't know if this counts as a different one. But kind of a related area in Kubernetes, at least, is batch workloads on Kubernetes.

ABDEL SGHIOUAR: Yeah.

KASLIN FIELDS: I know a lot of the folks that I talked to in the Kubernetes space have been interested in batch workloads for a while. And I'm seeing that continue, I think. I think that'll continue in 2023-- a lot of folks running a lot of jobs in Kubernetes. So that's pretty common with AIML, too. So it's kind of related to the ChatGPT thing, maybe another reason why it'll be popular this year.

ABDEL SGHIOUAR: Yeah, I think that when Kubernetes was released in 2015, it was kind of mainly designed to run web apps, I think. If you really look at the basic building blocks, it's mainly for long-running apps that are exposed onto a port. But yeah, batch became important because there is an API for it now and because more people are realizing you can use Kubernetes as just a platform as an abstraction layer on top of hardware. And then you can-- or you should be able, at least, to run all sorts of workloads, not only web stuff.

KASLIN FIELDS: Speaking of all sorts of workloads, another thing that's going to be my major focus in 2023 and which I've seen gaining a lot of momentum in the community is stateful workloads on Kubernetes. This is an area that I've been interested in for a long time because I come from a storage background.

And early on with Kubernetes, we would talk about stateless applications a lot. But then they started developing the capabilities to use persistent volumes. And so the capabilities for stateful workloads in Kubernetes have been growing--

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: --over the years. And I'm seeing a lot of workloads being run on Kubernetes these days, a lot of stateful workloads. So I think that'll be a thing in 2023 as well.

ABDEL SGHIOUAR: Yeah, if you would have asked me four years ago, should I run a database on Kubernetes, I would have said, hell, no. But now I think it's different.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: Now I think that there are a lot of database providers that have very mature operators-- even Postgres, MySQL, et cetera.

KASLIN FIELDS: And one reason that I've heard for some stateful workloads on Kubernetes is actually a lack of managed services. A lot of folks want to use a managed service for the stateful workloads that they're running. But if there isn't a managed service available, an operator in Kubernetes is kind of like the next best thing because the creators of that workload are telling Kubernetes exactly how to run it. So you can run it with best practices, so to speak--

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: --built into it without too much effort and without too much upscaling to be able to do that. So--

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: --stable workloads, very interesting space.

ABDEL SGHIOUAR: I was actually talking to a customer yesterday. And the more I'm talking to people, the more I get this impression that everybody wants to use Kubernetes with the Kubernetes Resource Model. So people seem to want to have everything done the Kubernetes way, which is essentially, write a YAML file or whatever is your favorite way of doing templates. And just deploy that. So even your database is--

KASLIN FIELDS: Declarative.

ABDEL SGHIOUAR: Yeah, exactly, declarative. Yeah, so it's going to be-- I think also, everything around the declarative model and the KRM is going to be big this year.

KASLIN FIELDS: Yeah, makes a lot of sense to me. And for our episode this time, we are going to be talking about some cool updates going onto Kubernetes, the open source project, and kind of how Kubernetes as an open source project works with the awesome Benjamin Elder, which I'm very excited about. But first, let's get to the news.

[OPTIMISTIC MUSIC]

ABDEL SGHIOUAR: Linkerd published their 2022 recap. Some highlights from the recap are that the number of stable Kubernetes clusters right in Linkerd have doubled in 2022. The Kubernetes Gateway API seems to be a great match for service mesh patterns.

EBPF was an optimization rather than a game changer. And, as it will probably be not a surprise, the order in which containers start continues to be a problem in Kubernetes. And it seems like the community is looking into it.

KASLIN FIELDS: The CNCF launched a cloud native maturity model. This new project aims to be a companion for organizations that are starting, or are in the middle of, or have just been considering a cloud native transformation. So rather than a model for the maturity of projects, this new cloud native maturity model is a model for organizations.

ABDEL SGHIOUAR: And some news from Amazon-- if you are using EKS but you are a Google Workspace customer, what we used to call G Suite, you can now use your Workspace identities to authenticate to your EKS cluster. Check the link in the show notes for guidance on how to set it up.

KASLIN FIELDS: The Cloud Native Computing Foundation, or CNCF, has revamped their CNCF ambassadors program. Ambassadors are an extension of the CNCF, furthering the mission of making cloud native ubiquitous through community leadership and mentorship. Applications for the Ambassador 2.0 program are open until February 26, 2023. If you're a community leader in the cloud native space, you might want to consider applying to become a CNCF ambassador.

ABDEL SGHIOUAR: CloudNativeSecurityCon North America will take place in Seattle February 1 and 2 this year. This new standalone event is a spinoff of the highly popular security con co-located event previously held at KubeCon. The goal is not just to propose solutions that incrementally improve what has came before but to give room to breakthrough technology and advances in modern security approaches. Be on the lookout for an upcoming episode featuring interviews from this event.

KASLIN FIELDS: Support for Istio 1.14 has ended as of December 27, 2022. The first patch release for Istio 1.16, making it 1.16.1, was released on December 12.

ABDEL SGHIOUAR: And since the release of Kubernetes 1.26 and the interview we've done with Leonard last year, a lot of blog posts have been published describing what's new with this version, especially the new APIs that have been either introduced or transitioned to GA. We will leave some links in the show notes for you to check out.

KASLIN FIELDS: As of January 13, 2023, Kubernetes version 1.26 is now available for GKE users on the Rapid release channel. Make sure you check out our last episode, where we interviewed the 1.26 release lead, for more information about what's new in the latest version of Kubernetes.

ABDEL SGHIOUAR: The CNCF has announced important updates for KubeCon + CloudNativeCon 2023 and co-located events. In recent years, KubeCon has extended its day zero co-located event to two whole days before the three days of the conference itself.

After community feedback, the organization is making some changes in 2023. For KubeCon EU in Amsterdam, co-located events will be held only on Tuesday of KubeCon, with the main content of the event being held Wednesday through Friday. Be sure to check out the CNCF blog on the changes for more details.

KASLIN FIELDS: And that's the news.

[OPTIMISTIC MUSIC]

All right. I am excited today to be talking to Benjamin Elder. Would you like to introduce yourself?

BENJAMIN ELDER: Hi, everyone. I'm Benjamin Elder. I'm a senior software engineer at Google. I'm Kubernetes' SIG testing chair and tech lead and a Kubernetes Steering Committee member, amongst other things.

KASLIN FIELDS: I feel like folks in leadership in Kubernetes end up wearing so many different hats. You've got a good number of titles going on there. [LAUGHS]

BENJAMIN ELDER: Yeah, I think it is pretty common. Most of the people in the Steering Committee have come from some other part of the project's leadership previously and may still be holding some of those roles down.

KASLIN FIELDS: Which seems like something that is as it should be.

BENJAMIN ELDER: I think it's more or less as expected.

KASLIN FIELDS: Speaking of the Steering Committee, you are newly elected to the Steering Committee, right? That finished up in--

BENJAMIN ELDER: Yes.

KASLIN FIELDS: I was literally one of the officers of that. And I'm still not sure. I think it was November.

BENJAMIN ELDER: Yeah, that sounds right.

KASLIN FIELDS: So a relatively new member of the Steering Committee. Could you tell us a little bit about what the Kubernetes Steering Committee is?

BENJAMIN ELDER: Yeah, so the Kubernetes Steering Committee is the top-level governance body for the Kubernetes organization, the project. When we say that, we mean-- because the Kubernetes project has established documents and governance for how everything is handled from-- the CNCF owns the project, who they delegate to Steering. Then Steering delegates most things down to special interest groups for different horizontals or verticals in the project.

So Steering doesn't make technical decisions. But Steering does help be an escalation point for things and interface to the Cloud Native Computing Foundation, who actually hosts and owns the project and all the code and trademarks and everything. So we sort of advocate for the project with our host organization and partners. And we establish the governance for the rest of the project. But things like how a particular portion of the project is administered are handed off to the SIG groups.

KASLIN FIELDS: And you were on the podcast, actually, several years ago. So this is an exciting new change from the last time we talked to you as being part of the Steering Committee.

BENJAMIN ELDER: Yeah, I was on-- I think it was episode 69. I was just looking the other day. It's been a while.

KASLIN FIELDS: It's been a while.

BENJAMIN ELDER: It was a different host and everything. We were talking about kind mostly, which has been a really fun project building local cluster tooling for testing Kubernetes and for other use cases eventually.

KASLIN FIELDS: And this is episode 196. So if it was 69 and now you're on 196, moving the numbers around-- but more about the Steering Committee. So I think a lot of folks really struggle to understand what the Steering Committee entails.

I kind of think of it as-- I don't know what a good way to say it is. But they kind of deal with the politics that go into running such a large project. There are so many things that you have to deal with around the project existence itself that I think the Steering Committee kind of handles.

BENJAMIN ELDER: That's a lot of how I put it. I'd say if you're trying to get a better idea, people are wanting to reach out, or if you're interested in running for Steering when the elections come around, which are annual-- we have rolling seats.

So we have four one cycle and three the next cycle out of the seven so that we don't have complete turnover every time. And those are two-year terms, two-term limits. The Steering Committee will run an AMA session for potential candidates. And that's a good opportunity to learn more. We've also run a number of them at KubeCon.

There's a lot of nuance to what it is. And a lot of what they're doing-- we're doing now-- isn't super visible. I've been learning a lot more about that myself going through the past archives and talking more to their members about just what all exactly happened.

There's a surprising amount of things that, despite being an open source organization with open governance, aren't so easy to go talking about publicly-- legal issues that come up with projects or people raising concerns about how to handle certain conduct. And we have a separate conduct organization. But Steering also engages with them and elects them.

So there's a lot of fun politics and legal things. And a lot of it is kind of not so easy to just go talking about. I know somebody raised a compliance issue with something. And we're like, well, we probably don't want to go yelling about that while we figure out who should handle that and how it should be handled.

But in the meantime, yeah, we're talking to people and going, hmm, we think this needs to get fixed. Who's the right person for this? It's just sort of a general escalation path for things. Even if something like that that's maybe a little bit technical comes up, if there's any question of whose responsibility is this, it tends to get punted up our way to at least reroute.

We have one right now where the Security Response Committee, which is another piece of our governance outside of the SIGs-- and we have a couple of committees like this that handle critical factors like code of conduct. The Security Response Committee handles incoming CVEs and security reports and delegates that out through the project. And they're working on a new policy for how they govern new members, and rotating people out, and trying to improve how that works. And so again, that goes up to Steering.

The feedback that I've gotten from other members-- and I'm already feeling a little bit-- is that it can be hard to make progress on the other things that you thought you wanted to improve. Because there's always something like that coming up that's like, oh, yeah, that's important. We should deal with that. And you get a little squirreled off into the latest drama constantly.

KASLIN FIELDS: So the Steering Committee is kind of a very broad "buck stops here" kind of leadership group within the project.

BENJAMIN ELDER: Right, working with the community to establish a governance for how it works for all the other things. In most cases, Steering isn't going to directly handle things. They're going to get handed off.

But there's a decent amount of, hmm, we're not sure where this fits. Or maybe we need to update some rules around this. There are-- all of the things that don't directly involve one of the other groups, which-- the other groups do pretty much everything, all the technical work, whatever.

But then you still have, OK, we need to go talk to our parent foundation, the CNCF, about funding for the infrastructure for the project, which is something that I've been involved in. Things like that, there is a SIG that's involved there. But they need that formal escalation path to underscore things and to discuss with that.

We have also, for example, had a governing board seat for the project. So the governing board of the CNCF is actually a totally closed meeting with some representation for this very large project. Kubernetes has-- I think last time I looked, it was 156 repos that were--

KASLIN FIELDS: Just a couple.

BENJAMIN ELDER: --in the org.

KASLIN FIELDS: You know, it's fine.

BENJAMIN ELDER: And I'm sure it's grown since then, even with some of them being retired out. We have a lot of sub tactics and a ton going on in the organization. So it's important that people go discuss these things with even the foundation level.

And so we have someone go do that. They've typically been a member of Steering. I think they're not required to be. But usually when it's worked out, someone has taken one for the team and taken on that role as well.

Like you mentioned earlier, also you'll find that most people who seem to get elected, they get there because-- or at least, I'd like to think based on everyone else and hopefully myself-- they're there because people have seen them doing leadership in other parts of the project. But it's not so easy to just drop those. So between that and a day job, it can be a pretty heavy time crunch.

KASLIN FIELDS: I think there's a really important point there where you're talking about there are things that you wanted to do and becoming a member of the Steering Committee. And oftentimes, you don't get to focus on those, which would be kind of more of the "fun," quote unquote, things because they're things you wanted to do.

But there's all these other things that come up all the time. So you end up kind of missing out on those. But it's important during the election to have those things that you want to do so that people can see what you're all about.

BENJAMIN ELDER: Absolutely. And a lot of those things are themselves kind of fun things. But they're still fun to do. I know something that's really been on our minds has been-- we've already touched on this here, but how people end up into it. You'll note one thing since Paris stepped down-- Pittman, who was on Steering--

KASLIN FIELDS: Huge leader in the community-- does all sorts of things.

BENJAMIN ELDER: --it's now an all-male body. Paris is still our governing board seat and doing tons for the project, but not only that. It really highlights that Steering is a seven-person body. For myself personally, touching on those topics, I'm looking at trying to avoid being part of some of the other parts of the problem and getting out of some of my existing roles. Towards the end of last year, I was able to promote two new tech leads and a chair for SIG testing.

But I still need to improve the number of active ones. And then I'm going to be looking at dropping at least one of those roles as we have other new people rotated in. I think in both of those positions, until I had some new people added last year, I was the most junior member.

But some senior members are kind of occupied with other things now and are de facto moved on, so not really. And I think it's very important. Because it's also part of how Steering gets elected, I think, is these other leadership roles are kind of like a pipeline. So that's been a hot topic for a while is, what can we do to help ensure that new people are getting big knowledge roles in the project?

There are some places that are really good about it. And there are some other places that are not. You can look to things like the release team, where people get very visible leadership positions. There's a decent amount of turnover. And there's a lot of new faces that come up through that.

I did it before myself. I shadowed for the release lead. And it was a really good experience. And I think we need more of that in the project. So in my own little corner of forensic testing, I'm trying to make sure we get some new people rotated in and look at rotating myself out so that we can bring some new people up and so I can devote more time to Steering while I'm on it.

KASLIN FIELDS: There's a lot there that I want to dig deeper into. But before we get too deep into leadership, and growing the project, and all of these really cool things, I want to mention something that you talked about earlier is financing things within the project and approving budget requests from SIGs and things. In my own work as a member of SIG ContribEx, we have needed tools.

And we've had to request funding. And that has gone up to the Steering Committee. And there's been a recent change for the whole community-- not just contributors, but for users-- that's really, really important that also has to do with funding. We are moving the registry where we put the Kubernetes images to this new registry at registry.k8s.io. And it's really an interesting project technically. So I was wondering if we could dive into that a little bit.

BENJAMIN ELDER: Sure, yeah. I've been pretty involved in that project. I worked on the current design and some of the code with a lot of other people to help push that forward. It's an interesting little project. Essentially, we set up a small application that allows us to split the load between different cloud providers depending on where the traffic is coming from-- nothing too surprising there. But we couldn't actually find anything off the shelf for the container registry specifically.

We're taking advantage of some properties of how the container registry works to ensure that we have one location that we've already been using that has all of the critical metadata that controls the images and ensures their security. But then we can offload the bulk of the bandwidth to multiple simpler hosts that just have a copy of all the layers. In this case, we're actually just using S3 buckets and a bunch of locations, copying the image layers over and doing our best to pass along traffic to those locations.

One of the things that we have when we fund the project is a lot of the project's funding isn't money. It's credits from cloud providers. We've been getting $3 million a year from Google. And we're going to get, hopefully, a matching application from Amazon. I think they just gave us some of the credits.

The specifics are a little bit different but more or less the same idea that, OK, so you can spend this on AWS. You need to use it there. This is not a budget to go buy an external CDN or something like that. So you might have asked, well, why don't you just throw "insert CDN here" at it?

Well, but we need some way to pay for it. And most of what we have for really large resources in the project is this cloud provider credits. So we're also trying to get more providers involved and say, look, under this model, your users get faster pulls.

And it's actually relatively cheap because we'll detect that this is your user's traffic based on the address and only offload it then. So you're not paying egress fees. The Kubernetes project spends a ton of money on egress fees.

KASLIN FIELDS: Any cloud engineers out there, I'm sure, can relate to the pain of egress fees in the cloud.

BENJAMIN ELDER: I think we're unique in that it's a project that underpins a lot of cloud infrastructure. So you have a ton of bandwidth. Instead of going to end users scattered around the globe, it's going specifically to the cloud providers.

And so if you're hosting primarily out of one-- we've hosted things out of Google for a long time because that's where we've had resources. But now you've got to send traffic from that cloud to all the other clouds. And it can get pretty expensive.

So we're looking at a number of ways to get that. And the registry project for the past year is one of the big ones. It's also about putting the project deeper in control. Google spent up this effort years back to try to migrate all this infrastructure that had just been stood up inside of Google to be something that the community actually has direct access to and can control and govern like everything else in the project.

It's tricky. You fund it. And you have to shift things over without disrupting people too much. So a while back, they had the Google Container Registry team work with them to do a more effective multi-regional k8s.gcr.io.

I don't know if people remember it used to be gcr.io Google containers. Because for a while, various things in the project were just named Google containers and didn't have a real name yet. That was the freenode IRC channel in the early days.

KASLIN FIELDS: I think I may have used that once. And wow, hadn't thought about that. [LAUGHS]

BENJAMIN ELDER: I used it a long time ago in '15 when I was a Summer of Code student. And yeah, it's a very different experience. But to that end, even though by that point in time, they well had multiple different companies working on it and they had a name and everything, it's just hard to say, OK, we're shutting down the channel and moving.

So there were a lot of things lingering in that. So the project moved from Google containers hosted internally at Google but could be pushed to, by external tooling, to, OK, these are registries that were actually set up by the project. But because of cost, and some other things, and we didn't have any additional infrastructure, they set up k8s.gcr.io, which is actually a Google-controlled domain name.

They especially handled listening to a couple of regional copies to get decent performance and cost there for such a large-scale usage as opposed to some internal customer running their own applications or something like that. So a big deal here is that we're moving to a community-controlled domain, registry.k8s.io. And community control, I really mean that.

SIG k8s Infra is one of the parts of the project that has been really, really tight about things in Git. Almost, to the extent possible, if you want to add a subdomain to the Kubernetes project, you go file a pull request to a little YAML file. And just add your subdomain and what you want it to point to. And so we add all kinds of stuff there.

So by having it there, it's up to the broader project to control these things and help solve for-- and a pattern we've seen across multiple companies is someone sends up some small piece of infrastructure. The project starts using it. That person moves on.

Turns out it's no one's actual job to keep after that. And then we're begging people that we know that work somewhere adjacently at that company to go, hey, what is this infrastructure? Where is it? How do we fix it? Something's wrong with it.

Having open governance also means that it's sort of up to the community. And that can be hard. But it also means that someone else can take over. And everything is hopefully, to the extent possible, in source control and has Git history. And we know what's going on.

KASLIN FIELDS: When you were talking about the Steering Committee, one of the things that I've learned since I became a regular contributor to Kubernetes is that it surprises me sometimes how much it feels like being part of a different company. You have to make funding requests. And they go up to the top. And maybe you'll get it. Maybe you have to provide justification.

BENJAMIN ELDER: Yeah, I mean, we have to work with what we're given. And nothing is free. So Steering is, when we get those funding requests, we turn around and interface with the Cloud Native Computing Foundation-- similar thing.

It's just like as a matter of scale, if I am looking for the CNCF to, through one of their services, handle something like providing some of their funding for some small-scale thing or something that they commonly do as direct services, like the legal side of things, looking at licensing for-- maybe a project is being donated. So when we do those things, we have a service desk that we file tickets with. But as a matter of them being able to scale, they give it to a certain set of members from each of their projects.

And so for Kubernetes, our maintainers of record are the Steering Committee. So the Steering Committee themselves, actually, just has a process that's like you file a GitHub issue so that we can publicly discuss this is what we're looking for. But then we turn around and we go to the service desk most of the time.

And we say, OK, I have a service ticket for this GitHub issue. Can you handle this? I think it keeps things a bit manageable for the CNCF. But it does add some layer of bureaucracy. It feels a little bit more like a corporate environment even though it's external.

KASLIN FIELDS: As an open source project, like you were saying, anyone should be able to participate. We try to make it as open to anyone as possible, both people from a variety of companies around the world and people who are just interested in the project.

There are all sorts of places that people can come from to get involved with the project. So that's one reason that it's really exciting to see the registry moving to a community-owned domain. And like you were saying earlier, one of the aspects of diversity that folks might not think about with the project is corporate diversity-- having a variety of different perspectives represented throughout the project.

BENJAMIN ELDER: I don't think this is the most important one. But it is kind of like an essential thing for if you're trying to do what Kubernetes is doing in shipping a vendor neutral API, end user confidence is going to depend a bit on saying, oh, well, this isn't just so-and-so's companies thing that's just completely at their whim. It has the backing of multiple players. And it will have longevity and will not just be contorted towards one company.

It's important that Kubernetes isn't just a Google thing but the source. I sort of wish we had, just off the top of my head, a different term for that because I do think it's a very different thing from what people are usually talking about with diversity.

And that's important. And it's something that the project could certainly do better at. I think we do relatively well at the corporate angle. Where it's really essential in places like Steering, we mandate it. But for other things, things like race or gender, we don't and we aren't so good about that.

And it's something that the project probably should be talking about more. I know it did actually come up quite a lot when we had our AMA session for Steering at the last KubeCon. And it was almost challenging to have that on the AMA panel because everyone wanted to say something. It's something that we've all been thinking about a lot.

KASLIN FIELDS: I think these challenges that we're talking about here are all pointing to a problem that we talk about constantly within the community, which is bringing in the next generation of leaders. Like you were saying, the project had its origins within Google. And so a lot of the folks who have been with the project since the very beginning have a very specific background.

But now the project is so much more. It's so much bigger than I think anyone could have dreamed when it started. And so it needs that kind of broad perspective in order to keep going forward. So I know a lot of the folks in leadership positions that I talk to are always talking about, how do we get folks in who represent what the project is today and can help lead it into what it will become?

BENJAMIN ELDER: Yeah, I feel like [INAUDIBLE] for me is that we can improve some of those pathways and then dead end some of them. It's kind of weird. Because at this point, I've been at Google for a few years. But I still kind of see myself as one of the younger, newer folks compared to some of the people that actually started the project when it was going 1.0 as a college student.

And I wound up at Google because it's a great, supportive place to work on the project if you can make that work. But we have got to include people from other places. And we've got to rotate in new folks. So I said I'm going to start some more of my SIG. I've been talking to some leaders of some other SIGs about how we can do that.

And Steering is talking about it a lot. What can we do? I think there's a line with how heavy-handed you can actually go about things and whether or not that's productive. But something has to give. And it's not just the top of the leadership.

If you look at who's taking care of portions of the project, who approves changes to code, and things like that, there's a number of places that have kind of dwindled off on us. And there's a few people that are kind of doing everything. And we haven't done a great job bringing in new folks. I think, also, as the project scaled so much, as much as I feel like the community has stayed very welcoming, it can be very hard to get started because areas are swamped.

And if you're depending on me specifically to look at something you're doing so that you can participate, I might not get to it because now I'm doing too many things. So I think we also have to push people to take it upon themselves to spend a little bit more time getting out of doing everything and getting new people taking care of some of the things that they were doing.

KASLIN FIELDS: I like to tell folks that I coach that you're not a senior engineer unless you are-- if you're always too busy to help out others, then that's not a senior engineer quality. You've got to be able to start bringing more people in. And as you mentioned earlier, you are a chair and a tech lead, both, of SIG testing, which is part of how you ended up in the Steering Committee. You go from one to the other. So I'm sure that you're thinking a lot about how to bring people into both of those roles, which don't have to be the same person in the future.

BENJAMIN ELDER: Yes.

KASLIN FIELDS: I know I personally am currently shadowing for the chair positions in Contributor Experience. And one of the chairs there is also wearing many hats. And so that can be an intimidating thing for shadows, I will say. [LAUGHS]

BENJAMIN ELDER: Yeah, and that's actually one of the examples of something we're looking for in the project is, we're looking at, can we do this pattern everywhere? Kubernetes SIGs used to just be-- they had the chair position. And that was all the root technical leadership, and the administrative, and people leadership, and organizational work that the chairs do that's very important. And if we can split up that workload with more people-- so I was a chair when SIG testing split it.

But since then, it's kind of been like, well, even though we have in tech leads because it's such a sort of broad space that touches on so many things as more horizontal SIG as opposed to a vertical on a specific area of the project, we've wound up with, well, but we still need to be able to escalate to a couple of tech leads that have not fully overlapping sub portions of the project that they're officially handling. So I've kept that hat. But then in Chair for a while, I was the only particularly active one.

And someone has to run the meetings and things. But I'm trying to get out of that business and make sure-- and so Michelle Shepardson, shout-out to her, has been doing a fantastic job handling a lot of the Chair work. And Antonio Ojea and Patrick Ohly have stepped up as new tech leads that are taking over some of the areas that I used to spend more time on, like the E2E framework, that I really haven't been able to keep after, where we've had important contributions for how we actually do the tests.

So yeah, I think people are doing too much. And if you're doing too much, then you should be looking at, how can I reduce that? Steering should be looking at how we can encourage that. And we are.

KASLIN FIELDS: And leadership, especially in open source and Kubernetes, I think, is such a living and shifting thing, like you're saying. It started as one thing. And it's changed into something a little bit different at this point. And it's going to continue to change in the future.

I think a lot of folks who are getting started who are trying to find ways to grow their careers often ask folks in leadership positions, how do I get to where you are? And every time I hear that question in these types of meetings, I always hear the person say, well, I don't know how I got here. It just kind of happened.

It wasn't really a clear path. I don't know how this worked out. And I think that a lot of it is because the leadership positions that folks are aiming for today will shift by the time that they get there. And the world that they live in will be a little bit different, just like the Kubernetes project itself has grown and changed.

BENJAMIN ELDER: Absolutely. We even have whole-- the SIGs, the groups that own [INAUDIBLE] the project, we have them spun up and shut down as things change, shift, and where people are working, and what's happening in the project, and what makes sense. I think the one consistent thing that people can do is being consistent. It's very hard.

You mentioned how it feels kind of like a company sometimes. But one of the things that a company has is they have hiring and HR. And you have some level of expectation that you can trust your coworkers, hopefully. Hopefully, this is a good company.

You feel like you can trust your coworkers. And you have kind of a clear delegation to some degree if this group owns this thing. And this group owns this thing. And we'll go talk to them, whereas that's a lot harder in open source. We don't have hiring and HR. And you have to get to know people.

And the best way that you can build that trust is to just keep showing up and being helpful, even if it can get a bit frustrating sometimes that some particular area isn't moving forward. Maybe pick up something else. Just try to keep [INAUDIBLE].

You should keep showing up and being consistently helpful and eventually notice-- one of my leads is like that. When Antonio first contributed to the project, I believe he was a manager at SUSE working on some other things entirely. And he just started picking up some work around some of the networking things that he has some expertise in.

And over time, he was consistently showing up, helping with the kind project as well. And somewhere in there, hey, you should be a tech lead in the SIG because you're actually doing all this stuff. And even if this isn't your main focus, you'll get at it. And you have more expertise than most and a lot of trust in the community-- I think you won multiple SIG awards at the Contributor Summit.

KASLIN FIELDS: He did. I'm pretty sure.

BENJAMIN ELDER: It just happens over time and just keeps showing up. So those are things that a project needs to do. But for individuals who want to get in, yeah, it's going to shift a lot, where the specific thing is, and so much so that I think you'll just have to keep looking until you find where you hit your niche and you fit in.

I notice when people keep showing up and keep helping. And over time, you start to go, hey, I think I can trust this person. They're not just some random username on the internet. I kind of know them. And why don't we promote them to owning some portion of the project?

KASLIN FIELDS: That's exactly what I was going to ask you about is advice for folks who want to move up toward leadership positions within the project. And it sounds like your advice is identical to my favorite advice to give to people, which is to be stubborn. [LAUGHS]

BENJAMIN ELDER: Yes, be very persistent.

KASLIN FIELDS: The moving and shifting we're talking about can feel a little scary. But it's actually a huge opportunity for folks. As long as you keep showing up, you'll suddenly be part of it.

BENJAMIN ELDER: Absolutely. In fact, some of the areas that are hardest to contribute to are great areas. Because if you keep persisting in that and people realize, hey, this isn't really being handled, and we have someone else that has shown up and is working on it and trying to take care of it, that first party's going to be frustrating, where inherently, this area that isn't already being really well-handled and is a good spot for someone to come in and take over. Because it isn't so well-handled right now, it's going to be hard to get the initial work done.

And people are going to be busy. It's just such a huge project. We have so many issues open, and features, and everything. So it's really hard to keep on top of everything happening here. You can't really do all of it. And even if you try to pick some area, they expand over time.

But if you could just keep coming back, be persistent about it, people will notice. And I think a lot of open source contributors will also start to go a little bit more out of their way to try to help you specifically if they notice that you've really been trying and trying to make sure that you're not stuck forever. It's hard to figure out what is even fair for how much time do I spend reviewing things, and where do I spend it, and things like that.

But definitely one of the things that I know I pick up on is when someone just keeps coming back. I'm like, well, I feel for them. I don't want them to be stuck forever. They've been trying so hard. Let me make sure that I'm getting to some of that.

KASLIN FIELDS: And I think today, we've covered a huge variety, a great breadth of the Kubernetes project, which can be really hard to find when folks are first starting to understand the Kubernetes project. I think it's just so much to take in that it can be really difficult. So I think we've got some advice in here for folks who want to get involved with the project, who want to understand how the project has grown over time, how the project is led. And it's not just advice for contributors either.

Really important thing that I want to come back to right here at the end is the registry change. For folks out there who are using Kubernetes, if you pull images, container images, to run Kubernetes yourself, make sure that you check out the change to registry.k8s.io like we've been talking about. It's very exciting. because it's going to be a community-owned domain.

It's going to improve performance, hopefully, for a lot of folks in different cloud providers. So it's a really exciting change. And I want to make sure that people check out registry.k8s.io and start using it. Any other final notes that you want to add?

BENJAMIN ELDER: Thanks for that plug and for having me on. If you visit registry.k8s.io-- that's k, the numeral 8, s, .io-- in your browser, that will take you to more about that project and more that you need to know there.

KASLIN FIELDS: So definitely check it out. Thanks so much for being on today.

BENJAMIN ELDER: Thank you, Kaslin.

[OPTIMISTIC MUSIC]

ABDEL SGHIOUAR: Kaslin, that was a great episode. Your interview with Ben was quite interesting.

KASLIN FIELDS: Yeah, I had a lot of fun talking with him. He has so much knowledge about the Kubernetes project. There's so much that we could talk about. It was kind of hard to pick just a few topics to go into.

ABDEL SGHIOUAR: Yeah, and he was very involved with kind, or is still very involved with kind, which is actually one of my favorite tools.

KASLIN FIELDS: That's true, which we didn't even talk about.

ABDEL SGHIOUAR: Yeah, but it is one of my favorite tools. I really like being kind.

KASLIN FIELDS: I think a lot of people do-- very popular.

ABDEL SGHIOUAR: Yeah, I think maybe behind the scenes, we use it with Anthos. When you are deploying Anthos with the command line, it spins off a local kind cluster.

KASLIN FIELDS: Huh, I don't know that I knew that.

ABDEL SGHIOUAR: Yeah.

[LAUGHTER]

It leverages-- because Anthos, specifically the VMware version, leverages the controller pattern in Kubernetes. So it spins off a local kind cluster, deploys the operators on it, and that's how it bootstraps the cluster itself remotely.

KASLIN FIELDS: Custom Resource Definitions.

ABDEL SGHIOUAR: CRDs, exactly.

KASLIN FIELDS: Custom controllers-- so useful. They do so many things.

ABDEL SGHIOUAR: Yeah. But from the interview, there were a couple of things that were interesting, starting with the technical aspects of it, or at least the migration of the registry. As a recap, Kubernetes starting in 1.26 are migrating or moving away from k8s.gcr.io to registry.k8s.io, which is the new registry. But it's not technically a new registry. It's more like a router, like a proxy that just routes requests.

KASLIN FIELDS: Like a load balancer.

ABDEL SGHIOUAR: It kind of-- yeah, exactly. I looked into the source code of that little proxy. And I looked into the document that explains the context, the background.

Why have they decided to do that? And obviously, one of them is basically reducing egress costs and providing people with a faster download time. But in the document, which we will link in the show notes, one interesting thing was when they analyzed the traffic, they realized that about 40% or 50% of traffic hitting the old registry was coming from AWS.

KASLIN FIELDS: Yep.

ABDEL SGHIOUAR: So I was curious if you know, is that people deploying Kubernetes themselves and they download the images from the registry?

KASLIN FIELDS: There's a lot we don't know about it. We just have the basic information of these IP addresses, which we know are in Amazon-pulled images. And that's about it for what we can learn.

But it's so fascinating to me that this is such an important change that affects users. And the project needs to do it for financial reasons to be able to sustain the project. So it's a very interesting, very public look at how the project works, I think.

ABDEL SGHIOUAR: Exactly. And it seems like-- and you have covered this in the interview. It seems like one of these things was when they set it up in the past on GCR, they hadn't really thought about it that much initially until they looked at the billing. And they were like, oh, this is going to cost a lot of money, right?

KASLIN FIELDS: Right. This is a thing that people pull from and thus, egress costs.

ABDEL SGHIOUAR: Exactly, exactly.

KASLIN FIELDS: I'm sure some cloud engineers out there listening have probably encountered something similar.

ABDEL SGHIOUAR: Yeah, especially in these interesting times we're going through. Cost optimization seems to be important for people. The other topics you discussed during the interview were the community. Because Ben is a Steering Committee member. And he is also a SIG testing lead?

KASLIN FIELDS: Yeah, chair and tech lead, both.

ABDEL SGHIOUAR: Yeah.

KASLIN FIELDS: They're trying to make that separate jobs, separate roles within the special interest groups. But for a lot of special interest groups, I think they're currently the same person. We talked about trying to get new leaders in. And part of that is defining new roles that are a little bit more approachable for folks who are getting started with the project.

ABDEL SGHIOUAR: Yeah, as I was listening to the episode, I was actually going and looking into the CNCF page. Because I've never really-- I have heard the TOC before. I have heard the tech leads, and the SIGs, and the committee. But I really never understood what they did. What's the difference?

And one interesting thing I learned is that what Ben does is purely not technical. The stuff that the Steering Committee does is not technical. It's more the governance aspects of the projects itself.

KASLIN FIELDS: Yeah, they're very much like a leadership group that handles-- I don't know if this is bad to say, but handles the bureaucracy of running a giant project.

ABDEL SGHIOUAR: That's how it sounds like, yeah. And this actually means that if you are interested in contributing but you don't have technical knowledge, you don't have to have technical knowledge.

KASLIN FIELDS: Yeah, absolutely. That's something that I'm always telling folks. The group that I run within-- the special interest group for contributor experience, we mostly don't really write any code. We have a few projects where folks want to write code. There are opportunities there.

But we also have work where you don't have to have technical knowledge really at all. And you can learn it while you're working with us. So join us.

ABDEL SGHIOUAR: To a large extent, the CNCF community and Kubernetes community itself seems to be organized like a company. It would have technical people. And you have people with other skills that are not just necessarily technical, right? You need people to organize things and budgets, and approve budgets, and probably do marketing, and all these kinds of things, which-- kind of like how companies work.

KASLIN FIELDS: Exactly. That's something that kind of surprised me when I started working with open source. I thought, oh, this is such a different world. I know so little about open source. But there's a lot that's very similar to just working within a company.

ABDEL SGHIOUAR: Yeah, exactly. I think it was a very interesting interview. I think a lot of people will probably learn a lot of new things. I certainly learned a lot of new things, especially from the organization aspect of the CNCF, not the technical part. But I think this will be interesting to a lot of folks.

KASLIN FIELDS: Awesome. And if you're out there listening and you want to get involved with the community, come say hello to us on the SIG ContribEx channel within the Kubernetes Slack, which is k8s.slack.io-- slack.k8s.io? I can never remember which one comes first. But we'll put the link in the show notes.

ABDEL SGHIOUAR: Yes, we will.

KASLIN FIELDS: Thanks for joining us. Catch you next time.

ABDEL SGHIOUAR: See you.

[OPTIMISTIC MUSIC]

That brings us to the end of another episode. If you enjoyed the show, please help us spread the word. And tell a friend. If you have any feedback for us, you can find us on Twitter @kubernetespod. Or reach us by email at kubernetespodcast@google.com.

You can also check out our website, kubernetespodcast.com, where you will find transcripts and show notes as well as links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thank you for listening. And we'll see you next time.

[OPTIMISTIC MUSIC]