KASLIN FIELDS: Hello and welcome to The Kubernetes Podcast from Google. I'm your host, Kaslin Fields.

ABDEL SGHIOUAR: And I am Abdel Sghiouar.

[MUSIC PLAYING]

KASLIN FIELDS: In this episode, we talk with Kat Cosgrove and Natali Vlatko. Kat and Natali are both members of the Kubernetes Steering Committee and leads of SIGs, release for Kat and docs for Natali. The Kubernetes Steering Committee recently published a new policy on AI usage for Kubernetes contribution. We dive into that AI usage policy and explore the challenges and experiences AI has brought to Kubernetes maintainers. But first, let's get to the news.

ABDEL SGHIOUAR: Apple has shipped version 1.0 of its native container tool for Mac OS. Built in Swift specifically for Apple silicon, it departs from traditional shared VM setups like Docker desktop by isolating every single Linux container inside its own dedicated micro VM Using the native Mac OS virtualization framework.

KASLIN FIELDS: Google launched OpenRL, a new open source project designed to streamline the training and reinforcement learning loops of large language models. The tool brings declarative Kubernetes style resource orchestration concepts to the messy process of AI model fine tuning.

ABDEL SGHIOUAR: At KubeCon CloudNativeCon India, the CNCF announced they added 14 new members and users and nonprofit organizations. This highlights the continued growth of the cloud native ecosystem. One of the new members is Lovable, who was a recent guest on the show. We highly recommend you go listen to episode 268 about the Asian sandbox.

KASLIN FIELDS: Is a pod the right deployment unit for an AI agent? Lin Sun from Solo published a community post on the CNCF blog, questioning whether the classic Kubernetes pod primitive is still the best abstraction for hosting autonomous, long running AI agents and introducing Agent Substrate, a project attempting to bring a solution to the table. And that's the news.

Hello and welcome to the Kubernetes Podcast from Google. I'm super excited today to be talking about AI in Kubernetes. And whenever I say this, people always think that I mean running AI on Kubernetes. There's so many different angles that you can take for it.

But today I want to talk about how AI is affecting the open source community. So for contributors, how they using AI? What is the community thinking about how AI is changing, how we do contribution, all that kind of stuff. And to join me to talk about that, I have Kat Cosgrove and Natali Vlatko. Would you like to introduce yourselves? Let's go with Kat first.

KAT COSGROVE: Sure. During the day, my actual job, I am the village sorcerer at Village SQL. But in Kubernetes, I am a technical lead for SIG docs. I own the release team subproject for SIG Release, and I'm a member of the Steering Committee.

KASLIN FIELDS: Wonderful. And welcome to the show. And Natali.

NATALI VLATKO: Thanks for having me. In my day job, I run the Open Source Program Office, or OSPO for short, at a little known company called Cisco, which is a lot of fun. But in Kubernetes specifically, I am one of the co-chairs of SIG docs as well. And in terms of other open source fun, I'm also currently a steering committee member for the TODO group, which is a project of the Linux Foundation that gets open source program office professionals and enthusiasts together to basically do good open source in the world.

KASLIN FIELDS: I'm so glad that I asked you to introduce yourself, because I didn't know anything about that, and now I do. [LAUGHS] I had not heard of the TODO group.

NATALI VLATKO: I'm more than happy to have you in the community. You don't have to be a member in order to join the Slack community that we have, to ask questions, or to use our resources. Happy to provide all of that later on.

KASLIN FIELDS: We'll include those links in the show notes. If you would, please, send those over.

NATALI VLATKO: Definitely.

KASLIN FIELDS: And both of you have had some experiences as very open source involved folks with AI in open source and particularly in Kubernetes, though, if you have insights from other communities, I'd be interested in those as well. But let's start off by diving right into it.

There is actually a policy from Kubernetes about how AI is allowed to be used in the project. I think a lot of people probably don't know that. You should check out the policy. I will link it in the show notes. But with how fast everything is changing, all the different tools that people are trying to use and different ways that people are trying to use AI, I think it's really important to have some guidelines. And we've also heard interesting news stories about how AI is affecting open source.

So let's start off by talking about what is in this relatively new-- it came out a couple months ago, relatively new AI policy for Kubernetes. And Kat, you published it. So do you want to kick us off by giving us a little summary?

KAT COSGROVE: I sure did publish, and it was less dramatic to publish that than I initially expected. I thought we would get a lot more pushback. But generally, the vibe is you are welcome to use AI in the assistance of authoring a PR. That is fine. You can do that. We don't object to it.

But you have to disclose it in the description of your PR. You cannot list the AI tooling as the co-author. You can't co-sign a commit with an AI tool. And that's a legal thing, right? Your AI tool cannot sign a CLA, so it cannot co-author a PR or co-sign a commit. You also can't use the assisted by or co-developed or whatever similar commit trailer for your disclosure.

That is sort of a marketing thing. We expect hyperscalers to behave. We don't expect necessarily that some smaller, shady tools would behave. That's to prevent somebody from pointing at the Kubernetes Git history and saying, oh look, the Kubernetes Project endorses our tool and slap our logo on their website. So put your AI disclosure in the PR description please.

But also your commit messages can't be generated by AI. You need to write those yourself. And regular rules about character count and being descriptive apply. When responding to review comments, you have to do that without relying on AI at all. We want to talk to you, the person. We want to talk to you about your code and make sure that you understand what you're submitting.

And that includes maybe non-native English. It's OK if English is not your first language and you think your English is bad. We would way rather handle that than talk to your LLM, because then we can't ensure that you actually understand what you're submitting. So please talk to us. We love talking to you, the person. But otherwise, you are responsible for your AI code.

KASLIN FIELDS: And that's what it boils down to, this AI policy. We're not saying don't use AI. [LAUGHS] Sure, it can cause some problems, which I think we'll get into in a little bit. But you can use AI, but there are some rules that we have to adhere to.

And remember, this is arguably-- the CNCF said it in one thing that I can find, that the Kubernetes Project is the second largest open source project in the world. Regardless of that, this is a very, very large project with people all around the globe. So there's a lot of legal and kind of bureaucratic rules that we have to think about when we make policies and when we adopt new tools. And so I love the way that this policy is worded, because I think it acknowledges that without being overly wordy and describing every little thing. It sets out the rules and you can just kind of roll with that.

But the first one that you mentioned that I want to point out especially is that we are not allowing folks to co-author their commits with AI tools. You can use AI tools, but you can't specifically use the term co-authoring. So you describe that a little bit, but I want to reiterate that for folks, because I think it's really important.

KAT COSGROVE: Yeah, and when we say that, that means like, you know, how copilot can like commit directly to a PR and it shows up as a co-author? Can't have that. Hard no.

KASLIN FIELDS: Like you said, for legal reasons, because Copilot or whatever AI tool you're using cannot sign the-- what does the CLA actually stand for?

KAT COSGROVE: Contributor License Agreement.

KASLIN FIELDS: Contributor License Agreement. There we go.

NATALI VLATKO: And I want to stipulate that the reason that we say it cannot sign, not that it isn't able to technically sign it, because obviously it's a click on a button to verify your identity via your email address, and so on. But what it actually means in a legal sense is that your agent, your LLM, your any kind of AI assistant tool isn't a legal person in the definition of the contributor license agreement. That's what that actually means. So we're not talking about the technicality of not being able to sign. We're talking about the legality of it's supposed to be a person that is human.

KASLIN FIELDS: And it also lends to the overarching idea of the policy of you need to understand what you're submitting. You, as the human, are responsible for this code that's going into this open code base.

NATALI VLATKO: Right. And I think the important part on that too, going on to the next area of why we want specifically you to disclose that you've used AI as part of the policy as well, we've definitely heard in the community since introducing the policy about how there is a possibility of or maybe people are worried about bias coming in that you must disclose if you used AI or not. And it is definitely linked to the idea that you're the human responsible for the contribution.

And so us having the signal that you've used AI could possibly determine whether you're adhering to that accountability aspect of do you understand the contribution you're making, or have you reviewed the contribution that's taking place? And being up front to say that you've used AI helps us understand whether are we able to review this PR with you or not, because people don't read policy quite often as well.

And us asking once again in the PR to disclose. And I often when I'm asking say, can you please disclose if you've used AI so that we can review this PR better together? I give context as to why that question is asked so that people understand that it's not about me putting your review at the end of my pile because it's AI assisted, but it's more so to understand are you fully understanding of and have you fully reviewed the contribution that you're making?

KASLIN FIELDS: AI is a tool for improvement, not a replacement. And another really important aspect of this policy that relates to some experiences I've had in the open source community. When I've talked with other co-chairs, other maintainers of Kubernetes about AI, there was a meeting where I asked folks, are you seeing a huge influx of AI written, low quality PRs that are overwhelming you? How's that going?

And surprisingly to me, the maintainers actually said not really. One, because they were just as overwhelmed by the AI PRs as they were by the human PRs, because we were already overwhelmed. We already had too few reviewers for the number of PRs coming in. So just as overwhelmed as before.

But the biggest thing that they mentioned that they were frustrated by with the use of AI in PRs was the comments, because they were getting these AI written pages and pages, long comments that told them nothing about the code that was actually being proposed. And so they couldn't review the PRs properly. So to your point, Natali, we want to review PRs with folks, right? So do you want to share some more about that?

NATALI VLATKO: Yeah, definitely. I've definitely come across PRs that I reviewed that the AI tool that they've used had hallucinated an issue that is being fixed that doesn't exist. And I click on the issues that are referred to in PR descriptions to see what it's fixing, and I've come across 404's

And it's just something that a couple of extra small bits of review, even if you are using that tooling to generate your PR description, and so on. If you're not checking those things, it tells me that you care maybe a little bit less than I do about how this review process is going to go. And so I really want to make sure that I'm spending my time with folks who are also wanting to spend their time productively when they're contributing as well.

Another interesting policy that we have on the doc side that also never gets read, but is absolutely heightened in contributions coming in because of AI, is that we have a trivial edits policy where we say, hey, you want to fix a small spell check error, a spelling error, should I say, in this specific document? And that's the entirety of your contribution.

Have you looked through the rest of this document to see if there are other improvements that could be made? Because that's kind of like a trivial edit, and maybe you can chuck that in with a couple of other improvements to make that review worthwhile time wise for your contribution and my input. And I will say that the amount of trivial edits we get is definitely on the rise now with the AI contributions.

However, interestingly, that is where AI shines in terms of it being less error prone, because the changes that are being presented are so small and so scoped that that actually isn't a horrible idea, using AI as this advanced linter for documentation, for example, just like we would possibly use that in the same way as an advanced linter for reviewing a PR. So that's an interesting kind of like-- how to say? Something that supports the policy, but also counters it at the same time, because of the increase in the contributions coming via that tooling.

KASLIN FIELDS: Kind of a dichotomy.

KAT COSGROVE: Docs gets hit harder by the AI stuff than other SIGs do. Docs has a PR Wrangler rotation, which I have to do as well as a tech lead. And the number of PRs I review and then ultimately close because the actual content of the AI generated PR does not at all match the content of the AI generated PR description is outrageous.

And that takes time for me to find. That is completely time wasted. Those PRs get closed immediately for me. That's not a conversation. That's just a, hey, this is in violation of the AI use policy. Closed immediately. No discussion. But I would rather be spending my time reviewing an actually useful contribution.

Because it's unfortunate, because sometimes the content of the PR is something that could be helpful. It is a genuine improvement to a piece of documentation. But the whole thing is AI generated. The PR description is AI generated and does not match the content. The comments from the user are AI generated. I at this point suspect that this is actually an agent that somebody has logged into the account and signed the CLA for it and turned it loose.

And I have to spend 10 minutes figuring that out. And in that 10 minutes, I could have cranked through two or three other actually useful PRs. So it's obnoxious. It's a waste of time. The AI use policy doesn't actually slow those kinds of contributions down much, but it does give us something to easily point at that is written down as our public policy so that we don't have to fight about it when we close the PR, which is a different kind of time saver, I guess.

KASLIN FIELDS: That was really challenging in the early days of AI, where different folks within the community had different views about using AI tools and how they should be used, and how it was affecting their specific area of the project. And there were some challenges on, well, do I just straight out reject this? It is causing me problems, but it's just hard to tell if it's good or not, and I'm going to have to spend more time on it.

I think your point about you're wasting all this time looking at these PRs that are not going to be high quality contributions to the project also lends itself to a point that I like to tell new contributors about all the time. I like to say that good first issues are a trap. Because everyone has to have a first issue. It's important for us to note issues as good things for new contributors to do. But historically, good first issues would get snapped up and then not completed.

And then when they don't get completed, the reviewers have to go back and triage those again, and they're spending the time triaging those that they could be spending reviewing PRs that people are actually making and trying to get into the project.

And AI kind of accelerates that, in that now not only are people snapping up good first issues like they're candy, they're also submitting something with them that is perhaps low quality, perhaps it's really wordy. It's going to take even more time for the triagers to review that. Plus, the volume of things that they have to review has increased. So it's kind of this vicious cycle that it creates for maintainers.

NATALI VLATKO: Yeah, definitely. And that cycle, interestingly, also comes from this area of folks wanting to use AI as this learning tool, which is great in some contexts. But Kubernetes as a project, in terms of how complex it can be, it's worth figuring out whether using it as a new to Kubernetes person, as in new to the community and new to contributing, versus new to the technology overall, and how to even learn what to do with it, I think that's also worth new contributors asking themselves.

Because contributor to open source generally without AI has often been for new contributors quite daunting. They often are not sure where to start. And we in our respective SIGs, and Kaslin of contributor experience and docs, have a lot of really great guidelines and references of how to take that first step on the contributing side. But an AI can read those stocks too and help out. So maybe applying the tool in a different way in order to make that first contribution is something folks can take away from this.

KASLIN FIELDS: I love when folks say that AI is an enhancer. If you understood something and are trying to create something with your knowledge, AI is going to enhance your ability to do that. If you do not know the thing and are trying to use AI to do the thing, it may just enhance your lack of knowledge of the thing and do things wrong, and then you won't know that it's doing them wrong. And then the reviewers have to catch that and that just spends more of their time.

KAT COSGROVE: Yeah, and we're already kind of crispy. But you know where is a great place to start if you want to contribute to Kubernetes or open source in general? It's your first time contributing to open source, and you're understandably real nervous and you want somebody to hold your hand. SIG docs.

[LAUGHTER]

SIG docs. You can come shadow one of us on a PR wrangler rotation, and we will hold your hand through your first contribution to an open source project.

NATALI VLATKO: Definitely. We also actually have issue wrangling as well, where we actually are trying to make sure we're creating better issues. I don't want to say a good first issue, because I also don't believe that that's always a good thing.

But being able to do issue triage where we're creating, or should I say, fixing up issues that already exist to be ready for solving by a possible new contributor and then self assigning is also something that we try and do with folks who aren't necessarily new to Kubernetes or docs, but they've been around for a little bit and feel like they want to contribute in some way that's beyond reviewing or sending in different kind of docs amendments and stuff. So yeah, definitely, if you're interested in helping out in a way that's maybe a little different but also helpful specifically to newer contributors, issue wrangling might be the way to go.

KASLIN FIELDS: And I want to dive a little bit deeper into that. I've been doing a whole bunch of workshops talking with folks in person. These have been for my day job, Google workshops. But I always mention, of course, that I'm deeply involved in open source Kubernetes. And if you want to talk about Kubernetes or open source contribution, I would love to talk with you about that. And I always get a pretty big group of folks who are asking me about making their first contributions, which I love.

So if you wouldn't mind-- and each of us are leaders of SIGs that I think are, in some ways, particularly welcoming to newer folks. But I'd like to dive into that a little bit more. Docs has several different areas where it's pretty welcoming to new folks. You mentioned PR wrangling, but there are a few others. Do you want to give us a list, Natali?

NATALI VLATKO: So yeah, firstly, I think we often overindex this idea of a first contribution. But in my opinion, it's just getting involved. And that is such a vague kind of thing, depending on what is available or good for you in the way that you're working or how you like to possibly introduce yourself to a room of online strangers, which is kind of basically what open source is.

And so for me, I often first focus on getting involved by attending a meeting or lurking in a channel. That is the first, first step of what you should be doing. And then in those places is where you're going to find the links to the different resources that we have about how contributing actually takes place.

You will often get folks that will come in and say, hi, team, I am so-and-so and I'm so happy to be hopefully contributing to Kubernetes soon. Please guide me and let me know what you want me to do. We love the motivation, but we want you to have a bit more initiative in finding those resources yourself, because we have placed them in areas that are very findable in all of the bookmark locations, in our channels, in our areas, on our website.

They are constantly mentioned and referred to in our meetings and in our agenda documentation. And so attending a meeting or going into a channel to then be exposed to this information is step one, in my opinion, every time as your first foray in contribution.

But then on top of that, how you then get involved. So Kat had already mentioned PR wrangling, which is great. All of us approvers in SIG docs do that on a weekly basis, and we can always have one or two shadowers that will help out. Issue wrangling is another place, and specifically folks can get in contact with me. That is kind of my job in the SIG to wrangle the wranglers on the issue side.

And then the other area to be able to start getting involved on the SIG docs, and we also have the blog. We have a couple of blogs, actually, Kaslin. You folks in ContribEx have a blog that you own, and then SIG docs also has a blog. And I mention this because it also coincides specifically with a lot of work that happens on the release team side with the release communications and feature blogs that happen that come out every release. We want blog reviewers. We want blog editors.

KASLIN FIELDS: Blog reviewers, please. Please, please, please.

NATALI VLATKO: If you want to write a blog, that's great. We have a lot of blog writers, I will say. But what we don't have is writing buddies to team up with those writers to help them out. And that's actually something that we've recently introduced as a bit of a kind of like a rule of when you submit a blog. Also, when you submit a feature blog as part of a Kubernetes release as well. Where we team you up with a writing buddy, who is often an author of a different blog, and you review each other's work so that you're helping out with the workload on the SIG docs blog team, which is also quite small and would love some more help.

KASLIN FIELDS: I love that approach to it of pairing up folks who are writing blogs to review each other's work. I've said before and will say again, writing blogs, there's a lot of glory in that. You get your name on something. It's public. People love writing blogs. Reviewing blogs is one of those less visible things that is so, so important and so valued by the community. So we would really love to have folks who are willing to review blogs.

Even if you're not-- we've had folks come in who are not deeply technical, who do not maybe understand Kubernetes super deeply, but they can review English language writing. And so that can be super helpful. And just getting different people's perspectives on how you would word something or if something comes across clearly.

If you are someone who maybe English is not your first language, seeing if that writing makes sense to you. You are part of the target audience, so your review is still really, really valuable. So I want to highlight the blog review opportunity for folks. We would love to have more folks doing that.

NATALI VLATKO: We actually also specifically have a style guide. And so even for folks who are not really sure how certain technical code blocks or technical aspects of the blog should actually be formatted, our entire style guide actually addresses this. So learning our style guide, you're going to be able to review any single blog that comes in through the blog team of either ContribEx or SIG docs. So we can definitely also in the specific notes for this podcast share the style guide as well.

KASLIN FIELDS: Absolutely. And we bring that up in comms meetings a lot. We have a lot of folks who are interested in writing blogs, and we always encourage them to also review blogs and point them to SIG docs's resources on learning how to do that.

And I love how you framed it of your first thing should be just joining a meeting, lurking in a Slack channel. When we do new contributor orientation, I always say those things are contributions. You don't usually think about it, but anything you do to support the community is a contribution. That includes sitting around and just listening and trying to learn. That's one of the most valuable things that you can do to be part of the community.

And then picking up something like PR wrangling or starting to dive into a specific area is kind of day two. And then I would say release team has one of the coolest, I would say, day three options. We can discuss why I think that. But Kat, do you want to talk a little bit about release team and the opportunities available there?

KAT COSGROVE: I am so happy to. So first of all, I agree that the release team is more of a day three thing. While we very much do have released team members who have never contributed to Kubernetes before or never contributed to open source before, it is the minority number of team members on any given release, and it is harder to get selected that way.

So the Kubernetes release cycle runs for roughly 14 weeks. It kind of varies depending on which release of the year it is. We do this three times a year. And it is made up of four subteams led by a release lead and their shadows. The subteams you can apply for are release signal, which is the merged CI signal and release signal teams or bug triage teams. The docs subteam, the comms subteam, and the enhancement subteam. All of those teams are fairly technical. The most technical of them is release signal.

None of them require an intimate understanding of the internal workings of Kubernetes, which is why this is a good day three thing. It is helpful if you understand the lay of the land, you know what a SIG is, you know what a working group is, you know what a subproject is, you understand how GitHub works and how we're organized. That is helpful, not a hard requirement. We do onboard all of the shadows at the beginning of the cycle and cover those things.

But it is a wide open application process about two weeks before the beginning of any given cycle. So if you are contributing a little bit in other areas of the project or not, you are welcome to apply for the next release cycle. If you don't get in, which is likely, unfortunately. We get around 2, 250 applicants per release cycle for about 20 spots, maybe 25.

KASLIN FIELDS: It's grown so much.

KAT COSGROVE: Yeah, it's huge and it's competitive. That does not mean that you're bad. It doesn't mean your application sucks. It doesn't mean that we don't want you. It just means that we are blessed with an abundance of very, very good choices in the release team. And you can still participate. Not being on the team does not mean that you can't help out with a release.

All of our meetings are public. You can come to those meetings. We have the regular release team meeting on Wednesdays. And then towards the end of the cycle, we start doing burndown meetings daily. The Monday and Friday ones are on Zoom as regular. The Tuesday and Thursday ones are usually async on Slack.

But there will still be stuff you can do, because we still never have enough people reviewing docs or blogs. Even with a fully staffed team of shadows, it's not enough people to handle the huge influx of documentation at the docs ready for review, freeze deadline, or the feature blog deadline. We still need more than those four people and whoever we can pull together from SIG docs.

So as a not team member, you are still welcome to help us review blogs and documentation. We can't include you in the roster. We can't assign work to you the way that we do shadows. But if you just hang around in the website repo, I promise you there will be so much to do and you are welcome to message SIG docs or me with my SIG release hat on for some direction figuring out where to go.

But yeah, it's a very popular and high visibility way to get org membership and start contributing to Kubernetes, but it is like drinking from a fire hose. You will be busy from day one for 14 weeks. So it is not the best choice, in my opinion, for a first contribution. It was my second contribution to Kubernetes.

KASLIN FIELDS: And this is something that over 200 people applied to. And if you get it, you need to be focused on it.

KAT COSGROVE: Oh yeah. Lock in.

KASLIN FIELDS: Yeah. Kat implemented some really important changes to the release shadow program that we talked about in your interview with Uwubernetes, which I was going to try to check before this. I think it was our highest listened episode since Abdel and I took over this podcast. So people love Uwubernetes.

KAT COSGROVE: I'm so glad people love Uwubernetes, but it is end of life now, and you have got to upgrade, please. Unless you're on an LTS thing with Microsoft or whatever, it's end of life. Please upgrade off of Uwubernetes.

KASLIN FIELDS: And so people love the structure of the release team. But Uwubernetes, Natali.

NATALI VLATKO: I was about to say, I wonder why people like it. Is there a correlation between anime and tech somehow? I'm not sure. I often like to semi brag that I work on every release, because I'm part of SIG docs and naturally help the release docs and release comms teams every cycle as my role of co-chair of SIG docs and reviewer of PRs and blogs.

And so I think a lot of folks are able to possibly claim that, too, if they want to maybe jump in and help out, because they are interested in doing that, or haven't even been able to get a spot on the release team. Yeah, myself, well, Kat obviously can brag about it, given she's the sub project lead. But yeah, I work on every release, a small part of it. But still, the reviewing part is just as important and we always need more folks.

KAT COSGROVE: Yeah, I think no two SIGs have a more symbiotic relationship than release and docs. Release and architecture are obviously heavily intertwined too, but we don't actually interact with them much throughout the cycle aside from PRR freeze, and they kind of handle that entirely on their own. SIG docs and SIG release have to work.

KASLIN FIELDS: But docs has its own freeze.

KAT COSGROVE: Docs has its own freeze that was not a hard freeze before I took over, and now it is, because I got tired of SIG docs having to bear the brunt of people not respecting soft deadlines for docs within the release. So it is a hard deadline with an exception process, just like code freeze now.

NATALI VLATKO: And which we have tooling to help out to get some of your docs first time provisioned and put together, and then the human can come in and review that after using this AI tool. That's the part that some of this tooling is possibly making easy.

One other aspect I wanted to mention about the AIness of Kubernetes that is often happening that doesn't get talked about a lot, but it's actually the use of AI note takers in meetings.

KASLIN FIELDS: Oh boy, I have opinions.

NATALI VLATKO: Yes. That the fact that all of us reacted in a certain way, either with sounds or body language, as I said that. We had a huge influx when they became a thing, and it's dropped off a little now. But I want to tell everyone out there listening that if you're not able to attend a meeting, I will kick out your AI note taking bot, because we don't want it there.

KASLIN FIELDS: Same.

KAT COSGROVE: We report them to Zoom every time. Get out of here with that. The fireflies.ai thing? Stop that. That's surveillance state behavior. I don't like it. We're not sending our robot assistants to open source meetings for us.

NATALI VLATKO: Especially because, and people may not know this, but SIG meetings are recorded, or they should be by the folks moderating them, and they're uploaded to YouTube playlists, and they're available to watch later. So they are public meetings. And if you're not able to attend, you can definitely watch over them later. You're able to read the agenda and the notes that are public for them too.

But AI note taking in meetings is definitely something that's a no-no. All of us different SIG leads chat in our own chairs and tech leads channel. We tell each other about the new read.ai tooling or things like that that are coming in that we now need to kick and block. So heads up, don't send them, please.

KASLIN FIELDS: Yeah, we talk about that a lot in contributor experience, because we own Zoom and a lot of how Kubernetes does its meetings and things. So we're working on some things there. Join some SIG ContribEx meetings if you want to hear all the gory details about what we're thinking on that. But a good point there is that all the meetings are on YouTube.

Feel free to use your AI to summarize the recording on YouTube and to review that yourself. But then once you've reviewed it and you're like, wow, I had something to say about that, join the meeting to say it yourself.

KAT COSGROVE: Yeah, definitely.

NATALI VLATKO: It's also one of those things that can always be chatted about async as well. I think folks, sometimes we talk about attending meetings and coming into the channels too. We know that we are not going to always have a perfect timed meeting for everyone that's available. So we work as async, obviously, as possible too. You can always start discussions in our channels as well based on what you've heard in a meeting or watched in a recording after.

Another place you can possibly contribute that likely needs more folks. Is a YouTube video not being uploaded quick enough? Maybe you can join the YouTube admins as part of ContribEx to help out with that effort.

KASLIN FIELDS: Yeah, join us. I say us, but I mostly leave that work to the rest of the YouTube admin team. One more thing that I want to come back to on AI in Kubernetes. I've found that when folks join Kubernetes these days, one question that I've gotten quite a bit that I find really funny is, are we allowed to use AI here or is Kubernetes too old for that kind of thing or too bureaucratic for that kind of thing?

And I hope we've answered a lot of that and you can see that you can use AI. We have guidelines around how we're using it. There are some blockages that we're working on with using AI tools, because we have a very large project. We've got to think about privacy and security of everybody. So there's that. But follow the guidelines and things will be OK.

But then as folks get into that day two, whether they're using AI tools or not, folks start asking me, my problem was that the reviewer didn't get to my PR in time.

KAT COSGROVE: Oh, absolutely not.

KASLIN FIELDS: How are we going to use AI to improve the work of maintainers?

KAT COSGROVE: Absolutely not. No. No, so here's a thing that I would love for new contributors to open source to really internalize. The overwhelming majority of us are working for free in our spare time for the love of the game. Natali and I are very lucky and very unique in that this is part of our jobs. That's not true for most maintainers. It's certainly not true for most approvers or contributors. We're outliers.

So we can afford to respond relatively quickly. The average contributor cannot. And nobody gets to demand that we work on anybody's time than our own. That applies to a random individual contributor as much as it applies to literally Google trying to exercise some strength to get us to move at a certain speed.

So no, we will get to a review in order it was received or in the order that we feel like it. You are not going to demand that we move any faster than that. It would be great if we had like some kind of tooling that allowed us to work more efficiently, like some kind of smart AI powered pre-review linter that does some of the regular things that we all check every time that feel a little bit like toil. But the human labor involved in reviewing a PR is always going to happen on our own time scale and nobody else's.

I'm going to only speak for myself on this and not the rest of the contributors or Natali. But personally, if you hound me, if you send me a bunch of kind of rude DMs trying to get me to review something more quickly, I am going to wait as long as I can to review that out of spite. Is that mean? Maybe. But so is sending me like six or seven DMs in a row because you tagged me in a PR and I didn't respond within 30 minutes. You know? I don't love that. It feels disrespectful. It's super rude.

KASLIN FIELDS: I think there's a good reflection of the AI policy there. And number one, respect for humans. [LAUGHS] And number two, the human is still responsible. That's true whether you are submitting a PR or whether you are a reviewer responsible for approving a PR to go into the project. Maintainers take their responsibilities very, very seriously. And so they will put some human effort and thought into the work that they do, whether they're using AI tools or not. So that's very important to consider.

That said, we are, like you said, interested in AI tools for maintainers that can help improve their processes. But when we talk about it as maintainers, we do take that kind of human centric view of the AI is a tool that's going to improve our workflows, not replace the maintainer in the process, which I think is very important for folks to hear. Natali, do you want to add anything about AI and maintainership?

NATALI VLATKO: I would like to add that you had made a comment before that actually kind of made me laugh, Kaslin, about folks asking, is Kubernetes too old to have AI assistance in terms of contributions or so on? And I was just sitting there thinking, wow, are we--

KAT COSGROVE: We're geriatric.

NATALI VLATKO: No, no. We are the millennials of open source, because Linux were the boomers. That's the boomer project.

KAT COSGROVE: I work on MySQL, for God's sake. MySQL is old.

NATALI VLATKO: And you've got Kubernetes is the millennial open source project, which it totally lines up with this idea of are we too old? And we're all sitting there freaking out. Like, oh my God. Existential crisis. Is Kubernetes too old to get on board with new state of the art tech?

And the answer is no and, asterisk, we have different ways that we need to work with some of this stuff. And part of it is because we are so large and so adopted, it would actually be super not great for us as maintainers and stewards of this project to bring in tooling that we didn't know was absolutely not only right for us as maintainers in terms of reviews and how contributions come in, but also didn't fit well along with the guidelines of the foundation that we're under, which is exactly why I'm OK always saying no on the AI signed by x AI tool because of the legality of that. And we're doing our due diligence and our open source duty for that as well.

So I think that's another part to keep in consideration, that there is always going to be interesting, fun, new state of the art technology in any of those boxes that folks that want to assess, that they're always going to wonder, is Kubernetes getting on top of this or not? And the bottom line is that we are also a community of maintainers and people who get together and help to make those decisions.

There's no Mr. Kubernetes out there making all the calls. It's a group of us who, as Kat mentioned, are largely doing this voluntarily. Even though I can do some of this for my job, a lot of this is after hours for me as well, because my job is also very busy. And so we are trying to also keep up with these different kinds of technological trends and understand what's good for our project and make good decisions that will also help advance its maintainability and security.

KAT COSGROVE: Yeah, we're not geriatric. We're preventative Botox age, I think.

NATALI VLATKO: You heard it here first, folks.

[LAUGHTER]

KAT COSGROVE: I say that as somebody with fresh Botox. I just got redone like a week ago.

KASLIN FIELDS: Well, I think that is a beautiful point to end on, that Kubernetes is perhaps not geriatric, but just millennial.

NATALI VLATKO: Botox age.

KASLIN FIELDS: Botox. And so we are looking at adopting AI tools. We have a variety of ways that we're adopting AI tools now. And if you would like to join us, I hope that you've learned something about the different ways that you can contribute and get involved in the community. Thank you, Kat and Natali, for joining me today.

KAT COSGROVE: You're welcome.

NATALI VLATKO: Thank you.

KASLIN FIELDS: I'm so excited that we got to do this interview with Kat and Natali. As a Kubernetes maintainer myself, I am having interesting experiences with AI in the open source ecosystem. But I am a lead of a SIG that is about contributor experience. I lead the communications subproject. A lot of the things that I do, we do some things in GitHub, we also do a lot of things in other tools. And docs also, they get a lot of AI contributions for docs. So it's not always code.

But talking with other maintainers about their experiences, whether it's code or docs or whatever part of the project, is just so interesting. So I love that angle, and I think a lot of people think about running AI on Kubernetes. But how AI is affecting Kubernetes's open source project is also really interesting.

ABDEL SGHIOUAR: Yeah, I mean, I didn't listen to the episode, but I am assuming that as any other open source project, they are probably struggling a little bit with the amount of AI slop they receive as maintainers.

KASLIN FIELDS: Yeah, I know you hadn't had a chance yet, but I wanted to talk with you about this anyway, because I just love it so much.

ABDEL SGHIOUAR: Well, the topic is very interesting.

KASLIN FIELDS: It's so good.

ABDEL SGHIOUAR: The impact of AI on open source in general is felt across the industry. But Kubernetes specifically is a very large project, right?

KASLIN FIELDS: Yes. And interestingly, very recently I saw a comment from Linus Torvalds about AI usage in Linux. And their opinion, his opinion at least, kind of mimics what I'm seeing in the Kubernetes community is that we see AI as a tool for engineers to use, and it can be a very useful one. And so we want to enable engineers contributing, anyone, contributing to Kubernetes to be able to use this tool.

But we do still have to think about guidelines around licensing. That was one of the major things we talked about in the episode is that there's a contributor license agreement and the AI can't sign it. So we can't technically have people saying that something was co-authored by an agent, because the agent isn't allowed to be an author legally.

ABDEL SGHIOUAR: Yeah, that's an interesting one.

KASLIN FIELDS: So we have things like that. And also we're a global project. We have folks all around the world. So privacy laws vary and security laws vary. So adopting new AI tools.

ABDEL SGHIOUAR: Cost.

KASLIN FIELDS: Yeah. We have to think about these things.

ABDEL SGHIOUAR: Yeah. I think I saw that at the post that you mentioned by Linus Torvalds. And I just remembered a few months ago, he also published something around the line that he wrote a tool. He wrote like an app in Python using AI. And the internet went absolutely bananas because they were like, see, even Linus Torvalds is using AI.

KASLIN FIELDS: I see that happening.

ABDEL SGHIOUAR: And he had to come back and say like, no, no, no, calm down. I'm not using AI to write the Linux kernel. This is not C++. This is just an extra nice to have. But yeah, I agree. I think a lot of people are hyping up AI, but if you think about it as a tool that can help you do stuff that you don't want to do, then it could be a very useful tool, considering we sort out the details.

KASLIN FIELDS: And the point in this AI usage policy that we're talking about, and what I'm hearing from other open source maintainers and other projects sometimes too, is the responsibility stays with the human.

ABDEL SGHIOUAR: Yeah, of course. Yeah.

KASLIN FIELDS: You hear that all across the industry, that it's really important for the human to understand what the code is. Another thing we talk about in the episode, and which I wanted to mention, about AI slop, contributions to projects. When I asked other maintainers about this, something they said is that they're not being so overwhelmed with new AI PRs, because they were already overwhelmed. So they're just as overwhelmed as they were before.

But they are being frustrated and slowed down by AI created comments on PRs, because they will just post pages and pages and it's unparsable, unreadable, and it can make things up. And they just can't trust the comment to help them understand the code that's being proposed. So they really want to have conversations with humans in the comments, even if AI was used to help write the code.

ABDEL SGHIOUAR: Yeah, I can see how this can become a problem. I think it's the same thing happening in the security space. When I was chatting with Michelle [? Shubrika ?] to bring her to the podcast for another episode.

KASLIN FIELDS: Look forward to that.

ABDEL SGHIOUAR: Yeah, we were talking about AI slop for security people. If you use AI to find vulnerabilities and report them, then at the end of the day, there is still a human being that has to go and read through all this slop to identify is this actually real, or is this just a hypothetical thing that an AI hallucinated? But we also see it, I think, in our day work. I'm working on Antigravity. A lot of times, it will just invent function names that doesn't exist.

KASLIN FIELDS: Yeah. And you can do I as a judge to try to make sure that you're not-- your output, your final thing is not hallucinating and all of that. But once again, the AI is also fallible. And the AI is not responsible is what it comes down to.

ABDEL SGHIOUAR: I know a developer, I was talking to one of the founders of the Devoxx, the conference. And Stefan, he was telling me that he had three different subscription, three different models. So he will have one model generate code, another model review the code that the first model generated, and then another model reviewing the code.

And then I was just asking him, how much are you paying per month for all these subscriptions? Must be insane, right? Because it's like three different providers. It's like Google, Claude, and whatever. If you are using the Mac subscription of all of them, that must be at least $1,000 a month just in subscription fees.

KASLIN FIELDS: Gotta be intense.

ABDEL SGHIOUAR: And I guess also that for a project like Kubernetes, they have to think also about this in the sense of like cost, right? Because if you build AI to the pipelines, how much tokens is this thing going to consume?

KASLIN FIELDS: Also an interesting thing that I didn't mention in this interview is that I've heard from other Kubernetes maintainers a few months ago, I don't know if it's still true right now, but when we were talking about an increase in AI contributions and stuff, we also saw an increase in CICD pipeline attacks.

ABDEL SGHIOUAR: Oh yeah, of course.

KASLIN FIELDS: Because people will attack our CICD pipeline where you create a PR in GitHub, and we have automation and bots that pick up that PR and run tests on them. So we have a flag that says OK to test, and we have to have a reviewer say that the code is OK to run through our testing infrastructure, because people will attack our CICD pipeline to try to get the compute that we're running for our CICD and use it for their own purposes.

ABDEL SGHIOUAR: Yeah, or just to financially hurt you by making you consume resources.

KASLIN FIELDS: Because, boy, does the Kubernetes Project use a lot of money for an open source project that doesn't directly bring in any money.

ABDEL SGHIOUAR: Oh yes. I mean, I think it's, what, $3 million from Google, $3 million from Amazon, and I think Microsoft throws a couple of millions as well.

KASLIN FIELDS: It's a lot. I feel like it was $100,000, not million, but I could be wrong on that. I think we have a blog post about it. Definitely check it out, though.

ABDEL SGHIOUAR: I might be wrong, but I'm quite confident it's in the millions.

KASLIN FIELDS: It might be.

ABDEL SGHIOUAR: Because you remember a couple of years ago when they had that issue, they were running out of credits because they were serving too much artifacts?

KASLIN FIELDS: Actually, it happens every year, but it was a big deal that year.

ABDEL SGHIOUAR: That year was a big deal. Yeah. I remember it was billions. It's millions of dollars they get in cloud credits to be able to run the entire test infrastructure and stuff.

KASLIN FIELDS: I don't know if we've had someone on from SIG architecture to talk about how much it costs to run Kubernetes architecture, but we should think about that.

ABDEL SGHIOUAR: Actually, it would be interesting to bring somebody to talk about the CI thing. They have a CI tool that use--

KASLIN FIELDS: Prow. It's technically within my SIG.

ABDEL SGHIOUAR: Prow, yeah, correct.

KASLIN FIELDS: We have a few different tools, but Prow is our main bot. Prow bot.

ABDEL SGHIOUAR: Yeah. It's interesting, this whole thing of AI in open source is an interesting topic.

KASLIN FIELDS: We hope you enjoyed listening to the episode today. Let us know if you want more.

ABDEL SGHIOUAR: Thank you for taking the time to do the episode.

KASLIN FIELDS: Thanks for being here, Abdel.

ABDEL SGHIOUAR: Thank you.

KASLIN FIELDS: See you next time.

ABDEL SGHIOUAR: See you.

KASLIN FIELDS: That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @KubernetesPod or reach us by email <kubernetespodcast@google.com>.

You can also check out the website at kubernetespodcast.com, where you'll find transcripts, show notes, and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]