KASLIN FIELDS: Hello, and welcome to the "Kubernetes Podcast from Google." I'm your host, Kaslin Fields.

ABDEL SGHIOUAR: And I'm Abdel Sghiouar.

[MUSIC PLAYING]

Welcome to the second episode of the four part special series for the Kubernetes 10 years anniversary. In this episode, we spoke to two very influential people in Kubernetes history, Kelsey Hightower and Tim Hockin. Both have been involved with the project since its inception, and both had and continue to have impact on the project and the community.

KASLIN FIELDS: Now, before we get to the news, happy 10 year anniversary to Kubernetes!

ABDEL SGHIOUAR: Yay!

KASLIN FIELDS: Yay!

ABDEL SGHIOUAR: What are your plans, Kaslin?

KASLIN FIELDS: If everything goes as planned, this should come out on the anniversary, June 6, 2024. [LAUGHS]

ABDEL SGHIOUAR: Hope for the best, plan for the worst.

KASLIN FIELDS: Yeah, and assuming that we worked this out right, then as this is releasing, I'm probably at the KuberTENes birthday bash in Mountain View, California.

ABDEL SGHIOUAR: And as this is releasing, I am on my way to Bergen in Norway for a small meetup, slightly less elaborate than the one you're attending. But small community, passionate people I'm hoping, and I'm hoping a cake. We'll see.

KASLIN FIELDS: And you're doing two, right? You're doing two birthday meetups. One today.

ABDEL SGHIOUAR: So one today, and one next week on Monday 10, in Aarhus, Denmark.

KASLIN FIELDS: So nice, and there are so many parties.

ABDEL SGHIOUAR: I'm going to be with Casper.

KASLIN FIELDS: You're going to be with Casper Nielsen?

ABDEL SGHIOUAR: Yeah, because he lives in Aarhus. That's the city where he lives.

KASLIN FIELDS: Nice. Casper is awesome and such an influential member of the community, so that's exciting.

ABDEL SGHIOUAR: Yeah, I'm looking forward to it. I know that one will have a cake for sure.

KASLIN FIELDS: Nice. I wonder how many of the 50-some-odd events that are happening all across the world have cake. [LAUGHS]

ABDEL SGHIOUAR: That's a very good question. So if you have an opportunity to check any of them, there is a website. We'll leave a link in the show notes. If you find an event near you and you go, take pictures. If there is a cake, definitely take a picture and tag our podcast on social media, @KubernetesPod on Twitter, and the "Kubernetes Podcast by Google" on LinkedIn.

KASLIN FIELDS: And this is hopefully releasing on June 6, but the parties are happening all week. [LAUGHS] They're kind of spread out. The first one started, I think, on June 1. So there are some that have already happened, but there may be some near you that still haven't happened yet. So definitely check out the website if you're like within the June 6 week-ish.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: There might still be some that you could go to. So Kubernetes is 10 years old. Yay!

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: Which, by the way, means that it is not allowed to have a Twitter account, which I learned the hard way last week. [LAUGHS]

ABDEL SGHIOUAR: I heard that. What happened?

KASLIN FIELDS: Yeah, so I'm a co-lead of the subproject that runs comms for the open-source project, and so owning our Twitter accounts and various other social media accounts is part of the job. And we've recently got ownership of the KubernetesIO account, which we are so excited about because that used to be just owned and run by the CNCF. And now, the community gets a say in what-- well, a bigger say, in what goes on there.

Before it was a little bit harder to get to the CNCF folks if you wanted to put something on there, and basically, we didn't do it. But now, it's run by the community, which is exciting. But I messed that up. [LAUGHS] And I got the account locked for being under age.

Luckily it was resolved in just two days, even though it was a holiday weekend in the US. So the account is back up now, and I still need to go in and set that birthday to something that is not exactly the birthday of the project.

ABDEL SGHIOUAR: 6 of June, 2014.

KASLIN FIELDS: So that it will get those balloons. So if you see balloons on Twitter, know that those were hard won.

ABDEL SGHIOUAR: Yes, yes, yes. I remember I was with Mario in Vilnius, Lithuania. And when that happened, we were laughing. That was funny. [LAUGHS]

KASLIN FIELDS: Yeah, it was pretty funny, I must admit.

ABDEL SGHIOUAR: Stuff happens.

KASLIN FIELDS: If you're going to make a mistake, at least hopefully it'll be funny.

ABDEL SGHIOUAR: Yes, that's a good strategy.

KASLIN FIELDS: And so with that, let's get to the news.

[MUSIC PLAYING]

The Kubernetes project introduced Hydrophone, a simple single binary to execute conformance tests using the official conformance image released by the Kubernetes maintainers. Conformance testing is a way to validate if a Kubernetes environment meets the community standards. Previously, this required running scripts and third party tools.

Hydrophone aims to streamline this by providing a single tool that runs as a pod inside the conformance namespace and outputs test results. You can choose to run end-to-end or single tests, or anything in between. Check out more on the Kubernetes IO blog linked in the show notes.

ABDEL SGHIOUAR: At Microsoft Build 2024, the company announced AKS Automatic, a fully managed version of Azure Kubernetes services. The new offering exposes the Kubernetes API, while managing the life cycle of cluster resources automatically.

KASLIN FIELDS: The Certified Kubernetes Security Specialist, CKS for short, certification is getting an update. Starting September 12, 2024, new domain knowledge will be added and the percent weight for several of the domains will be changed. This means if you're planning to take the exam after September 12, you should consider looking at those upcoming changes.

ABDEL SGHIOUAR: KubeCon and CloudNativeCon North America Call For Proposal is closing on June 9, 2024. The event will be held in Salt Lake City, Utah, on the 12th to the 15th of November. So hurry up, you only have few days left to send your proposals.

KASLIN FIELDS: The Call for Proposals for KubeCon North America's co-located events is also open now and closes July 14. ColoEvents at KubeCon in 2024 include Platform Engineering Day, Cloud Native and Kubernetes AI Day, Data on Kubernetes Day, Observability Day, and many more. Check out the links in the show notes for the list of co-located events and more info about the Colo CFP. And that's the news.

On this, the 10 year anniversary of Kubernetes, we are so excited today to have with us Tim Hockin. Tim is a distinguished engineer at Google, where he works on Kubernetes and Google Kubernetes Engine. He's been part of the Kubernetes project since before it was publicly announced, and mostly pays attention to topics such as APIs, networking, nodes, resource management, and multi-cluster. He occasionally dabbles in API machinery.

Before Kubernetes, he worked on a number of infrastructure projects at Google, including the Borg and Omega projects, as well as efforts to manage Google's fleet of servers, all the way down to the bios and hardware. Welcome to the show, Tim.

TIM HOCKIN: Thanks for having me. Good to see you.

KASLIN FIELDS: We're so excited to have you on. It's been quite the journey, hasn't it? You've been with Kubernetes all the way since the beginning. [LAUGHS] Before the beginning.

TIM HOCKIN: Yeah, it has been quite a journey. 10 years, and still going. I'm excited to say that it is moving. It is as interesting as it was 10 years ago.

KASLIN FIELDS: That is an interesting comment. [LAUGHS] Are you prepared to defend that comment, that Kubernetes is as interesting as it was 10 years ago? I feel like a lot of people would argue.

TIM HOCKIN: It's a very subjective statement, isn't it? I mean, it's interesting to me as it was 10 years ago. I'm having a ton of fun playing with systems that affect real people and getting to solve hard problems and deal with things that are new to me, which is always fun.

KASLIN FIELDS: And that means a lot coming from someone who is actively working on the project and has been since its early days. So even if as a user, Kubernetes might have gotten boring for you, it's still exciting to the contributors.

TIM HOCKIN: Well, I think there's a difference between exciting and interesting. Let's be careful.

KASLIN FIELDS: Fair point. Fair point. So let's kick all of this off with the beginnings of your involvement in the project. How did you get here? How did all of this start for you?

TIM HOCKIN: Well, you mentioned that I was working on Borg and Omega before. I was working on our internal infrastructure at Google and when this Docker thing came to light, it happened right about the same time that we were thinking about how would we publish more of what we do around cluster management and container management.

In fact, we were working on a project called Let Me Contain That For You at the time, which was also open-sourced, sort of within months of Docker open-sourcing and a very different approach to containerization. And I couldn't not get involved in this. It looked so exciting and I cut my teeth on open-source programming.

I learned to program by hacking on Linux and Red Hat. And so the chance to go back to working on open source software and working on something in the public eye was far too tempting for me to walk away from.

KASLIN FIELDS: Does this Let Us Containerize That For You-- what was it called? Does that still exist?

TIM HOCKIN: Let Me Contain That For You.

KASLIN FIELDS: Let Me Contain That For You. I like that. [LAUGHS]

TIM HOCKIN: I think you can find the code up on GitHub still, but it's been archived. It hasn't really been maintained. It was a joke. A while back, there was a site, Let Me Google That For You.

KASLIN FIELDS: Yup.

TIM HOCKIN: And when your parents would ask you a question, you would just type it in to Let Me Google That For You and send them the link. And then it would show them. They could just type it into Google.

So Let Me Contain That For You was first and foremost, evidence that you should not let me name things. But it was a fun way to take literally the code we were using inside Google, inside Borg, refactor it into a way that we could publish to the world and use it as a starting point for conversations. It wasn't intended to be a competitor to Docker. It wasn't really even intended to be something that everybody would use on their servers or desktops every day.

It was more of a proof point for conversations with kernel teams, with the Linux kernel community, where we could say, we need this feature because-- and then we could show real working examples of, if you have a VM with the standard kernel, it behaves this way, and a VM with our patched kernel, it behaves that way. And that's why we need these sorts of features. It just happened to come a few months after Docker and, well, the rest is history, isn't it?

KASLIN FIELDS: Yeah. [LAUGHS] And in those early days when I was trying to explain containers to people, containers weren't a very well known concept when Docker first came out. And then, the way that I like to explain it to people is that the usability of Docker is what really catapulted containers into their current common usage.

But before that, something I always took joy in explaining to people, is that containers existed long before Docker. It's just that they were kind of difficult to use. You had to understand those underlying kernel technologies a lot better to be able to use them and put them together in interesting ways.

TIM HOCKIN: Yeah, for sure. Linux still doesn't have a concept called container. Like that's not a proper noun in the Linux kernel or anywhere. It's a grouping of a bunch of different APIs that you can assemble the concept of a container with. But we didn't invent it. It existed in other operating systems before.

KASLIN FIELDS: I feel a parallel here.

TIM HOCKIN: Yeah, we all stand on the shoulders of giants, right?

KASLIN FIELDS: Mhm. I feel a parallel here between containers not existing as an object in the Linux kernel and sidecar containers not existing as a concept in Kubernetes. We've had some conversations about that on the show.

TIM HOCKIN: Yeah, the analogy is not lost on me.

KASLIN FIELDS: [LAUGHS] You were also deeply involved with that cap, right? To get sidecar functionality in.

TIM HOCKIN: Yeah, that was a really challenging one to figure out how to fit something so large into the working framework without breaking anybody.

KASLIN FIELDS: Yup. I love that story. And I'll put some links in the show notes for folks who want to go dive into that, but we'll go on past that for now. We're talking about the 10 year anniversary of Kubernetes. So tell me a little bit about how you think Kubernetes has changed the world of infrastructure automation.

TIM HOCKIN: I think, again, things we didn't invent. We didn't invent declarative, but I think we had a play in making it a more popular way of doing things. I think it makes it very easy for people to think about what they want, and it frees them from having to think about how do I get there?

And all the weird little corner cases of imperative sorts of infrastructure. So, you know, again, we didn't invent it, but we'll take a little bit of the credit. I think that has really shifted how people think about what they want from their infrastructure. And at the same time that Kubernetes was happening, Terraform was being born.

And Terraform going after sort of traditional infrastructure machines and networks and those sorts of things and Kubernetes is coming after workloads are different approaches to the same problem. And I appreciate that we came up with some of the same ideas.

KASLIN FIELDS: I definitely think the declarative model has kind of taken on a life of its own. It was a term that I don't think many people had heard much or understood, definitely in context, when Kubernetes first started using it. But now, it's very common. Like you said, with Terraform, Kubernetes, the concept of GitOps, there are all sorts of implementation chains that we have now of, you should be able to define what your infrastructure looks like, and then it should just happen.

TIM HOCKIN: Yeah, you know, even coming into declarative, when I first started to become aware of the concept, was weird for me. Coming from a programming background, I was used to saying do A, then do B, then do C. And if B failed, then return early and don't do C.

And declarative, I encountered many more of these APIs that were like, ensure that X is true. Not make X true, but ensure that it is true and stays true, which was a very different way of thinking about things and really shifted how I looked at systems design. And obviously, it was very fundamental to how Kubernetes is built.

KASLIN FIELDS: And over the last decade, Kubernetes has been through a lot of changes and you've been right there along with it. So how do you think Kubernetes itself has changed over the last decade?

TIM HOCKIN: I think we, the maintainers and the developers of Kubernetes, have matured a lot. We have a much more sophisticated sense of what people are doing with stuff. Once upon a time, we thought that it was going to be OK if we would just deprecate APIs and get rid of stuff. And we learned pretty quickly that was just not going to fly.

It was too much of a burden on too many people, and the cost of success here is that everything we do is used by somebody. And so we can't just rip those things out as much as we would like to. And so even still, if you look at some of the documentation, we talk about the deprecation policies and the ability to get rid of things, but we've made a pretty public statement that we're never going to actually invoke those policies.

We should probably just go back and change them. And it's sometimes disconcerting to new contributors who step up and say, well, this feature was bad. We should get rid of it. Well, we can't get rid of it. We can't even change defaults, which is sometimes very frustrating.

You mentioned the sidecar thing earlier, we can't go back and change APIs. Those things are out there in the wild and we can't do anything that would put those workloads in jeopardy. And I always try to think of myself as the poor cluster administrator whose job was working fine on Monday when they left work, and when they come in Tuesday morning, it's all broken.

And the only thing that changed was their Kubernetes cluster got upgraded. I'd be a pretty unhappy person if that was me, and I never want to make somebody unhappy like that.

KASLIN FIELDS: Understandable. I feel like a lot of the time when I'm in a conversation around contribution to Kubernetes and you're there, folks will mention something that feels very fundamental to Kubernetes and you're like, eh, we should probably change that. [LAUGHS] It's your experience in the project showing.

TIM HOCKIN: We made a lot of mistakes, like we were all younger versions of ourselves. We all had a younger understanding of what people needed and what they're doing with the system. And so there were lots of mistakes. And I love to point out my own mistakes because it means that we've learned since then. So there's plenty of things that we should totally fix. We should rip things apart and fix the way they work. Just don't break users.

KASLIN FIELDS: And I feel like that's a core tenant of how the Kubernetes community works these days is, I think all over the project, we try to make it OK to understand that we make mistakes. And when we make decisions, we need to have pathways to change them as best we can. But we need to understand also that there are users relying on all of this.

TIM HOCKIN: Absolutely.

KASLIN FIELDS: I feel like we try to make that a thing across the project.

TIM HOCKIN: Yeah, I mean, one of the things that is easy to forget when you're just staring at a pile of software is time-to-market is actually a feature. And sometimes, that's a feature that comes at the cost of other features. And Kubernetes, if it had been a year earlier or a year later, might not have mattered at all. Docker, same thing, right?

If Docker had launched in 2015 instead of 2013 and '14, would it have mattered? I don't know. Maybe, but it would have been different. And we try to be cognizant of that as we're designing APIs and things that are going to go into Kubernetes.

We could navel gaze indefinitely about what the perfect API is, or we could actually ship something, get it in people's hands, let them use it and point out where we made mistakes, and then live with those mistakes and try to fix it.

KASLIN FIELDS: Speaking of things changing and making mistakes and fixing them, I feel like this is an exercise you do publicly a lot. If you had a magic wand to change one thing in Kubernetes, what would it be?

TIM HOCKIN: Whew, only one thing?

KASLIN FIELDS: [LAUGHS] You can do a list. I would be interested.

TIM HOCKIN: I have a long list of things that I would love to see us fix when time machines finally get invented. If I had to pick a single thing-- wow, I've never actually really thought about what the one thing that I would change would be. Let me start with a list and see if that points anything out.

I think some of the concepts are too loosely coupled. Loose coupling is a very strong property of Kubernetes, but there are places where I think we make it harder on ourselves for no real advantage. For example, the relationship between deployments and services is usually pretty well understood. I think we could maybe have made life easier for people if we had made those concepts a little more mutually aware.

I think early on, we didn't pay enough attention to the idea of identity. And so things like service accounts are in some places an afterthought. And I wish maybe we could go back and teach more things like network policies about service accounts. And maybe we still can, but I wish we would have maybe started with that.

I'm on record publicly saying how much I hate the services API, despite the importance of it and what it has done for things. It's a terrible API. And I wouldn't say go back and not do services, but maybe I would have advised myself to think a little harder about how it was going to expand. Yeah, there's a few things you can start with.

KASLIN FIELDS: That was a fantastic lightning talk, by the way. I'll make sure to include a link to that in the show notes.

TIM HOCKIN: Thank you.

KASLIN FIELDS: There was a lightning talk-- was it at KubeCon Chicago?

TIM HOCKIN: Yes.

KASLIN FIELDS: North America 2023, where Tim went into why services is a terrible API, and I had never thought about the way that you put it there. I think kind of the gist of it that I got from it is, services became too many things. It's doing too many jobs at once and none of them particularly well.

TIM HOCKIN: Yeah, and we didn't leave ourselves enough avenues for growth, so that we sort of glued a bunch of concepts together. And then the intersection of those things is cool, but the sort of partial space around that is poorly defined.

KASLIN FIELDS: And I'm sure a lot of folks out there wish that they could become Kubernetes contributors. I feel like a lot of folks are like, someday I'm going to do that. And hearing you talk about some of these problems or bugs perhaps that the Kubernetes project would like to fix, maybe folks are thinking about that a bit. What advice would you have for folks out there listening who might want to get involved with the Kubernetes project?

TIM HOCKIN: I do talk to a lot of people who say they'd like to get involved but it's daunting. It seems like an enormously complicated system to just jump into and start participating. And I understand and I appreciate that. But at the same time, it's just a bunch of code.

And if you understand how programs work, you can pick up any Kubernetes component and start reading the code and you'll understand what's going on. There's not that much magic going on in there. And, I encourage people to just put aside the fear of it and just say, I can do this. This is just a program.

Somebody gave me that advice a long time ago when I was learning about operating systems and I was intimidated by the idea of working on a kernel. And they said, eh, it's just a big C program. You know how to write C programs, just go read the kernel.

And so I did. And you know, it's not exactly true. There are buried bits of magic inside Kubernetes and buried bits of magic inside Linux, but most people don't have to deal with the magic. They just get to deal with the logic that does pretty straightforward things.

You know, the kubelet, for example, it's a complicated piece of code. But at the end of the day, it's a bunch of linear logic that is managing some stuff on the machine. It's not that complicated. So it sounds a little bit trite, but I hope that people who are interested in participating in Kubernetes just come do it.

Just put aside a couple of hours to crack open the code and start reading or fiddling. If you show up with a particular itch you want to scratch, that's usually the best path to success. But if you're looking-- I just want to get involved and I don't know where to start, there's lots of people here who would love to help you get going.

KASLIN FIELDS: And I do think one thing that gets in a lot of folks way when they start contributing is just how interconnected all of the various parts of Kubernetes are. So sometimes you'll pick up something that's labeled good first issue, and you start reading into it and you start creating a solution. And you realize that it's connected to these other things and you have to dive deeper into the project. So something for folks to be aware of.

TIM HOCKIN: Those sorts of things probably shouldn't be labeled good first issues.

KASLIN FIELDS: Yeah.

TIM HOCKIN: I'm guilty of having mislabeled that a few times in the past and I've been chastised, because what is a reasonable issue to me, who has 10 years of experience, is not always a reasonable issue to somebody who's new to the project. We try to be more responsible that way.

What I have found is there's an army of people out there who are dying to help if you just show them how they can help. And that means filing a good bug with reproduction steps and some instructions about where you think the problem might be. And there are people out there who want to fix those bugs.

KASLIN FIELDS: Speaking of good bugs, you had a very popular talk at KubeCon EU in Paris. I believe the title was Kubernetes Maintainers Read Mean Comments, Mean Posts, Mean Tweets, where you and Dims, who we had in another episode during the 10 year anniversary series, read mean comments from GitHub. And I loved how you posed it as, we're going to teach you how to interact with Kubernetes maintainers through counterexamples.

TIM HOCKIN: Yeah, we had this idea for a long time. Dims and I use each other as a vent for frustration sometimes. And when people post these rude comments, we send them back and forth and have a little chuckle about it.

And a couple of years ago, we said, you know, we should do a talk. We should just collect all of these and show them all. And then we said, this would be great. We should save it for the closer to the 10 year anniversary. So we've been putting those aside for quite a long time, and I was happy that we could finally do it.

It wasn't exactly the delivery that I had originally envisioned. Originally, I wanted it to be part of the keynotes, where they would put up like one or two-- like the Jimmy Kimmel read mean tweets thing. Some maintainer would read a mean comment and deadpan to the camera and we would put that up in between the keynotes, and I thought it would be really funny.

It wasn't practical for a lot of reasons, so I'm glad that we could do it as a real talk. And I'm super happy that people enjoyed it. Like they laughed at the right places and that was the most gratifying thing.

KASLIN FIELDS: It is highly enjoyable. I was lucky enough to get to be there in person. We will make sure to include the link to the recording in the show notes. Highly recommend that you check that out if you haven't.

And one more question I wanted to ask you. We talked a little bit in here about the Linux kernel and how a lot of your experience has been in learning how to contribute to the kernel and kind of moving into Kubernetes. And I understand that you still use Linux machines primarily today. So is it the year of the Linux desktop?

TIM HOCKIN: Ha! It is always the year of the Linux desktop. I've had Linux on my desktop almost exclusively since about '98, and before then partially. But you know, I took a little detour back through Windows when I was doing some BIOS level work, because a lot of those tools were Windows-based.

But I've used Linux on the desktop for a very long time. And for the most part, I love it. It works great. I have my gripes with the way certain desktop environments are evolving. I feel like an old dinosaur when I yell at the cloud and-- they're taking away my focus follows mouse.

The one thing I will complain about still is Bluetooth. My Bluetooth headset, I love, but every day I have to reset it at least two or three times because somehow Linux just loses it.

KASLIN FIELDS: And don't worry, folks, he only had one minor issue with his camera when we got started recording this and we only record audio, so it's fine. [LAUGHS] So closing back up with Kubernetes here, on the 10 year anniversary, what would you like to see next for Kubernetes?

TIM HOCKIN: One of the things I think we really need to focus on coming up in Kubernetes is how new hard problems can be solved without changing the core of the system fundamentally. The rise of AI in particular, has really highlighted some of the assumptions that we've made throughout the last 10 years, around what most applications and most workloads look like and the relationship they have with the machine.

And I think AI workloads right now are very, very different than that. And it shows me, very starkly, where we've made unstated assumptions. And so finding ways to make those things work without having to go back through the bowels of the system to change heuristics, which is what we're doing right now, which is, you know, fine, but we can't do that forever. And so we've got to find better ways to make more extensibility.

We put an enormous amount of energy into things like custom resources. They continue to get better. I love the work that the API machinery sig is doing around that. I think there's more. I think we have more opportunities to make extensibility better, to make more of the heuristics that we've built less hardcoded and more self-tuning, and finding ways to let people be successful without coming through the core of the project.

KASLIN FIELDS: And you also had a keynote at KubeCon U-- no, Chicago, wasn't it?-- about the 10 year anniversary of Kubernetes and what's next. And we'll include a link to that in the show notes as well.

One of my favorite things that you mentioned in that was also kind of around the concepts that you were talking about, around the core and its extensibility. But one of my favorite things that you said was that we need to manage the scope of Kubernetes very carefully in the next 10 years, as we head toward 20 years.

TIM HOCKIN: Yeah, 20-- I will be very happy to have the 20 year podcast.

KASLIN FIELDS: Yeah. [LAUGHS]

TIM HOCKIN: But it's true, and we've said this for a long time, but it's as true as it ever was, maybe more. Kubernetes can't do everything and it can't serve every purpose. And we need to find out how we can let people be successful and get out of their way without them having to send us a pull request for everything they want to do, because that's just not going to scale.

And the complexity of the system, there's a budget. And eventually, it is so complex that nobody can keep working with it. And we can't let that happen. So we've got to find ways to streamline, simplify, outsource logic. The recent changes to move the cloud provider logic out of the tree were a huge-- I mean, they took a long time to get there, but they were a huge step forward.

Millions of lines of code were deleted, which means the core project can reclaim some of that complexity budget. New things that we can think about. It didn't make the problems go away, but it did compartmentalize them a little more.

KASLIN FIELDS: Yeah, make it a little bit easier for users to avoid the places where they don't need to know about things, and we can maybe help them down more smooth paths.

TIM HOCKIN: For sure. And you know, ultimately we don't know what we don't know. And if you had asked me five years ago what Kubernetes would be doing with AI and ML, I would have told you, I don't know. It's not interesting. It's too niche or it barely existed, really. It was not on our radar.

And today, it is the top thing that we hear about from users of Kubernetes. How do we do better for those workloads? And so who knows what will be the thing that is on the top of people's minds in five years or 10 years?

We've got to get out of the way so that those people can do what they want to do, or we relegate ourselves to just serving, quote unquote, "traditional workloads," and something else will come along for these other things. And I don't think that's what I want.

KASLIN FIELDS: And we'll have another episode on that later in this series, where we'll dive deeper into how AI/ML workloads have gone from this niche thing to something that the Kubernetes community has a lot more work to do around and specifically, the Kubernetes perspective on that. So we've mentioned it a couple of times that it kind of changes the way that we think about how we do the distributed system aspect of Kubernetes, the whole thing, in some ways.

TIM HOCKIN: Absolutely. It's a totally different world right now than it was just a few years ago. The equations that we usually think about in terms of trade-offs are completely upside down.

KASLIN FIELDS: And we'll dive deeper into that another time. Thank you so much for being with us today, Tim, and we'll make sure to include the links to all of those awesome resources in the show notes. And I look forward to seeing you on the 20 year anniversary of Kubernetes.

TIM HOCKIN: You said it. See you there. Nice to talk to you. Thanks for having me.

KASLIN FIELDS: And I'm very excited today to have with us Kelsey Hightower. Welcome to the show, Kelsey.

KELSEY HIGHTOWER: All right. Happy to be here. I think this might be my first or second time on the podcast.

KASLIN FIELDS: I think you have been on one other time, before the host change. So it's the first time that we've had you on at least. Would you like to give a little introduction to who you are for anyone who might not know?

KELSEY HIGHTOWER: Hey, I'm Kelsey Hightower. You may know me from Kubernetes The Hard Way. Former Googler, now retired, but still deep in the open-source community. So happy to be on the podcast today.

KASLIN FIELDS: Welcome to the show. I do want to go into Kubernetes The Hard Way in a second, but first off, you retired recently, and that seems very exciting. How has retired life been? It seems like you're still very busy.

KELSEY HIGHTOWER: Yeah, I'm coming up on my one year retirement anniversary, so I retired last year in July. And really, retirement is probably not the best word to use for what I've been doing for the last year.

KASLIN FIELDS: Yeah, not what one normally thinks of.

KELSEY HIGHTOWER: Yeah, I'm just not working a typical 9-to-5, but I do-- you know, there's this success criteria. The best thing that you can do in your career is work with the people you want to work with on the things you want to work on. And so I just get time to do that.

And so I picked up a couple of hobbies, like doing projects around the house, but also, I'm still tightly integrated into the open-source community. So that's still attending the conferences, speaking at those conferences, but also spending a lot more time advising startups.

So all those cool companies behind those open-source projects, that's a big go-to for me right now, even including the venture capitalists involved with funding those companies. So there's a little bit of everything, maybe a little less pressure. No OKRs. [LAUGHS] No quarterly planning, but still helping people ship things.

KASLIN FIELDS: Nice. And I don't think you're involved with open-source Kubernetes even anymore, even though you were very involved, of course, in the early days of open-source Kubernetes. But like you said, you're still very involved in the community. Are there any particular projects that you're working with right now?

KELSEY HIGHTOWER: Yeah, at this point, every startup that I advise, they probably have some type of Kubernetes integration. A couple of months ago I did update Kubernetes The Hard Way to be completely ARM-based. So I think I'll always try to make sure that the educational things that I maintain are up to date.

But I think now, Kubernetes is more like Linux, right? I think a lot of people, whether directly or indirectly, are involved with Kubernetes, especially if they're doing cloud-like deployments. The project that I'm most interested in today are Q. Like QLabs is a company behind this configuration system that allows people to do verification.

So think alternative to Terraform, more general purpose. The other project that I'm interested in from an open-source perspective is some of the tools that Red Panda makes. So it's a Kafka clone written in C++. Crossplane is another one, which is the whole operator style pattern for Kubernetes as a framework.

Other than that, I'm just like, yeah, I'm just paying attention to all the other stuff, things like Llama 3 from Meta. That's super interesting, the fact that they're challenging the ideas of open-source and what does it mean for AI? And of course, watching what happens with all of our proper projects, like Terraform and their licensing models. So paying attention to the culture around open-source, not just the projects.

KASLIN FIELDS: It's definitely a very interesting time in open-source, I feel like. Like you were saying with the new open models, there's been a lot of debate, at least that I've been involved in, where people are debating how do we talk about these. Because a lot of the time, it's a model that has been trained. And then what's open-sourced are the weights of the different nodes within the model.

And so that is how it does what it does. But it's not fully open-source in the traditional sense, so we have to create new terminology for this new world we live in.

KELSEY HIGHTOWER: Yeah, I think the struggle is that's almost the equivalent of getting like an ELF binary and saying, hey, this thing is open-source. It's like, well, no, it's more like free to use, but you really can't change it. You don't have the inputs to tweak it. And you truly don't know what's going on there. So I think we've got to be careful with the signing open-source to things in that situation.

Maybe they'll come up with a new term. But I do think what we have today is worth protecting, this concept of I can see the source code, I can use the source code, and I can modify it and make derivative things. We should protect that. If there needs to be something else for AI, then we should do something else for AI. But let's not confuse the two.

KASLIN FIELDS: Yeah, I like it. And I was talking with someone recently, actually, about the current atmosphere around open-source and closed-source businesses. How do businesses these days use open-source?

So since you're in the startup world, advising a lot of companies on how they're approaching open-source, I would imagine you have some thoughts on open-source strategy in the modern era. I didn't put this on our list of questions to ask, but if you have any thoughts, I would love to hear it.

KELSEY HIGHTOWER: You know what's challenging about all of this? There's multiple parties involved in this. There's, of course, the companies who make and sell the software. There's a community who tries their best to support it, add new features, make suggestions.

But then there's the customer. And the role of the customer is to exchange money for something that brings them value. And in the case of open-source, a lot of us are so used to all of it being $0. And if that's the new price point, then there is no way, it's not mathematically possible, to have a company who pays people actual money to continue to build this stuff.

And so one approach would be, only the big companies can do open-source, right? Google, Amazon, Microsoft, and that's it. And so far, they've been really good stewards. I actually love Chrome. They have decades of doing good work around Chrome and even competitors use it. So it's not impossible.

But there is still this value where companies like Docker would not exist, right? They brought new ideas to the table. They were not a big cloud provider when they started. And they gave us a new way to think about how we package and deliver software.

I want that to do it on a continue. And so I think on the customer side, which is slightly different than the community side and the company side, we have to get back into the habit of paying for things we want to exist. And so what's the best way to do that? And I think that's the core to your question.

There's this open-core model, which is maybe you make the base, the core components, open-source, download it for free, have fun. And then maybe there's a layer on top, whether that can be a web UI, Active Directory integration. You know, something that you have to pay for.

And I'm an advisor to companies like Docker, and Docker desktop is in that unique situation, right? Docker desktop is something you pay for and then Docker, the runtime, is something you can use for free. And they layer on top of each other. So based on what I'm seeing, I think the open-core model will win out from a business perspective.

It's probably fair to have companies figure out where to draw the line between, you don't have to pay us anything, enjoy the product, and then that layer where you have to pay. And so one thing we could do though is again, for the customer side, if you like these projects, even from an indie developer, one or two people really making use of their time and giving you what you want, figure out how to pay them and then they can keep doing that.

And that's the thing I think we've kind of-- I don't know if we've ever finished that part when we were defining what open-source would be in terms of the dominant way we think about software, because that doesn't necessarily mean you end up with a successful business. And we got to figure that part out.

KASLIN FIELDS: Definitely, I've had some really interesting conversations with folks lately about how open-source can be such a strong way to create a standard. You create some technology and then if it's open-source, it's a lot easier to adopt in various environments and various use cases. And you can get a lot of adoption and a lot of expansion of understanding of who your user base is by doing that.

But without the business aspect of it, there are so many challenges that can come with that. One thing we've been talking about on social media a lot lately, at least I have, is the challenge of burnout in open-source, something that we in the open-source community are always talking about. And someone asked me recently, what is the difference between the risks of burnout in open-source versus in businesses?

And I was telling them that I feel like the biggest difference is that if you're an employee of a company, you can take time off. They might have systems in place where they can get feedback from users and understand if something's gone wrong. And they have systems in place for how that's handled. Maybe there's a layer of support before it gets to whoever's on call or something like that.

KELSEY HIGHTOWER: They have another thing. They have money.

KASLIN FIELDS: And they have money, basically. [LAUGHS]

KELSEY HIGHTOWER: Yeah, they can hire additional people, so it's not just one person totally responsible for the thing. They can move it between different teams. There's someone to talk to when it's understaffed. And I think the reason why we can get a lot of burnout in open-source is because we don't have those things, right?

KASLIN FIELDS: Mhm.

KELSEY HIGHTOWER: And also in open-source, we're not really good at defining finished.

KASLIN FIELDS: That's very true.

KELSEY HIGHTOWER: Right? This thing is done. We are not adding any more features. When you go to the store and you buy a pair of headphones, they're finished. You're not adding more features to those. You've got to wait till the next version ships.

And so that delta between the thing we have and the thing that's coming later, it would be nice for a more open-source projects to draw a line and say, hey, I know we could go in any direction, but we're choosing not to.

KASLIN FIELDS: Yep. And so there's a lot of benefits to the open-source. I think the core model, like you were saying, makes a lot of sense, where you create a technology, you create a standard, but then having some way to monetize it so that we can build in those safeguards, not just for the customer but also for the people working on the product, is really important.

KELSEY HIGHTOWER: I'm going to call out one other example, and maybe we're getting to the truth where all of these projects don't need companies. And I think that's the tough pill to swallow. Because you create all this value and we've all thought that we should monetize the value we create.

But when you take an example like Let's Encrypt, for example, man, what a great project, right? SSL certificates used to be like $99 to $400 a year, and you would have to get them all the time. They were very clunky to deal with.

And along comes this non-profit that has enough investment to keep the things going, keeping people employed. And that model has been great because if you go look at the stats, I'm pretty sure it's near over 50% of the internet is being secured by Let's Encrypt SSL certificates. And that model seems like a good model for the type of public good that we're actually making.

So not everything needs to be a company that will end up getting bought by Cisco or hopefully going IPO. Maybe some of these projects just really need a strong non-profit foundation so they can focus on the technology and not necessarily the profit.

KASLIN FIELDS: But still have some element there where they can kind of fund the work and create these systems to reduce the risk to users and to contributors.

KELSEY HIGHTOWER: 100%.

KASLIN FIELDS: And you've been doing work in this space for quite some time. This is the birthday of Kubernetes that we're hopefully releasing this episode on. That's the plan. [LAUGHS] So let's talk a little bit about how this has gone for Kubernetes. Might make sense to start with-- should we start with your contributions to Kubernetes or should we start with Kubernetes The Hard Way, as many do? [LAUGHS]

KELSEY HIGHTOWER: When I first heard about Kubernetes, we got a call from Google and Google is like, we're going to release this thing and it's called Kubernetes. I don't even think they gave us the name. Maybe they did. And I knew about it like the night before the DockerCon keynote and press release.

And so I'm at CoreOS at this time. Maybe this is like 2015. And we were like, what are we going to do? Like we got this stuff we're working on. We have etcd. Apparently they're using that. We have this thing called flannel, which became the foundation for CNI. Apparently they may use that.

But nothing else means that we actually get to participate so we're in wait and see mode. And we also had our own runtime at that time called Fleet, which was scheduling containers using systemd across a fleet of servers, hence the name. And so when I got GitHub access, I'm like, oh, look at this thing, it's written in Go. That's a good start.

But the instructions were like, good luck. Like if you can figure out what this is, good luck. And I remember I spent all night trying to get that thing to run on CoreOS itself. Not in the cloud, not assuming anything about GCP, but just VMware on my Mac laptop. And some things didn't build correctly for me, so I'm tweaking and patching and tweaking and patching. Then I got it to work.

And so by the morning-- and I remember Alex Polvi and Brandon Phillips and Melissa, she was the head of marketing. I was like, I think we should have a blog post with a tutorial so that way when Google announces this, we can say, here's how you do it on CoreOS.

And I remember when that press release got launched, it was like number one on Hacker News for a moment. And boy, did CoreOS, when we put out our blog post, like, hey, that's cool, but here's how you actually run it, right now, on your own machine. And it felt really good seeing people kind of put the puzzle together.

CoreOS plus Kubernetes equals this vision that we had at Core for a long time, which was, Google's Infrastructure For Everyone Else, this thing called GIFEE. That was our tagline. And so that was kind of my first beginnings of getting involved with the Kubernetes project.

And then my contributions-- because at that moment, we were not necessarily convinced that this was going to be the winner. You had Mesos, you had Docker Swarm. There was no reason to believe that this time it was going to knock it out of the park.

So everyone was kind of hedging their bets still, and I'm just pulling nights and weekends contributing on the side, being very excited. So every meetup, every conference was like, hey, let me show you this thing that I'm working on. Not sure if it's going to be the thing that wins out, but I think it's also cool, so let me show it to you.

KASLIN FIELDS: That's so awesome. And my favorite fun fact with Kubernetes The Hard Way is that your first commit, creating the guide on GitHub that this would eventually turn into, says, let the pain begin. [LAUGHS]

KELSEY HIGHTOWER: The origins of Kubernetes The Hard way, I joined Google later that year, I think in like November. And we were at DockerCon and I was sitting next to Brian Grant in the audience. And this is-- the early DockerCons were crazy. It was just like, rock concert.

KASLIN FIELDS: Actually got to go to some, so yeah. [LAUGHS]

KELSEY HIGHTOWER: Yeah, and so we're sitting in the audience and they announced Docker Swarm version 2 or version 3, and it had a lot of Kubernetes-like features. We were just like, oh my god, they've taken like the good parts of Kube and then made them user friendly.

And we held the first empathy session when we all got back to campus. And I think we may had an engineering offsite. And everyone was like, I don't think we can compete with Docker in terms of developer usability. I mean, they already have the lead here.

And I was like, you know what? I think it's not necessarily just about ease of use, right? We had this install script called kubed.sh. You ran it, it did magic under the covers, and out came a cluster. And I was like, you know, I don't think that's the problem.

And we did this empathy session where I asked all the engineers, you need to install Kubernetes from scratch without using any of the scripts. And boy, did they struggle. Man, I don't think anyone actually pulled it off. And these are all the principal and distinguished engineers working on Kubernetes.

And I'm just having this good time laughing like, you all can't do it. And I was like, you know what? Two things came out of that. One, kubeadmin was born. Mike Denise was like, you know what? I think we can build something better here to fix this.

And then there was another thing that we ended up doing, was Kubernetes The Hard Way. I said, you know what? Given my operations background, when people don't know how things work, they perceive them to be overly complex. So let me put this guy together.

And you're right. I think that first commit, I was like, you know what? It was painful for me.

[LAUGHTER]

And I knew it was going to be painful for everyone else that was going to go through it. But I remember just meeting people in the community saying, that's what I needed. I needed to see how all the LEGO pieces fit together so I can understand what I was looking at.

KASLIN FIELDS: Yup, that is how I got my start in Kubernetes, is with Kubernetes The Hard Way. So thank you for that. [LAUGHS] I was a big fan, honestly, of Docker Swarm in the beginning. Like you said, Docker was so good at making technology user-friendly.

I was like, you know, this is some good stuff over here. But in the end, Kubernetes won out. [LAUGHS] And it was really interesting. You mentioned kubeadmin as well. In the early days, when I first started with Kubernetes, before kubeadmin, you had to learn all of these different pieces of the Linux kernel and how Kubernetes itself is put together.

And recently, I was talking to someone, and we got on the topic of how complex Kubernetes is, which is still something that people talk about, of course. To this day, there are still a lot of challenges. It can be a hill to get over to learn.

And they said, you know, I don't think it's very complicated at all. And I was like, do tell. Why would you think so? And they came from a Linux sysadmin background. So if you're familiar with the Linux kernel, like you were saying, it makes a lot of sense in that world. The different tools that it uses and the different ways that it puts them together makes a lot of sense.

And another thing that you did with Kubernetes is create the "Kubernetes, Up and Running" book alongside, who has that in the early days? Joe Beda and--

KELSEY HIGHTOWER: Brendan Burns, Joe Beda.

KASLIN FIELDS: Brendan, yeah. Also one of the early resources that I found really helpful. So you were doing all of this work, creating these resources to help people get involved with Kubernetes. Is that the main part of the work that you were doing with the project, or were there other pieces where you were getting involved?

KELSEY HIGHTOWER: I think the first commits were really about little bug fixes. I mean, if you were to saw the Go code base, it looked a lot like Java. We used to jokingly call it Guava, you know, Go written in the style of Java.

KASLIN FIELDS: [LAUGHS] I did not know that.

KELSEY HIGHTOWER: And so there's a lot of work just moving things around, trying to make it more Go-like. And also, some of the work was also getting Kubernetes to run well on CoreOS. So back then, outside of a cloud provider, there is no automation for what a node does, like how it gets registered, how it kind of configures itself.

And so a lot of the work was building a cloud provider for CoreOS. So GKE has one of these. AKS has one of these. But if you're on bare metal, there's nothing. What do you do?

And so there's a component in Kubernetes where if you're a node, you can register yourself and tell Kubernetes that you exist. And so building some of that out for the CoreOS ecosystem was a big part of that work. Another component was CNI, some early contributions into defining what that thing was and how it worked.

And so one of the first things I did was write the IPAM, so the thing that gives IP addresses. So we had this thing called flannel, which uses etcd. It's a little complicated. You have to run daemons. And there was a guy named Eugene who probably did the bulk of the work.

And Eugene was like, hey, how should the interface work on the networking layer? And I was like, look, Docker had a networking abstraction, but it was so Docker specific, you really couldn't do all the networking things.

For our API, let's just make CNI pretty open-ended. So the way it works is, a container starts and it calls out the CNI. And CNI's only job is to say, give me the network namespace. I'll configure some stuff and give you back an IP address.

And so that meant if you were Cisco or Juniper or anybody doing something complex, you can still do those complex things without the API limiting the information or access to the underlying container. And for this to work, we want it to have a way to manage IP addresses.

And so one easy way was to say, hey, how about on every node we have a thing that just manages IPs? And there's a scheme you can use. You can look at your own IP, do a bit of math and figure out a subnet that doesn't conflict with the other IPs on the nodes. And so that was the first implementation.

It was just storing your IP addresses and files. When a new container would start, we would go see if there was a free IP, pick the next one. Containers would stop. We removed the file. And that's how it was done.

And so that was some early work at the low levels. And I can go on and on, but most of the work was around, let's make this thing usable so that it can actually fit in existing ecosystems.

KASLIN FIELDS: Yeah, when you get down to it in Kubernetes, you always come back to the networking because it's distributed computing at its core. It's about how things talk to each other. So it's inevitable that you get to the networking layer at some point. And the point you were mentioning there with bringing nodes into kind of a cluster, so to speak, is also really interesting.

I think it's easy to forget that this was also kind of the era of OpenStack, and like it was a very big thing to try to figure out how are we going to take all of these computers that we have all over the world and combine them into systems where we have more compute and more resources available to our applications and can kind of spread them across them more easily. So Kubernetes, Mesos, all these things.

KELSEY HIGHTOWER: Yeah, one of the node designs-- and I remember we had like this first kind of contributor summit. It was very small. And I remember we were on a whiteboard. Clayton Coleman from Red Hat was there, Brian Grant, Tim Hockin. And I was an engineer at Puppet Labs.

And the way Puppet used to do its node registration, the node would come up, generate its own private public key pair, and then submit it to the Puppet master for signing. And you could turn on this thing called auto sign, and that would speed up getting nodes in and out of your Puppet cluster.

And so I recommend that be the way we tackle this node problem, because nodes didn't have really identity at that point. You could just join a cluster. And so this really gave us the TLS encryption so we can pass things around, and it gave us strong identity that can then be signed by the Puppet agent. So that was one big design.

I was like, ooh, this thing from configuration management days is also useful during that time. So I think that design actually landed, probably got tweaked and improved upon, but that was another thing from making the nodes work because nodes were very fragile.

KASLIN FIELDS: Yeah, this makes a lot of sense. Because I remember the first time that I did Kubernetes The Hard Way, those certs on the nodes were kind of a sticking point for me, because I'd never worked with something like that before. So thank you very much for Kubernetes The Hard Way and showing me how to do that.

And then later, when I tried to explain to people like, here's how you actually run Kubernetes and kubeadm came out, kubeadmin, it was a very different approach to that piece, and you didn't have to worry quite as much about doing those certs anymore. kubeadmin kind of handled a lot of that. So it's all coming together here.

KELSEY HIGHTOWER: Here's why I thought it was so important to show people the certs. There were people who would stand up Kubernetes using these setup scripts. Right on the mark, one year, things would stop working and they wouldn't know why. And it turns out their certificates were expiring, but they knew nothing about the certificates because they never created the certificates and they didn't even certificates were involved.

KASLIN FIELDS: Exactly.

KELSEY HIGHTOWER: So I wanted to make certificates so front and center, look, the current design is beautiful, those things should auto regenerate themselves and repopulate. But they were so hidden, but they're so critical.

And when those things expire, you have no idea. You just see authentication failures. Everything was fine yesterday. You didn't change anything, you promise, and now it's broken. And that was it.

KASLIN FIELDS: The magic of certs, for good or for evil. [LAUGHS] For the individual to decide. So this is such a wonderful look at the early days of Kubernetes. Of course, this being the 10 year anniversary of it, I would love to get some of your thoughts on where you think Kubernetes and kind of open-source in general is going now.

KELSEY HIGHTOWER: Kubernetes is halfway there. I think it became at least obvious to me about two or three years in, this is going to be a 20-year technology. And at that time, the reference point was Linux. Like what does it take to be 20-year technology?

To be popular like OpenStack was isn't enough, because hydrocycles come and go. And it was the extensibility of Kubernetes that made me believe that it would be around for a long time. And when Brendan Burns really pushed the idea around Custom Resource Definitions, commonly called CRDs, that felt like the extension module system that Linux has.

So whatever the core project didn't do, you can extend on your own. And we got lots of extensions, right? People were building operators. People were doing things like Crossplane now. Even Istio took the CRD approach.

And at that moment, I was like, you know what? Kubernetes is a universal control plane. And for a lot of people, it wasn't about big scale clusters. It was about better configuration management. And so instead of Puppet, Chef, and Ansible, there were some people that were just like, man, this Kubernetes model, where-- and I think I was introducing people to this term-- infrastructure as data.

Infrastructure as code, you can do a lot of fancy stuff, but it gets even more complicated the more code you write. But in the Kubernetes world, even though people say they hate YAML, I don't believe that. I think people struggle with the syntax. Maybe people struggle remembering all the schema attributes.

But it's a very beautiful thing knowing that there is an actual schema. There's a way to verify that schema. There's a way to generate that schema from any programming language you want.

And the other beautiful part about having infrastructure as data is this piece where you can have mission controllers that manipulate this thing to adhere to your security policies. You can add components like update a container image from CI/CD. That's really hard when you're doing infrastructure as code, so data makes that possible.

So now that we hit this 10-year mark, this thing is hyper mature. You know, I recall going to-- last year, I gave a keynote at the VMware conference, right? And so I'm at this VMware conference and they were talking about Kubernetes, a lot.

And a lot of people-- we're on year eight at this point, or year nine-- and they're like, yeah, man, Kubernetes, I'm just learning about Kubernetes. And those people were on the sidelines, like, this might be a fad like other projects. But it's not.

And so I think for them, it's like, all right, it's mature enough. There's enough people in production. There's enough books. There's enough support. And my favorite vendor, that could be Oracle, that could be VMware, they now support Kubernetes, and I'm all in.

So that's how I know we're at the halfway point. And we got another 10 years before I think as much attention we give Kubernetes should go away.

And I know a lot of people get sad with that, with like Kubernetes goes away-- it won't go away, like we don't care about it anymore. It will go away because it's doing such a good job that it doesn't have to be in the way. And I think that is the perfect transition for any infrastructure project to mature to the point where it just gets out of the way.

And I think that's the next leg of work. It won't be as fun, it won't be as sexy as some of the other work. But it's so important because if we do it right, it will feel a lot like Linux. That's just there, does what it does, and largely stays out the way.

KASLIN FIELDS: And I think that's such a great point, which I've often had trouble communicating to folks who have been in the Kubernetes space for a long time. There are so many people who are just starting out with Kubernetes. It's just now gotten to a point where they need it or where they're comfortable pursuing it, or whatever the case may be.

And at KubeCon, still 50% of the audience is new, each KubeCon. And I feel like I've finally gotten through to a few of the folks that I talked to at work, like, we really need more beginner content still. We're 10 years in, but we still need beginner content. I like the view that we're halfway there.

We're not quite at the point where Kubernetes can just get out of the way, like you were saying, but it's at a point where it's solid. It's something that you should feel comfortable using, and it'll just keep getting better, hopefully. That's what we want in open-source at least.

KELSEY HIGHTOWER: And I think a lot of people will start borrowing ideas. I mean, at some point, and there probably already is one, there's going to be a Control Plane that implements Kubernetes style APIs without Kubernetes. I think Cloud Run is probably one of the best examples of this.

Cloud Run has a Kubernetes like API, and Control Plane, but it doesn't necessarily leverage Kubernetes. And so that Kubernetes Resource Model, the KRM, that may become the HTTP of infrastructure, where people-- if you're going to design some infrastructure-- we all love APIs, right? Curl something, get a response back.

But what's even better, though, is when you have a really clean schema, that I can just submit data to it. Your Control Plane does the entire life cycle. We went from bash scripting, where you run it and hopefully it works, maybe it doesn't, but that's it, to something like Terraform that can integrate really well with APIs. But then it's not clear who owns the resource.

Is it the cloud provider or is it your Terraform code? But that Kubernetes Resource Model, I think it completes the puzzle. You own the life cycle. I just want to submit to you what it should be.

In 24 hours, you should always be getting to where the data has telling you to get to and have that little status section if we're not quite there yet and if there's something that I need to do. That model, I think will outlive a lot of the things that we're working on now. And that's a net good thing.

KASLIN FIELDS: Wonderful. And one other thing that I want to come back to that you mentioned is custom resource definitions and operators, and the extensibility of Kubernetes. And that has been such a big thing throughout its life cycle, as you were saying.

And now, I feel like there's even an extension of that that's happening in the community with things like the Gateway API and the removal of the cloud provider code from entry Kubernetes. We're seeing more definition of standards that work alongside Kubernetes that are not implemented into Kubernetes themselves, and that's that extensibility kind of evolving to the next level, I think.

KELSEY HIGHTOWER: Yeah, and one thing we didn't touch on, and I'm pretty sure you've touched on with other guests, but I meet all the time people who are release engineers for the Kubernetes projects. Most people don't know who they are. They never met these folks.

They don't all get their chance on the keynote stage, and they work on every aspect of Kubernetes-- the YAML parser, the release, the versioning, when to do a new patch. I mean, there's so much work that goes on under the covers that those group of people, they are the people that will bring the next generation of Kubernetes to bear.

So you don't always get the same credit as the founders of a project, but you're equally as important right now, given this moment, because most of the early folks that worked on Kubernetes don't work on it anymore. And that's not necessarily a bad thing. Honestly, that's a sign of maturity that they were able to pass the baton.

So if you're holding that baton right now, I just want to say, thank you. But that race still needs to be run. And if this is going to go from a 10-year to a 20-year to a 30-year project, it will be your turn to pass that baton too.

So don't think that this is all on you to bring it to the next generation as well. You're just doing your part for this segment of the race. And start identifying that next generation so this project will be around for a very long time.

KASLIN FIELDS: And that is a perfect lead-in to my last question for you today, which is, what advice would you give to folks who are starting out in the cloud native space today?

KELSEY HIGHTOWER: I think if you're starting out in cloud native today, it's almost like cloud native doesn't really mean anything anymore, because we needed that word as scaffolding to talk about the difference between what we were doing 10 years ago and how it was different. And those words pushed together kind of represented opportunity.

There's an opportunity to rethink the way we're doing things. But today, all the observability vendors, from Datadog to Splunk, probably support OpenTelemetry. And so is that a cloud native thing or is it just a thing? And at this point, a lot of these ideas are just the way we do things now.

And so don't get so confused trying to figure out where you fit on the cloud native litmus test. More than likely, the things you're using have adopted patterns from the cloud era, and we're entering a new era now. So the cloud era, for some people, will be considered legacy soon, like in five or six more years.

You know, I think I've already started reading the term legacy cloud. It's like, what? We're already at legacy cloud? And so I think now, it's just about those techniques. So if you're just getting started today, learn the techniques that are in front of you, and then go back in time and see how things evolved.

And you're going to see this really nice pattern reveal itself. We go forward, and we loop back again. And a lot of things that were old become new again. And so if you just replace those words with just continuously learning how to get better at this stuff, you're going to be totally fine. So you're not behind, even if you're just getting started today.

KASLIN FIELDS: Wonderful. Thank you so much, Kelsey, for being here today. And I look forward to having more folks joining the ecosystem and helping them out along the way.

KELSEY HIGHTOWER: Awesome. Thanks for having me, Kaslin.

KASLIN FIELDS: And happy birthday to Kubernetes. [LAUGHS]

ABDEL SGHIOUAR: Well, thank you, Kaslin, for that interview. Listening to Kelsey and Tim is always a pleasure.

KASLIN FIELDS: Such a delight to actually get to interview them. Sometimes, I'm really excited that we're doing the podcast, you know.

ABDEL SGHIOUAR: Just for the sake of talking to these people?

KASLIN FIELDS: When you get to talk to awesome people--

ABDEL SGHIOUAR: Yes, yeah.

KASLIN FIELDS: --really feels like a privilege.

ABDEL SGHIOUAR: It's true that we have an added privilege that Tim is closer to us, so it's kind of easy to get Tim, but Kelsey--

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: I mean, we're hanging out with Tim at KubeCon in Paris, right?

KASLIN FIELDS: Yeah, by the way, there is a 10 year anniversary T-shirt that we designed for contributors, so only available at contributor summits for Kubernetes org members. But I got to design that with Tim, and Tim designed the original Kubernetes logo. And when I interviewed him, we had a little conversation beforehand.

Because he was like in his office at home and there were all these art supplies around him because his wife also does art. A lot of them were hers, but it was very-- very excited to get to work on the 10 year anniversary contributor T-shirt design with him. So cool.

ABDEL SGHIOUAR: Yeah, and the T-shirt is really cool and I'm sad that I'm not a contributor so I cannot get one.

KASLIN FIELDS: You just got to join the org.

ABDEL SGHIOUAR: Yeah, that's what I need to do.

KASLIN FIELDS: Do some work. Do some PRs.

ABDEL SGHIOUAR: Actually, to be honest with you, I got inspired by all these interviews we've been doing for the 10 years anniversary. There is quite a lot of interesting things going on, and I'm seriously thinking about picking some stuff up.

KASLIN FIELDS: Me too, which is weird to say because I'm already very deeply involved.

ABDEL SGHIOUAR: You're already involved, yeah.

KASLIN FIELDS: I'm already co-chair [LAUGHS] and I'm already a project lead and stuff, but I'm kind of in one area of the project, mostly, is where I have been. But it's very, very common for contributors to work in several different special interest groups, different areas of the project. And they've been making me feel very inspired to go check out some other areas of the project as well.

ABDEL SGHIOUAR: Yeah. Yeah, I mean, case in point, Nikita. I finished the interview with Nikita, which is coming next, and she has been involved in a lot of areas in the projects--

KASLIN FIELDS: Oh yeah.

ABDEL SGHIOUAR: --that she's been doing a lot.

KASLIN FIELDS: A little preview. Yeah, look forward to that one.

ABDEL SGHIOUAR: A little preview, yes. Spoiler alerts. All right, so let's start with Kelsey. Kelsey is retired but not retired.

KASLIN FIELDS: Is there any such thing as retirement for Kelsey Hightower, really? He's always going to be Kelsey Hightower.

ABDEL SGHIOUAR: Of course. Probably not. No, because I've seen him going around like doing conferences, which is kind of expected. When he announced his retirement, I kind of knew that doesn't mean not doing anything anymore.

Just because people will want to have him on different shows and different conferences and hear what he has to say. But it's still quite interesting to see that he found this space where he's just kind of working for himself, but it's still retirement, which is kind of interesting.

KASLIN FIELDS: I do think that he'll probably slow down on the events to some degree in the next several years. But he loves this stuff so much.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: Any time you get to talk to Kelsey Hightower, it's obvious that he just really loves the technology. And so I don't think he'll ever be able to fully pull himself away from it because he doesn't want to. He wants to keep working on it, but maybe just not for someone else.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: I love how he's been describing his retirement in social media and when we talked about how it's just about doing what you want for yourself and not for a company.

ABDEL SGHIOUAR: Yeah, not what other people want you to do.

KASLIN FIELDS: Mhm.

ABDEL SGHIOUAR: I mean, of course, I knew Kelsey since I got started on Kubernetes. I knew him from Kubernetes The Hard Way, which was funny when he was describing it because he said that he pushed the first pull request and he was like, yeah, this is actually hard, right?

KASLIN FIELDS: Mhm. I've heard him tell that story a few times, which is really cool, about how, like, he just tried to deploy the thing and that was hard enough as it was. And then he created this whole concept of customer empathy sessions, is a concept within Google where someone runs a workshop basically, with the engineers building a thing to show them how hard something about that thing is.

Like, get them to be users of their own product and give them a task here. Here's your tool. Here's what you need to do. You're not allowed to use any internal resources. Go try and do it.

And so based on that experience, he created that whole concept within Google, which is still used today, theoretically. I would like to run some of those.

ABDEL SGHIOUAR: Yes. I mean, I think what Kaslin is trying to say is, like shame people. Just take a bunch of engineers, put them in area, in a spot where they feel shame, and then they're like, well, maybe we need to fix this.

KASLIN FIELDS: Yeah. [LAUGHS]

ABDEL SGHIOUAR: Shame works in magical ways.

KASLIN FIELDS: Being an engineer, developing a thing, is not the same as being a user of the thing. I think we all learn that at some point in our careers.

ABDEL SGHIOUAR: Yes, and to be very fair, not to-- I mean, we're joking, but to be very fair, a project like Kubernetes is so big that it's very difficult to have everybody knowing everything, right?

KASLIN FIELDS: And of course, in the early days, it was a slightly different problem, right?

ABDEL SGHIOUAR: Of course.

KASLIN FIELDS: It wasn't that huge, but there were very few people working on it, and they were working on "it." Making it exist.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: And so documentation was not the top of the list for a lot of the folks in the very, very early days.

ABDEL SGHIOUAR: Yeah, and as the project got more complex and more separated into special interest groups, it's very hard to find people that know everything. Well, of course, there is people of the ranks of Tim and Clayton, who have been involved in a lot of stuff.

But the engineers working on very specific areas, they might not know everything. And when you ask them to do stuff, then that's where things get interesting. I did some empathy sessions, some engineering empathy sessions or customer empathic session. They're very interesting, actually.

KASLIN FIELDS: It's true in open-source too. The folks who are developing the thing are not necessarily the users of the thing, which is why communication from all of you out there, who are Kubernetes users, is very important to us in open-source Kubernetes. We would love to find more ways to interact with our user communities.

It's one reason that we're so excited to have the KubernetesIO account on Twitter now, is that that's an easier way for us to reach our end-user audience and hopefully hear from them.

ABDEL SGHIOUAR: Yeah. You talked about a lot of topics, but one that was very interesting for me at least, is his thoughts on the economical viability of open-source, generally speaking.

KASLIN FIELDS: Yes, very interesting topic.

ABDEL SGHIOUAR: That was really cool to hear him share his thoughts on that.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: Especially that he's now in a position where he can advise other companies and this whole freemium model, and that was pretty cool.

KASLIN FIELDS: I feel like that's always an interesting topic and especially right now, with a lot of the interesting things that have been going on in the open-source space. I mentioned in the interview that I've had some interesting conversations with folks lately about how do we do open-source for business.

So Kelsey, I think, has such fantastic perspective on that, having been involved with open source, especially something as large as Kubernetes from its early days and then now, advising companies on their strategy. I would take his advice very seriously if you are considering an open-source business.

ABDEL SGHIOUAR: And this came in a very interesting time because when we published the last episode, which is the first episode in the special series, in the interview with Dims, we talked about this whole concept of the Kubernetes projects costing millions and millions of dollars. And people within Kubernetes, even people who are using Kubernetes, are not aware of that necessarily.

They just expect that images will be available, artifacts will be available, and the code source will be available. So what's happened to me is I was in a conference last week from whenever we are recording this, and I was on a live podcast with a developer advocate at Broadcom called Josh Long. He's a very known person in the Spring world.

KASLIN FIELDS: I got to do a keynote with him at Google Cloud Next.

ABDEL SGHIOUAR: Yes, yes. I forgot that, yes. Thank you for reminding me. And we were talking about it and I was like, guess how much the Kubernetes project costs? And he was like, how much? And I was like, it's $6 million, and he was very, very surprised.

Like people within the community doesn't necessarily know, but then people outside doesn't know how much money these things cost to run, if we don't include the cost of development.

KASLIN FIELDS: It's a business.

ABDEL SGHIOUAR: It is a business, exactly. It is run like a business.

KASLIN FIELDS: A very different kind of business. But-- [LAUGHS]

ABDEL SGHIOUAR: Yeah, it's called open-source. Cost money. Anyway. Yes, so that was cool. And then, Tim, I liked the last comment from Tim, which was-- what was it, like the next 10 years? Like we are in the middle of-- the next 10 years are going to be as interesting as the previous 10 years. And you were surprised that he has this vision for 10 more years.

KASLIN FIELDS: Oh, yeah, yeah, yeah. Yes. I hope that he's still around when we hit the 20 year anniversary.

ABDEL SGHIOUAR: I hope so.

KASLIN FIELDS: Still contributing.

ABDEL SGHIOUAR: Yeah.

KASLIN FIELDS: Maybe don't hope for that for him. I don't know. [LAUGHS]

ABDEL SGHIOUAR: [LAUGHS] I hope that they will not be getting the same mean comments in the next 10 years as they've been getting in the previous 10.

KASLIN FIELDS: Oh, yes. Yeah, maybe some other people will be in the kinds of roles that he's in the project, but hopefully he can still hang out.

ABDEL SGHIOUAR: Yes, I think that's fundamentally one thing that is interesting in what Tim and people like him does, is like very, very fundamental core work. I remember one of our episodes, we covered this migrating the whole code base to Go workspaces. That's a very massive undertaking. That is not something that a single contributor that is a new contributor at least can do. You have to be involved for a very long time.

KASLIN FIELDS: Yep, and that's something that we talk about with new contributors a lot, is that there is a lot going on in the project that you have to just be here for a while to understand the context and how everything works. Being part of the community means getting involved in the community.

ABDEL SGHIOUAR: Exactly. Exactly. Now, that was really cool. Well, thank you for those interviews. That was super cool.

KASLIN FIELDS: Thank you so much to Tim and Kelsey for being on with us. We really appreciate them being here for the 10 year anniversary of Kubernetes!

ABDEL SGHIOUAR: Yes. So I hope people would have enjoyed today. If there is any participation in any of the ongoing activities, I hope you enjoy your Mountain View little party.

KASLIN FIELDS: Hop on to Twitter and wish us a happy 10th birthday, because that account does exist. [LAUGHS]

ABDEL SGHIOUAR: Yes, and by Twitter, Kaslin means the KubernetesIO Twitter, not the podcast Twitter.

KASLIN FIELDS: Oh, yes, and I mean X.com, obviously.

ABDEL SGHIOUAR: Yeah, whatever. I refuse to say X.com. I'll just continue saying Twitter.

KASLIN FIELDS: Yeah, anyway.

ABDEL SGHIOUAR: OK, yeah. All right. Thank you very much, everyone.

KASLIN FIELDS: Thank you, everyone.

ABDEL SGHIOUAR: That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have feedback for us, you can find us on social media at Kubernetespod, or reach us by email at <kubernetespodcast@google.com>.

You can also check our website at kubernetespodcast.com, where you will find transcripts and show notes and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]