CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google. I'm Craig Box.

ADAM GLICK: And I'm Adam Glick.

[THEME MUSIC PLAYING]

CRAIG BOX: Sad news about Myspace this week.

ADAM GLICK: It is. I don't know if you were ever a big user, but I certainly enjoyed my time there. And now all of those recordings will be lost to the times of history, as will the prolific use of the sparkle tag.

CRAIG BOX: What recordings did you have on Myspace?

ADAM GLICK: Oh, the world will never know now. That's the beauty of it.

CRAIG BOX: Aye, was it rips of Limp Bizkit and Justin Timberlake MP3s?

ADAM GLICK: [LAUGHS] No, I think I actually had a bunch of spoken word pieces that I had posted up there.

CRAIG BOX: Wow.

ADAM GLICK: How about you?

CRAIG BOX: I've been a little bit more offline last week. I've spent a couple of days with presentations and customers and so on, but also a couple of days walking in the bush. I should point out that "bush" is the New Zealand term for forest.

I have found that in small-town New Zealand, there is almost always a really nice cafe. It never used to be the case, as I'm sure is the case with other small towns around the world. But I think what happens is people move to the big city and they're like, oh, that's great. I'm going to move somewhere else. And they move somewhere smaller, and they open a lovely cafe. And so all these little towns where previously you were lucky if you could get a pie or deep fried food, now you can find a lovely little cafe to have lunch as you go off on your bush walk for the afternoon.

ADAM GLICK: Any particular one that you want to log roll for as a great one if anyone's ever in town?

CRAIG BOX: Sure. So this is going to go super deep-cut. But if you're ever in a place called Peel Forest, which is a podocarp forest of mostly native trees in the South Island, mid-Canterbury of New Zealand, there's a lovely cafe called The Green Man where we have a, I think, an English gentleman, who'd moved to New Zealand a few years ago, and his partner. And they have a lovely cafe. I had a lovely chicken pizza there. And I can thoroughly recommend it.

ADAM GLICK: Excellent. Hopefully, you didn't have to walk a Green Mile to get there.

[MUSIC PLAYING]

We're happy to announce our guest for the Kubernetes podcast live at Google Cloud Next. We'll be hosting a fireside chat with Eric Brewer, the man who announced Kubernetes to the world almost five years ago. Eric is VP of compute infrastructure at Google Cloud, author of the CAP theorem, a professor of computer science, and a thoroughly nice guy to boot. Prior to Google, he started a search company-- whoever would want to do a thing like that-- and worked with President Clinton in creating USA.gov.

CRAIG BOX: Even better news-- if you would like to attend Google Cloud Next and you promise to sit in the front row of the podcast recording, we've managed to score 10 free full conference tickets from our friends at the Next events team. You'll have to be in San Francisco, or able to get yourself there and put yourself up, from April 9th to 11th. First come, first serve. Some restrictions apply. We're sorry to our friends around the world, but unfortunately this offer is only available to residents of the US. Please check our show notes for details.

And now the news.

[MUSIC PLAYING]

ADAM GLICK: Kubernetes 1.14 is out. This release of the kernel of the cluster operating system brings Windows container support to GA, as well as pod readiness gates and pod priority and preemption, reaching the stable milestone. New features include server-side kubectl apply in alpha, and ingress moving from the extensions to the networking API group in readiness for promotion to GA in a future release. You'll hear all about this release later in the show.

CRAIG BOX: But perhaps you don't need Kubernetes at all. Maybe you're listening to the wrong podcast. Contrarian engineer, Matthias Endler of Trivago, sparked debate with his "Maybe You Don't Need Kubernetes" post. Endler says that Nomad provided his team 20% of the functionality of Kubernetes, while misunderstanding the role of Helm and ConfigMap objects and simultaneously pointing out that other teams at Trivago were very happy with Kubernetes. I don't have to mkconfig on my Android phone. And as the tooling improves, you don't need to care about the underlying details of Kubernetes. But the community and ecosystem matters, and so we think, yes, you do need Kubernetes.

ADAM GLICK: Gravitational has released Gravity 5.5. Gravity is designed to provide application portability for complex cloud-native applications that require multi-node clusters. It's an open-source solution that enables users to package multiple cloud-native applications into a Kubernetes cluster and build a dependency-free cluster image from that. You can take your image and publish them to app catalogs or file stores. And since you have an image of a full cluster, you can create full replicas of the original cluster, and you can remotely manage all those cluster replicas. The headline feature in version 5.5 is that Gravity now uses helm charts as its manifest format.

No comment was available from Gravity co-founder, Isaac Newton.

CRAIG BOX: Check out a security post-mortem from JW Player on how a cryptocurrency miner made its way onto their internal Kubernetes clusters. Spoiler-- our monitoring application was configured to have privileged access, and a load balancer was accidentally exposed to the internet. The full story of how it happened and how it could have been prevented is in their write-up.

ADAM GLICK: Malte Isberner of StackRox posted a really educational article about Kubernetes and mission controllers this week. In his post to the Kubernetes blog, he talks about why admission controllers were created-- the security, governance, and configuration management benefits of using admission controllers, and some neat details to let you know that you may already be using admission controllers in your cluster and not even know it. He also goes through creating a couple of different kinds of admission controllers and how to test them.

CRAIG BOX: Do you run tests against your helm charts or the YAML they generate? Are you confident all your spaces are in the right places? Gruntwork have taken their Terratest tool for Terraform manifests, and extended it to support Helm, with both unit and integration tests. Now your configuration as code can be even more code-like.

ADAM GLICK: Patrick Ohly of Intel this week posted a blog talking about how to do end-to-end testing with Kubernetes. He points out that more and more parts of your cluster are likely not part of core Kubernetes. And he used the container storage interface as his example.

He then goes through a new framework that came about in 1.13 to help better complete end-to-end testing while working with pieces that are not part of base Kubernetes technology. He also covers some new pieces that are part of 1.14. So if you're building tests for your Kubernetes-based applications, this could be a very useful article to read.

CRAIG BOX: Finally, cloud providers are great-- at least we think so-- but there are some locations where the big ones don't offer service. For those occasions, the joy of Kubernetes is you can run the open-source pieces yourself and enjoy the same API in all your locations. Cloud Ops went through this process while migrating a client to Google Cloud Platform in 2018, and write about the experience in their blog. Because GKE On-Prem wasn't available in 2018, they picked Rancher Kubernetes Engine and installed it in a data center in Russia. Their write-up talks about how they used open-source storage and logging technologies to bridge the gaps to what you get in a cloud provider.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

ADAM GLICK: Aaron Crickenberger is a senior test engineer with Google Cloud. He co-founded the Kubernetes Testing SIG, has participated in every Kubernetes release since version 1.4, has served on the Kubernetes steering committee since its inception in 2017, and most recently served as the Kubernetes 1.14 release lead. He also has a mighty beard, which makes regular appearances in the Kubernetes community. Welcome to the show, Aaron.

AARON CRICKENBERGER: Happy to be here.

CRAIG BOX: We like to start with our guests into digging into their backgrounds a little bit. Kubernetes is built from contributors from many different companies. You worked on Kubernetes at Samsung SDS before joining Google. Does anything change in your position in the community and the work you do when you change companies?

AARON CRICKENBERGER: Largely, no. I think the food's a little bit better at the current company. But by and large, I have gotten to work with basically the same people doing basically the same thing. I cared about the community first and Google second before I joined Google, and I kind of still operate that way mostly because I believe that Google's success depends upon the community's success, as does everybody else who depends upon Kubernetes. A good and healthy upstream makes a good and healthy downstream.

So that was largely why Samsung had me working on Kubernetes in the first place was because we thought the technology was legit. But we needed to make sure that the community and project as a whole was also legit. And so that's why you've seen me continue to advocate for transparency and community empowerment throughout my tenure in Kubernetes.

ADAM GLICK: You co-founded the Testing SIG. How did you decide that that was needed? And at what stage in the process did you come to that?

AARON CRICKENBERGER: So this was very early on in the Kubernetes project. I'm actually a little hazy on specifically when it happened. But at the time, my boss, Bob Wise, worked with some folks within Google to co-found the Scalability SIG.

If you remember way, way back when Kubernetes first started, there was concern over whether or not Kubernetes was performance enough. Like, I believe it officially supported something on the order of 100 nodes. And there were some who thought, that's silly. I mean, come on, Google can do way more than that. And who in their right mind is going to use a container orchestrator that only supports 100 nodes?

And of course the thing is we're being super-conservative. We're trying to iterate, ship early and often. And so we helped push the boundaries to make sure that Kubernetes could prove that it worked up to a thousand nodes before it was even officially supported to say, look, it already does this, we're just trying to make sure we have all of the nuts and bolts tightened.

OK, so great. We decided we needed to create a thing called a SIG in the very first place to talk about these things and make sure that we were moving in the right direction. I then turned my personal attention to testing as the next thing that I believe needed a SIG. So I believe that testing was the second SIG ever to be created for Kubernetes. It was co-founded initially with Ike McCreary who, at the time I believe, was an SRE for Google, and then eventually it was handed over to some folks who work in the engineering productivity part of Google where I think it aligned really well with testings interests.

Like I don't know what you people are trying to write here with Kubernetes, but I want to help you write it better, faster, and stronger. And so I want to make sure we, as a community and as a project, are making it easier for you to write tests, easier for you to run tests, and most importantly, easier for you to act based on those test results.

So that came down to, let's make sure that Kubernetes gets tested on more than just Google Cloud. That was super important to me, as somebody who operated not in Google Cloud but in other clouds. I think it really helped sell the story and build confidence in Kubernetes as something that worked effectively on multiple clouds. And I also thought it was really helpful to see SIG testing in the communities advocacy move us to a world today we can use test grids so that everybody see the same set of test results to understand what is allowed to prevent Kubernetes from going out the door.

So the process was basically just saying, let's do it. The process was finding people who were motivated and suggesting that we meet on a recurring basis and we try to rally around a common set of work. This was sort of well before SIG governance was an official thing. And we gradually, after about a year, I think, settled on the pattern that most SIGs follow where you try to make sure you have a meeting agenda, you have a Slack channel, you have a mailing list, you discuss everything out in the open, you try to use sort a consistent set of milestones and move forward.

CRAIG BOX: A couple of things I wanted to ask about your life before Kubernetes. Why is there a Black Hawk flight simulator in a shipping container?

AARON CRICKENBERGER: As you may imagine, Black Hawk helicopters are flown in a variety of places around the world, not just next to a building that happens to have a parking lot next to it. And so in order to keep your pilots fresh, you may want to make sure they have good training hours and flight time, without spending fuel to fly an actual helicopter.

So I was involved in helping make what's called a operation simulator, to train pilots on a bunch of the procedures using the same exact hardware that was deployed in Black Hawk helicopters, complete with motion seats that would shake to simulate movement and a full-fidelity visual system. This was all packed up in two shipping containers so that the simulator could be deployed wherever needed.

I definitely had a really fun experience working on this simulator in the field at an Air Force base prior to a conference where I got to experience F-16s doing takeoff drills, which was amazing. They would get off the runway, and then just slam the afterburners to max and go straight up into the air. And I got to work on graphic simulation bugs. It was really cool.

CRAIG BOX: And for a lot of people, when you click on the web page they have listed in the GitHub link, you get their resume, or you get the list of open source projects they work on. In your case, there's a SoundCloud page. What do people find on that page?

AARON CRICKENBERGER: They get to see me living my whole life. I find that music is a very important part of my life. It's a non-verbal voice that I have developed over time. And I needed some place to host that. And then it came down between SoundCloud and Bandcamp. And SoundCloud was a much easier place to host my recordings.

So you get to hear the results of me having picked up a guitar and noodling with that about five years ago. You get to hear what I've learned messing around with Ableton Live. You get to hear some mixes that I've done of ambient music. And I haven't posted anything in a while there because I'm trying to get my recording of drums just right.

So if you go to my YouTube channel, mostly what you'll see are recordings of the various SIG meetings that I've participated in. But if you go back a little bit earlier than that, you'll see that I do, in fact, play the drums. I'm trying to get those folded into my next songs.

CRAIG BOX: Do you know who Hugh Padgham is?

AARON CRICKENBERGER: I do not.

CRAIG BOX: Hugh Padgham was the recording engineer who did the gated reverb drum sound that basically defined Phil Collins in the 1980s. I think you should call him up if you're having problems with your drum sound.

AARON CRICKENBERGER: That is awesome.

ADAM GLICK: So you mentioned you can also find videos of the work that you're doing with the SIG. How did you become the release manager for 1.14?

AARON CRICKENBERGER: So I've been involved in the Kubernetes release process way back in the 1.4 days. I started out as somebody who tried to help figure out, how do you write release notes for this thing? How do you take this whole mess and try to describe it in a sane way that makes sense to end users and developers? And I gradually became involved in other aspects of the release over time.

So I helped out with CI Signal. I helped out with issue triage. When I helped out with CI Signal, I wrote the very first playbook to describe what it is I do around here. That's the model that has since been used for the rest of the release team where every role describes what they do in a playbook that is used not just for their own benefit, but to help them train other people.

So formally how I became release lead was I served as release shadow in 1.13. And when release leads are looking to figure out who's going to lead the next release, they turn around and they look at their shadows, because those are who they have been helping out and training.

CRAIG BOX: If they don't have a shadow, do they have to wait another three months and do a release again?

AARON CRICKENBERGER: They do not. The way it works is the release lead can look at their shadows, then they take a look at the rest of their release team leads to see if there is sufficient experience there. And then if not, they consult with the chairs of SIG release.

So for example, for Kubernetes v1.15, I ended up in an unfortunate situation where neither of my shadows were available to step up and become the leads for 1.15. So I consulted with Claire Lawrence, who was my enhancements lead for 1.14 and who was on the release team for two quarters, and so met the requirements to become a release lead that way. So she will be the release lead for v1.15.

CRAIG BOX: That was a fantastic answer to a throwaway Groundhog Day joke. I appreciate that.

AARON CRICKENBERGER: [LAUGHS]

ADAM GLICK: You can ask it again and see what the answer is, and then another time, and see how it evolves over time.

AARON CRICKENBERGER: I'm short on my Groundhog Day riffs. I'll come back to you.

ADAM GLICK: So what are your responsibilities as the release lead?

AARON CRICKENBERGER: Don't panic. I mean, essentially, a release lead's job is to make the final call, and then hold the line by making the final call. So what you shouldn't be doing as a release lead is attempting to dive in and fix all of the things, or do all of the things, or second-guess anybody else's work. You are there principally and primarily to listen to everybody else's advice and help them make the best decision. And only in the situations where there's not a clear consensus do you wade in and make the call yourself.

I feel like I was helped out by a very capable team in this regard, this release cycle. So it was super helpful. But as somebody who has what I like to call an "accomplishment monkey" on my back, it can be very difficult to resist the urge to dive right in and help out, because I have been there before. I have the boots-on-the-ground experience.

The release lead's job is not to be the boots on the ground, but to help make sure that everybody who is boots on the ground is actually doing what they need to do and unblocked IN doing what they need to do. It also involves doing songs, and dances and making funny pictures. So I view it more as like it's about effective communication. And doing a lot of songs, and dances, and funny pictures, and memes is one way that I do that.

So one way that I thought it would help people pay attention to the release updates that I gave every week at the Kubernetes community meeting was to make sure that I wore a different cat T-shirt each week. After people riffed and joked out my first cat T-shirt where I said, I really need coffee right meow, and somebody asked if I got that coffee from a purr-colator, I decided to up the ante.

And I've heard that people will await those cat T-shirts. They want to know what the latest one is. I even got a special cat T-shirt just to signify that code freeze was coming.

We also decided that instead of imposing this crazy process that involved a lot of milestones, and labels, and whatnot that would cause the machinery to impose a bunch of additional friction, I would just post a lot of memes to Twitter about code freeze coming. And that seems to have worked out really well. So by and large, the release lead's job is communication, unblocking, and then doing nothing for as much as possible.

It's really kind of difficult and terrifying because you always have this feeling that you may have missed something, or that you're just not seeing something that's out there. So I'm sitting in this position with a release that has been extremely stable, and I spent a lot of time thinking, OK, what am I missing? Like, this looks too good. This is too quiet. There's usually something that blows up. Come on, what is it, what is it, what is it? And it's an exercise in keeping that all in and not sharing it with everybody until the release is over.

ADAM GLICK: He is here in a cat T-shirt, as well. We'll try and get a picture for the show notes.

When a new US president takes over the office, it's customary that the outgoing president leaves them a note with advice in it. Aside from the shadow team, is there something similar that exists with Kubernetes release management?

AARON CRICKENBERGER: Yeah, I would say there's a very special-- I don't know what the word is I'm looking for here-- bond, relationship, or something where people who have been release leads in the past are very empathetic and very supportive of those who step into the role as release lead.

You know, I talked about release lead being a lot of uncertainty and second-guessing yourself, while on the outside you have to pretend like everything is OK. And having the support of people who have been there and who have gone through that experience is tremendously helpful.

So I was able to reach out to a previous release lead. Not to pull the game with-- what is it, like two envelopes? The first envelope, you blame the outgoing president. The second envelope, you write two letters. It's not quite like that.

I am totally happy to be blamed for all of the changes we made to the release process that didn't go well, but I'm also happy to help support my successor. I feel like my job as a release lead is, number one, make sure the release gets out the door, number two, make sure I set up my successor for success.

So I've already been meeting with Claire to describe what I would do as the introductory steps. And I plan on continuing to consult with Claire throughout the release process to make sure that things are going well.

CRAIG BOX: And if you want to hear the perspective from some previous release leads, check out episode 10 where we interview Josh Berkus and Tim Pepper.

ADAM GLICK: What do you plan to put into that set of notes for Claire?

AARON CRICKENBERGER: That's a really good question. I would tell Claire to trust her team first and trust her gut second. Like I said, I think it is super important to establish trust with your team, because the release is this superhuman effort that involves consuming, or otherwise fielding, or shepherding the work of hundreds of contributors.

And your team is made up of at least 13 people. You could go all the way up to 40 or 50, if you include all of the people that are being trained by those people. There's so much work out there. It's just more work than any one person can possibly handle.

It's honestly the same thing I will tell new contributors to Kubernetes is that there's no way you can possibly understand all of it. You will not understand the shape of Kubernetes. You will never be the expert who knows literally all of the things, and that's OK. The important part is to make sure that you have people who, when you don't know the answer, you know who to ask for the answer. And it is really helpful if your team are those people.

CRAIG BOX: The specific version that you've been working on and the release that's just come out is Kubernetes 1.14. What are some of the new things in this release?

AARON CRICKENBERGER: So this release of Kubernetes contains more stable enhancements than any other release of Kubernetes ever. And I'm pretty proud of that fact. I know in the past you may have heard other release leads talk about, like, this is the stability release, or this time we're really making things a little more mature. But I feel a lot of confidence in saying that this time around.

Like, I stood in a room, and it was a leadership summit, I think, back in 2017 where we said, look, we're really going to try and make Kubernetes more stable. And we're going to focus on sort of hardening the core of Kubernetes and defining what the core of Kubernetes is. And we're not going to accept a bunch of new features. And then we kind of went and accepted a bunch of new features. And that was a while ago. And here we are today.

But I think we are finally starting to see the results of work that was started back then. So Windows Server Container Support is probably the biggest one. You can hear Michael Michael tell stories about how SIG Windows was started about three years ago. And today, they can finally announce that Windows Server containers have gone GA. That's a huge accomplishment.

A lot of the heavy lifting for this, I believe, came at the end. It started with a conversation in Kubernetes 1.13, and was really wrapped up this release where we define, what are Windows Server containers, exactly? How do they differ from Docker containers or other container runtimes that run on Linux?

Because today so much of the assumptions people make about the functionality that Kubernetes offers are also baked in with the functionality that Linux-based containers offer. And so we wanted to enable people to use the awesome Kubernetes orchestration capabilities that they have come to love, but to also use that to orchestrate some applications or capabilities that are only available on Windows.

So we put together what's called a Kubernetes Enhancement Proposal process, or a KEP, for short. And we said that we're going to use these KEPs to describe exactly what the criteria are to call something alpha, or beta, or stable. And so the Windows feature allowed us to use a KEP-- or in getting Windows in here, we used the KEP to describe everything that would and would not work for Windows Server containers. That was super huge. And that really, I think, helped us better understand or define what Kubernetes is in that context.

But OK, I've spent most of the time answering your question with just one single stable feature.

CRAIG BOX: Well, let's dig a little bit in to the KEP process there, because this is the first release where there's a new rule. It says, all proposed enhancements for this release must have an associated KEP. So that's a Kubernetes Enhancement Proposal, a one-page document that describes it. What has the process been like of A, getting engineers on-board with using that, and then B, building something based on these documents?

AARON CRICKENBERGER: It is a process of continued improvement. So it is by no means done, but it honestly required a lot of talking, and saying the same thing over and over to the same people or to different people, as is often the case when it comes to things that involve communication and process changes. But by and large, everybody was pretty much on-board with this.

There was a little bit of confusion, though, over how high the bar would be set and how rigorously or rigidly we would be enforcing these criteria. And that's where I feel like we have room to iterate and improve on. But we have collectively agreed that, yeah, we do like having all of the information about a particular enhancement in one place. Right?

Like the way the world used to operate before is we would throw around Google Docs, that were these design proposals, and then we'd comment on those a bunch. And then eventually, those were turned into markdown files. And those would end up in the community repo,

And then we'd have a bunch of associated issues that talked about that. And then maybe somebody would open up another issue that they'd call an umbrella issue. And then a bunch of comments would be put there. And then there's lots of discussion that goes on in the PRs. There's like seven different things that I just rattled off there.

So KEPs are about focusing all of the discussion about the design and implementation and reasoning behind enhancements in one single place. And I think there, we are fully on board. Do we have room to improve? Absolutely. Humans are involved, and it's a messy process. We could definitely find places to automate this better, structure it better. And I look forward to seeing those improvements happen.

You know, I think another one of the big things was a lot of these KEPs were mired across three different SIGs. There was sort of SIG architecture who had the technical vision for these. There was SIG PM, who-- you know, pick your P of choice-- product, project, process, program, people who are better about how to shepherd things forward, and then SIG release, who just wanted to figure out, what's landing in the release, and why, and how, and why is it important? And so taking the responsibilities across all of those three SIGs and putting it in the right place, which is SIG PM, I think really will help us iterate properly, moving forward.

CRAIG BOX: The other change in this release is that there is no code slush. What's a code slush, and why don't we have one anymore?

AARON CRICKENBERGER: That's a really good question. I had 10 different people ask me that question over the past couple of months, quarters, years. Take your pick. And so I finally decided, if nobody knows what a code slush is, why do we even have it?

CRAIG BOX: It's like a thawed freeze, but possibly with sugar?

AARON CRICKENBERGER: [LAUGHING] So code slush is about-- we want to slow the rate of change prior to code freeze. Like, let's accept code freeze as this big deadline where nothing's going to happen after a code freeze.

So while I really want to assume and aspire to live in a world where developers are super productive, and start their changes early, and get them done when they're done, today, I happen to live in a world where developers are driven by deadlines. And they get distracted. And there's other stuff going on. And then suddenly, they realize there's a code freeze ahead of them.

And this wonderful feature that they've been thinking about implementing over the past two months, they now have to get done in two weeks. And so suddenly, all sorts of code starts to fly in super fast and super quickly. And OK, that's great. I love empowering people to be productive.

But what we don't want to have happen is somebody decide to land some massive feature or enhancement that changes absolutely everything. Or maybe they decided they want to refactor the world. And if they do that, then they make everybody else's life super difficult because of merge conflicts and rebases. Or maybe all of the test signal that we had kind of grown accustomed to and gotten used to completely changes.

So code slush was about reminding people, hey, don't be jerks. Be kind of responsible. Please try not to land anything super huge at the last minute. But the way that we enforced this was with, like, make sure your PR has a milestone. And make sure that it has priority critical urgent. In times past, we were like, make sure there is a label called status approved for milestone.

We were like, what do all these things even mean? People became obsessed with all the labels, and the milestones, and the process. And they never really paid attention to why we're asking people to pay attention to the fact that code freeze was coming soon.

ADAM GLICK: Process for process, they could start to build on top of each other. You mentioned that there is a number of other things in the release. Do you want to talk about some of the other pieces that are in there?

AARON CRICKENBERGER: Sure. I think two of the other stable features that I believe other people will find to be exciting are runtime gates and Pod priority and preemption. So runtime gates are where-- today, Pods have the concept of liveliness and readiness. So a live Pod has an application running in it, but it might not be ready to do anything. And so when a Pod is ready, that means it's ready to receive traffic.

So if you're thinking of some big application that's scaled out everywhere, you want to make sure your Pods are only handling traffic when they're good and ready to do so. But prior to 1.14, the only ways you could verify that were by using either TCP probes, HTTP probes, or exact probes. So either make sure that ports are open inside of the container, or run a command inside of the container and see what that command says.

And then you can definitely customize a fair amount there, but that requires that you put all of that information inside of the Pod. And it might be really useful for some cluster operators to signify some more overarching concerns that they have before a Pod could be ready. So just-- I don't know-- make sure a Pod has registered with some other system to make sure that it is authorized to serve traffic, or something of that nature. And so Pod readiness gates allow that sort of capability to happen-- to transparently extend the conditions that you use to figure out whether a Pod is ready for traffic. We believe this will enable more sophisticated orchestration and deployment mechanisms for people who are trying to manage their applications and services.

Pod priority and preemption, I feel like, will be more interesting to consumers who like to attempt to oversubscribe their Kubernetes clusters. So instead of assuming everything is the same size and is the same priority and first Pods win, you can now say that certain Pods are more important than other Pods so that they get scheduled before other Pods, and maybe even so that they kick out other Pods to make room for the really important Pods.

You could think of it as if you have any super important agents or daemons that have to run on your cluster. Those should always be there. Now, you can describe them as high-priority to make sure that they are definitely always there and always scheduled before anything else is.

ADAM GLICK: Are there any other new features that are in alpha or beta that you're keeping your eye on?

AARON CRICKENBERGER: Yeah. So I feel like, on the beta side of things, a lot of what I am interested in-- if I go back to my theme of maturity, and stability, and defining the core of Kubernetes, I think that the storage SIG has been doing amazing work. They continue to ship out, quarter, after quarter, after quarter, after quarter, new and progressive enhancements to storage-- mostly these days through the CSI, Container Storage Interface project, which is fantastic. It allows you to plug in arbitrary pieces of storage functionality.

They have a number of things related to that that are in beta this time around, such as topology support. So you're going to be able to more accurately express how and where your CSI volumes need to live relative to your application. Block storage support is something I've heard a number of people asking for, as well as the ability to define durable local volumes.

So today, if you want to-- like, let's say you're running a Pod on a node, and you want to make sure it's writing directly to the node's local volumes. And that way, it coulc be super performant. Cool. Give it an emptydir. It'll be fine.

But if you destroy the Pod, then you lose all the data that the Pod wrote. And so again, I go back to the example of maybe it's an agent, and it's writing a bunch of useful, stateful information to disk. And you'd love for the agent to be able to go away and something to replace it and be able to get all of that information off of disk. And so local durable volumes allow you to do that. And you get to do that in the same way that you're used to specifying durable or persistent volumes that are given to you by a cloud provider, for example.

Since I did co-found SIG testing, I think I have to call out a testing feature that I like. It's really tiny and silly, but it has always bugged me that when you try to download the tests, you download something that's over a gigabyte in size. That's the way things used work for Kubernetes back in the old days for Kubernetes client and reverse stuff as well. And we have since broken that up into-- you only need to download the binaries that makes sense for your platform.

So say I'm developing Kubernetes on my MacBook. I probably don't need to download the Linux test binaries, or the Windows test binaries, or the ARM64 test binaries, or the s390x test binaries. Did I mention Kubernetes supports a lot of different architectures?

CRAIG BOX: I hadn't noticed 390 is a supported platform until now.

AARON CRICKENBERGER: It is definitely something that we build binaries for. I'm not sure if we've actually seen a certified conformant Kubernetes that runs on s390, but it is definitely one of the things we build Kubernetes against.

So not having to download an entire gigabyte plus of binaries just to run some tests is super great. I like to live in a world where I don't have to build the tests from scratch. Can I please just run a program that has all the tests? Maybe I can use that to spoke test or sanity test my cluster to make sure that everything is OK. And downloading just the thing that I need is super great.

CRAIG BOX: You're talking about the idea of Kubernetes having a core and the idea of releases and stability. If you think back to Linux distributions maybe even 10 years ago, we didn't care so much about the version number releases of the kernel anymore, but we cared when there was a new DB in a Red Hat release. Do you think we're getting to that point with Kubernetes at the moment?

AARON CRICKENBERGER: I think that is one model that people really hope to see Kubernetes move toward. I'm not sure if it is the model that we will move toward, but I think it is an ongoing discussion. So you know, we've created a working group called WG LTS. I like to call it by its longer name-- WG to LTS, or not to LTS. What does LTS even mean? What are we trying to release and support?

Because I think that when people think about distributions, they do naturally gravitate towards some distributions have higher velocity release cadences, and others have slower release cadences. And that's cool and great for people who want to live on a piece of software that never ever changes. But those of us who run software at scale find that you can't actually prevent change from happening. There will always be pieces of your infrastructure, or your environment, or your software, that are not under your control.

And so anything we can do to achieve what I like to call a dynamic stability is probably better for everybody involved. Make the cost of change as low as you possibly can. Make the pain of changing and upgrade as low as you possibly can, and accept that everything will always be changing all the time.

So yeah. Maybe that's where Linux lives, where the Kernel is always changing. And you can either care about that, or not. And you can go with a distribution that is super up-to-date with the Linux Kernel, or maybe has a slightly longer upgrade cadence. But I think it's about enabling both of those options. Because I think if we try to live in a world where there are only distributions and nothing else, that's going to actually harm everybody in the long term and maybe bring us away from all of these cloud-native ideals that we have, trying to accept change as a constant.

ADAM GLICK: We can't let you go without talking about the Beard. What is SIG Beard, and how critical was it in you becoming the 1.14 release manager?

AARON CRICKENBERGER: I feel like it's a new requirement for all release leads to be a member of SIG Beard. So SIG Beard happened because, one day, I realized I had gotten lazy, and I had this just ginormous and magnificent beard. It was really flattering to have Brendan Burns up on stage at KubeCon Seattle compliment my beard in front of an audience of thousands of people. I cannot tell you what that feels like.

But to be serious for a moment, like OK, I'm a dude. I have a beard. There are a lot of dudes who work in tech, and many dudes are bearded. And this is by no means a way of being exclusionary, or calling that out, or anything like that. It was just noticing that while I was on camera, there seemed to be more beard than face at times. And what is that about?

And I had somebody start referring to me as "The Beard" in my company. It turns out they read Neil Stevenson's "Cryptonomicon," If you're familiar with that book at all. It talks about how--

ADAM GLICK: It's a great book.

AARON CRICKENBERGER: Yeah. You have the beard, and you have the suit. And so the suit is the person who's responsible for doing all the talking. And the beard is responsible for doing all the walking. And I guess I have gained a reputation for doing an awful lot of walking and showing up in an awful lot of places. And so I thought I would embrace that.

When I showed up to Google my first day at work where I was looking for the name tag that shows what desk is mine, and my name tag was SIG Beard. And I don't know who did it, but I was like, all right, I'm running with it. And so I referred to myself as "Aaron of SIG Beard" from then on.

And so to me, the beard is not so much about being bearded on my face, but being bearded at heart-- being welcoming, being fun, embracing this community for all of the awesomeness that it has, and encouraging other people to do the same. So in that regard, I would like to see more people be members of SIG Beard. I'm trying to figure out ways to make that happen. And yeah, it's great.

CRAIG BOX: All right, Aaron. Congratulations on the release. And thank you so much for joining us today.

AARON CRICKENBERGER: Thanks a bunch. It's been a blast.

CRAIG BOX: You can find Aaron on Twitter @spiffXP-- S-P-I-F-F XP-- and find Kubernetes 1.14 wherever your favorite distribution is assembled.

[MUSIC PLAYING]

Thanks for listening. As always, if you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter @kubernetespod, or reach us by email at kubernetespodcast@google.com.

ADAM GLICK: You can also check out our website at kubernetespodcast.com for show notes and full transcripts of each episode. Until next time, take care.

CRAIG BOX: See you next week.

[MUSIC PLAYING]