ADAM GLICK: Hi welcome to the Kubernetes Podcast from Google. I'm Adam Glick.

CRAIG BOX: And I'm Craig Box.

[MUSIC PLAYING]

ADAM GLICK: So it's been an eventful week. I've completed most of the first layer of Westeros. I know you're all very concerned about how that's coming together.

CRAIG BOX: I had almost forgotten to ask for the jigsaw puzzle update.

ADAM GLICK: It's going much better than the 5k update that I mentioned a while back. So turns out that-- we'll focus on those positive things. I also noticed on the box of baby wipes that we have, it says that it's 99% naturally derived. Which made me think that, since basically everything that's on the earth is naturally derived in some way, shape, or form, that somehow every box of baby wipes is 1% space dust. Which I just thought was amazing. They're really kind of underselling that in the wrong way. And that was just the highlight of my week, noticing that.

CRAIG BOX: First of all, fantastic segue. On the topic of what things are made of, it's always been one of those things that scientists say, people talk about things being organic. And organic means carbon based. So pretty much everything, all life is organic, or all things organic. I'm not sure if that applies here. But a pack of baby wipes that I've seen suggested that it contains 99.9% water and a drop of fruit extract. And that makes it sound a little bit more like homeopathic medicine to me. Like, isn't there meant to be some sort of paper, or some sort of wiping material in the process as well?

ADAM GLICK: I'd be very happy with actually like a baby bidet. And it's just you plug a hose to the side of the container and pump it up and just spray away. I think that we've got an idea there.

CRAIG BOX: You say that. But the situations that you probably want to use baby wipes, you might want something a little bit more of a delicate touch. Wouldn't you think?

ADAM GLICK: Perhaps. Shall we get to the news?

CRAIG BOX: Let's get to the news.

[MUSIC PLAYING]

Red Hat hosted their virtual summit experience last week, attracting over 80,000 people. Announcements included OpenShift 4.4 built on Kubernetes 1.17, which brings OpenShift Serverless, built on Knative, to general availability. The release also adds OpenShift virtualization, derived from the KubeVirt open source project, to preview.

Red Hat also preannounced a new management solution, Red Hat advanced cluster management for Kubernetes. Built on technology by IBM, ACM for K will provide the single pane of glass for managing OpenShift clusters at scale, promising policy driven governance and application lifecycle management when available.

ADAM GLICK: Red Hat partners also made a number of announcements. Microsoft and Red Hat's turnkey OpenShift on Azure has now been updated to version 4 after being on version 3.11 since its launch, one day before OpenShift V4 was released last year. OpenShift 4.3 has just been announced as available on IBM power systems for those that want to run Kubernetes on mainframes. And finally, Red Hat announced a partner software marketplace operated by IBM and a variety of free training courses and extended product life cycles.

CRAIG BOX: Microsoft announced the general availability of three features on Azure Kubernetes service-- private clusters, managed identities, and Windows Server containers. Private clusters live on a private network, never on the internet. And managed identities allow you to interact securely with other Azure services without having to manage service principles or rotate credentials. Windows Server containers, which we covered on episode 70, are now GA on AKS as well as on GKE.

ADAM GLICK: Ingress for Anthos is now generally available. This feature provides a Kubernetes native interface to deploy Ingress resources for internet traffic across multiple Anthos GKE clusters in multiple regions. It provides a global and stable Anycast virtual IP independent of cluster back ends, which can be used to provide multi region and multi cluster high availability, low latency serving of traffic to the closest cluster, and intelligent traffic management across many clusters.

Ingress for Anthos is available as part of Anthos on GCP, which can be purchased on an hourly per VCPU basis or as a long term subscription. Ingress for Anthos is free until August 1. Standard pricing applies to traffic and load balancers created through the Ingress. Data science company Kaggle, a division of Google Cloud, wrote a post on how they use Ingress for Anthos to manage a GRPC application across many clusters.

CRAIG BOX: Google Cloud would like to invite you to explore Anthos with a new sample deployment. Available from the Google Cloud Marketplace, you can now launch a test environment with multiple clusters and various Anthos components. It also includes a sample microservices application that runs in the multi cluster environment so you can explore all of Anthos's advanced capabilities.

This announcement also includes the news that Anthos on GCP is free of charge for non-production use, up to 100 VCPUs per month through June the 30th. This promotion covers Anthos licensing, but you are still responsible for the cost of underlying compute, network, and storage resources.

ADAM GLICK: Helm has officially graduated from the CNCF, becoming the 10th project to reach that milestone. If you'd like to learn more about Helm, just keep listening to the interview in this episode.

CRAIG BOX: The Deis team, authors of Helm, has also been working on writing Kubernetes extensions in Rust. And they talk about the security implications in the post the Microsoft security response entered this week. Engineer Tyler Thomas writes that as the team wrote code for managing web assembly modules, the Rust compiler caught cases that would have resulted in unsafe or concurrent data access problems.

It also prevented them from using objects that could have already been accessed by previous parts of the code, cutting off potential security bugs before that could be attacked. A Helm bug that would have been caught by the Rust tool chain was offered as a comparison. Downsides of Rust include the lack of a garbage collector and a hard learning curve.

ADAM GLICK: The Vitess project, the subject of episode 81, has released version 6.0 as generally available. In the three months since their previous release, the project has added improved SQL support, switched to using Kubernetes as the topology service, and made the V replication feature generally available.

CRAIG BOX: If you prefer NoSQL, version 2.0 of the Couchbase autonomous operator is out. Major changes include a new custom resource model, decoupling the single Couchbase cluster resource from 1.0 into many constituent parts, which can have RBAC rules applied to them separately. It also adds monitoring through Prometheus and scheduled backups.

ADAM GLICK: Also changing the resource model is Kong. In this week's release of version 0.8 of their API gateway integration for Kubernetes, they've changed their plug-in CRD to work cluster wide and added support for acting as an Ingress for Knative.

CRAIG BOX: Tern, with an E, is a container inspection tool originally written by VMware and now governed by the Automated Compliance Tooling workgroup at the Linux Foundation. The project has just released version 2.0, introducing a feature called Dockerfile lock. This can take a Dockerfile and, before building a container with it, expand out things like environment variables and the latest tag. That way, you will be able to audit exactly what the state was at the time the package was built. The release also adds integration with the Scancode tool to detect licenses and copyrights of containers.

ADAM GLICK: With the Dutch government's banning of all events until at least September, KubeCon EU 2020 has officially gone virtual. It will now be held from August 17 to the 20th, and the CNCF are currently working through the logistics of the event. People who paid to attend in person will receive a refund, minus a $75 virtual event fee. And the event will be opened up to additional attendees in the future.

CRAIG BOX: Alcide has continued their periodic blogs comparing the Kubernetes services from Google Cloud, Azure, and AWS. Their latest blog does a nice job of comparing multiple factors of hosted Kubernetes services. In particular, they call out Google and Azure as having the most up to date versions available, while Google won the upgrades, maintenance, scaling, and performance categories. Finally, AWS got top marks in security due to their requirement for pod security policies.

ADAM GLICK: Google Cloud is continuing their deep dive blog series around Anthos. This week, we see a dive into Anthos Service Mesh and its ability to manage across environments, provide a visual map, deliver automatic mTLS with certificate management, and a call out to upcoming VM-based service management.

CRAIG BOX: Finally, analyst Enrico Signoretti has posted a GigaOm Radar for data storage for Kubernetes. The Radar, a circular infographic which can best be described as the lovechild of a magic quadrant and a dart board, weighs up features versus platforms and innovation versus maturity. The full report is available to subscribers, but you can see that their leaders are Portworx, Maya Data, and Diamante. This follows a recent Radar for hosted Kubernetes solutions, where their identified leaders were Google, Microsoft, and Platform9, and Federated Kubernetes, where the leaders were Google and Rancher.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

ADAM GLICK: Matt Butcher is a Principal Software Development Engineer at Microsoft, and the creator and core maintainer of the Helm project. Welcome to the show, Matt.

MATT BUTCHER: Thanks!

CRAIG BOX: Your bio says you have a PhD in philosophy-- but isn't that what the Ph stands for?

MATT BUTCHER: Yeah, it's sort of redundant. Got a Doctor of Philosophy in philosophy. In fact, that's what they actually print on it, on the piece of whatever parchment.

CRAIG BOX: It's very deep.

MATT BUTCHER: Yeah, it feels like it. It's kind of meta, right?

CRAIG BOX: How do you get from philosophy to cloud native software?

MATT BUTCHER: [LAUGHS] You know, that's the kind of question where you're supposed to have this really good story. But the fact of the matter is, I had been doing software development since I was about 16, professionally, and basically always wanted to be a philosopher. And so, from high school, went to college, and paid my way through by writing really bad web pages and stuff like that. Made it through that, went on to grad school. Paid my way through grad school by writing more web pages and more software. And it kind of incrementally-- more and more challenging at the end.

Got through all of that, and academia just was not my thing. It moved at a very slow pace, it was kind of isolating, and at the end of the day, I went, I think I might actually like software development more than being a professor. And so that relaunched me back into this as a career choice.

I like to tell people that I feel like philosophy prepared me for becoming the kind of engineer I am today, because a lot of philosophy is just learning two things-- how to think about systems as complete entities and look and try and understand how all the things fit together. And second, logic is a core discipline in philosophy. And so when you pair those two together, you end up taking that kind of approach to problem solving.

And I will never be the one to optimize a sorting algorithm. And thankfully, I work with a lot of brilliant engineers who are great at all of that stuff. The thing I hope that I bring to the table from the philosophy background is being able to say, look, we've got this big system that's kind of evolving around us. It looks like there are some open spaces here, and here. How can we start to fit in and connect those kinds of things?

Which, actually, is how we stumbled on the idea for Helm, was by doing an exercise similar to that. And saying, this Kubernetes thing, how do we want to conceptualize it? How do we see Kubernetes filling a gap, and how do we articulate to people what it's doing? And at the time, we said, Kubernetes is like an operating system. Where my local operating system is the place that I'm executing my native binary executables and doing stuff, Kubernetes opened up for us this idea that we could take our Docker containers and run them in a platform, analogously to the way we manage our binaries on bare metal servers, or our binaries on our local workstations.

And so, walking through that exercise, we went, well, what are the things that are pain points about using Kubernetes in this way? And one of them was, it's awfully hard to install stuff on Kubernetes when you have to write three dozen YAML manifests to get the whole application done. Well, analogously, how did they do that on Linux, or Mac, or Windows? Well, they have these installers that manage all the various artifacts, and put them in the right place, and give you some cues as to what you need to be able to provide as input. And that was how we ended up going with the nascent idea that became Helm.

CRAIG BOX: You said that you didn't have a pre-prepared answer to how you get from philosophy to software, but it does sound like that story has a little bit to it, there.

MATT BUTCHER: [LAUGHS] Yeah. I should also say, there are really three things you learn in philosophy that are applicable. One is the architectural view, the other is logic, and the third one is that you can pretty much, if you talk long enough, turn even the most BS answer into something that sounds like it might actually have been well thought out.

ADAM GLICK: By the way, if anyone is listening from O'Reilly, or Simon and Schuster, I believe the book rights are still available for the story.

MATT BUTCHER: [LAUGHS]

ADAM GLICK: Congratulations on the graduation of Helm. It's probably well known to many of the people in the community, but how would you describe it to someone who is new?

MATT BUTCHER: We describe Helm as the package manager for Kubernetes. The whole idea being, it fills that same kind of role that apt-get does for Debian, or yum for the Red Hat ecosystem, or Homebrew, or Chocolatey, or all these kinds of things. We like to try and express it as, this is the way you get from an empty cluster that's running nothing to having your first couple of applications installed and running there.

ADAM GLICK: So for people of my generation, it's kind of like a floppy disk?

MATT BUTCHER: Oh, I like that. I'm going to ask if we can change our logo to, like, a floppy disk with steering nubs on the side. I don't know anything about nautical terminology, so I think steering nub is what you call the things on the side of a Helm. I am not really sure.

ADAM GLICK: I like to think of it as handlebars, if an octopus were driving.

[LAUGHTER]

MATT BUTCHER: That sounds good, too.

CRAIG BOX: Helm started out as a hackathon project at Deis, where you worked before its acquisition by Microsoft. I hear there were a few different things that came out of that same hackathon?

MATT BUTCHER: We were doing an all-hands meeting. So Deis was part of Engine Yard at the time, and we did this big all-hands thing. Everybody flew out to the office in Boulder, holed up in the Boulderado Hotel, and we just spent two, three days in brainstorming sessions, in presentations from different teams. And then we did a hackathon in the late afternoons to chill out and cool off a little bit.

So out of that, that was the time when Deis, which our premier platform at that time was called Deis Workflow-- that was when we committed then that we were going to re-platform and build Deis Workflow on top of Kubernetes. So the ship really turned at that point, from focusing on PaaS to focusing on Kubernetes oriented development.

Then I was asked to explain Kubernetes to everybody at the company after lunch. And I'm going, wait, so, we're talking like, everybody, like all the engineers? And they said, no, like finance, marketing, everybody. You've got to explain to everybody, if we're going to pivot, what this thing is.

ADAM GLICK: They give you one sentence to do it in? Just a quick, one sentence, what is it?

MATT BUTCHER: Yeah. More than one sentence, but less than a half an hour. So I took a bunch of my kid's stuffed animals, and took pictures of them, and wrote this dumb children's-y story kind of thing called "The Illustrated Children's Guide to Kubernetes," telling this story of what Kubernetes was, from the point of a little application who was getting migrated into a Kubernetes cluster.

And it worked. I mean, people did not fall asleep, which was actually the goal of giving a presentation on Kubernetes to people who were not terribly interested in it. But afterward, Karen came up-- Karen Chu is the community manager of Helm and a colleague of mine from the Deis days onward-- and she said, you know, maybe we should see if we could actually make a little children's illustrated book. Wouldn't that be funny to go put that out there? You'd be able to give that away at conferences and stuff like that. And so she found an illustrator, and that kicked off the whole process that became the Children's Illustrated Guide.

And then, again, at that same all hands, a team of three of us sat down and did that exercise of where are the empty spaces in Kubernetes, and ended up with this little demo project called k8splace, K-8-S, k8splace, that showed a little rinky dink package manager thing. We were really proud of it. We won a $75 gift card, which we split three ways. So you know, I won't be buying my private island yet-- unless it's about two centimeters by two centimeters.

So the day after we finished the big meeting, the CEO and Gabe Monroy, the CTO-- Beau and Gabe called me up on the phone. And I'm like, uh oh, what did I do? And they said, so we've been thinking about this overnight, and we think maybe we should try to do that package manager thing as a product. Just one thing, we need a new name. [LAUGHS]

So, about the day after that, we officially kicked it off. Jack Francis, one of the co-inventors with me-- Jack Francis and Rimas [Mocevicius] and I were the three who did it-- Jack and I sat around with a nautical dictionary, because I know nothing about nautical terminology, and just reading each other names back and forth. And in my version of the story, he came up with the name. In his version of the story, we simultaneously said Helm at the same time. So you can pick which of those you like, but I still think it was Jack's idea.

CRAIG BOX: Once Helm was released and became tooling that was in use for a subset of the small Kubernetes community at the time, there were other people who were also looking at packaging in this space. The evolution between Helm 1 and Helm 2, tell us about that story. I remember a meetup that laid the groundwork for what would become Helm 2.

MATT BUTCHER: Right, right. So I think we first announced Helm at the inaugural KubeCon, which would have been late 2015, I think-- late in the year. And announced it at our booths, talked to a bunch of people. At the same time, there was a team at Google who was working on porting the Deployment Manager over to the Kubernetes ecosystem. And we kind of chatted with them here and there over the next month or two. And in January, they invited us to come visit the campus in Seattle.

And so we flew up there, hung out, drank a lot of coffee, and brainstormed about what kinds of things we could do. And this led us to the idea that, if we just kind of did something rarely done in open source projects and merged two projects and just played up to the strengths of each of the two, then we would be able to build something better. I know, right, shock and awe. Merging two projects and not forking things? So we spent about a day kind of white boarding out the design. And then we all kind of went back to our respective places.

We started with the Deployment Manager repo in Kubernetes and just began kind of layering the two together. As I'm sure you could imagine, there were technical hiccups as we got started. But we ended up bringing in Skippbox and then Bitnami, and we ended up with this very well rounded view from all the different developers, ranging from the large enterprise view of Google to startup-y us. Bitnami had been building images for a long time, and Skippbox was way out on the forefront of deploying workloads in Kubernetes. So it ended up really gelling very well in the end, and Helm 2 came out of that particular effort.

At the end of last year, we released-- the days are flying by right now-- released Helm 3. Helm 3 was a long, arduous development process, and I learned a lot during this one. Because the transition from Helm 1 to Helm 2 impacted probably in the neighborhood of about 100 people or less. Kubernetes was very new, and all the people that we were working with at that point were in the move fast, break stuff kind of phase of life.

Now we fast forward a few years, and Kubernetes's uptake is astronomical. I mean, it just boggles my mind every time I see the numbers. I can't comprehend from the however-- the couple hundred people that I met at the first KubeCon, to a conference where I can't even see the back of the room at the last KubeCon North America. And the responsibility weighed very heavily on us that we needed to do a really good job of building something that was going to address the shortcomings of Helm 2, but do it in such a way that people would not have to rewrite a lot, would not have to relearn a lot of the sort of concepts or metaphors that we'd been using all along. So it took us a long time to do it. Consequently, we were very proud when we finally got that release out there.

And one of my favorite things was somebody sent me a message soon after it was released saying, "well, it doesn't really feel like you changed much. You just got rid of Tiller. Doesn't feel like you changed much at all". And, at first, I wanted to say, oh. [GRUMBLES] And then I thought, no, actually that might be the highest form of praise we can actually get, because we rewrote tens of thousands of lines of code.

And we went through everything with Adam, who's the current architect-- Adam Reese is the current architect of Helm for Helm 3-- and he went through the code with a fine tooth comb. And he was just trying to micro adjust things to make sure that nothing had that cascading effect. That is a much harder, much more difficult way to write software than to just say, all right well, we'll nuke this package and start over from the ground up. So to me, it was a big achievement because it reset a foundation that I think Helm 4 will be a much easier transition because we did all the big reorganization during the Helm 3 time frame.

ADAM GLICK: So you're saying that Helm 4 might come in a little bit shorter cycle time than Helm 3?

MATT BUTCHER: Yes, that is our goal. One of the key learnings from Helm 2 was that we really, really emphasized not breaking compatibility from Helm 2.0 to Helm 2.16, and still going, 2.16.5 or something is our current release.

CRAIG BOX: Well, that's because you work at Microsoft.

MATT BUTCHER: [LAUGHS] Yeah, I do now. Like that video of starting with Windows 3.1 and upgrading, upgrading, upgrading until you end up at the-- that is kind of what we wanted with Helm. And we have not succeeded as well as other projects have, but we were very dedicated to that. But the lesson that we learned was that, while that is good, you don't want to fall too far behind the cutting edge. And Kubernetes was moving very, very quickly, and there were things that felt more and more like patches and add-ons that should have been more at the core.

When Helm was created, there was no such thing as ThirdPartyResources, which was the precursor to CRDs-- CustomResourceDefinitions. And so we had to glue on TPR support, then rip that out and glue on CRD support, except we couldn't get rid of TPR support so it was still kind of in there, and consequently it never really felt right. If we had been a little more reactive, and iterated maybe a little bit faster, I think it would've been a smoother transition for users to the newer technology stuff, as RBAC kind of landed and got concrete, and CRDs landed and got concrete, it would have felt like a more natural experience. So the takeaway is, Helm 3 was a big reset. Helm 4 will be more of an incremental change where we're just kind of keeping pace with the evolution of the community.

CRAIG BOX: Helm uses charts, which manifest files that you can template in order to make changes to a predefined environment to reflect what it is that you have to install locally. People can download those charts from a repository, or they can create their own. A lot of software has adopted Helm as the packaging format-- or the descriptive format, perhaps, more than the package I'd say, to describe the software installation. Do you see a pattern in how people use Helm? Do you think that more people are using it to download third party software that already exists, open source packages, versus creating their own things? Or do you find across the board that it's everything to everyone?

MATT BUTCHER: There are a couple of very interesting patterns that we have observed. When we initially built Helm, we were knee deep in over architecting things. Actually, this would've been right at the beginning of the Helm 2 thing. We were kind of knee deep in over architecting everything, which of course, as engineers, is our tendency.

And I remember, we're kind of at the whiteboard, we're scribbling stuff out, and Michelle Noorali-- who is another one of the early contributors, like day two contributors to Helm-- kind of pauses the conversation. And she says, is what we're trying to build designed to handle the most expert case, or are we trying to make something that's going to help the newcomer to Kubernetes? And we kind of paused.

And she says, what are the other successful projects that we like? And we started listing them. Rails, obviously apt-get, which was a huge part of my inspiration. And then she says, so the common denominator I see here is that they got people working very quickly on the thing they were trying to learn, or experiment with, or use. And we all went, yeah.

So we ended up writing this phrase on the whiteboard, right at the top, that said something like, we want the users' experience to be five minutes to endorphins-- zero to endorphins in five minutes or less. So they get their Kubernetes cluster stood up, and within five minutes, they have their first application. That became our guiding principle for how we wanted to do Helm 2.

So that was story number one. Story number two, which was the former story number one, was that we wanted an experience that was like apt-get. Where you go out there, and you find the packages you want, and you install them very easily, like Homebrew, or Chocolatey, or any of those things. And so, when we came out of the gate, our assumption was that one group of people would build packages, and another group of people would consume the packages.

The surprising thing has been that there has been a big movement to do a lot of internal enterprise stuff, where people are writing their packages and they are the sole consumer of these packages. And we hear lots of stories about how people are managing all of their deployments using Helm charts, which is a surprise to us. A pleasant one, but it was a surprise to us because that was not the story we came into it thinking. And it has caused us to make some subtle shifts here and there in the way Helm works, in order to better facilitate the kind of continual deployment model that a lot of organizations use.

But I think from the beginning, the kind of core idea was that what we really wanted was to provide people an easy entry point to Kubernetes that allowed them to back into understanding. So I want to learn how Kubernetes works. I install the WordPress chart. Oh, great, now I have WordPress running in Kubernetes. And now I can go poking around at all that stuff that the Helm chart just installed, and try and understand. What's this deployment thing? What's this service thing? And that was what we liked about technologies like Rails, is that you experience success rapidly, but then success merely opened the door to be able to learn about the underlying technology that you were using and relying upon.

CRAIG BOX: With a repository of charts published online, there is the chance that people could maliciously edit those charts. There is now a software supply chain problem. How does Helm deal with that, either at the charts level or at the application level?

MATT BUTCHER: That is a good question, and one that we spend a lot of time on. The original design for that has been that-- even if you go back to the early days of Helm 2, there's a provenance model in there. One of the really fruitful things that came out of our early collaborations with Google was that we all kind of combined to talk about different ways we do software supply chain.

And so that one was based largely on a sort of GPG model, where you have trusted signing keys, you sign your charts, you only use charts that are signed by a trusted key holder, and the system in general was pretty much on par with what you would see from just about any package manager out there. The tragic thing about it, and as we learn more about all of our peers in similar package managing industries, is that way too many people disable those features, don't use those features, find them overly burdensome.

CRAIG BOX: It's like SELinux.

MATT BUTCHER: Right, yeah. And so we have been talking a lot about what the next version of this is going to look like. And we have been working really closely with the TUF and in-toto teams, other CNCF projects. In-toto is fascinating, and is, I think, going to end up being the future of open source software supply chain security. And we have been working a lot-- so in Helm 3 we introduced a pilot where, as one of our experimental features, you can store Helm charts in OCI registries, Docker registries. And a huge part of that is sort of, as the beginning of our testing grounds for how we're going to do chart security moving forward so that we can establish full, end-to-end chain of trust.

And in-toto-- if you haven't heard of it, in-toto is this great technology that provides an outline of what your process from source code to artifact is. And you build up this kind of reproducible sequence of events with signatures all along the way based on who did each individual step or what automated component did an individual step. And then you can sign this whole bundle. And then, at any given point, the person who is so inclined can rerun the entire software supply chain. And they can verify that it was compiled with the right compiler, that it was built with the right image building tools, that it was pushed to the repo by a trusted individual.

And this gives us a level of security in package management that just kind of completely blows away the idea that you sign something and you release the signature along with the artifact and hope that nowhere else in there did somebody maliciously tamper with a Docker container that the developer didn't notice, and things like that. So we think that's the future of it. Now, in the present, we still support the provenance model that we've had since Helm 2. And we're just starting to push into this new territory, both in Helm and in one of Helm's sister projects, CNAB.

CRAIG BOX: If you reduce the problem of installing applications to, here's the set of stuff you have to install, and say here's the set of parameters, and you squint at the model, you can kind of say that Helm charts and templates and the operator pattern are somewhat similar. You can define, here's the set of things I want to install and configure. And the mechanics are slightly different.

This is a topic which obviously close to your heart-- you wrote a blog post about recently. A lot of people see this as an either/or, like there should be a winner, there can be only one. But you have a more collaborative suggestion. How do you think these two things will coexist?

MATT BUTCHER: I think I hinted at it a little bit earlier, when I said that one of the things that surprised us about Helm usage was that we assumed people would be using this as, you have people that produced packages, people that consumed packages, and that would be kind of the ecosystem. And what surprised us was seeing a lot of internal usage of charts as a tool in deployment. And that pushed Helm to make some small changes to better handle deployments.

But there's been a lot of pressure to push Helm from package manager into a deployment tool. And that to me is exactly the point where we can zero in and say, OK, clearly, Helm is not a deployment tool now. It is a piece in a deployment tool toolkit, a piece in the deployment tool box of equipment. But that's not what it does well. So put that on hold for a second.

Operators. Operators, the core idea when Brandon Phillips introduced the idea was, we can now encapsulated the operational knowledge of how to keep something running, and how to carefully manage its lifecycle. We can encapsulate that in code that can run alongside of it, inside of the cluster. Well, if you look at those two things, you start to say, oh. Oh, I see. So if we combine these two ideas together, we're actually starting to build the kind of thing that's going to be useful to enterprises large and small, who want to be able to say, I want to easily deploy this thing, I want to easily upgrade it, I want to easily delete it. OK, Helm can do that.

But I also want to make sure, that during its lifecycle, when something critical happens, it knows enough about its environment to say, OK, I can try this repair strategy. Or it knows enough about its environment to say, OK, I need to do a database update on this particular thing, or update the schema of something, or stop relying on that cache.

So if we can figure out a good way to combine the core philosophy that Brandon talked about in his operators blog and that CoreOS really pioneered, and combine that with the idea of an installable, upgradable, deletable artifact that can be managed just like any other package, that's where I think we're going to get a really powerful and compelling solution that says here is where Kubernetes can really shine. No more operators having to retype the same couple dozen commands into their terminals anymore-- we can delegate that into the code level and operators can focus on all the million other things that they really would rather be tackling.

CRAIG BOX: The human operator.

MATT BUTCHER: Yeah, the human operator no longer has to manage all of the day to day rerunning of the same shell commands over and over again. They can start to focus on the millions of other things that they really want to be spending their time on and allow the codified operator, the operator toolkit, to take care of those kinds of day to day operations. And they can rely on those things to handle the 80% cases. And then the dev side of it becomes generating an operator that can do these kinds of things. And the operations side really dwindles down to the more emergency stuff or the 20% cases. And then, they can start focusing on preparing the next versions of the release, and really refining how the SRE lifecycle is going to go from there.

CRAIG BOX: Delivering business value.

MATT BUTCHER: Oh, yeah. There we go. If you have a C in your title, you might prefer "delivering business value". [LAUGHS] But yeah.

CRAIG BOX: Helm started out as a sub-project of Kubernetes, and then it moved to a top level project in the CNCF. What was the thought process there? Was the idea that it would do more than Kubernetes, that it would need to be separate in some fashion?

MATT BUTCHER: We had talked about it for a long, long time, how things should work. And I think the watershed moment for me was, at the very first Helm Summit, Brian Grant, the primary architect of Kubernetes at the time-- and I think still today-- Brian Grant came to the Helm Summit. And I got up early one morning, went to grab some coffee. Brian went to the same coffee shop. We walked in the door at about the same time, and we just sat down and talked. It was like, great to see you Brian, great to see you Matt. This summit thing's going great, yeah.

And then we just kind of drove into Brian's vision of how Kubernetes would work, what the constraints of the project were. We talked about Unix tooling, and what we liked about the way the Unix architecture and Unix philosophy had done for the world of computing. And the conversation sort of naturally led us to talk about, do we want to turn Kubernetes into the gigantic, behemoth project that has all the little things in it, or do we want to start talking about Kubernetes as one kind of core technology, and growing other technologies along the periphery of Kubernetes?

And of course, as you've seen, the answer really ended up being, we want to grow core Kubernetes as one thing, and then start breaking tooling out in different ways. Helm was an easy, obvious candidate for breaking out of that. And Bryan really saw it as a chance to take something that was high enough profile to have its own summit, and say, look-- going back to our discussion of deployment manager-- we know when it's right to merge two projects together. But we also know when it's right to keep two projects separate, in that kind of Unix philosophy style of saying, Helm is going to be really good at doing this one thing. We don't want to glom all of that into Kubernetes and then inflate the burden of maintaining Kubernetes. And indicate to all the other people who want to do things differently that they're not allowed to do it because Helm is part of Kubernetes.

One nice thing about working in open source is the term competition doesn't really apply the same way as it would if we had a financial stake in the success of Helm. And so giving the signal that Helm was a thing in its own right that welcomed other approaches to deploying things onto Kubernetes, it was not the official and only way to deploy things onto Kubernetes, that worked well for Kubernetes' overall story.

For us, it worked well because it kind of said, OK, here you've got freedom to do things in your own way. It greatly simplified the governance model. It was getting very complicated to try and have a governance model that could handle everything from core Kubernetes kubelet stuff all the way up to package manager-y stuff. So it greatly simplified the governance model as well.

And I think, looking back on that, that coffee with Brian was just a fabulous experience. Because it was a chance to really sit down and almost accidentally say, let's ponder whether we want to make a big decision about this. And let's be really intentional about talking through the pros and cons and looking at what is best for the community as a whole, for the people who contribute, for the people who have the governance burden. And I look back on that, and it was also a very, very good coffee.

CRAIG BOX: Glad to hear that.

MATT BUTCHER: So, you know, the combination of that, it was a great day. [LAUGHS]

CRAIG BOX: And now, of course, Helm has become the 10th project to graduate the CNCF. What does it mean to be graduated?

MATT BUTCHER: The thing I'm most proud of about the graduation, the thing that is the most meaningful to me, again, given this kind of historical perspective I have on it, is that graduation signifies for us that we have successfully taken this project, and we have run the kind of marathon that, even now, I'm shocked to see. I thought I was running a sprint. Thought I had to make it a hundred meters.

Looking back on this and seeing the growth, seeing the community evolve, jumping over all of the hurdles of security audits, and writing out the full security assurance cases, and auditing all the code for reliability, maintainability, things like that-- we finished a really remarkable milestone. One that says volumes about the community, one that says volumes about CNCF's ability to identify and foster a promising project all the way through the point where they can say-- effectively, what graduation is, is a statement that we have now reached the level of stability where we believe this is ready for everyone from the startup who wants to move fast and break things, to the enterprise who has to be very careful and very risk-averse when they make their selections about technology.

And that, I think, is what makes me proudest. Is that, again, it looked like a sprint. It looked like we were trying to prove to the world that this was a little thing that could be done. And in the end, we look back and can't even-- can just barely see the starting line way, way back there. And instead, feel like we can say with a high degree of confidence, we built a thing that is ready for a bank to use, ready for governments to use. And that's something we very much take pride in.

CRAIG BOX: You have your logo on a coffee cup.

MATT BUTCHER: [LAUGHS] My dream has come true. Now all I need is Phippy on socks, and I can retire and buy my two centimeter by two centimeter island.

CRAIG BOX: I do have a couple of pairs of Deis socks. I am also in possession of-- and I have no idea how rare it is, but I have one of the original, first edition printed copies of your children's book.

MATT BUTCHER: Really? Really, I am impressed.

CRAIG BOX: I do. It was a gift from Gabe, along with a satchel, with a little Tide pen for clearing out stains on clothing. I'm not 100% sure what the Tide pen had to do with everything, but the book definitely made sense.

MATT BUTCHER: We had some hilarious stories about the Tide pens, the socks, and things like that because we were giving those away at conferences largely when the things that were being given away were t-shirts and stickers. And people would come up to us and say, I had forgotten to pack socks on this trip, and so I'm so thankful that you've got socks. And can I take a pair for tomorrow as well? [LAUGHS] I just spilled something on my shirt, can I grab one of those Tide pens real quick? You'd be surprised at the uptake on this.

I do, incidentally, have one of the very, very few hardcover editions of the original "Illustrated Children's Guide."

CRAIG BOX: As well you should.

MATT BUTCHER: Signed by the author.

[LAUGHTER]

ADAM GLICK: What I really want to know, is why is there so much love for Phippy and not for Captain Kube.

MATT BUTCHER: I don't know. I mean, come on.

ADAM GLICK: Is it just because people hate owls? Is it the scowl on his face?

MATT BUTCHER: I think it's the kind of perma-scowl that he's got. He's the kind of guy that you kind of keep at a distance. The guy you check in with every Tuesday, but you don't invite him out for drinks after work.

CRAIG BOX: In fairness to you, you wrote the book, but you didn't illustrate it. Did you have any oversight of the illustrations?

MATT BUTCHER: Accidentally, yes. And then, directly, yes.

CRAIG BOX: So we will blame you, then.

MATT BUTCHER: The original version was done as a Google slideshow with pictures of my daughter's stuffed animals. And then the artist and Karen, Bailey and Karen, went with that as the inspiration. And then Bailey did these super cute illustrations that are all sort of roughly inspired by it. And then I only got veto power at the end to say, no.

CRAIG BOX: So do you have a children's toy with a scowl on its face?

MATT BUTCHER: Apparently, it was a little plastic owl, and the angle I took the picture at apparently made it look like it was scowling.

CRAIG BOX: Helm obviously started while you were at Deis, which was acquired by Microsoft three years ago to, roughly, the month. Since then, you've been part of the Deis Labs team, which works on a bunch of open source technologies. Helm, as well you mentioned CNAB before, and you've also been working on event programming with technology called Brigade, and recently an announcement about running web assembly on top of Kubernetes with Krustlet. Is there a pattern to these?

MATT BUTCHER: Yes. Our core guiding principle as Deis Labs, which is our name under Microsoft, has been we want to build open source tooling that takes the cloud native technologies as a foundation, and pushes the limit to see what kinds of things we can build in order to enable developers, DevOps, and SREs to build a new class of things. So if you look at Helm, that one kind of self-explanatory, almost.

Brigade was our attempt to bring event-based scripting into the Kubernetes cluster in a kind of way where-- we would call it shell scripting for Kubernetes, where in typical shell scripting you're tying a bunch of binaries together to build something bigger with just Bash and call outs to other things. And we wanted to bring that kind of mentality to the cluster.

Krustlet was our attempt to do two things. To say, hey, there's a big place where Rust can make a dent in the Kubernetes ecosystem for the better. As a systems language, it's phenomenal for that kind of thing. But also, and far more importantly for us, we see WebAssembly as a technology that's really going to burst onto the scene in the cloud native landscape, and enable developers to use a different set of tools to very rapidly produce microservices and other kinds of applications. And so we wanted to experiment and say, so how do we enable those people to be able to test the waters and see if this is a thing they're going to like.

So, I like to think of it as one part practical intuition, what are people going to do, and two parts crazy experimentation, where we kind of throw spaghetti at the wall and see which things stick. And it's been an amazing space to get to work in. Sometimes, we've had failures, some of which never even made it to GitHub because we were too ashamed to post them there. But a lot of times we stumble across things, large and small, that we're quite proud of, and that we are happy to see other people use and find helpful.

CRAIG BOX: Your GitHub handle is technosophos which is derived, I guess, from a Greek word. So you're a little ahead of the curve there. Was that a philosophy thing, or was that a Kubernetes thing?

MATT BUTCHER: Oh, totally. I'm at heart an Aristotle nerd. And Aristotle used the word "techne" to mean artisanship This is the kind of creating that is different from artistic creating, in the sense that you're creating not just for aesthetic value, but also for utility. So techne, and "sophia", of course, is the wisdom. And the idea, an ancient philosophy, is that one pursues wisdom as a goal of life-- the idea that you can really suck the marrow from life when you take that kind of deep approach to it.

And so I wanted to combine those two ideas. Sophos, sophia really coming from the philosophy background, the techne coming from what I really think is awesome about software. Software is this brilliant place where we can create, and we can experiment, and we can talk about beautiful code. But in the very concept of beautiful code is encapsulated that idea of usefulness, and readability, and utility.

CRAIG BOX: So given all of that, and given what everybody else has done, how come you haven't named anything with an obscure Greek word?

MATT BUTCHER: [LAUGHS] OK. Confessional time, I suppose this is. I originally, well prior to working at Deis, had tried naming several projects with obscure Greek words. And someone much older and wiser than me, who had been in the industry a good decade or two longer than I did, said, Matt, I appreciate the subtlety and nuances of the names of your projects, but nobody can pronounce them. And unless you can pronounce it, nobody is going to use it. [LAUGHS]

CRAIG BOX: And that feels like a good place to wrap up. So thank you very much for joining us today, Matt.

MATT BUTCHER: Thank you. Thanks for having me.

CRAIG BOX: You can find Matt on Twitter @technosophos, and you can find the Helm project at helm.sh.

[MUSIC PLAYING]

CRAIG BOX: Thanks for listening. As always, if you've enjoyed the show, please help us spread the word, and tell a friend. If you have any feedback for us, please tweet us @kubernetespod, or email us at kubernetespodcast@google.com.

ADAM GLICK: You can also check out our website at kubernetespodcast.com, where you'll find transcripts and show notes. Until next time, take care.

CRAIG BOX: See you later.

[MUSIC PLAYING]