ADAM GLICK: Hi, and welcome to the Kubernetes Podcast from Google. I'm Adam Glick.

CRAIG BOX: And I'm Craig Box.

[MUSIC PLAYING]

ADAM GLICK: Hey, Craig, what's been going on this week?

CRAIG BOX: Last week, we talked about the impending purchase of GitHub. And it's obviously become real.

ADAM GLICK: Indeed.

CRAIG BOX: It's been interesting to see the response of people who worry about Microsoft based on their opinions from the past. But I would say that-- especially the new CEO, Nat Friedman, who has been a longtime Linux and open source user and company builder, building Ximian, who were the GNOME company, and working on Mono and things like the Xamarin platform based on that for mobile app development, which was eventually acquired by Microsoft. I would say that I'm optimistic about the GitHub platform under his leadership. And it'll be interesting to see where things go.

ADAM GLICK: Yeah, I'm really hopeful that Microsoft will become more prolific in the open source community, and contribute much more to it. And I hope that this is part of an ongoing trend for them to really get more involved. I think that would be great for everybody. So I'm hopeful.

CRAIG BOX: Absolutely. And their involvement with Kubernetes has obviously been a good sign in that direction.

ADAM GLICK: Indeed. Craig, you want to go over the news?

CRAIG BOX: Do I what?

[MUSIC PLAYING]

CRAIG BOX: Regional clusters and now generally available and Google Kubernetes engine. One of the most commonly requested enterprise features, regional clusters create a multi-master, highly available Kubernetes environment that spreads both the control plane and the nodes across multiple zones in the region. This raises the cluster SLA to 99.95%. In addition to the increased availability, regional clusters give you a zero-downtime upgrade experience so that your cluster always available for deployments.

The great news is that regional clusters are free to deploy-- the same as zonal clusters. And in both cases, you only pay for the nodes you run.

Google Cloud also released kubemci which allows you to use the Google Cloud load balancer for multi-cluster environments. kubemci allows you to use a Kubernetes Ingress definition to connect the cloud load balancer to multiple Kubernetes engine clusters running at regions around the world using a single Anycast IP address. This allows you to serve traffic from the closest cluster, taking advantage of GCP's 100-plus points of presence in global network. Kubemci is a command line tool, and a controller-based, multi-cluster Ingress solution based on it is in the works.

The Helm project has moved from a subproject of Kubernetes to an incubation project in the Cloud Native Computing Foundation. Today's Helm project started as a merger between what is now Helm Classic, originally developed by Deis, and the Kubernetes Deployment Manager, built by Google Cloud. This move allows Kubernetes to continue to scope its focus to its core componentry, and acknowledges the growth in the Helm community, with more than 300 contributors to the code and 800 contributors to the chart's repository.

One user of Helm is Etsy, the craft marketplace, who deploy to Google Kubernetes Engine. This week, they published a blog post about their deployment workflow, and you can find a link to that post in the show notes.

Microsoft, after last week acquiring the Atom code editor for $7 and 1/2 billion has this week announced tools for Kubernetes and its old a half-sibling, Visual Studio. While VS Code, the cross-platform editor which is very similar to Atom, has had Kubernetes tools for some time, this is the first such support for the Visual Studio platform, which was first released under that name in 1997. The tools for Visual Studio include support for generating Docker files and Helm charts, as well as integrated deployments to Azure's Kubernetes service.

Speaking of clouds and control planes, Amazon's Elastic Containers Service for Kubernetes, or EKS, is now generally available. EKS is a managed control plane for Kubernetes, as opposed to a fully managed Kubernetes service, as you have to bring, manage, and update your own nodes. Congratulations to Amazon for their launch, and we look forward to seeing contributions from AWS to the upstream Kubernetes community.

Finally, in our first episode, we told you about Sandeep Dinesh's Kubernetes Best Practice blog and video series. The final episode in the series has just been published, coincidentally on upgrading clusters with zero downtime using GKE regional clusters and node pools. You'll find a link to this and all the other episodes in the show notes. That's this week's news.

[MUSIC PLAYING]

ADAM GLICK: Today's guest is Phillip Wittrock, software engineer with Google Cloud. Phillip is one of the creators of the Kustomize project. Welcome, Phillip.

PHILLIP WITTROCK: Hi, thanks for having me.

ADAM GLICK: It's great to have you. For the people listening, can you tell us what is Kustomize, spelled with a K?

PHILLIP WITTROCK: Kustomize is a command line tool that's used with Kubernetes YAML files to help run workloads on a Kubernetes cluster.

ADAM GLICK: And where did you come up with the idea for Kustomize?

PHILLIP WITTROCK: The idea for Kustomize came from Brian Grant. Brian Grant evaluated a number of different options for writing YAML config. And really, the problem Kustomize is trying to solve is how to allow users to more easily write the YAML config that kubectl Apply uses to manage a Kubernetes cluster. And it came from him doing a lot of evaluation of different internal solutions we had inside Google, and coming up with a new approach that we hadn't really seen before.

ADAM GLICK: How is the Kustomize solution different than just using YAML files that people are familiar with today?

PHILLIP WITTROCK: Excellent question. So it tries to be as similar as possible to that workflow. There are a number of points of friction when using just raw YAML files, and this is where Kustomize comes in. One of the first pieces of friction is that there are a number of things that are just cumbersome to do with YAML files, such as creating a secret. The secret requires [INAUDIBLE] encoding some binary file. And so this is an extra step that one would normally need to write some [INAUDIBLE] for that does pre-processing on some binary file before ending up with the YAML files. A human wouldn't write that.

Another sort of UX friction issue is wanting to do meta operations across a bunch of YAML files. So you want all the YAML files to have this set of labels applied to all the objects. Or you want them all to have this annotation, or you want them all to have this particular name prefix, or these sorts of things. So it allows you to manage a directory or a collection full of YAML files, and make sure they all have these specific pieces to them.

The third thing that's different about using Kustomize, and kind of the third point of friction, is reuse, and essentially saying, how do I compose a new solution out of existing YAML files. And I want to vary it by a small piece. So I have my base, my SQL config that everyone uses, but you know what? I want to give it a little more memory, or I want to change this, or I want to change that. How do we make that possible while still seeming like you're just editing YAML files. And so Kustomize provides some solutions for basically saying, take these YAMLs, and then make these small tweaks to them.

ADAM GLICK: Would it be fair to think of it as kind of a meta layer on top of the YAML files to help organize them, and also to help provide certain parameters so you can have a kind of templated use, but not do the same exact deployment to scale up and down, say, for production environments versus dev test environments, and be able to alter that without having to come up with completely separate scripts or collections of YAML files?

PHILLIP WITTROCK: Yeah, I'd say that's fair. It's a meta layer alongside the YAML files next to it.

ADAM GLICK: How is that related to the Kubernetes project overall?

PHILLIP WITTROCK: The Kustomize tool is developed by a number of maintainers in SIG CLI. And that is the set of maintainers that maintain the kubectl tool. And so it's published as a subproject of the SIG CLI group. And we are currently evaluating its success and considering whether it should be moved into the kubectl tool.

ADAM GLICK: Gotcha. In the documentation, it mentions that Kustomize solves several problems using kubectl in a declarative fashion. Is there any reason you didn't fix those problems just directly in kubectl?

PHILLIP WITTROCK: Yeah, we haven't done that yet. I think Kustomize is the way we are hoping to do that. The reason we decide to develop Kustomize as a separate project instead of directly under kubectl-- one, this is somewhat of an experiment both in a process and the solution. The solution is completely novel. It's different than providing a templating mechanism. It's different than providing a new language on top that has special features in it, or something like that.

So because it was kind of a novel approach, and because we wanted to iterate on it quickly, we decided to create a new repo with its own code source and build it there. And this has been a really, actually, great experiment for us. It's been working really well, because what it allows us to do is, one, it allows us to push code and release it whenever we want.

We can do releases as often as we want. We can fix bugs often as we want. You can file a bug, and we can't have it pushed. And so you can run that version of Kustomize with the fix within a day, which is just simply impossible to do if you were building it in the Kubernetes repo. And while we're iterating so quickly, initially it seemed to just make sense to do that until it became stable, and we-- seeing fewer changes at that point, moving into the main Kubernetes repo.

The other really nice aspect of this is it is much more approachable to new contributors because they don't need to go through all the Kubernetes setup environment for the development environment, and figuring out the pull requests, and code freezes and code slushes and [INAUDIBLE] and all this sort of thing. And really, you can just clone the Git repo, install Go, make a PR, send the PR, and get it merged, as long as you coordinate on what it is you intend to do with the maintainers.

ADAM GLICK: Awesome, it's kind of like the microservices model in building applications applied to software development.

PHILLIP WITTROCK: Yes, it's the microservices of Kubernetes development.

ADAM GLICK: Awesome. Is this based off a set of best practices that you've used in other places? Or was this just kind of a brilliant idea of Brian Grant and yourself?

PHILLIP WITTROCK: I'll give all the credit to Brian Grant.

ADAM GLICK: Fair enough.

PHILLIP WITTROCK: I'd say it's a reaction to many of the solutions that we develop internally at Google and tried to use and found somewhat limiting. One of our experiences developing internally at Google was by developing an abstraction layer that has different parameters, and then each team exposing this new abstraction layer as part of the package. So if you're giving me a way of running a Java app, but you're also giving me a completely foreign abstraction layer with it, become very difficult to use this, because every time I want to use a new package, I have to figure out what the special way of using that package is.

And so the focus here was to say, OK, how can we make it so that we're not introducing this completely new way of using the system with a new set of parameters for each new package that we're providing. And so we said, well, what if we could just use the APIs directly. And instead of providing a layer on top of the APIs, we just provide a way of transforming those same APIs with some very simple mechanisms.

ADAM GLICK: Gotcha. Some people may be familiar with the Helm-based workflows, and using Helm charts in order to kind of do their Kubernetes deployments. How would you compare a Kustomize-based workflow versus a Helm-based workflow?

PHILLIP WITTROCK: The Helm-based workflow-- I want to say there's several pieces to it, and that tool solves a number of different concerns for users. And so there is an overlap with everything that Helm does. There is one aspect of what Helm tries to solve is allowing reuse of YAML config, and also allowing meta operations on YAML config to maybe change three or four things at once through a specific parameter.

That is a problem that Kustomize is trying to solve. In fact, that is basically the only problem Kustomize is trying to solve, is how do we get reuse, how do we get composition, how do we make authoring of the config simpler. The Kustomize approach works directly with the Kubernetes APIs. And so this means that instead of having someone who writes a Go template in Helm package decide what are the right things to expose to the end user, and maybe document them well, or not document them well, or name them well, or not name them well.

You're working directly with the Kubernetes APIs. And so there's no choice on what's exposed, necessarily. Now, this means they have to be written slightly differently. So the package that's provided should actually just spin up and work without any customization at all. And then the mix-in should be applied.

So that means the experience is a more native Kubernetes API experience, whereas the Helm experience tends to be at a higher level abstraction layer that someone else exposes a different set of configuration that they think is what's important.

ADAM GLICK: You've talked about a number of the ways that you use and kind of designed Kustomize for. Is there a particular set of scenarios that you'd say that people should use Kustomize for? And are there certain scenarios where you don't think Kustomize is the right solution for people?

PHILLIP WITTROCK: Yes. So we're trying to figure this out now. We have some idea based on our own experiences with the tool. But we're also looking for customer engagement and feedback to figure out what doesn't it do well now, but maybe it should do well and seems to be within its scope, and what is it not targeted at.

And kind of back to the Helm piece, it's not targeted at basically anything other than the customization of some YAML configs. And it's not targeted at exposing a simple interface. And sometimes that is appropriate, where you want one or two knobs, like you just want to specify the MySQL image, and you don't want anything else, right? And that's the appropriate user view.

The other case, and similar to that, is the case where you really want a native Kubernetes API. And you want maybe a CRD or an operator or a controller, this sort of space. You may want to build something out there. And what that approach allows one to do is actually manage the workload once it's been pushed. So you can do maybe rollforwards, and rollbacks, and backups, and this sort of stuff. And so the API extension pattern is much more appropriate for that.

ADAM GLICK: With all the changes that you're doing, how fast do Kustomize updates come? What's the release cadence that you're targeting?

PHILLIP WITTROCK: Well, right now you can just build right off of head. So the notion of a release is not totally fixed. We are tagging releases about every other week. So every two weeks, we say, OK, if you want a specific tag to pull down so that you can reference it for bugs and that sort of stuff, we're pushing those.

We are seeing a lot of additions in contributions. We did a blog post recently, and so we've seen, actually, a number of new contributors that we hadn't seen before coming and adding new features, which has been great, and picking up issues. We have been merging changes daily, sometimes more than one or two changes a day, which we're pretty happy with, given the scope of the tool, to have that level of additions.

ADAM GLICK: That's really great. As you're looking forward with this, what are the things that are next on the list that you want to build into Kustomize?

PHILLIP WITTROCK: Yeah, so there's two things that we've come across that we know folks want. We considered holding off our launch of our blog post until we produced those features, and decided actually it's more appropriate just to put it in user's hands and actually let the users tell us if those are the most important things, or if there's actually something else they want to see. But it turns out the things that were on our list are actually the two first things users encountered and said, yes, we want those things, which is always great to have that sanity check, that you have some notion of what you need to do.

And so those two things are-- the first is modifying namespaces. So I mentioned that one of the things that Kustomize tries to address is, how do I manage a change across a bunch of YAML files. And when I add a new YAML file, that YAML file gets the change, and not worry about, oh, did I update the label on everything, or did I forget one of the files?

And so one of those changes is namespaces. It's a pretty common use case to say, hey, I have this directory, and now all the stuff in this directory really should run in this namespace. So having Kustomize be able to say, OK, any YAML file that I am customizing should get this namespace addition to it.

The second one is something we're referring to as using the downward API, because that's kind of the internal technical solution of the problem. But the specific problem is, in your YAML config, oftentimes users reference other YAML. So an example of this is you have a secret in one YAML config, and then you have a command line argument that references that secret in your deployment config, right?

And so, when you make a change to, say, the secret name, for instance, how do you then make sure that everything that's referencing that secret name also gets updated, right? This would be a problem if you were just using regular YAML files as well. It's not unique to Kustomize. But Kustomize does make it easier to make changes to your files. And its goal is to make it so you can do these high level meta transformations.

And so the problem we're trying to solve is that when you apply a certain change or transformation, that we're able to understand every place that would be impacted by that change, and propagate the change there.

ADAM GLICK: Cool. So if I wanted to go learn more about Kustomize, how to use it, try it out myself, get involved in the project, where should I go?

PHILLIP WITTROCK: The GitHub repo is the first place. It's in a Kubernetes SIGs repo. So that's Kubernetes-SIGS in GitHub. Kustomize with a K. I'm not going to try and spell it out. I'm sure I would fumble that one.

ADAM GLICK: We'll put that into the show notes, so anyone who's curious can just look at the show notes. There'll be a link to the repo.

PHILLIP WITTROCK: Excellent. That would be the first place to engage upon and look at the open issues. If you're interested in contributing, look at the open issues, and comment on one. You can engage-- I think there's a Slack channel for Kustomize. You can direct message or @ mention myself, PWittrock. You can probably add my GitHub up handle to the notes as well. Or Jeff Regan is one of the other maintainers who would be a great person to reach out to.

Engage with us. That's if you want to contribute. If you just want to use the tool and play around with the code or see what's going on, go right there, clone the repo, build it, and have some fun.

ADAM GLICK: Thanks, Phil. It was really great having you on to talk about Kustomize.

PHILLIP WITTROCK: Thank you.

ADAM GLICK: Take care.

PHILLIP WITTROCK: Bye.

[MUSIC PLAYING]

ADAM GLICK: That's about all we have time for this week. If you want to learn more about Kustomize, you can check out github.com/kubernetes-sigs/kustomize.

CRAIG BOX: Thanks for listening. As always, if you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at @KubernetesPod, or reach us by email at kubernetespodcast@google.com.

ADAM GLICK: You can also check out our website at kubernetespodcast.com. Until next time, take care.

CRAIG BOX: Bye-bye.

[MUSIC PLAYING]