#6 June 6, 2018

Skaffold, with Matt Rickard

Hosts: Craig Box, Adam Glick

On this weeks Kubernetes Podcast, Adam and Craig talk to Matt Rickard about Skaffold.

Do you have something cool to share? Some questions? Let us know:

News of the week

CRAIG BOX: Thanks for downloading the Kubernetes Podcast from Google. I'm Craig Box.

ADAM GLICK: And I'm Adam Glick.


CRAIG BOX: Adam, you're back in Seattle now. What are your thoughts after your whistle-stop tour through Europe and the Middle East?

ADAM GLICK: Oh, it was a great chance to be out and meet with a lot of interesting folks. The summit last week in Tel Aviv was great. It is fantastic to see how many people are not only interested in Kubernetes, but how many of them are actually up and using it. I got to meet with a lot of great folks, so thanks for those of you who came out. I'm also happy to be heading back home, back to Seattle, and preparing for Next, which is coming up in July. How about you, Craig?

CRAIG BOX: I will be doing much of the same. The last week here in New Zealand, I've been back to London for five weeks, which will basically be heads down preparing for the next conference, which is July 24 in San Francisco. We'll have some fun new features to announce, and I'm having a great time working on their installation, which will let you experience some of them in person.

ADAM GLICK: Awesome. Why don't we jump into the news?


One of the big rumors that is just floating around as we're recording is that GitHub may be being acquired potentially by Microsoft. GitHub is the home, of course, of Kubernetes' source code and development process, as well as almost all open-source projects of note aside from the Linux kernel. The latest developments in the story you can find in our show notes.

CRAIG BOX: The Istio team this week announced the release of 0.8 of their open-source service mesh. Istio is an open framework for connecting, securing, managing, and monitoring microservices. Version 0.8 is a long-term support release and hopes to contain the last breaking changes before the upcoming release of 1.0. It fixes many issues that were blocking early adopters from moving into production. The major change in 0.8 is a new traffic management model based on feedback from the community. Ingress objects in Istio have been decoupled from Kubernetes and are now called gateways.

A new virtual service object replaces the previous route rule as a method for specifying traffic routing. A virtual service object can now contain multiple routes to one or more actual services. Those routes don't even have to be to the same host, which can be useful when decomposing monolithic services, as you don't require existing consumers of that service to adapt to the transition. You can also add entries to the Istio Service Registry with a new service entry object, which replaces the functionality of egress rules. These services can either be internal or external to the mesh.

The new Traffic Management API fixes scalability issues and has significantly more functionality, but unfortunately does break backward compatibility. A tool has been published to migrate configurations from the old format to the new, and subsequent Istio versions will not support the original configuration format. Istio 0.8 also introduces support for a mesh expanding across multiple Kubernetes clusters. And other new features, such as Istio using Istio itself to provide telemetry for the control plane components and support for the new v2 Envoy API.

ADAM GLICK: Another new enterprise feature was announced in Google Kubernetes Engine this week. VPC-native clusters allow you to use Google Cloud's IP Alias feature to replace network routes, which means that pod IPs become first-class citizens to your Google Cloud virtual network. This means that your GKE clusters no longer are subject to the limits or quotas based upon network routes and your GKE node subnets can now be advertised over a VPN or interconnect using the Google Cloud router. Also, because the IP addresses are known to the VPC, you can enable anti-spoofing protection to your GKE nodes, as well as ensuring you don't accidentally dual-allocate an IP range to two clusters on the same network.

CRAIG BOX: Google this week announced Kustomize, a tool built by Googlers in the Kubernetes Command-line Special Interest Group. Kustomize, with a K, brings declarative configuration to the problem of reusing and customizing, with a C, YAML files. Rather than having to adopt a completely separate language for templating Kubernetes configurations, with Kustomize, you are able to express configuration and a set of changes that might need to be made to it for a different environment, like labels you might set differently or replica counts that might change between staging and production environments.

Kustomize addresses a number of longstanding issues with the declarative workflow using kubectl, like how to update configs. If you directly entered a config map, the deployments that refer to it do not know it is changed and thus don't trigger an update. With Kustomize, your changes generate a new config map and patch the deployment to point to this new config map, which will then cause a rolling update and your application to update as expected.

ADAM GLICK: After the success of KubeCon Europe in Copenhagen last month, the CNCF has issued their call for papers to upcoming KubeCon events in Shanghai and Seattle on November 14 and December 11, respectively. If you're looking to submit to either event, the CNCF has published some handy guidelines, including the following. First, make sure you have a clear and compelling title and description. Focus all of your work on an open-source project, whether it's a CNCF project or a new project that adds value to the cloud native ecosystem. Don't just submit a sales or marketing pitch for your product or service even if it's open source and no matter how compelling you feel it may be.

Be sure to pick your target audience. Conference goers range from top experts to total beginners, and talks are accepted across the whole range of attendees. You should also try and choose a unique topic that is relevant and unlikely to have been covered by multiple people. This is a great way to help your submission stand out.

Along the same lines, don't submit something that's the same as a previous talk, as you can find all previous talks from KubeCon on YouTube. If you're covering something that has been talked about before, make sure you point out how your presentation is new and different. KubeCon is also explicitly interested in increasing the voice of those who have been traditionally under-represented in tech. So if you're a voice that we don't hear from enough in the community, please submit your talk.

And finally, don't miss the deadline. KubeCon Shanghai, the deadline is Friday, July 6 for talks. And for KubeCon Seattle, it's Sunday, August 12.

CRAIG BOX: And that's the news.


ADAM GLICK: Our guest today is Matt Rickard, who is a software engineer on the Container Tools Team at Google, lead engineer on the Skaffold project. Welcome to the show.

MATT RICKARD: Thanks for having me.

ADAM GLICK: What is Skaffold?

MATT RICKARD: Skaffold is a command-line tool that really targets the Kubernetes developer experience. So that's everything from the inner development loop of build and deploy to also taking that tool and having something usable in a CI/CD pipeline.

ADAM GLICK: And can you run that locally as well as with a cluster that you may be running remotely in the cloud?

MATT RICKARD: Absolutely. So it works with local clusters like Minikube or Docker for desktop. But it also works exactly the same with the GKE cluster or any sort of cloud cluster.

CRAIG BOX: If I've built an application and I have a normal local development cycle, how am I going to adopt that to Skaffold?

MATT RICKARD: So the entry point here is basically a skaffold.yaml file. And it's a really simple YAML file. It's not as crazy as kind of some Kubernetes YAML files. And the idea behind it is it just kind of combines the metadata from your build part of your workflow and your deploy part of your workflow. So that means things like, how are you actually building your image? Skaffold supports a lot of ways to build your images not only with Docker, but with Bazel or with other ways that you can build this container image. And it also supports many ways to deploy your Kubernetes resources, such as using Helm, or using kubectl, or using kind of other higher level abstractions around the deploy process.

ADAM GLICK: Does it have particular opinions about which of these tools it prefers?

MATT RICKARD: No, so there is actually no opinion. Of course, we've chosen to implement a few that we're familiar with at first, but we gladly accept contributions. And these APIs are public, and we're accepting contributions.

ADAM GLICK: You launched Skaffold relatively recently and put it out there. What's the reaction that you've seen from people as they've started to pick up the tool and use it?

MATT RICKARD: Yeah, so the tool has definitely scratched some sort of itch in the community with the developer workflow. It's something that, as Kubernetes becomes more stable and the APIs become more stable, we're actually able to target those APIs with our tooling. And people are just starting to write really, really complex applications on Kubernetes, and they need some sort of iterative development flow to work on those apps.

CRAIG BOX: And does that development flow work for multiple apps that are deployed together and the interaction between them in the microservices environment?

MATT RICKARD: Yeah, absolutely. So one of the things that I thought a lot about when I was designing Skaffold was that in this world of microservices and multiple applications, you really need a tool where you can work on many services at once. So Skaffold is smart enough to build and deploy multiple images and multiple deployments, and it will only rebuild the ones where the source code changes.

ADAM GLICK: How does it keep those separate if you're working on multiple different projects, multiple different pieces?

MATT RICKARD: Yeah, that's a good question. So we actually do some sort of smart introspection on your build and deploy resources. So we can actually tell exactly what source code dependencies correspond to which images or which deployments. So we know when we need to build and when we need to deploy.

CRAIG BOX: How do you relate that to services that you need to consume but you're not necessarily building yourself?

MATT RICKARD: Skaffold also has a mode where you can start developing on things that you've already deployed to your cluster. So let's say you don't actually have the manifest locally, you don't have them in some repository. You can actually point Skaffold to an already deployed resource and start developing on that right away.

CRAIG BOX: The story of Docker is, obviously, you can build locally and take the same artifact and deploy that anywhere else. When we start talking about clustered environments, you have Minikube, which is only a single node, and most people's production clusters will be multiple nodes. Is there a use for Skaffold in being able to work through the dependencies on these environments? Should you deploy locally to a one-node environment if your production environment's going to be more than that?

MATT RICKARD: Sure. So a little background. I'm also a [INAUDIBLE] of Minikube, but I do kind of firmly believe that the future is developing on these remote clusters and developing on something like GKE, because as your applications and as your stack gets larger, it's not going to be able to fit on your laptop. You're not going to be able to emulate all of the multi-node distributed environment that you would have in a real production cluster. So I think tools like Skaffold are really bridging the gap between the kind of traditional, build a binary, deploy it on localhost, hit that endpoint, and kind of taking that nice experience, and bringing it to remote clustering Kubernetes.

CRAIG BOX: What was the impetus to start the Skaffold project?

MATT RICKARD: A lot of it was my personal frustration with developing [INAUDIBLE] on top of Kubernetes. Working on Minikube, I thought a lot about what that kind of developer experience looks like. And in Minikube, we started to build some features out like Skaffold, but we quickly realized that these belonged in kind of a separate tool. And this workflow was something that everyone had trouble with. And it could also be applied to any sort of cluster.

ADAM GLICK: So how is that different from-- there's a number of tools that people use today to accomplish their CI/CD pipeline pieces. What is it specific about Skaffold that makes it particularly useful in the Kubernetes ecosystem?

MATT RICKARD: Where Skaffold is useful is kind of removing a lot of the boilerplate around build and deploy. A lot of people have kind of reinvented the wheel in terms of building their Docker image on some sort of Jenkins pipeline and then writing some custom Bash scripts to template the Kubernetes manifest and deploy them. Skaffold gets rid of all of that, and it also gives you the benefit of using alternative implementations.

Our team open-sourced a tool called Calico, which is unclustered Docker builds. So the idea is that you can do something like that in your CI/CD pipeline with Skaffold and a very similar Skaffold YAML to do your builds on cluster and have a very nice CI/CD pipeline.

CRAIG BOX: Does that mean Skaffold, as a tool, carries through and you'll use the same technology to do the deployment to production?

MATT RICKARD: We haven't thought much about production yet, but the idea is that you don't need to rewrite this for CI/CD. I'm not sure if it's exactly ready for production use, but I could imagine, at some point, it might be.

CRAIG BOX: How does Skaffold integrate with Spinnaker and tools that are designed for the actual deployment of artifacts to production environments?

MATT RICKARD: Skaffold really has a separation of concerns, where Skaffold is not going to watch your Git repository. It's not going to figure. It's not going to do remote execution on different machines. It's really meant to be a step in a Spinnaker pipeline or a step in a Jenkins pipeline. Because we know that people already have the CI/CD infrastructure. We want to get rid of the boilerplate that they're writing inside these steps.

ADAM GLICK: So if you think about the tools that people plug in with this, talk me through what the tool chain would be and where this fits along that pipeline. So if you were using-- you start your code, let's say, in GitHub, and then you'd be using something like Bazel to do your build.

MATT RICKARD: Yeah. Absolutely. So I guess I could approach this from [INAUDIBLE] first the build side. So we obviously [INAUDIBLE] your regular, old, local Docker builds. If you want to use something like Google Cloud Builder to actually build your images with support using that, we also support building without a Docker file entirely, so using something like Bazel. And I think users will kind of gravitate towards these solutions that are actually much faster and more incremental than a Docker file for the build side of things.

And I think, historically, it's been very hard to get that inner development loop with building Docker images just by the fact that the Docker file is not really made for that kind of inner development loop. And you need to craft it very well to even get good build times. But I think as we looked at alternatives-- and Skaffold can implement those, and give users kind of a way to try those and integrate them into their workflow.

CRAIG BOX: Skaffold addresses what you call the inner development loop. Tell me what that means to you.

MATT RICKARD: The inner developmental loop, to me, is really build and deploy. We're starting to think about where something like Test fits in here, because it's not something that you want to run. You don't want to run Test on every single source file change, but you might want to run it as part of your kind of Skaffold run, where you're running through the pipeline once before you actually deploy your artifact. So right now it's build and deploy, but Skaffold is really meant to be flexible. And it's meant to kind of grow as we figure out really what this development workflow looks like for cloud-native workflows.

CRAIG BOX: If I'm working in a team, will I be deploying my own builds before I'm committing the code, pushing it to a central repository, or is this something that the team will be doing once they've collected their code? If developers are working on something locally, they will work on changes, only then sort of push them up, they'll be in the integration step after that. Like, is Skaffold solely for the part before that? Or is there a case for using it once you-- like, in the repository when things are committed centrally? It feels to me like we talk about it as if it's what you run on your laptop. And if there's more to it than that, I want to be able to present that.

MATT RICKARD: We built Skaffold with the kind of DevOps workflow in mind, so that's really you have your version-controlled repository as your source of truth. And it contains your configuration, it contains your source code, and everything you need to kind of deploy.

And Skaffold interacts with this through a few ways. So one, we're able to kind of add metadata to your container and also your Kubernetes deployments with getMetadata such as the tags, that commit SHA, any information that can help you kind of trace back what code was used to build these artifacts. So that's one way we kind of support the GitOps flow.

The other way is through a service like Google Cloud Builder, where you can set up GitHub triggers and pull request triggers. Then you can run the Skaffold run as part of your cloud build.YAML and you can have this kind of build/deploy cycle typed to your productions or events.

ADAM GLICK: What do you have on your roadmap next to start building into Skaffold?

MATT RICKARD: We want to obviously add more builders and deployers as we start to figure out what these even are. There's been a ton of activity in the space with kind of alternative ways to build your Docker file. And there's also been a ton of work in the space of, how do you actually define your application and how do you-- we have higher level abstraction around, what is then an application on Kubernetes? So Skaffold is meant to be pluggable. It's meant to kind of support those higher level abstractions. So I think that will be much, much better for users.

ADAM GLICK: And if I wanted to try it out, where do I go get it?

MATT RICKARD: So you can get Skaffold on GitHub. There's lots and lots of examples that you can run through. The documentation is always improving. But it's on GitHub under GoogleContainerTools/Skaffold.

CRAIG BOX: And since you've released open source, what contributions have the community brought to Skaffold?

MATT RICKARD: Yeah, so the community has been super helpful. I mean, we've been blessed to have so many contributions already. We've gotten lots and lots of bug fixes, and there are lots more bugs to fix. But we've also had builders and deployers be implemented by the community already with users who are really familiar with these kind of specific tools. Personally, I'm familiar with a lot of them, but we really need the expertise of the community to help integrate these into Skaffold.

ADAM GLICK: And if people want to come and help you work on it, since you're putting out the clarion call of, like, come help work on this, is there a list of issues and items up on GitHub where they can just take a look and start claiming those pieces to work on them?

MATT RICKARD: Yeah, absolutely. So all of the issues are tagged as Good First Issue for new contributors. And those are great places to start. A lot of them are really fully documented, with the idea that we could probably do a lot of these if we had the bandwidth. But we really want to get people involved and we want people to contribute.

ADAM GLICK: And what language skill would they need?

MATT RICKARD: Skaffold is written in Go lang, like most of the container tooling.

ADAM GLICK: Great choice.

MATT RICKARD: Yeah. So we love Go, but no Go expertise is needed. We'll help you kind of get started with Skaffold. The code base is fairly readable.

CRAIG BOX: All right, Matt. Thank you so much for joining us today.

MATT RICKARD: Thanks for having me.

ADAM GLICK: Take care.


CRAIG BOX: That marks the end of our show for another week. If you want to learn more about Skaffold, you can check out GitHub.com/GoogleContainerTools/Skaffold with a K or check the links in our show notes.

ADAM GLICK: Thank you for listening, and if you have suggestions, questions, or other feedback, you can reach us on Twitter at @KubernetesPod or by email at kubernetespodcast@google.com. And if you liked the show, please help us spread the word and tell a friend.

CRAIG BOX: You can also check out our website and find our show notes at kubernetespodcast.com.

ADAM GLICK: Until next time, take care.

CRAIG BOX: Bye, bye.