CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google. I'm Craig Box.

ADAM GLICK: And I'm Adam Glick.

[MUSIC PLAYING]

CRAIG BOX: Is it sunny in California?

ADAM GLICK: It is. It is wonderful. I caught a flight down here, and California, as always when I come down here, is bright and sunny. I had a great trip down playing a little game called Majotori, which is kind of a trivia game tied in with story and narrative. Basically, a witch shows up, offers you a trivia game. And depending on if you win or lose that game, the story of your character then either gets better or worse depending on winning it.

CRAIG BOX: Answer me these riddles three.

ADAM GLICK: Something along those lines. And also, they have pluots down here, which is like if you take a plum and an apricot and you mash them together-- well, not mash them together, but breed them together-- and they're absolutely delicious. If you haven't had one, totally something to go and enjoy.

CRAIG BOX: I haven't, but I will be in California myself in a couple of weeks' time. I think we're just going to miss each other, unfortunately. But I'll have to check out the pluot situation.

ADAM GLICK: How are the things growing in the UK?

CRAIG BOX: Well, I don't think we grow a lot of the fruit that we eat here in the UK in the UK, but we do have a lot of wildlife. And I must say, a few people have been asking to follow up on the fox family that were living in our backyard.

We put a bit of food out every now and then, and it's gone the next morning. And we see scratch marks on the ground, and then very occasionally fox droppings. Very comically, actually, the fox went through our recycling bin, and got out a plastic container, and then proceeded to do his business in the plastic container on the back porch, which was polite in a strange kind of way.

ADAM GLICK: That was very kind of them.

CRAIG BOX: And we did see one sitting on the neighbor's shed recently. But overall, they're going to go and find their own thing. I think the chances of seeing more than one or two of them at a time isn't that much anymore. But we trust they're around. And if they're not going through our garbage, soon enough, they'll be going through somebody else's.

ADAM GLICK: Out of curiosity, what does the fox say?

CRAIG BOX: You know, I never asked. I should have really been paying attention.

ADAM GLICK: Let's get to the news.

[MUSIC PLAYING]

CRAIG BOX: Last week's security theme continued with the open sourcing of a third-party security audit of Kubernetes sponsored by the CNCF, a working group was set up to run the audit earlier this year. And a team comprised of two external vendors was chosen. Their reports were made public last week.

In all, they found 37 vulnerabilities and made a myriad of recommendations on fixes that should be applied long and short term in areas of networking, cryptography, authentication, authorization, secrets management, and multi-tenancy. Along with the findings, they published a general guide on attacking and protecting Kubernetes, which is an enumeration of all the things Ian Coldwater mentioned last week.

ADAM GLICK: Cyber Arc have started a blog series on the topic of penetration testing Kubernetes. The first part refers to the role-based access control system and how you can find a role with too many permissions and pivot to running a pod using that account.

CRAIG BOX: When you do get a malicious pod running, what can you do with it? Rory McEwen has a suggestion. Set up a reverse shell. A reverse shell is a command shell which connects out to a host you have waiting for it on a network. You can do this with any platform that lets you run arbitrary docker file steps, like a hosted build service, or even start a DaemonSet and have a reverse shell running on every node in the cluster.

ADAM GLICK: Google Cloud Security Scanner was first launched for App Engine in 2015 and has now added support for GKE and CGEVMs. Cloud Security Scanner is a web application vulnerability scanner that can alert you to things like cross-site scripting vulnerabilities, flash vulnerabilities, mixed content that could be exploited by a man-in-the-middle attack, or other common vulnerabilities. It will also scan for header conflicts and unencrypted passwords. This is offered as part of Google Cloud's Security Command Center and is available today for use.

CRAIG BOX: Now we've got everything secured, let's move on. The Knative Project has announced version 0.8, containing the requisite fixes for a point release and new features around scaling and route readiness. 0.8 is notable for being an API release candidate. If all goes well and continues as planned, the API will stabilize and be released as V1 in the next Knative release.

ADAM GLICK: Ever wonder how Pinterest runs its Kubernetes deployments? If so, you're in luck. In a Medium post this week, Pinterest Engineering posted about how they've configured their clusters, how they do their deployments, and how they handle a massive scale internet service. They also talked about their goals with Kubernetes being focused on developer productivity, service reliability, and infrastructure efficiency. If you want to know how they handle stable work, custom resources, or testing, you might enjoy this post.

CRAIG BOX: Brian Lyles from VMware, our guest in episode 54, has launched Octant. Octant is a tool for visually experiencing what's going on in your Kubernetes environment. It's a dashboard for debugging with a difference in that it runs locally rather than in your cluster. Octant includes resource and log viewing, port forwarding, and has a plugin interface to add components on top of existing views.

ADAM GLICK: Each year, the CNCF does a report on the state of Kubernetes in the cloud called the Cloud Native Survey. This year's survey has just kicked off, and the CNCF would like your input. As the landscape of cloud native apps has changed, so has the survey, and this year's results will expand to cover new topics like service meshes, including Istio, and CI/CD development pipelines.

The CNCF is also offering a drawing for a gift card for those that fill out the survey. The results of the survey will be released before KubeCon San Diego this year, and we can't wait to see how the community has grown and evolved.

CRAIG BOX: Finally, a funny thing happened on the way to the forum-- or should I say the summit? If you were excited to head to the Kubernetes Summits in Seoul and Sydney that we mentioned last week, be aware that CNCF has renamed these events to Kubernetes Forums. The change has been made to avoid confusion with a contributor summit.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Ahmet Alp Balkan is a developer advocate at Google Cloud working on GKE, Cloud Run, and open source developer tools. Luk Burchard is a former intern at Google Cloud Developer Relations. He's a computer science student in Berlin, currently finishing his last semester. Ahmet, welcome to the show.

AHMET ALP BALKAN: Hey. Glad to be here.

CRAIG BOX: Luk, welcome.

LUK BURCHARD: Hey. Nice to meet you.

ADAM GLICK: Most Kubernetes users are familiar with kubectl as a way to control your clusters, and that's been extended with plugin support. When did that come, and why?

AHMET ALP BALKAN: Kubectl plugins are sub-commands that you can add to kubectl in your developer machine. For example, 'kubectl get' is a kubectl official sub-command, but if you go and develop, say, a command called Foo, you can have this show up on kubectl. And this will be a plugin that you wrote. And it does not ship with kubectl. You basically have it on your system.

So a little bit history on how kubectl plugins came to be. They were an alpha feature in 1.8, so that's sometime in 2017. And we had the feature sitting around for a while. And it was not in great shape, mostly because it required a lot of configuration, and it was such a big hassle to use it. And we turned the feature to-- well, we reactivated the whole thing around, I think, kubectl 1.12. So that was sometime last year. And now the feature is actually generally available as of 1.14.

CRAIG BOX: kustomize was a command all of itself. I believe now it's a subcommand. Is it implemented as a plugin?

AHMET ALP BALKAN: Well, yeah. So technically, anything that could be a kubectl subcommand could be a plugin as well. The main benefit of having plugins is that you can have extra commands in kubectl that you don't have to develop in the Kubernetes tree. So that comes with a long process of getting the command accepted. The command may not be applicable to most use cases. In that case, there will be a lot of deliberations about whether this command should exist in kubectl or not. So plugins are an easy way to get around this and have your command show up in kubectl.

CRAIG BOX: What are some of the most popular plugins in kubectl today?

AHMET ALP BALKAN: The ecosystem is pretty new. The feature just became GA. Therefore, there are not a lot of plugins. I would say there are maybe some-- like, 60 plugins out there today.

CRAIG BOX: That's a lot.

AHMET ALP BALKAN: Yeah, that's quite a bit. But the discovery around those plugins are not great right now. And we're working on those now. So we're trying to get more visibility for the plugins that are available out there.

CRAIG BOX: What is an example of a kubectl plugin?

ADAM GLICK: There is a command called 'kubectl access-matrix'. So this command is implemented as a plugin by one of our contributors in the community. And when you run kubectl access-matrix, you're able to see what you can do on each namespace on each resource type. So you can say, can I create pods? Can I delete namespaces? And so it gives you this table with checkmarks and so on that gives you a visual way of representing what are your abilities on a cluster, or as a user, what resources do you have control on.

So this is a good example of a plugin that, as you know, could be part of kubectl, but it's not, because it's visual. And it gives you this UI that's not necessarily consistent with the rest of kubectl. But within itself, it makes perfect sense, and it's a great utility to have. So yeah, this is a great example of a plugin.

CRAIG BOX: What is the actual thing that I'm distributing? Am I distributing a different binary that simply gets executed when I type kubectl plugin-name?

LUK BURCHARD: So a kubectl plugin basically is a binary that is being called by kubectl itself. And we make this binary available through your path.

CRAIG BOX: And so when you called kubectl plugin-name with some parameters, they passed on just as parameters to that binary?

LUK BURCHARD: Yes. In the background, the binary would be called, and then all the parameters are getting passed to this binary.

CRAIG BOX: I can take from that that it's not necessary that the binary be written in Go?

LUK BURCHARD: No. You can write the binary in any language you would like. You can also use a shell script if you want to write a small kubectl plugin. Basically, it just needs one entry point, and it needs to be self-contained.

ADAM GLICK: The two of you work together on a project called krew.

AHMET ALP BALKAN: Yeah, that's right.

ADAM GLICK: What is krew?

LUK BURCHARD: Krew is a kubectl plugin manager which helps you with discovering and installing plugins.

CRAIG BOX: Is it fair to say krew is a package manager in the way that we think about it for yum and APT?

AHMET ALP BALKAN: It is pretty close, but krew is not trying to be a package manager. Writing a package manager is probably not a great idea. So krew strives to be simple. What it does is it gets your plugin from the internet, and it extracts it out of an archive file and places it somewhere in your path so that it can recognize as a kubectl plugin. So in that sense, krew is a-- I would say a glorified bash script, but it's not, obviously. It has subcommands and all this logic around it.

But at the end of the day, it's trying to do something very simple-- finding plugins, installing them, and keeping them up to date. So in that sense, it looks like a package manager, but it does not go and try to address the complicated problems the package managers are solving. For example, it doesn't manage dependencies, or it doesn't let you install a particular version of a plugin. It always installs the latest version. So it's taking a lot of shortcuts and avoids becoming a full package manager.

ADAM GLICK: Where did the name krew come from?

AHMET ALP BALKAN: The name came from Homebrew, the package manager for macOS. And we did what everyone else is doing, which is replacing the first letter with k for Kubernetes, obviously.

LUK BURCHARD: Also, the great thing about the name krew is that krew is like a daemon ship. So now we have something that connects with Kubernetes.

ADAM GLICK: One of the things that various package management tools do is get the version that you want from a particular location. There are lots of tools that do this. How does krew know where to look for that? Where does it store that? Do you have a centralized repository? Can you have private repositories where it only pulls from local ones? How does this work?

AHMET ALP BALKAN: Today, krew has a centralized plugin index. When you a develop a plugin as a developer, you go and submit your plugin manifest to the centralized index repository so that it's centrally discoverable by everyone. This is one plugin repository that is global.

ADAM GLICK: Kind of like Helm charts or Docker Hub?

AHMET ALP BALKAN: It's pretty close to what Helm charts are. There is a single repository on GitHub where all the charts are listed for Helm. Similarly, there is a krew index repository that has a list of all the plugins. The community feedback that we've been getting is people have a bunch of reasons to distribute the plugins themselves, like they want to have control over their release cycles, or they're developing plugins that are not necessarily eligible for the centralized plugin index.

So we want to give them the ability to go and tap into their repositories and install the plugins directly from there so that they don't have to come into the centralized repository and follow the guidelines, necessarily. We want to give that freedom to the developers.

ADAM GLICK: So the tool gives you an index, but doesn't do the hosting.

AHMET ALP BALKAN: Correct. We only host the lightweight manifest files, which only tells the krew process where the plug-in is.

ADAM GLICK: It's a pointer.

AHMET ALP BALKAN: Yeah, it's a pointer to essentially a package on the internet which could be anywhere, like on Google Cloud Storage or GitHub releases as a tarball.

ADAM GLICK: And if I wanted to do that for private commands within my organization rather than publicly, do I have to do that and then just put the private address that other people can't access, or is there a way to point it at a private index?

AHMET ALP BALKAN: Today there is no way to point it to a private index, or rather even a custom index. But this is the feedback that we've been getting. So one of the use cases for kubectl plugins is creating reusable commands that are useful within your company. Let's say you have a command called debug, and the way it works is very opinionated and it's customizable for your company.

So ideally, we want these people to be able to tap into the private index repository in their companies. Like imagine this is a Git repository that all your engineers have access to. So you could just tap into that. And your company machines are configured to get the plugins automatically from there. So this is where we actually want to get to, but today we're not there.

CRAIG BOX: If I developed a kubectl plugin and I wanted to distribute it globally and publicly, what's the process to submit it to your index?

LUK BURCHARD: The first thing someone would need to do is to archive a plugin. So put everything that is contained within the plugin-- put it into a ZIP or a TAR archive, and then make it publicly available-- for example, through GitHub hosting or through Google Cloud Storage. The next thing-- you would need to write a plugin manifest. So this manifest describes where the tarball is located, which version it currently hosts. So you would need to provide a hash for that.

And then you can fill in some meta information. For example, what's the name of plugin? And you can provide OS and an architecture. So this basically tells krew, OK, so there is something that can be used on Windows and on an AMD64 machine, for example.

Then you can test this plugin locally. We have developer comments for that. And later, when everything is right, you follow the guidelines. Then you can submit this plugin manifest to our centralized repository. If you've done everything right, your plugin has probably been accepted.

CRAIG BOX: And who is in charge of accepting plugins to that repository?

AHMET ALP BALKAN: Today, krew is a Kubernets SIG-CLI sub-project, so we're following the Kubernetes open source project standards. We have a set of maintainers that are in Kubernetes SIG-CLI that are looking at the plugins and seeing whether they're working correctly. We're doing some basic validation, but we don't want to go and enforce everything that a command does. We want to leave room for creativity.

But other than that, today, me, myself, and a few other maintainers are going and approving these pool requests and accepting plugins for the repository.

ADAM GLICK: Is krew itself a kubectl plugin?

LUK BURCHARD: Yes, krew itself is a kubectl plugin. The only thing you need to do is install it, and then through kubectl, you can install all of your plugins. So you don't need another command you need to know. So everything works through kubectl. You can also upgrade krew through krew.

CRAIG BOX: But you need to download it with just cURL to start with or something?

LUK BURCHARD: Yes. So that would be an initial phase where you would need to copy and paste a single line which installs it.

ADAM GLICK: How did you get involved in the krew project?

LUK BURCHARD: I wrote krew during my internship at Google in the summer of 2018.

CRAIG BOX: What was the process like of becoming an intern at Google?

LUK BURCHARD: One day during lectures, I was browsing through Twitter, as any good student [does]. Then I saw Ahmet's tweet, and I immediately wrote him a message on Twitter. He replied, and then I sent my CV to the HR department.

ADAM GLICK: That's quite a tweet. What was the tweet?

AHMET ALP BALKAN: We were just looking for interns. So I was trying to recruit some people. And I think Twitter seemed like the best place to find people who know about Kubernetes and can write some cool developer tools.

ADAM GLICK: Did you have the project in mind when you were looking for an intern?

LUK BURCHARD: Yeah. We've been thinking about this idea since, I think, late 2016. And we've been thinking about that because as the plugin system evolves, we realize that the fact that kubectl itself does not allow installing plugins and it doesn't make it any easier to package and distribute your plugins for all these different platforms, like Linux, Mac, and Windows, it would be a problem. And we realize that if we go and write a utility that makes it easier for users to install the plugins, keep them up to date, and cleanly remove the ones that they didn't like, that will be great for the users.

And for the developers, the packaging is a big problem. Let's say today you wrote a plugin, and you have to go and package it for apps, DNF, or yum, or macOS, you have to write up a Homebrew package. For Windows, you have to write a Chocolatey package.

So to capture all that, we wanted to simplify this plugin packaging process and allow a centralized index so that plugins are discoverable. Now back to your original question. We've been playing with this idea since 2016. But the fact that Luk had joined as an intern in Google Cloud Developer Relations, that gave us the time to actually go and do it.

CRAIG BOX: Did you start the process with the Kubernetes SIG CLI, or did you start the process of building a tool and then present it to them as something that you'd built?

LUK BURCHARD: Krew itself was designed as a standalone binary. We later made it into a plugin. And we thought that as the project matured, that we could contribute this to the Kubernetes SIG-CLI. And in the end of my intern project, we wrote a KP, which is a Kubernetes enhancement proposal, where we suggested to add krew to the kubectl command.

CRAIG BOX: And you'd mentioned, obviously, that it's a plugin. Was the thought to build krew into kubectl natively, so you don't need to do that initial step?

LUK BURCHARD: Yeah. So the fact that we developed krew as a plugin gave us an ability to demo krew as if it was part of kubectl. So usually, we actually-- the way you call krew is you type kubectl krew install. But back in the day, there was a command called, let's say, install directly. So you would type kubectl plugin install directly. And this gave us that opportunity to go and show the krew's capabilities. And we gave various demos here and there to SIG CLI and other places like meetups. And having the plugin mechanism in place let us show that krew itself can be a subcommand, potentially, someday.

ADAM GLICK: Is there a path for people to take plugins and actually make them part of the full, official kubectl tool?

AHMET ALP BALKAN: Yeah. I think plugins are a great prototyping mechanism. And if you distribute your plugin and prove that they are actually useful, the plugin can be turned into an official subcommand by proposing this to the Kubernetes project. I think it's a great way to get started if you have a great idea that you think it will be accepted in kubectl. And you can just start, and distribute your plugin, and get some feedback on it, and then you can submit it to Kubernetes.

ADAM GLICK: What's been the adoption of krew?

AHMET ALP BALKAN: So far, we actually don't know how many users we have. I think opportunities like KubeCon are great opportunities for us to come to the conference floor and see if people are using it. We gave a talk about kubectl plugins and krew in the first day of KubeCon, and the room was almost full. There were-- I think a few hundreds of people were actually interested in writing plugins.

We don't know how many times plugins are downloaded. We don't know how many times krew is downloaded. We do not collect any analytics. But I can tell that in the past six months or so, where we have announced krew and started to get plugins, we have reached a place where we have over 35 plugins right now. And the number is increasing almost every week. Every week, someone is coming and saying, hey, I developed this cool plugin. Can we please distribute it on krew? We're like, sure.

CRAIG BOX: Is it a conscious decision not to do any analytics?

AHMET ALP BALKAN: Yeah. We wanted to kind of keep it open, and we didn't want to do any analytics because it's an open source project, and we don't want to have an opt out instead of an opt in. Opt in wouldn't work, and opt out would be rather also like-- rather an invasion of privacy. So we wanted to just go with no analytics.

ADAM GLICK: In terms of people creating plugins for krew, does someone submit that just as a pull request into that GitHub repository, or do they need to go to SIG-CLI, or is this just something that people contact you two personally? What's the right way that people go about actually submitting a plugin?

AHMET ALP BALKAN: Today you only need to write a plugin manifest and submit this to the krew index repository as a pull request. You don't need to go through the SIG-CLI approval process or anything like that. The way we're running the centralized repo is on a case-by-case basis. If we look at a plugin, and if we think that's going to be useful to people, we right now accept it. And this is actually allowing us to figure out what kind of plugins people are writing so that we can create some guidelines around this.

And we actually need community help here. We don't know what will be the future for the plugins. We don't know where this thing is going. So we want to understand first what kind of plugins will we be receiving in the future, and can we create some guidelines for the acceptance into the centralized repository, and how will that look like. For example, if you wrote a plugin that is just a one-line bash script, should that be a plugin, or should that be just bash alias that people put on there like command line scripts? We don't know that yet. I think we're going to have an answer within the next six months, and we need the community help there to show us the way.

ADAM GLICK: Who is the 'we' in that case that they're submitting to?

AHMET ALP BALKAN: This is the krew index maintainers, which includes myself and a few other members in the open source community.

ADAM GLICK: So when you say you're not sure about the one line script-- so my plugin for no-op is not likely to be accepted?

AHMET ALP BALKAN: I don't know. It depends on how much you want to pay me.

ADAM GLICK: [LAUGHS]

CRAIG BOX: What other things would you like the community to help you with?

AHMET ALP BALKAN: Today we're trying to figure out what constitutes a good plugin. I think the key to writing a good plugin is actually figuring out, is it actually useful out of the box? Does it follow the kubectl conventions? And when I say that, does it follow the same naming centers within the subcommand? Does it follow the same command line arguments-- for example, dash namespace, or the same command line flag could actually be dash directly.

So we want these plugins to follow these centers. And we have a few utilities for that. If you go to the github.com/kubernetes/sample-cli-plugin, you'll be able to find a sample CLI plugin there. And we have a project called CLI Runtime, also in Kubernetes SIG CLI that offers you some Go packages for writing plugins in Go. If you're using Go to write plugins, you should probably use this package to get all these generic command line arguments or printers that allows you to put output in YAML, or in table format, or in JSON path format. So these sort of utilities let people write plugins that are more coherent.

Another thing that we need help from the community is actually, as I said before, figuring out what is the gold standard of acceptance criteria for the plugin centralized repository. And we don't know whether we should even have a centralized index going forward. Right now, we do have a centralized index. Maybe the future of kubectl plugins is not that. And again, all these decisions, we want the community input to shape these decisions.

CRAIG BOX: Luk, where would you like to see krew go next?

LUK BURCHARD: We would like to see krew being further adopted and krew being the central part of kubectl. Also, it would be great if the community could help us to write more plugins so we could see where kubectl can actually go. Currently, krew has a centralized repository. We would like to split this into more decentralized repositories so that, for example, when I work in a company, and I have some private plugins that I don't want to publish to the central plugin index, that I can have my own repository which I can maintain by my own and which no one else can see.

CRAIG BOX: And, Ahmet, what was it like having Luk as an intern?

AHMET ALP BALKAN: I think it was great. Luk is a very low-maintenance intern. Well, he's pretty much done the whole project by himself. So we just had this idea, and we designed it for a few weeks, and then I kind of let him be. And he came back with krew, which was amazing, in a matter of a few weeks.

So we actually didn't know what to do with the most part of his internship, like the whole three-month internship. It was kind of-- OK, we have this now. So we had a lot of time to actually chat on the various parts of the krew, like the manifest file. And we had to go really-- dig into details and go polish that experience. So most of the krew that you see today is actually what Luk has written maybe over a year ago at this point.

CRAIG BOX: There are many interesting things you could do with a spare three months in a Google office.

LUK BURCHARD: Yeah. We have a great ping pong table in the Fremont office. And also, Seattle is a great place to visit. And now I'm fluent in espresso.

ADAM GLICK: Thank you both for coming on the show.

AHMET ALP BALKAN: Well, thank you for having us.

LUK BURCHARD: Thank you.

ADAM GLICK: You can find Ahmet on Twitter, @ahmetb, and Luk on Twitter, @lukburchard.

[MUSIC PLAYING]

CRAIG BOX: Thanks for listening. As always, if you've enjoyed the show, please help us spread the word, and tell a friend. If you have any feedback for us, you can find us on Twitter, @KubernetesPod. Please follow us, or reach us by email at kubernetespodcast@google.com.

ADAM GLICK: You can also check out our website at kubernetespodcast.com, where you'll find transcripts and show notes. Until next time, take care.

CRAIG BOX: See you next week.

[MUSIC PLAYING]