ADAM GLICK: Hi, and welcome to the Kubernetes podcast from Google. I'm Adam Glick.

CRAIG BOX: And I'm Craig Box.

[MUSIC PLAYING]

ADAM GLICK: Well, I hear it's not just the temperature over in England that's been baking recently.

CRAIG BOX: Well, yes. So first point there, it was the third hottest day ever in the UK last Friday. And for some reason, we decided last week was baking week. And the couple of recipes that I've got in the show notes, we made some banana bread and we made a caramel slice, which in the UK is actually known as millionaire's shortbread, but it's a silly name. It's very tasty nonetheless.

ADAM GLICK: I looked at those. Those look pretty yummy. I do have to say I might be clicking on a couple of those to make them, especially the caramel slice, which I've never seen but looks amazing.

CRAIG BOX: How does it compare to the caramel peanut ice cream pie?

ADAM GLICK: I'll have to make it and let you know.

CRAIG BOX: Now, over in Washington, I hear it's time to exercise one's democratic rights?

ADAM GLICK: Yes, yes. It is the primary election here in Washington state. And as a voter here, one of the things I'm always interested in is who are all the people running and doing my research on it. And Washington state is a state that allows anyone who really wants to run can run. There's not a high bar to put yourself on the ballot as part of the primary. And as such, there are a lot of people who are running for office.

We have governor, we have something like 20 people running for the position. Lieutenant governor is something like 10 or 11. So there's a lot of people on it. And there's just a broad diversity of people that are running, and it's always interesting to read some of these.

Because some of them are perennial candidates, people who just always run for office. Sometimes people have used it as a promotional opportunity. Not on this time, but previously there was someone who just was a real estate agent, used it as free ad space, a couple people pitching their book. So it's always just kind of fun to see who's taking it seriously and who's just having fun with the process. And it gave me a few giggles as I went through it this week.

CRAIG BOX: Something that I think did make the news worldwide was in the last couple of elections we've had a candidate here in the UK named Lord Buckethead who has been on stage with-- who was the prime minister at the time? I remember it was Theresa May won her seat, and Lord Buckethead may have come fourth or fifth or something like that. But he's there in the background.

There's a "Monty Python" sketch which talks about the silly party and the slightly silly party. He's very much taking that idea to heart and that some of these candidates, I think, are possibly going the same way. But it is good to see everyone exercising their democratic right to stand. And I'm sure a lot of people will be exercising their right not to vote for everybody.

ADAM GLICK: Shall we get to the news.

CRAIG BOX: Let's get to the news.

[MUSIC PLAYING]

ADAM GLICK: There's a new collaboration in the Linux Foundation, and this one is focused on security. The Open Source Security Foundation, or OpenSSF, has been set up to consolidate industry efforts to improve the security of open-source software. The foundation has a who's who of Kubernetes, cloud, and customer companies and brings together the Linux Foundation's Core Infrastructure Initiative and the Open Source Security Coalition, amongst others. No new projects were announced with the new foundation, but all of the governance model details are available in their GitHub repo.

CRAIG BOX: Kubernetes updates come every quarter. But what about your Helm charts? This is the problem that Nova from Fairwinds is designed to fix. Nova is an open-source command line tool that will compare the Helm chart you deployed for an application with the most recent one available. It provides you a list of the deployed and most recent versions so you can see if there is a discrepancy you want to address. Helm 2 and 3 are both supported, as are a number of chart repositories. Roadmap features include support for managed Kubernetes services in the cloud.

ADAM GLICK: Gustav Westling has released an alpha version of Lifebelt, an application monitoring tool you can install in your cluster. Lifebelt seeks to achieve higher uptime and improved security and utilizes kube-score, another project by Westling. Current features include a dashboard to monitor state and the ability to export metrics to Prometheus. It's currently free but expected to cost $20 per cluster when generally available.

CRAIG BOX: The CNCF Sandbox has just made room for another project. Chaos Mesh is an open-source Chaos Engineering platform that works at the pod, network, file system, and even the kernel level. The team at Chaos Mesh are currently working on their 1.0 GA release and are working on multiple improvements, including bringing chaos to your own application. If you want to learn more about the concept of Chaos Engineering, check out Episode 82 with Ana Medina.

ADAM GLICK: Another new project, Serverless Workflow, is also keeping the CNCF Sandbox growing. The project is a specification for defining declarative workflow models that orchestrate event-driven serverless applications. The project is currently available in version 0.1 and provides SDKs for Java and Go.

CRAIG BOX: Vitess continues its journey towards full MySQL compatibility with the release of version 7. This release brings replicate transactions, save point support, and the ability to set system variables per session. The team has also rewritten key systems to remove technical debt and make new features possible in upcoming releases. There are even new tutorials to help you learn and use for tests. There are a few incompatibilities with previous versions, and these are detailed in the release notes.

ADAM GLICK: Armory, a company selling an enterprise-ready variant of the Spinnaker deployment tool, has announced the general availability of an open-source Spinnaker Operator. The Operator replaces the custom Halyard tool that Spinnaker uses and adds Kubernetes-native features like storing Spinnaker secrets in Kubernetes secrets.

CRAIG BOX: This week's Azure Kubernetes Service release brings Azure Active Directory integration to general availability, automating a process that used to have to be done manually by cluster admins. You can also bring your own managed identity to the control plane as a preview feature, allowing features like custom VNets.

ADAM GLICK: Google Kubernetes Engine has changed its default node type to the new E2 family which is comparable to but 31% cheaper than the N1 VMs that were previously default. The savings comes at the expense of certain features which are no longer supported. If you want to use GPUs, local SSDs, or sole tenancy VMs, you need to continue to select a compatible VM type.

CRAIG BOX: Amazon's Elastic Container Registry has gained support for server-side encryption with user-specified keys. Previously S3s default encryption at rest using Amazon keys secured your images. But users can now use a master key provided by themselves using the key management store or cloud HSN.

ADAM GLICK: The CNCF has continued their set of project journey reports with the latest covering Jaeger. Since joining the foundation, Jaeger has enjoyed a 917% increase in contributing companies, a 603% increase in individual contributions, and the number of folks adding to the docs has gone up 496%. Congratulations to the team on such great growth and on their upcoming fifth birthday. If you want to learn more about the project, check out Episode 97 with the Yaeger's creator Yuri Shkuro.

CRAIG BOX: Finally, Alexey Ivanov and Oleg Guba from Dropbox have outlined how they moved their traffic infrastructure from NGINX to Envoy. Dropbox had to use lots of custom Lua in their NGINX deck to be able to observe it. And as this is built into Envoy out of the box, they say they were able to reduce their server fleet by 60%. In terms of extensions, modern C++ is cited as as easy to write as Go or Python, and the team are excited to see WebAssembly support, as they see this as the direction the industry will be going, especially at the edge.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

ADAM GLICK: Thomas Stromberg is a Minikube maintainer who also manages the container developer experience team at Google. In his 14 years at Google, he has been part of the hardware operations, site reliability engineering, security, and cloud teams. In his free time, he likes to build bicycles out of bamboo. Welcome to the show, Thomas.

THOMAS STROMBERG: Thank you, it's a pleasure to be here.

CRAIG BOX: Quite often, we'll talk to someone who invented or pioneered a technology or a project. You're a little unique in that you came to Kubernetes relatively recently. And that'll be interesting because we can talk through how the getting started experience was to someone who has gone through it and then who is responsible for making it better. So let's start at the top. How did you get started with the Kubernetes project?

THOMAS STROMBERG: A couple of years ago, I guess, 2018, I had been watching the uptake of Kubernetes in the open source space and eyeing it as, I should probably know this. I had a great deal of experience at Google working with our job control systems there, Borg. I'd also worked with many job control systems previously, everything from Sun Grid Engine to Condor, all sorts of archaic technologies. So I put in my New Year's resolution that I was going to learn Kubernetes.

ADAM GLICK: And which year was this New Year's resolution?

THOMAS STROMBERG: This was 2018.

ADAM GLICK: OK.

THOMAS STROMBERG: So I figured it wasn't something I was necessarily going to use at Google, but I should know what's going on outside of Google. So a couple of months in, I saw a job posting internally from Dan Lorenc, who you interviewed about Minikube last year asking for people to join the team to work on Kubernetes development tools. I was like, yeah, that sounds like a good way to fulfill a New Year's resolution. Let's just join a team that does it.

And I tried Minikube before applying for the job just because it seemed like the easiest way to get started with Kubernetes. And four hours later, I am swearing at my computer, like, what the heck is going on here? Why do I have to learn all these things? Why do I have to know what can KVM is and all these other things? I just want a Kubernetes cluster to work on this machine. It shouldn't be this hard.

CRAIG BOX: That must've made for an interesting interview with the guy who wrote it.

THOMAS STROMBERG: Yeah, well I didn't realize it was the guy who wrote it.

CRAIG BOX: Who you'd be doing a job interview with?

THOMAS STROMBERG: Yes, so I hadn't quite caught on with that yet. But I did describe my frustrating experience. And I said, I think there's room for improvement here, and I think I can help.

ADAM GLICK: And you got the job.

THOMAS STROMBERG: And I got the job. And I think part of what made it very easy for me is I had developed a very similar system at Google for doing local simulations and distributed simulations of Google's production environment, which was a very ambitious project for us in many ways because, with a Kubernetes cluster, you define, here are all of my applications. Here are all of my microservices. And you've got all the discovery built in for how services should talk to one another. Everything's just built in once you have Kubernetes set up.

But at Google, a lot of these technologies hadn't been set up constantly. And so some of them had only been, like, ever installed once. And so we had a great deal of complication trying to describe, how do you build Google from scratch? And so a lot of my time in developing this production simulation was actually going with a very similar model to Kubernetes, where you have great piles of YAML and say, this is what my intent is. Build it out.

ADAM GLICK: I love that. I want to hear someone start defining their application as, I have great piles of the YAML.

CRAIG BOX: Goodness gracious. I have had some conversations, which we probably can't talk about on this show, with the team who were looking at what must be considered the ultimate DiRT exercise is, what happens if we don't have Google in order to start Google again?

THOMAS STROMBERG: Yes, so I was part of that discovery process at Google.

ADAM GLICK: Do you want to explain what DiRT is for those who might not be aware?

THOMAS STROMBERG: DiRT is an exercise where you try to simulate what would happen if parts of Google's infrastructure go down. And for instance, you cut connectivity links between a data center and a corporate office, or you take down a random service. And people scramble for that week. I remember my first DiRT exercise because I was on-call. It was my first on-call at Google.

CRAIG BOX: A good week to join.

THOMAS STROMBERG: And yeah, I was all of a sudden told that one of my production systems in a particular cluster, that zombies had taken the machines away. So somebody actually did go and unplug the machine just to be thorough. This was a very different Google. This was a 2007-era Google, so things worked a little differently then. But--

ADAM GLICK: Were they in zombie costume when they did this just for full reenactment?

THOMAS STROMBERG: Unfortunately not, but we did recover from the zombie invasion, and it worked out quite well. It's a great learning exercise. Something that we practice more often on teams is this great thing called wheel of misfortune. It's like a mini DiRT exercise for every team throughout the year. And you basically get put in the hot seat to solve production issues. So I've gotten very good at debugging, and it's actually helped immensely for Minikube.

CRAIG BOX: It might be worth digging into that a little. You have a background in systems administration. And one thing that I find is the thing that you mentioned before, people who have to maintain a system are always the first to say, hey, that was a horrible experience. I'm going to improve it. Do you think systems administrators make the best Kubernetes core developers?

THOMAS STROMBERG: I wouldn't say the best, but I think having that knowledge and having that experience of being dragged through the coals trying to debug a system that you know nothing about in the middle of the night teaches you a certain discipline about, how do you make unexpected actions debuggable? Like, how do you leave the trail of breadcrumbs to help the poor person who's going to be paged later? How do you make behaviors discoverable? How do you make tools debuggable?

I think those insights help a lot. In fact, a lot of code review comments that I end up leaving is like, please add a debug line here because when it goes wrong, I want to be able to debug this later. So I think that it's certainly helpful.

I think the system administrators often times are extreme pessimists in that they have seen things go wrong. So there is always this expectation of what goes wrong. And I find a lot of times I don't know if we're necessarily the best engineers, but I think we might be some of the best code reviewers. Because we will look-- every line for us is like, how is this line going to shoot us in the foot later?

CRAIG BOX: Well, I've heard a quote which I will attribute to Miguel de Icaza, but it probably goes back a lot further than that, which talks about, there are people who have a tendency to see the negative in things or tend to be pessimistic are either great at or that can be caused by debugging, where you have-- your entire life is finding the one broken thing within a paragraph of code, for example.

THOMAS STROMBERG: When I learned how to use computers, I learned on a Timex Sinclair ZX81. And for those of you unaware, it had no persistent storage mechanism built in. So when I wanted to play games as a kid, I would get these books or magazines. And they would be filled with the basic code.

ADAM GLICK: No way, you wouldn't type those in on the membrane keyboard?

THOMAS STROMBERG: Yes.

ADAM GLICK: Oh, my god.

THOMAS STROMBERG: You would type these in, and you'd spend a day or two typing it in and then three or four more days debugging that one line that you had the typo in.

CRAIG BOX: Someone sell that kid a tape drive.

THOMAS STROMBERG: Yeah, I know. So my older brother got the tape drive. He had the fancier machine. He had like the spectrum. But I wasn't quite so lucky. So I lived in South Florida at the time, and there were power outages constantly in the summertime.

And so I got really fast at typing these in because I knew in the summer at best I'd have one week where I could leave the machine on without losing the game. And then I'd have to type it all over again and do the debugging. So it's this constant feedback loop of type in, debug. So I consider this my super power now as an engineer.

ADAM GLICK: That's great. You mentioned the first four hours in particular that you were using Minikube. What do you remember of that?

THOMAS STROMBERG: I'm a little fuzzy on the details now, but I remember KVM was involved. At the time, Minikube had two options on Linux for running. You could either do VirtualBox or KVM. And I knew KVM was native to Linux kernel, so of course I was going to default to it. Like, why install another piece of software?

And then all of a sudden, I realized Minikube actually doesn't work with KVM unless you download another tool, which is the KVM driver for Minikube. And I was like, this is dumb. And then I ran into all sorts of problems with libvirt. And you have to do all sorts of configuration to make, at least on the platform I was on at that time, which I don't remember which Linux distro it was-- like, libvirt was misconfigured.

And so I spent an hour or so trying to figure out, oh, I have to add myself to this group. And it's this constant debugging session just to get a VM started. And when I finally got down to it, then after a lengthy, lengthy download where Minikube told me nothing about what it was doing, I basically sat and watched my screen do nothing for half an hour. And then eventually I got a timeout message about it failed to set up Kubernetes.

CRAIG BOX: You had one job.

THOMAS STROMBERG: Yeah, you had one job. It was, like, some sort of timeout with kubeadm, like, probably that the kubelet didn't start. And I was like, OK, now I get to learn what kubeadm is. Now I get to learn what a kubelet really is. Let's figure out if we could actually get this thing up and running.

And what I eventually found was the secret of the time for Minikube was if it doesn't work the first time, delete it and do it again. And by attempt number three, I actually got a running cluster. But I think just the length of the downloads were quite painful. And I didn't have a very good internet connection at the time.

And I didn't realize that in the background it was downloading, like, a CD's worth of information, 650-plus megs. Kubernetes is pretty big now. It was big then, and it's big now.

CRAIG BOX: When Minikube was started, was it explicitly targeted at new uses?

THOMAS STROMBERG: When it first started, pretty much all Kubernetes users were new users--

CRAIG BOX: Fair enough.

THOMAS STROMBERG: --back then. When I joined the project, we started discussing, what is the point of Minikube? Like, what are our principles? And the first and foremost primary goal of Minikube is to make it simple to run Kubernetes locally.

And when we've done our surveys, we find that 30% to 40% of our users are actually using this as their first Kubernetes experience. They are running Minikube to learn Kubernetes just like I was. And a lot of the other ones are, I want to develop an application locally. And so we try our damnedest to support both use cases.

ADAM GLICK: Is it still as difficult as when you first started on it?

THOMAS STROMBERG: Oh god, I hope not. It's hard to say without having the beginner mindset. But I would say really fundamentally what we have spent the last year and a half doing is making it easier to have a getting started experience.

Nowadays, we are pretty good about showing what the progress is. We are really good about auto-downloading things that we might need, like the KVM driver, for instance. We'll automatically download that based on my experience the first time. We have spent a lot of time trying to basically make it so that our ideal is that any machine in the world, someone downloads Minikube, they start it up, it downloads or installs all the dependencies required.

And we don't handhold enough yet, I think, for all people, but I think we do a really good job, especially in comparison to what we did before. Kubernetes as a community in the last years, it's grown significantly larger. It's a lot more diverse. And so we've been trying to make Minikube inclusive to all users from anywhere in the world.

So that means run on any platform, support local languages, and cater to people who are not necessarily technical yet. I don't want people's first experience with Kubernetes being, how do I debug Kubernetes? Because this thing is not working. That's a pretty rough transition.

CRAIG BOX: You gave a talk last year in China behind the so-called Great Firewall. And one of the things that came up in that talk was the fact that you couldn't actually download Minikube because it was hosted on Google Cloud, and that was banned. And so I think one of the things that came of that was obviously a need to move that ISO. But what are some of the examples of other things that you found through either being a user or simulating that experience or talking to users that have helped you improve the experience?

THOMAS STROMBERG: We have a survey that we review every two weeks people's results on. And one of the best questions on that survey to give us that empathy with users is, if you could fix any one thing with Minikube, what would it be? And we end up trying to prioritize the items that we see there for our milestones.

I was actually really surprised by how bad the experience was in China because we weren't seeing that user feedback and, I think in large part, because our surveys are in English only. They definitely lean themselves to an English-speaking audience. So we were missing this whole huge swath of feedback. Additionally, our survey was not available in China either.

CRAIG BOX: That's a good way to make sure you don't get any feedback.

THOMAS STROMBERG: Yeah, due to the Great Firewall of China. So it was really good to go there and listen. After my talk, I had a local university professor take me aside and say, I'm using Minikube in my classes, and everything you said in this talk was real. It really is painful. I'm really excited to help you fix it.

So we've had some great contributions from users in China since. We have relocated the ISO so that it's also hosted on GitHub. It can still be really tricky, though, because to this day, a lot of the Docker images for Kubernetes are hosted by GCR, which is not available in China. So depending on what you're doing, you might come into a case where you can't do something, like, hey, I want to enable the Kubernetes Dashboard. So we actually now-- we'll sneak those Docker images in for the Dashboard even if it isn't a required feature just so we have that smooth experience for the new users.

Other things that I've heard a lot about is, I'm on a remote island in the South Pacific, and your download behavior is terrible.

CRAIG BOX: That's because you keep adding Docker images and things like the Dashboard.

THOMAS STROMBERG: And so one of the problems is that we used to download everything in parallel. And so on these bad connections, nothing would ever complete. Everything would just basically swamp the connection, fail eventually, and not resume properly.

So there are a lot of things that we had to listen to for user feedback. Especially as we get more and more experience with our own tools, we forget what it's like to be a beginner. And if we are not consciously listening, we will fail to make tools that work for everybody.

ADAM GLICK: Speaking of making things accessible for everyone, you installed Minikube on a Linux distro. I've played around with it on my Mac. What proportion of the world's desktops are running those operating systems? Are there still people that we need to reach out to?

THOMAS STROMBERG: The proportions are really different depending on the country that you're talking about. I actually haven't looked at the global numbers lately. But one thing I found really fascinating was that the percentages for Kubernetes developers on different platforms is very different in America versus China, for instance.

When I looked in China at the stats, I think it was like roughly 85% of developers were on Windows. And if you look at the developer stats here in America, it's a very close number on Mac instead. And I think a lot of developers in America have transitioned to Unix-based operating systems.

But it is not the case in many other places. India also has higher percentages, for instance. Mac OS is-- you pay a price premium for it. And so in countries with less disposable income, you won't see Macs very commonly. But they're everywhere for America for developers.

ADAM GLICK: And we're still waiting for the year of the Linux desktop obviously. To that end, what are we looking at in terms of Windows support?

THOMAS STROMBERG: We had a big effort last year to make Windows a first-class citizen for Minikube. We've come a long way. I think we've basically called it done at this point as far as being a first-class citizen.

So we support Docker quite well on Windows. We support Hyper-V pretty decently. It's a little complicated with Hyper-V because of the privilege escalation issues. But all the feature sets should work on Windows out of the box now. And this has really become more important to us as we've had tool vendors come to us, for instance, IDEs who have said, I want to be able to use Minikube from inside my tool set, and I've got Windows users. Please make it work out of the box without Windows users having to configure anything.

And so we have come a long way there. So now it is one of our-- I wouldn't say it's as well supported because a lot of the Minikube core staff, I guess, maintainers are not Windows users. But we do have a couple now. So it's come a long way.

CRAIG BOX: Did you trade your Linux laptop in for a Windows machine?

THOMAS STROMBERG: Actually, as part of making this work better, I did for a little bit. I did switch to using Windows at work for quite some time to get a better sense of empathy. And all of a sudden-- like, one of the problems initially was we didn't even have build-- like, our makefile for Minikube didn't even work on Windows. You had to cross compile in Linux.

So we were like, OK, clearly we can't iterate very well on that. So we improved that. We made quite a bit of improvements.

That said, no, I don't use Windows full time right now. But I do have a KVM switch where Windows is now a button away. And so whenever somebody comes to me with, like, an obscure HyperV issue, I'm like, OK, now I can take a look at this.

CRAIG BOX: Since Minikube launched, there have been other solutions in this similar kind of space. We have K3s from Rancher, kind, MicroK8s from Canonical, for example. What have you learned from looking at those other distributions, and how does Minikube compare to them today?

THOMAS STROMBERG: I love that there's a lot of options now. When Minikube first started, it was the only option for running locally. A lot of them have different spins on things and different behavior sets.

So yeah, there's MicroK8s, as you mentioned. MicroK8s is quite similar to Minikube, but it only works natively on the snap-based distributions. And I've actually never had snap work on a machine to try it, unfortunately. But I do love their documentation. I definitely took inspiration with how smoothly they made the getting started experience on their documentation. Feature set-wise it seems very, very similar.

There is k3d which is running k3s in a Docker. Again, pretty similar, a little bit more limited in configuration, but it is very fast to start. And part of the reason why it's so fast is it doesn't use kubeadm. It bundles all of the Kubernetes services into a single binary. It throws that CD away. So it's very fast to start, but it is difficult to make it to match your production configuration unless your production configuration happens to be k3s already.

And then there is kind, which is also a Kubernetes SIG project. And we've taken great inspiration from them. They went the route to make a great experience with Docker. And this was also on our roadmap, to have a great VM-free experience.

Previously when Minikube-- for many years, the only VM-free experience was interacting with Docker on a Linux machine with root. And it was quite a painful experience honestly, definitely more for the people who already have provisioned a VM. And so we adopted a lot from kind.

In fact, when we use the Docker driver, you're actually using their boot image, for instance, at the moment. We owe a lot of credit to the kind folks. But ideally, Minikube is trying to work for everybody, no matter the configuration. So we do allow a lot of tuning that these other local Kubernetes solutions don't allow for, like swapping out your container runtime, for instance.

CRAIG BOX: Given your history, you might understand what I mean if I were to mention KNOPPIX.

THOMAS STROMBERG: Yes, absolutely. I loved KNOPPIX for whenever people would come to me with, oh my god, my machine is broken. I can't figure out how to get on it. That was the way to go.

CRAIG BOX: In the old days, you used to install Linux from a CD, which would boot up an installer and then partition your disk and so on. KNOPPIX was a live environment where you could boot off the CD and be running in a Linux environment that was basically a drive image on the disk. What you mentioned before about kubeadm makes me think about this. Why is the VM that is distributed with Minikube installing Kubernetes rather than just having a pre-built one that is like that live environment?

THOMAS STROMBERG: The reason for it is because we allow you to specify any Kubernetes release within at least the last six versions. So if your production environment is running Kubernetes 1.13.1, you can specify Minikube start, yadda, yadda, 1.13.1. There are a bunch of other flags that you can specify, like, hey, I don't want to use containerd. I want to use CRI-O.

So we have talked about having some of these pre-baked in combinations as something that you can quickly hydrate and have a faster starting experience. But we haven't really gotten there yet. What we have done, for instance, to speed things up is when you do select a Kubernetes version we download a tarball of, here are all of the containers that you're going to need. And it's already preloaded in that container runtime format.

And so we basically just download it and untar it into the ISO. And then, poof, you have all these images preloaded in. But we do still run kubeadm, which does cost us about 20 seconds. But so far for the flexibility, it's been worth it.

ADAM GLICK: It sounds like you've built a lot of features to try and make Minikube as closely resemble a full Kubernetes environment as possible when people are running them locally on a machine. Indeed, I've heard that you can actually take Minikube and put it in a cluster. When would someone want to do that?

THOMAS STROMBERG: When they're bored at home on a Friday night with a tub of ice cream in front of them. So technically, really, you can join any node from any Kubernetes cluster into a federated cluster as long as there's TCP connectivity between them and SSL certificates. So you can add a Minikube node to a kind node to a MicroK8s node. You can join them all together if you want. You're going to have a bad time, but you can do it. And it's not something I recommend unless you're just super curious about how Kubernetes and kubeadm work.

CRAIG BOX: Let's say you have a cluster of Raspberry Pis on your disk and you want to experiment with having more than one machine. Would Minikube fit that use case, or would you recommend something else?

THOMAS STROMBERG: I would personally recommend MicroK8s on this. I think they've done a great job of documenting how to bring multiple physical nodes into the same cluster. They've done a better job of anyone else in that area that I know of.

I haven't actually explored whether or not k3d has this documented on how to do this. But Minikube is, by design, running on a single physical machine. And so far, we're planning on keeping it that way until someone convinces us otherwise.

CRAIG BOX: We've talked a lot about the getting started experience and new Kubernetes uses. Another case that people need a local Kubernetes environment is when they are familiar with the technology but they're developing. They're working on local applications, and then they're going to deploy to a cluster somewhere else. Do you see a lot of people using Minikube for that particular use case?

THOMAS STROMBERG: Absolutely, and it's one that we optimize for. So I guess we see Minikube as being that local dev system for them that they develop their app. It looks good in Minikube. Then they push it to a test, looks good in test. Then they push it to production.

So one of the things that we really value, and a lot of the design decisions for Minikube have been around, how do I give developers the fastest iteration process possible? One of the special features for Minikube is that it will provide a Docker end point so that you can build images from your host inside of Minikube's Docker and do that quick iterative development and only stream the changes when you make a change locally. So if you're using a tool for iterative development in Kubernetes, like Skaffold, for instance, which is another tool my team works on, you're going to have the fastest experience if you're using Minikube underneath. Because it can talk directly to Docker.

And all the other local Kubernetes platforms have switched to containerD for their container runtime. But we've kind of stuck with Docker as the default because it has that feature. We do support containerD if you like it, though.

ADAM GLICK: We last talked about Minikube in February of 2019. What's new with the project since then?

THOMAS STROMBERG: Oh, god, there's so much. When you last spoke to us, we weren't even 1.0 yet at that point. I had just started on the project a couple of months ahead. We did this big push for 1.0 where we said our focus for 1.0 is going to be reliability.

There were a lot of rough edges and failure cases for Minikube. And so we doubled down on integration testing and reviewing all possible error scenarios, adding a lot of support for trying to detect errors and present them to users. One of the cool things that Minikube will do right now is that if it detects there's any kind of pressure, like disk pressure or memory pressure, it'll let you know up front. It'll go ahead and do those queries and give you warnings.

But otherwise, new things-- so we talked about trying to make it inclusive for all users around the world. So we localized to seven different languages, everything from Polish to Chinese. We did the auto-installation of drivers that we talked about before.

We added Docker and Podman support for people who don't want to have a separate VM running on their machine. That's been a huge change for us. The pause command is like one of my personal favorite things that we added to Minikube since then, which allows you to pause Kubernetes using one of the Linux features called freezing where you can freeze cgroups. So you can actually use the pause command to suspend Kubernetes but leave your apps running.

So if your laptop battery is draining on you or you hear the fan spinning, you just run Minikube pause. You can keep running your application. In fact, you can actually keep talking to the Docker daemon in it. So it basically gives you about 2% CPU overhead. And within 500 milliseconds, you can switch back to having Kubernetes on.

We added multi-node support. So on a single physical node, we can now provision multiple nodes. And then we did a lot to decrease startup latency and CPU overhead. A lot of our developers are on their laptops, so we worked a lot to lower CPU overhead. We've lowered it 50% since last year and decreased startup latency by 2 and 1/2 times. So for most users, it only takes about 30 seconds to start a Minikube cluster once you've actually done the initial download. We've added a lot to our add-on ecosystem. So easy addition-- you can easily add Istio or KubeVirt or MetalLB to Minikube if you like.

CRAIG BOX: I remember there was a little bit of a challenge getting Ingress running back then.

THOMAS STROMBERG: Oh, yes, it was. And Ingress is complicated. So we do have built-in load balancer support. We did have the tunnel command back then. But yeah, we've come a long way in making it easy to do the Ingress and supporting multiple configurations for that.

CRAIG BOX: There are a couple of open-source projects that have spun out of Minikube, things that you've built to help manage the program that are now used by other things in Kubernetes. Tell us a little bit about Triage Party and Slow Jam.

THOMAS STROMBERG: I mentioned the necessity for engineers to listen to their users. Like, GitHub issues, a lot of engineers, they hate triage. They see it as like, here is a chore that I have to do. But I see GitHub issues as our way of, as engineers, to learn from our users. And it's really important to have that open ear.

And we had such a problem with Minikube because we have all these brand new users. These people are brand new to Kubernetes, brand new to virtualization, and they would open up issues that were difficult to comprehend. And we would lose these issues.

Like, we would post a follow-up comment, and then we'd forget about it for three or four months. And by then, the users moved on. Maybe they're not even using Kubernetes anymore. We've lost them.

To work around this, we've built this triage process where we had this Google doc of, here all the GitHub queries to type in. And we got tired of typing all these in manually. So we developed Triage Party.

It's open source. We call it a massively multiplayer triage for GitHub. And it basically has these bookmark queries where you can say, here are the things I want to know about every day for Triage. Here are the things I want to know about every week for Triage. And you can put it in this multiplayer mode where you have a meeting with all your other contributors once a week.

CRAIG BOX: Every Friday night, you all get your tub of ice cream.

THOMAS STROMBERG: Exactly, we get our tub of ice cream, we go to this page, we can select which player number we are, and it'll divide up all the issues and PRs among the players where we can triage them collaboratively. And if somebody has a question, they might present on the screen like, hey, I've got a question about this thing. So that's been really good.

We've seen some good uptake on other projects. We've recently added some built-in support for helping plan your work in the future as well as triage. So that's been really great to see that happen.

ADAM GLICK: Sounds a little bit like parents playing the cleaning game with their kids. Hey, we're going to do the fun cleaning game. How many of those things can you put back where they belong?

THOMAS STROMBERG: So it totally is. Because the idea for Triage Party is that each of those pages should be empty at the end, and then you get your reward. So it's been adopted by a bunch of peripheral projects like Bazel and cert-manager as well as now kubectl. And we'll start to see it happen more within Kubernetes. We've had to do some tuning to make it work for some of the larger projects like Kubernetes Kubernetes. Expect to see it everywhere.

ADAM GLICK: What about Slow Jam?

THOMAS STROMBERG: Yeah, so I talked about the startup latency improvements. And part of the difficulty for startup latency was just not having a clear vision into what was causing-- what we were blocking on. And there's a bunch of CPU profilers for Go. There are memory profilers for Go. There are a great variety.

But there were no profilers that just let us see function latency at the time. So we built a profiler that lets us basically build a Gantt chart of here is what Minikube is doing, and here is what's blocking. So we could see, oh, loading the container images is taking 15 seconds of our time, and it's blocking the next step. Can we run these two steps in parallel instead?

ADAM GLICK: I love that you just threw Gantt chart in as kind of, like, oh, of course, a Gantt chart.

THOMAS STROMBERG: Yeah, well, to me it made sense.

CRAIG BOX: I love that the Slow Jam logo is clearly a take on the "Space Jam" logo.

THOMAS STROMBERG: Absolutely, different enough to avoid the copyright police, but definitely inspired.

ADAM GLICK: Michael Jordan won't be calling.

THOMAS STROMBERG: I hope not.

ADAM GLICK: Well, what comes next for Minikube?

THOMAS STROMBERG: Honestly, it's really up to the contributors. We have a road map of here are the things that we'd like to do. But a lot of it is just based on what contributors share with us. We definitely have a vision of where we'd like to be at the end of 2020.

Part of it is better documentation. It's no mystery that great documentation drives adoption. And I think honestly, great documentation is the best predictor for the long-term success of any open-source project. And we want to make sure that our documentation is there for everybody, that it's localized just like the rest of Kubernetes, and that it caters to people who aren't yet experts.

We do want to improve integration with other tools. We want to make sure that people can consume the output with JSON, for instance. We'd kind of like to refactor storage, especially now that we're talking about multi-node. Right now we've got this great host path provisioner, which is wonderful for provisioning on a single node. But we'd like to explore things like using NFS or CIFS or something like that for built-in storage provisioning within Minikube as well.

One of the other things that I'd really like to do is we demo'd at KubeCon San Diego this idea of having a menu bar UI. We want to make people feel comfortable with running Minikube in the background. And that's been a lot of why we have driven reducing CPU overhead and adding the pause command.

We'd like to make sure that, for your Mac or Windows user, it's sitting there in the tray and that you have quick ability to pause and switch contexts. So that's something that we would like to do. We'd like to finish that prototype and ship it.

ADAM GLICK: In the intro we talked that you like to create bamboo bikes. Why bamboo bikes?

THOMAS STROMBERG: I live in a tiny San Francisco apartment. And I've wanted to build a bike forever. And then I was cycling to work one day, and I kind of ran over myself in a way that I still can't quite explain. And I broke my foot.

And so I'm like, I can't cycle for a month or two. What am I going to do? So I was like, here's my perfect opportunity to build a bike. And I looked into my options. And I was like, yeah, I could get a TIG welder. And I'm like, you know, my landlord might not love that idea.

CRAIG BOX: You know people sell bikes pre-made.

THOMAS STROMBERG: Yeah, that's not so much fun. It's like anything else with engineering. You want to know how it's built. You want to--

ADAM GLICK: Yeah, do you want to build it, or do you want to buy? It's the build buy decision. And clearly, he's in the build camp here.

THOMAS STROMBERG: I mean if you really want to learn the failure cases of anything, you should try building it yourself.

CRAIG BOX: It's that system administrator mindset.

THOMAS STROMBERG: Absolutely. And I think my wife had once showed me this video of somebody riding a bamboo bike. And she said something like, doesn't this look interesting? I was like, yeah, but that's a lot of work.

CRAIG BOX: And it won't survive the upcoming panda apocalypse.

THOMAS STROMBERG: Exactly. So with a broken foot, I bought this kit from a company that-- there are several companies that will sell you the bamboo, the instruction kit, some of the metal parts you might need. I basically hobbled around the garage with a broken foot trying to navigate my way to the saw and the hole cutter and eventually set it aside until my foot healed up because honestly, it was kind of hard to get leverage. But I built a bike, and I was like, OK, this is fun.

It took me, like, six months to actually build it. Let's do it again and see if I can get faster. And so the next one took me about three or four months. And the next one took me about a month. So I'm getting better at it. It's kind of building a sense of craftsmanship. It's getting me away from the keyboard. And it lets me play with some neat ideas.

CRAIG BOX: All right, well thank you so much for joining us today, Thomas.

THOMAS STROMBERG: Yeah, thank you. It's a pleasure to be here.

CRAIG BOX: You can find Thomas on Twitter at @thomrstrom, or on the web at stromberg.org/t.

[MUSIC PLAYING]

ADAM GLICK: Thanks for listening. As always, if you've enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at @KubernetesPod or reach us by email at kubernetespodcast@google.com.

CRAIG BOX: Check out our website at kubernetespodcast.com where you will find transcripts and show notes. Until next time, take care.

ADAM GLICK: Catch you next week.

[MUSIC PLAYING]