ADAM GLICK: Hi, and welcome to the "Kubernetes Podcast from Google." I'm Adam Glick.

CRAIG BOX: And I'm Craig Box.

[MUSIC PLAYING]

ADAM GLICK: I had a wonderful week this past week. I got to take a few days off of work and spend it with the family, which was wonderful, and really had this just great moment to remind yourself that sometimes one little detail can just change your whole perception of something. I went out, and I got the mail, and then I'm going through the mail. And someone stuffed a flyer into our mailbox.

And lots of people are looking for additional ways to make income, and so they're talking about, hey, I can help mow your lawn, or walk your dog, or sit your cat. And I'm like, is someone really printing these things out and stuffing them into mailboxes? And I'm like, what's going on here?

I get to the end of it, and the last sentence just totally won me over. It says, "if interested, please call my mom", and then it has a phone number. And I was like, oh, so then the heart melts. You're like-- OK, my perception totally shifted on this thing.

And I was like, we don't have a cat or dog, but what is something that we could do just to help them out? And it was one of those things that just reminded me of there's a lot of ways to view any situation.

CRAIG BOX: Do you need any lemonade?

ADAM GLICK: Lemonade? Maybe if they set up a stand. I'm not sure how COVID-friendly that would be, but we'd probably buy a glass anyway.

CRAIG BOX: The thing that you did some people like to describe as a staycation, and that's a word which, again, perspective-wise the British people recently have been trying to use that word to mean a holiday in the country. And that just surprises me. I know this is an island, but the idea is that they think all holidays should be holidays abroad, and any holiday that's kept inside the country they're considering a staycation. That's not what the word should be used for. The word should be used for staying home and eating steak!

ADAM GLICK: [LAUGHS] Well, that's another possibility. That may be tough for the vegetarian and vegan members.

CRAIG BOX: That's a good question, though. Is steak a cut of a thing, or does a steak imply meat? You hear about beefsteak tomatoes, which are definitely disappointing when you go and eat at a restaurant named Beefsteak, and you don't know what you're getting in for.

ADAM GLICK: [LAUGHS] I believe I've eaten at that same restaurant, and I must say that I quite enjoyed it.

CRAIG BOX: I'd say you probably enjoyed it more than I did.

ADAM GLICK: Probably. I probably knew more of what I was getting into. You know, speaking of which, got a chance to do a little cooking this week, as well, and made nachos, which is just one of those things that-- it's not necessarily health food, I would say, but it certainly is delicious. And nachos and cheese, or you can use fake cheese if you want, and ground beef, or fake ground beef. And so all of ours is the various fake equivalents thereof, but it's just you bring out this tray of these things. And you turn on a game, and it's just this wonderful experience.

CRAIG BOX: Do you have real tortilla chips at least, or are they fake, too?

ADAM GLICK: Well, they're made with real corn, but they are not made out of tortillas, like presumably the namesake would imply, so perhaps.

CRAIG BOX: Should we get to the news?

ADAM GLICK: Let's get to the news.

[MUSIC PLAYING]

ADAM GLICK: Chaos Mesh has hit version 1.0 and is now generally available. The chaos testing project grew out of TiDB and you can learn more about both in episode 121. It's one of two chaos engineering projects currently in the CNCF sandbox. Congratulations to the team.

CRAIG BOX: Kubernetes news from Microsoft's Ignite conference last week included a preview of Azure Kubernetes service on Azure Stack HCI, which lets you run Microsoft to manage Kubernetes flavor in your own data centers. If you're running AKS on cloud, there are a number of preview features, including stopping and restarting clusters, confidential compute nodes, and Kubernetes 1.19. Bridge to Kubernetes, a Visual Studio plug-in that allows you to work locally on a micro service while it has connectivity to other services in a remote cluster, is now a GA.

ADAM GLICK: The first Istio Steering Committee election has concluded with four community representatives being elected to the project's governance group. Congratulations to Neeraj Poddar of Aspen Mesh, Zack Butcher of Tetrate, Christian Posta of Solo.io, and Zhonghu Zu of Huawei. They join representatives of Google, IBM, Red Hat, and Salesforce as the top contributors to the project in 2019.

CRAIG BOX: Microsoft's new Envoy-based service mesh, Open Service Mesh, has been accepted into the CNCF sandbox. The project aims to be lightweight, is semi compliant, and uses the XDS APIs. It becomes the fourth service mesh predict in the CNCF, with none yet having met the requirements to graduate.

ADAM GLICK: It appears "Deep Space Nine" made an impression on the team at Red Hat. The odo project is now GA at version 2.0.0. odo is a dev focused CLI for Kubernetes and is unrelated to the Odo Security company. Changes in version 2 include a switch to using a dev file for deployment. The operator hub is now out of experimental mode, and the debug mode is out of technical preview.

CRAIG BOX: Determined AI, the creators of an open source deep learning training platform, have released native support for running on Kubernetes. Determined aims to make model building fast and easy. And after realizing that most people were running Kubernetes for non-GPU workloads, they now provide easy installation on it with a help chart.

ADAM GLICK: Google Cloud has announced the beta of event support in Cloud Run for Anthos. The new feature announcement comes with example use cases like triggering code on changes in a Cloud Storage bucket, a BigQuery audit log, a Cloud Scheduler event, or even custom defined events.

CRAIG BOX: Looking to learn more about Kubernetes? VMware thinks they have the right information for you. Building on their previous KubeAcademy trainings, which offered five- to 15-minute introductory courses on upstream Kubernetes, the new KubeAcademy Pro tier offers in-depth sessions on concepts like networking and operations, along with access to virtual events, AMAs, and other goodies. The new Pro offering also comes with an assessment test so you can figure out which training you should take first. The Pro tier is available now for the cost of filling out a form with your contact information.

ADAM GLICK: The CNCF has released a conference transparency report for the KubeCon EU virtual event. The event had over 13,000 attendees, with over 9,400 people attending for the first time. Attendees from almost 8,000 companies joined from 130 countries and six continents. Once again, our community failed to entice someone from an Antarctic polar research facility to join.

61% of attendees spent over 10 hours at the virtual event. 9% of attendees were female, a slight drop from the last KubeCon, but 74% of keynote speakers identified as women or gender nonconforming, which was a significant increase. About 3% of attendees received a diversity scholarship. Congratulations to the CNCF team for putting on such a successful event.

CRAIG BOX: Speaking of diversity scholarships, the CNCF is once again providing opportunities for KubeCon and pre-day attendance for people coming from underrepresented groups and those with financial needs. Scholarships are available for KubeCon, ServiceMeshCon, and Cloud Native Security Day. Applications are due by November the 1st for the event starting on November the 17th.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Mike Danese is a software engineer at Google working on Kubernetes and GKE. He is the chair and tech lead for the Kubernetes auth SIG. Welcome to the show, Mike.

MIKE DANESE: Hi, thanks for having me.

ADAM GLICK: I heard you got your first programming job in what some might call a "non-standard" way. How did you get that job?

MIKE DANESE: Back my freshman year of college, I worked in a bio lab, and half the lab was a wet lab. Half the lab was a dry lab. I worked in the wet lab.

That means we were working under big chemical hoods pipetting. I ran these cloning cycles where we took bacteria called Klebsiella ammonia, injected some DNA, and then cloned it. And this whole process took three weeks. I have never been very attentive to details, so I was actually terrible at this job, but it turns out, halfway through the summer of my freshman year, I broke my hand. I slammed it in a door, and I had a big cast.

CRAIG BOX: Ow!

MIKE DANESE: Yeah, it was terrible. And I couldn't work in the wet lab anymore because I couldn't work the pipettes. So they picked me up and put me in the dry lab, where I worked on modeling of biological systems, so a lot of numeric Python, a lot of linear algebra. And I got hooked. I changed my major from biochemistry to math, and it was all programming from there. So that's how I got my start.

ADAM GLICK: You ever gotten a job that same way again, through a personal injury?

MIKE DANESE: No, fortunately.

CRAIG BOX: I'm saying, if you had, there's this guy you should call.

MIKE DANESE: Yeah, I mean it worked out. It worked out. No regrets.

ADAM GLICK: How did you get started working on Kubernetes?

MIKE DANESE: Back in maybe 2014, I was interested in the project. I was working in a startup in Seattle, and I started to contribute code. And Joe Beda merged my first pull request-- number 2000 or something like that-- and I continued to contribute for a little while. After maybe six or nine months, I took a position on the GKE team at Google, and I've been working on GKE at Google for over five years now. So that's all ancient history.

CRAIG BOX: Today we'd like to talk to you about some concepts which are low-level computer science concepts, but have serious applicability to Kubernetes. And to do this, it might help to jump back quite early in the stack. There's a lot of challenges in running a distributed system, so I want to look at some of these concepts, first of all, from the idea of running a single process. If you're running a process on a kernel, it can have multiple threads. What's the difference between a thread and a process?

MIKE DANESE: To be honest, not much, especially in Linux. I would say that, if somebody says thread, they are usually implying that multiple threads share the same virtual memory space. And if somebody says process, that would imply that the process lives in some virtual memory space, although that gets kind of fuzzy. All threads and processes have PIDs in Linux, and there's not much distinction between the two. And there's a lot of different flags that you can apply to either make virtual memory copy on write or forked and exec.

CRAIG BOX: If I've written some code and I want to have, perhaps, multiple threads in that code, and they're all going to touch a particular part of the system, how do I make sure that I don't have a problem with them trampling on each other's memory, for example?

MIKE DANESE: The way that this is often done-- it's not the only way, but the easiest, simplest, way-- the way that people are most familiar with is by using locks or mutexes. Locks and mutexes are synonyms. Mutex, the term, comes from a joining of the two words mutual exclusion. So you can use a lock or a mutex to guard a critical section of code that is modifying some shared resource and make sure that if a thread is in that critical section then it is exclusively in that critical section. And no other threads occupy that critical section at the same time.

ADAM GLICK: What's a way that people might be familiar of when that would be used?

MIKE DANESE: So you have a logging library. We have a multi-threaded application. Each thread wants to write application logs to some file. If we let all the threads write to a file at the same time, it would just be garbled nonsense. So what we do is a naive way to do this is to have each thread write directly to the file, but take a lock before they write their log entries. So they write their log entry completely before giving another thread the ability to acquire the log file and write their own logs.

ADAM GLICK: Would this be somewhat akin to a transaction in databases?

MIKE DANESE: Yes.

CRAIG BOX: And no.

[LAUGHS]

MIKE DANESE: They are. Although the transactions for databases-- this gets a lot more complicated as soon as you require transactions for some data stored on disk. So far, we've only been talking about transactional memory, like a mutex that lives in memory. Transactions for file rights are actually fairly more complicated.

CRAIG BOX: These locks, or mutexes-- are they implemented by the standard library of your language? Are they implemented by the kernel that you're running your software on?

MIKE DANESE: Maybe I'll give a brief example, very simple implementation, probably the simplest implementation of how you could write a lock. So CPUs have an atomic operation called compare and swap. Basically, imagine I have a byte somewhere living in memory, and it has a 0 value.

I can submit an instruction to the CPU that says compare that byte to 0, and if it is 0, switch it to 1. So we can use this to build a very naive lock, where if the value of this byte is 1, somebody has switched it on. That would be the acquire operation. Later threads that come and try to compare and swap that value to 1 will not be able to because their compare operation will fail. And that allows the acquirers to acquire the lock, and then waders to spin trying to run this compare and swap operation. That's called a spin lock.

CRAIG BOX: I've heard it compared to people going and getting the key for the gas station bathroom, and then you have to wait there until the person who went and got the key finishes up and returns it before you're able to get that physical lock undone.

MIKE DANESE: Yeah, so I would say that this is very similar to a gas station bathroom, where all the people at the gas station are incredibly rude.

CRAIG BOX: Gosh.

MIKE DANESE: So everybody rushes at the bathroom at the same time. First person gets in, locks the door. Everybody outside is rattling the door. Nobody is waiting in any line. Nobody is giving you any privacy. And you wash your hands. You unlock the door, jump out, and then the next person tries to get in as fast as possible.

So not a very efficient way to implement a lock, not a very fair way to implement a lock. So there's some improvements made in practice that involve interaction with the kernel and the scheduler, because you don't really want to have a CPU going at 100% trying to acquire a lock. That's a waste of electricity. That's a waste of resources. That CPU could be used to do something else.

So there's something called in Linux the futex, which is a fast user space mutex, which first tries to acquire an uncontended lock in the way that I described earlier. And if that fails, if the lock is already acquired, it will do a sys call, and the thread will be de-scheduled. And the kernel is notified that this thread is waiting to acquire a lock. And the thread manages-- yields the CPU to something else that can use it and tries to implement some sort of fairness between different waiters.

CRAIG BOX: In the more civilized country, you might imagine some kind of queue for the gas station bathroom. Is there an equivalent to that?

MIKE DANESE: Yes, exactly. So that gets to the concept of fairness of a lock. So different lock implementations implement fairness in different ways, but usually this is desirable in locks that are very high contention. So it's actually sometimes fair bit more expensive to implement fairness on a lock.

ADAM GLICK: When you say fairness, that word can mean many different things. Would it be the one that showed up first? Would it be the thing that needs it most? How is fairness defined when you decide who gets access to this virtual bathroom next, so to speak?

MIKE DANESE: The answer is it probably depends, and it depends a lot on what obviously the lock implementer needs. In general, we sacrifice some fairness in order to make locking operations as fast as possible. In the spin lock example, you can imagine 60 CPUs on a very big machine spinning to try to acquire this lock. And if you look at a latency distribution of how long each thread is waiting, you can see some large tail latency if something is getting unlucky. Maybe a CPU is located far away from the NUMA node, and CPUs that are closer to it-- the memory bus that holds the specific lock have the ability to acquire a contended lock with higher probability. So generally what we want is to bound that tail latency of acquired operations on very contended locks.

CRAIG BOX: So up until now, we've been talking about a single process running on a single kernel. When we start talking about distributed systems, we're now talking about multiple machines that are disconnected from each other coordinating over a network. And then we hear about a thing called a lock server. There was the paper from Google on the concept of Chubby. And there will be implementations, such as etcd in Kubernetes, which many people will be familiar with.

What is a lock of server? And is it comparable to how the kernels do locking in the discussion we've had so far?

MIKE DANESE: I would say that distributed locks are very similar in most ways to the locks that we have described just now on a single machine, although certain aspects of distributed systems make the problem a bit more challenging. So etcd and Chubby, like you just mentioned, which is the system that the Google paper is based on, implement a locking API that is very similar to our compare and swap example, although both of them allow you to store additional data along with that bit value, or sequence number they're usually called.

So we can use etcd or Chubby in the same way that we use that compare and swap operation on a single machine. However, certain problems arise. So if I acquire a lock in a single process in a single virtual memory space, I enter a critical section, and I do something that I shouldn't have done. And I get segfaulted, and I crash.

Well, generally, I'm in the same memory space as all these other threads. They are also killed along with me. However, it's not really the case with a distributed system. So if you have a distributed lock, you have multiple contenders running across a data center. If a single process enters that lock and crashes, nobody else will know.

So in general, in distributed systems, we expect communication to be unreliable, and we expect processes to fail independently. We are well aware of this as both developers and users of Kubernetes. So when we design systems that require some sort of mutual exclusion in a distributed environment, we need to take this into account.

ADAM GLICK: Kubernetes objects are backed by etcd. What happens when two calls try and update an object at the same time? When you have this same kind of one bathroom and two users want to access it problem, what happens while the first one is still being updated?

MIKE DANESE: Our example from earlier about the log file, where we acquire a lock, and then we write some data, that's an example of a pessimistic concurrency scheme. In Kubernetes, the Kubernetes API uses an optimistic concurrency scheme to do exactly what you just described. So if I want to update a pod, for example-- I'm a controller-- I'll read that pod. I'll make a modification, and I'll update it.

So maybe you've noticed this field. Maybe you haven't noticed this field. Inside every pod, there's that object metadata. That object metadata has a resource version field inside of it, which is just--

CRAIG BOX: A big number.

MIKE DANESE: --a big number, but you should treat it as an opaque string. So every time an update is committed, that resource version changes. So if I read the object, the object is updated by someone else, and I resubmit. That resource version will have changed. If those resource versions don't match, the update is not permitted, and I get a resource conflict error. So that's an example of an optimistic concurrency model. It's optimistic because everybody's trying to write all at once, rather than trying to acquire a mutual exclusion before writing.

CRAIG BOX: You have the API server, which is the part of Kubernetes that interacts with etcd. If you are running a cluster with a single control plane node, you only have one instance of the operating. And you also only have one scheduler running. If you are going to run a regional cluster on GKE, for example, you have multiple control plane components, such that you can survive one zone going down.

In that case now, you have multiple replicas of things. Which of them have this problem of needing to select a single leader, and which of them are able to run multiple replicas concurrently?

MIKE DANESE: We hope that all the components that we write would work fine, work correctly, even if acting at the same time as other components. So maybe a simple example is a control loop that reads an object and updates the status. So in that control loop, if we have two processes simultaneously trying to reconcile a single object, the first process enters, updates the object. The second process gets resource conflict, bails out.

So assuming we don't have any programming bugs, which is never the case, we have a situation where all we get by enforcing some sort of mutual exclusion on the actuator process is a performance improvement. There are other types of failure modes. So one-- we just described the inefficient failure mode. Duplicate work wasted.

Another failure mode is your system becomes unavailable. So maybe in a hypothetical system that is not the Kubernetes controller manager, two processes reconciling at the same time prevent anyone from making progress. So that's a slightly worse situation. And then a third type of failure mode is when two processes are reconciling, or multiple processes are reconciling at the same time, that creates incorrect results-- so maybe data corruption or something like that. So that's the failure mode where mutual exclusion is most critical. So in Kubernetes, we aim to write controllers that are only inefficient in the case where multiple processes are reconciling.

ADAM GLICK: Given that we'll undoubtedly have bugs, and you want to ensure that only one controller is writing at any one time, how does Kubernetes decide which one should be able to write?

MIKE DANESE: Kubernetes uses something called a leader election. So Kubernetes has a leader election client. It uses objects in the Kubernetes API as to the coordination mechanism. When, say, three schedulers start, they look at that same object. They try to acquire leadership and only a single process hopefully makes it through that process of acquiring leadership. And that becomes the active controller.

The other two processes wait, check the status of that leader election process periodically, and wait to jump in, wait for that primary process to either be preempted due to a management event-- maybe the node it's running on gets updated, or maybe it crashes. But basically they're waiting on hot standby for anything to go wrong.

ADAM GLICK: What happens when new schedulers are added in this example?

MIKE DANESE: It's not really a problem. A new scheduler starts up. There's already an active scheduler. That active scheduler has--

CRAIG BOX: The bathroom key.

MIKE DANESE: --the bathroom key. The leader election acquired. The new scheduler starts up and waits. So this is actually quite different and a bit easier than, probably, the consensus algorithms that people are familiar with-- Paxos, which they teach in all computer science classes, distributed system classes, and Raft. So these are a lot lower-level primitives than what Kubernetes depends on.

In fact, what Kubernetes depends on is backed by one of these primitives-- Raft. etcd uses Raft to implement its distributed consistent lock. These protocols have failure modes. They require, for example, an odd number of participants in the election group.

We can have two. We can have one process performing a leader election protocol with itself. No waiters.

CRAIG BOX: It will probably win.

MIKE DANESE: Yeah, it will probably win. So it's not a problem. It's a lot simpler, in general, for applications to use a lock server or lock service and compartmentalize the operational difficulty of running a distributed consensus protocol.

CRAIG BOX: When one of these controllers has taken the lock and becomes the leader, does it hold onto it indefinitely until it crashes or otherwise goes away, or does it release it after a period and let others have a chance, as well?

MIKE DANESE: While the leader election client does allow a process to release its leadership, no Kubernetes core process will do that ever. Scheduler and controller manager-- they acquire the lock until they either fail to renew it or shut down, which is another way of failing to renew it.

ADAM GLICK: So you're effectively leader for life in this electoral example. What happens with Kubernetes when the leader is lost?

CRAIG BOX: A period of cluster-wide mourning.

MIKE DANESE: Yes, so if we go back to our single process example, we were kind of touching on this. If a single thread that is in that critical section crashes, and for whatever reason the other threads don't shut down, you get into a conundrum called a deadlock where this lock is acquired and nobody is going to unacquire it. So that was kind of what I was getting at, where these independent failures and distributed systems make locking more challenging.

How we fix that problem, which works well for our inefficient failure mode style problems that we're trying to solve, is we not so much take a lock on that leadership, but we take a lease on it. So each process in an election group will have acquired that leadership for a lease duration, and they'll have a deadline to renew that leadership.

So maybe the lease duration is 15 seconds. If they aren't able to renew it in 10 seconds, they'll crash. Other processes are hoping to see an update to that lock record every 15 seconds. And if they don't see that, they will assume that the current leader is having trouble renewing that leadership and will then go try to acquire it.

CRAIG BOX: Can you end up in a split brain situation, where something has disconnected, and thinks it's still the leader, and is still making updates, but a different node has come along and claimed it, as well?

MIKE DANESE: Yes. In this example, and using the Kubernetes leader election client, there's no prevention against that. A couple things that maybe the astute listener has noticed-- I mentioned time, which is a big no-no when you're trying to build these correct consensus algorithms. The other thing is my excuse here is that, if we do have a split brain scenario, it will generally be a temporary scenario. And the worst thing that can happen in that scenario is maybe the API server has to return more resource conflicts during that duration.

So there are mechanisms for preventing that type of split brain scenario. Obviously, we really wouldn't want that in etcd. We wouldn't want two etcd nodes writing different records to their commit log. They solved that using a correct consensus protocol.

Another mechanism that's mentioned a lot for solving this problem is something called fencing, where each update is submitted with, for example, a fencing token or the current epoch or resource version of the leader election cycle. And the server or the resource owner verifies that that is actually the current leader before committing the update. So there are mechanisms for solving these. We don't necessarily need this for the failure modes that we are worried about in Kubernetes.

ADAM GLICK: Is this the way that you make sure that no one is cheating, or is it just implicitly trusting anyone who is making a vote is doing it correctly?

MIKE DANESE: In general, we assume that controllers that are participating in an election group are working cooperatively. So we don't guard against much beyond that resource version that updates are made with respect to the previous version of an object.

CRAIG BOX: The leader election library in Kubernetes is implemented as an annotation on the objects. I understand that, since that was built, there's also been a lease object introduced into Kubernetes. What can you tell us about that object?

MIKE DANESE: Every storage-backed object in Kubernetes has the basic primitives that we need to implement a leader election protocol on top of. However, some objects are better than others. Initially, leader election was built on top of the endpoints object.

This is conceptually kind of nice, because you have a set of end points, which are the members of that election group, and the leader is an attribute of that group. In practice, this turned out to be a very terrible idea. End points are one of the highest fan-out objects in Kubernetes. Every time you make a change to an end point, that sprays to every single Kube proxy running in the cluster.

So due to scalability reasons, we have tried to move away from using end points. The first direction we went was to optionally use config maps. This was later replaced with the lease object in the coordination API group, which is a dedicated object for this purpose. It's used optionally as a lock for the leader election client, and it's also used for node heartbeats.

ADAM GLICK: You've been working on Kubernetes for over five years, which is pretty amazing. Is there anything that you would change in the design retroactively if you could?

MIKE DANESE: The biggest thing is the one that we just talked about. That end point fan out has caused a number of issues, and a lot of people have spent a lot of time working on that. The challenge now is the migration. So looking back, I probably would have recommended going with a dedicated object sooner.

Other than that, what I've described is a pretty lax version of leader election. It hasn't been a source of too many issues. There have been a number of bugs that would have been problematic in the presence of changing membership. For example, a common bug that has happened, I would say, a handful of times is a controller using some sort of hash of an object to decide whether it should change the object field gets added when the Kubernetes version gets updated.

Two of the same controllers running at different versions patch to the object differently, and this can cause thrashing of an object. Maybe a deployment is getting switched back and forth. This happens occasionally, and this is kind of the bug that we assume won't happen in our inefficient failure mode model, because it does cause unavailability. So far, those scenarios have been few, and it hasn't been a big enough issue to implement any sort of stronger guarantee.

ADAM GLICK: Finally, you spent a lot of time coming up with technological solutions to leader election. Have you ever been inspired to work on governmental elections and reform?

MIKE DANESE: I don't think you would want me to. What I have described today is definitely not a democracy. We have a small set of etcd nodes deciding leadership for a separate set of controllers.

CRAIG BOX: And then holding onto it for life.

MIKE DANESE: Yeah, and then holding onto it for life. So yeah, I would definitely not recommend myself for that job.

ADAM GLICK: Challenges in a technocratic solution. Though I will say that the terms of 15 seconds are slightly shorter than the regular terms that we have.

MIKE DANESE: Yeah.

CRAIG BOX: All right, well, thank you very much for joining us today, Mike.

MIKE DANESE: Yeah, thanks for having me.

CRAIG BOX: You can find Mike on Twitter @MikeDanese, or on GitHub at github.com/MikeDanese.

[MUSIC PLAYING]

ADAM GLICK: Thanks for listening. As always, if you've enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter @KubernetesPod, or reach us by email at kubernetespodcast@google.com.

CRAIG BOX: Please send recipes, meat-free or meat-filled. You can also check out our website at kubernetespodcast.com, where you will find transcripts and show notes, as well as links to subscribe. Until next time, take care.

MIKE DANESE: Catch you next week.

[MUSIC PLAYING]