CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google, live from Google Cloud Next!

[APPLAUSE]

[MUSIC PLAYING]

CRAIG BOX: My name is Craig Box. I am the co-host of the Kubernetes Podcast. Thank you for joining us today. Adam will join us momentarily. Before we get started, who is a subscriber to the Kubernetes Podcast from Google?

[CHEERING]

Thank you very much. You have come to the right place. Ladies and gentlemen, please welcome Adam Glick.

[APPLAUSE]

For the benefit of those who are not here in the room with us, it is worth pointing out, at this point, that I am standing here physically, in person, in full three dimensions, and Adam is the giant floating head on the big screen.

ADAM GLICK: I hear I'm big in San Francisco.

CRAIG BOX: There is a very important reason why that is, which, Adam, I'd love it if you'd be able to share with us.

ADAM GLICK: Sure. We haven't talked about it much on the podcast, but Glick 2.0 is currently internal, and me and my wife are expecting our first child. And she is literally due any day now, and so I am here manning the bat phone in case she calls and goes into labor, so I wasn't able to be in San Francisco.

[APPLAUSE]

CRAIG BOX: Well, it is great that you were still able to join us through the wonders of modern technology. This is actually how we record the show most weeks. It might seem like we are right there, trading off each other in the same place, but unfortunately Adam was not able to get a month to tour the cafes of New Zealand-- which has been a surprisingly popular segment, as I've had feedback as we've gone throughout this conference.

But what we do is, we have a little chat for a couple of minutes every week. This is the only part, by the way, that my mum listens to. She likes to hear where I've been in the world because I'm not very good at keeping in touch with her. And then she sort of tunes out when we start talking about Kubernetes. So, Adam, let's talk baby prep because obviously that's a topic we've been keeping to ourselves for a while. What has that process been like for you?

ADAM GLICK: Oh, my gosh. It involves rearranging your entire life around a small little dictator that you're not quite sure what they want, but have to predict on it. And so all the usual things you do of getting-- recently moved to a new place-- so getting the place set up, making sure your-- nothing is lying out or that the baby can get to or crawl to anything, blocking electrical outlets, figuring out how to put the car seat in a car, which is a surprisingly complex process.

CRAIG BOX: Right. By cheer, who in the room has a child?

[CHEERING]

AUDIENCE: "Cheer!"

CRAIG BOX: OK. There's a lot of sort of mixed enthusiasm for that cheering. We do not have children. It is-- some people have different experiences with them than others. So I think some of those kids were saying, yeah, we have kids, and it's hard. So who thinks Adam's up for a good time?

[PAUSE, THEN LAUGHTER]

ERIC BREWER: Eventually.

ADAM GLICK: I hear that misery loves company.

CRAIG BOX: Eventually. Eventually, Adam will have a great time, I am sure. Well, anyway, our best wishes for you and to Mrs. Glick for that lovely surprise. So we do normally record an overview of the news of the week. And I think, again, we get a lot of feedback to say that is a very useful part. So who likes the news section?

[CHEERING]

Thank you.

ADAM GLICK: Let's get to the news.

[MUSIC]

ADAM GLICK: It's replica count, n plus one, for the Glick family as Glick 2.0 has moved from internal development to public beta. Early users, affectionately known as grandparents, are thrilled with the latest release, code-named "daughter," and the co-founders, Adam and Chante, are doing well, and both very proud. The launch happened later on the day of the recording of our live show.

CRAIG BOX: Adam is therefore excused from participation in today's interview. Adam's other recent launch, Anthos, generated a ton of tech press. We've linked a few of our favorites in the show notes.

ADAM GLICK: After 2 and 1/2 years in the CNCF, Fluentd has graduated to a top-level project. Fluentd, an open-source data collector normally used as part of a logging pipeline, becomes the sixth project to graduate from the CNCF. In announcing the release, Brian Grant from the technical oversight committee called out Fluentd's impressive growth, adoption, and numerous integrations with the broader community.

CRAIG BOX: As we approach KubeCon EU in Barcelona, the schedule for the following event, in Shanghai, has been announced. KubeCon China is co-located with the Open Source Summit this year, and attendees have a program put together by 75 reviewers under the guidance of co-chairs, Brian Lyles of VMware and Vicki Cheung of Lyft. Speakers come from all around the world and include a who's who of web-scale tech in China, including Alibaba, Baidu, Huawei, and Tencent.

ADAM GLICK: Call Ice Cube and tip your 40 for Eazy-E, because Azure Kubernetes Service is now on its way to South Central. Unfortunately, this is the South Central US region and not Compton. But Korea south and Korea central are also getting AKS, as the service is finally in GA for these regions for Azure, bringing more Kubernetes to more hoods around the globe.

CRAIG BOX: Lee Briggs from Apptio has been experimenting with AWS Fargate and posted his thoughts in a blog this week. Is it easier than Kubernetes? No, it's just different. He looks at the different learning curves for the two technologies and concludes the things you have to learn alongside Fargate during a production service bring as much complexity as you would have in Kubernetes, but without the power of the declarative, API-driven platform. The best of both worlds is the promised Fargate integration with Amazon EKS, which was initially promised in 2018 but has subsequently been delayed.

ADAM GLICK: OpenStack Stein, as in beer, has launched, announcing a number of improvements designed to help the 61% of OpenStack users who run Kubernetes. In particular, they claim that the Kubernetes Cluster launch times have come down 50%, from 10 minutes to 5 minutes, for a deployment. Additionally, the Stein release includes an update to the network service Neutrino that lets admins create a virtual network ports in bulk when containers are started. There's also an update to Ironic, which is like 10,000 spoons when all you need is a knife, but focuses on bare-metal provisioning.

CRAIG BOX: We brought you most of the announcements from Google Cloud Next on last week's show, but there were a few hidden in the sessions that warrant special mention. First, the beta of GKE Sandbox, a second layer of defense between containerized workloads and GKE based on gVisor. Next, Workload Identity, which enabled you to access Google Cloud Services from GKE without managing any keys or secrets at all.

And finally, GCP config connector, a feature that lets you use Kubernetes CRDs to control objects on Google Cloud Platform, like Pub/Sub queues and storage buckets. Each of these features has a Deep-Dive session with its product and engineering teams, and you can watch those videos to learn more about them.

ADAM GLICK: Derek Carr from Red Hat has a post up on a new feature and Kubernetes 1.14 meant to protect your clusters from process ID starvation. Pods can have applications that spin up a lot of processes, especially in the case of testing. These applications can try and take many, if not all, of the process IDs available. This can have an interesting side effect of the node not scheduling additional pods and moving the pod to a different host, thus spreading the resource exhaustion issue. The new feature is available in alpha, allowing admins to set aside a certain number of PIDs to avoid this exhaustion.

CRAIG BOX: And that's the news.

ADAM GLICK: Truly excited to do this. The first time we're doing a live recorded one with an audience. So thank you all for being a part of this and a part of the show. It's exciting stuff. We've got a great guest on today. Why don't we get to it?

CRAIG BOX: All right. So thank you very much, Adam.

[APPLAUSE]

CRAIG BOX: Our guest today is Eric Brewer. Eric was the person who announced Kubernetes to the world almost five years ago and has been working on Google's cluster and compute infrastructure since 2011. As a researcher, he has led projects on scalable servers, network infrastructure, IoT, and the CAP theorem. In business, he co-founded Inktomi and helped lead it onto the NASDAQ 100, as well as working with President Clinton to create usa.gov. It is my pleasure now to introduce our guest, Professor Eric Brewer.

[APPLAUSE]

ERIC BREWER: All right. Great to be here.

CRAIG BOX: Now, I'd like to start by asking you-- you've had a long distinguished career in computer science. What was it that got you interested in computers to start with?

ERIC BREWER: That's a good question. I guess originally, kind of electronics-- because computers weren't available at least till I was about 10 or so. Eventually, I got an Apple II. That was like a revolutionary change for me. I had done a little bit of programming before that, the Commodore 64-- nothing too exciting, but really liked it. And then, when the Mac came out in 1985 with the "1984" commercial, you might remember. It turns out, that was in a relatively different assembly language, the 68000.

And so it turns out, not many people knew how to program in that. So I learned that relatively quickly and became an assembly language programmer because it paid, literally, five times as much. And they would hire a 17-year-old, which they wouldn't otherwise do.

CRAIG BOX: Where were you living at the time?

ERIC BREWER: In a suburb of LA. I'm a valley dude, valley boy. I used to date valley girls. So we just-- we just called them girls-- went to, like, "Fast Times Ridgemont High." That was kind of like my high school, roughly, about exactly the same experience.

CRAIG BOX: And so you were doing all of this through school. So it seemed obvious to you that was something you were going to pursue to academia, to university?

ERIC BREWER: Yes. And so I knew I was going to be an undergrad in computer science. I didn't think about a PhD until later. What got me thinking about a PhD actually was joining the astronaut program.

CRAIG BOX: Whoa.

AUDIENCE: But that ended when the Challenger crashed, I guess, in 1987. And that really ended the manned space program for quite a long time. And I'm, like, well, I still like this PhD idea. And so I focused on computer science for that, generally, with MIT, and then returned to Berkeley as faculty.

CRAIG BOX: So once you finished your studies at MIT-- again, we're talking before the start-up era-- and it will come in as we start talking about your career. But were there choices at that point in time? Did you want to go continue your career as an academic, as a teacher? Or did you think about going to industry at that point?

ERIC BREWER: I did pursue both options. I had some decent offers, DEC's SRC was a big research lab at the time. I considered UC San Diego. I was almost going there for a while. And then I got a very late interview offer from Berkeley, actually, after they'd interviewed many other people. So I was definitely not their first choice. But it worked out. And I loved Berkeley as an undergrad, so I was honored to return.

CRAIG BOX: Are these weather-related choices as well? You wanted to come back to California?

ERIC BREWER: Weather-related-- I enjoyed living in Boston. And I think everyone should live in a different place for a while. But I'm a native Californian for real, and I was happy to return.

CRAIG BOX: OK. And now, while you were at UCB, what were you researching?

ERIC BREWER: My first set of projects was around this brand new thing that people actually didn't believe in much at the time called the web.

CRAIG BOX: Do we think the web will take off?

AUDIENCE: Yeah!

ERIC BREWER: In fact, people confuse it with the internet, even though they're quite different things. But they're so unified today. So I was looking at my background. My PhD was in, essentially, parallel computing-- so how to write large-scale programs on supercomputers. And I was thinking, well, if this web thing takes off, it's going to need a lot of computing power, at least for some sites.

And the obvious site that would need a lot of computing power was the search engine. So we built a search engine, not because we loved search, but really because it was the best example of a site that needed scalability and significant compute power.

CRAIG BOX: I think that's exactly why AltaVista built a search engine.

ERIC BREWER: It is. And in fact, the digital office I was going to work in was one of the two working on AltaVista

CRAIG BOX: OK.

ERIC BREWER: There's a funny story there. So I started Berkeley in the fall. The following summer, which is the summer of, I guess, '95, I go -- have an offer to work at the summer as a consultant for DEC in this lab, since they made me an offer, but I didn't accept it. But they let me, and said well, why don't you come join us for the summer?

It's common practice. It's a good way to learn about things. And so I get there, and it turns out they're working on AltaVista and I'm working on Inktomi, But we didn't know about each other. I was like, I can't stay here. This is a conflict of interest.

CRAIG BOX: But I can't tell you why.

ERIC BREWER: I told them why, but it was weird meeting the AltaVista team. And they're talking about scalable search as well. But I think, most interestingly, their approach was, buy the biggest DEC machine they make, and build search in that way. And our approach, I think somewhat famously turned out to be right, which is use a cluster of lots of small computers.

And that has lots of advantages-- obvious one being a larger total scale. But also, what we now have learned-- availability and incremental scalability. You can add nodes as you need to to increase your capacity, things we now called auto-scaling, except we had to do it with physical machines, which is much harder.

CRAIG BOX: Call-- it's an API call which basically rings Dell and then causes a machine to be ordered and then delivered to your loading dock three months later and then plugged in by--

ERIC BREWER: That's about the best case. Yeah.

CRAIG BOX: That is. So the product-- the thing that you built that became Inktomi, was that something that you just intended to be an academic project to start with? Was it always with a look to commercialization?

ERIC BREWER: No. I really-- it's actually somewhat foolish to do a startup early on in your tenure track faculty position. But I felt like this is a once in a lifetime chance. This web thing is real. I understand it more than most. And then Netscape went public in '95. Then it became obvious that it was real to everyone else.

CRAIG BOX: We should probably do this.

ERIC BREWER: So we started, I think, for real in the fall of '94. And we had a little LLC to raise money in maybe early '95, sometime in '95, and then formally took real funding in early '96.

CRAIG BOX: OK. Is Inktomi a Greek word?

ERIC BREWER: It is a Lakota word.

CRAIG BOX: Oh, what does it mean?

ERIC BREWER: So the Sioux Indian-- it is a trickster spider. It's hard to exactly capture its meaning. And I, in fact, heard conflicting reports. But roughly, there aren't many mythological spiders that are interesting. And that is one. It also, just turns out, one of my relatives grew up in South Dakota and was an honorary Lakota chief. And so I have some connection there, although it's loose.

CRAIG BOX: I always thought it might have been "link to me" but you just lost the L somewhere along the way.

ERIC BREWER: No, in fact, a lot of people think it's Japanese as well, which it's not. In fact, it was not intended to be a consumer name. So it's not a name you would expect to know. Because our strategy was much more to be the engine inside, more like Intel inside. And so only a few people actually needed to know and understand what we were doing.

CRAIG BOX: OK. So were you selling the technology to consumer-facing companies? Was that the business model--

ERIC BREWER: Yeah. So for a while, we were doing a large fraction of searches, maybe 80% or something, because we had Yahoo, Microsoft, and AOL all running on the same-- and HotBot, which was the original one we did, which was wired search engine, which actually was the best of the bunch at the time.

CRAIG BOX: And you took that company public to the NASDAQ?

ERIC BREWER: We did. That was a super fun year. That was in 1998. In fact, I went on the road show, which is where you go around and try to convince people to give you money. But I only went the first two stops, which I think were New York and, like, Philadelphia, because there were no technical questions. It was all about, what are your finances? How do you-- does this machine model makes sense? And so it was kind of pointless for me to be there. And it's a big hassle, actually, because you jump cities every day. But it was fun to do a couple.

CRAIG BOX: What was your position at the company at that time?

ERIC BREWER: I was, I guess, CTO.

CRAIG BOX: OK, so there was--

ERIC BREWER: I was originally founder

CRAIG BOX: --look after the business.

ERIC BREWER: Quickly gave-- I was even CEO for a tiny bit, but that was just until we could find a real CEO.

CRAIG BOX: OK. And once you'd listed, how did life change for you?

ERIC BREWER: I bought this watch. That's a true story. It's a nice watch. Well, I think-- it's interesting, because I had the full ride, as I say, for the era of the internet bubble. And Inktomi is not the worst offender of bubble companies, in the sense that we were profitable, and there were many that were never profitable.

In fact, because we were profitable, we were one of the very few-- I think Yahoo was the other-- that got actually added to the NASDAQ 100. So that means you're top 100 companies in NASDAQ. That's a big deal. Except we were only listed as NASDAQ 100, I think, for one or two quarters.

CRAIG BOX: Still, you can claim it. It's there, it's--

ERIC BREWER: We reached it. That's an achievement. But our revenue came from two sources. The search engine was a minority, about a third, and two-thirds came from a traffic server, which was basically a product you'd now think of as being like Akamai or a content delivery network. But this is very early in that era. And it was really this first kind of overlay networking. It actually saved the internet for a while. I can talk about that later.

But the problem is, once the internet bubble burst, it turns out the telecom sector also crashed. And it crashed for, like, three to five years. It didn't recover for a long time. And so we just lost two-thirds of our revenue. And we have a market drop-- it was called irrationally exuberant. Not quite the right phrase.

But it was too high. But when it falls, it doesn't fall to where it should be, it falls, actually, way below that. And then you have to kind of-- because the same things that drive it to be too high, which is kind of groupthink, drives it to be too low once it shifts. And on top of that, when you're on the board, you can't actually sell stock during that fall, right?

Because every few months or so, there's a material event, and you can't sell within 90 days of a material event. And I also got some bad advice on that front. So the short answer is, I rode it up, made a billion dollars on paper, rode it down, lost a billion dollars on paper. To answer your question, how did it affect me? Well, it was a fantastic ride up and a not-so-great ride down.

CRAIG BOX: And you have a nice watch now.

ERIC BREWER: I have a nice watch.

CRAIG BOX: Brilliant.

ERIC BREWER: So-- I did OK. I have no complaints. But it was certainly-- it's not-- the billions of dollars that I had on paper but never got to spend.

CRAIG BOX: So how was your research career going at this time? You mentioned you interrupted your tenure track it to take Inktomi to the road and then to the stock exchange. Were you able to-- obviously, you gained the professorship at some point. How did that happen in the timing?

ERIC BREWER: It was one of those things. I think, in retrospect, doing Inktomi was kind of doubling down on the tenure track. Because it's a big distraction from my official tenure research. But at same time, if it hits, they're going to give you tenure, right? Because it's too visible. And I did maintain a regular-- I had students that were doing well, and some of whom are faculty now at Berkeley and other places.

So I did enough on the actual tenure track part, and I did enough-- there's enough innovation in Inktomi itself-- well, actually, tons. And I was able to write about that. I gave keynotes at [ACM] SIGMOD among other places. So I was active enough to get tenure. But I was certainly helped by the fact that Inktomi did well.

CRAIG BOX: And then when Inktomi stopped being your full-time concern, were you back to full-time research? And how did the research that led to the publication of the work on the CAP theorem come about?

ERIC BREWER: CAP theorem actually came out of work while I was at Inktomi. I had two jobs, basically. So I was tenure track faculty and significant player at Inktomi for about seven years-- which was too long and burned me out. But it was certainly-- I didn't want to give up either one, and I didn't give up either one. So I returned to Berkeley only for real in, like, 2003-- so a seven-year run with Inktomi.

Came back, did a different kind of work. Mostly did work on technology for developing countries. But the CAP theorem came out of two things. It came out of building systems at Inktomi, especially the traffic sort of CDN stuff, where you're basically building, a widely distributed system and managing it, and how do you remotely control all these things-- which, by the way, is a lot like how Anthos works, which is Google essentially managing a whole bunch of distributed stuff that's very parallel, it turns out. Not that it shares any code or even any people other than me. But it's conceptually very similar.

And so when you're building that kind of system, you are making these trade-offs all the time about, how do I deal with the fact that I don't have perfect connectivity to all these things? And of course, connectivity was worse then than it is now. And it became-- I kind of had an intuitive sense for it at first. And eventually, when I was trying to teach it in class, is when I actually wrote it down for real. I first presented the CAP theorem in my OS class.

CRAIG BOX: OK. So again, to the audience, by way of noise, who here is familiar with the CAP theorem?

[CHEERING]

Would anyone like to have a go at explaining it? I say, I can explain the cat theorem, which is that videos of cats on the internet do better than videos of dogs. But I think there's a little bit of a change in trend recently, so we may have to revise that. But would you like to summarize the 30-second version of CAP theorem?

ERIC BREWER: Sure. 30-second version is that there are three properties you'd like to have in a distributed system. One is that the data is consistent, which means it looks the same from all views. Another is the system is highly available-- available for rights in particular, so you can update the data at any time. And you'd also like it to be tolerant of network partitions.

So if someone cuts your fiber cable, or you lose connectivity between two replicas of the database, it'd be nice you could still say your system is also available and consistent. And the answer is, you cannot. You can pick two of those three, but you can't have all three.

And it was important, I think, mostly because, at the time, the database community-- which, rightfully so, is very strong on consistency, the ACID properties in particular-- were saying things about availability that wasn't true. And I think they didn't really have the same view on availability that they have now, which is really that it is a spectrum.

And you can have choices that make your database more available at risk of some consistency or strictly consistent at risk of availability. And now we're having a real conversation on that front. And many of the companies you see in the last 10, 15 years are doing interesting trade-offs in that space. And I think that's great. That was the real point, was really, let's look at the whole space.

CRAIG BOX: So this podcast-- we obviously have a network partition at this point. I am here and Adam is no longer here. So are we CP or AP?

ERIC BREWER: Well, we're highly available--

CRAIG BOX: We are.

ERIC BREWER: So-- and we have the majority on our side, so we get to do writes, also.

CRAIG BOX: OK. So this work that you published, how did that lead to-- or, what is the connection? How do you get to Google?

ERIC BREWER: I get to Google, I think, for-- not related to the CAP theorem per se--

CRAIG BOX: Your distributed systems work?

ERIC BREWER: --a couple of different reasons. I think-- one thing I didn't know is a problem we had with-- right before the end of Inktomi, when it was bought by Yahoo, was that we had a ton of search technology that none of our partners were using. Because again, we built the engine, but we can't make them use it correctly.

And at the time, which is hard to understand pre-Google, there was a period, a few years where search was kind of one of many things you have in a portal. Search was no more important or less important than Yahoo Finance, or movie reviews, or directory. And the answer is that's not true. A search is vastly more important for lots of reasons. It's actually more useful. But also, it's tremendously better for advertising revenue. Anyway, so we had all this stuff kind of ready to make searches better, but we couldn't get our partners to use it because they didn't really value search.

Google came along, started making money. It wasn't just Google. The other one was GoTo, which you probably haven't heard of. Although, if you look at the Google IPO, they actually had to settle a lawsuit with Yahoo and GoTo about all this stuff. It's super interesting-- which I won't go into. But it was all cleaned up nicely. But the short version is, Google showed these companies that search was actually useful and profitable, and then there was interest in Inktomi. But by then, the bubble had burst, and it was too late to fix that.

But what happened was, after Yahoo bought Inktomi, we ended up-- now that we were controlling the Yahoo search engine, actually fixing-- or putting all of the stuff to use. And so somewhere, unbeknownst to me, around 2004, Google's internal measures of search quality were that Yahoo was suddenly better than them. And they were not pleased.

So there was the two-year war-room-- which I didn't know about at the time-- to go catch up to Yahoo, which is really Inktomi inside Yahoo. And so that event, even though Google came out and was able to catch up and fix it, which is not that surprising. They had more money and more investment.

CRAIG BOX: But it certainly brought Google to your attention.

ERIC BREWER: But certainly, they were well aware of the work we had done. And there's another, even going farther back. It turns out, the original Google cluster, which was famously badly engineered as a point of pride--

CRAIG BOX: It was made of Lego, in fairness.

ERIC BREWER: Made of Lego's and fans was in Santa Clara next to the Inktomi cluster.

CRAIG BOX: You never thought about just pulling the cable out? Accidentally trip over it?

ERIC BREWER: I didn't know they were there. They knew we were there. Anyway, so Urs, who was effectively my boss, and spoke recently, and I saw him yesterday, he was looking at our cluster. And he was also well aware of the work we were doing at the time. And so if you look at Google clusters now, they look way more like the Inktomi clusters than they do like Google clusters.

They're-- I would think you would say they're totally based on the Inktomi designs. Although they're much more advanced and much larger. I don't mean to say they just copied it. It's just like-- but as a starting point-- kind of clean, rack-based things with careful wiring is, in fact, the right way to do it.

CRAIG BOX: And did Urs reach out to you and say, we'd like you to come and pretty up our clusters?

ERIC BREWER: I actually started working at Google related to work on developing regents. I was helping google.org in work on Kenya and a few other places. And while I was there, a few different people kind of just asked me questions about stuff they were doing. And I guess they liked my answers, is the essence of it, and I was actually hired by Urs former boss, Bill Coughran. And he had led Bell Labs, he's a very interesting fellow anyway. And so he brought me in the technical side of Google. And that was in about 2012.

CRAIG BOX: I was talking to Brian Grant yesterday, who said that he was your Noogler buddy, basically. He was working with you on the clusters back in that time.

ERIC BREWER: Yes. Yes

CRAIG BOX: One of the things that you have done-- many things you've done at Google-- is work around the Spanner database system. And Spanner's famous for being able to operate globally, offer effective five nines availability. And one of the things that, obviously, your involvement was is, it says, this kind of bends the edges of the CAP theorem a little bit.

ERIC BREWER: It does.

CRAIG BOX: Let's talk a little bit about that.

ERIC BREWER: Yeah. So to be super clear, I didn't actually do the work I'm talking about, but I did help analyze it and explain it to the world, about how it fits in. There are two parts to it. The purely CAP part is that it doesn't violate it. So if there's a network partition, it's going to choose consistency and forfeit availability. However, after about five years of extensive work, the different zones of Google's data centers are actually very unlikely to partition, even though they're globally distributed, because there are now multiple paths.

And turns out, the main thing that was causing partitions was not that all the paths got cut. It was that-- well, it was, but I'll tell you how. It was that, when you'd push a configuration change, it would occasionally have bad changes. And it would break multiple paths with the same push. So if you actually want to have a highly available network that's globally distributed, you can't do that. And so after many years, we basically have stopped global pushes for networking, mostly, and we have better ways to test pushes before we operate them.

So the short answer is, it's a CP system that will forfeit availability in the face of partition. But its uptime is about five and a half nines. And that's because, basically, there aren't partitions in practice, only because we control all the data centers and all the fiber-connected data centers. There's no public internet involved. So we're kind of cheating on many fronts here.

Second part of the answer I won't go into is that there's also a connection to TrueTime, which is, how do you agree on global time in different spots? That also is a kind of thing that violates the CAP theorem. But that's really because time is essentially, in this case, read-only. We are broadcasting and agreeing on time.

But we don't really have to-- we can be off a little bit. We don't have to have perfect updates. We're not deciding what time it is. It's kind of globally decided and locally managed with atomic clocks. That's a whole other topic I don't have time for. But it's in-- there's a paper about it called "Spanner, TrueTime, and the CAP Theorem."

CRAIG BOX: We're coming up to five years since the public announcement of Kubernetes. And obviously, it was a few months before the announcement the various teams at Google started work on this-- on various implementations which congealed and conglomerated to become what we now know as Kubernetes. You were involved in the discussions around that, how to work with the communities that were starting at the time, specifically Docker. And you were also involved with the discussion to open-source that. What was the feeling like at the time?

ERIC BREWER: So you can kind of think of me as-- my role was two things. It was kind of, maybe, the exec sponsor for it. But also, I just felt it was a super important part of the future strategy for Google Cloud. And that's kind of my day job, is kind of core strategy for Google Cloud. And I would say a few things about what made me think that at the time.

The first one was that Google's infrastructure, because it's from 1998, doesn't really use virtual machines. Because 1988 candidate is also the year that VMware was founded. And Inktomi was the same way. Inktomi didn't use virtual machines either. Our notion of how to program highly available services was really Unix processes-- talking to each other using simple messaging over TCP.

And so Google built a whole infrastructure that didn't use virtual machines. But it still needed some notion of what we now call containers, although, Google's containers are much more about resource management-- so Linux containers in particular. And we had things like tarballs to do the packaging that Docker does.

Now, when Docker came out, and in particular, when I could see it was taking off, I'm like, well, this is an opportunity for us. Because even though I don't agree with everything they're doing, it's generally right. And more importantly, it's much better for Google to be in a container world than a VM world. Because, if we're playing in the container world, we have a huge trove of assets where we've been running containers for now 10, 15 years.

We have tools and logging and debugging stuff-- all the stuff that's built around, how do you build using containers? And so I'm like, well, what we really would like is to move the entire cloud from thinking about VMs and disks to thinking about containers and APIs and services. And if we make that change, we're playing on a field where we should win. That's the basic strategic bet here.

So there are two parts of that. One is, should be better containers. Because at the time, we had two products, App Engine and GCE, IaaS as a service. And this was introducing a third product in the middle, which is Container Engine. And that-- we were already feeling distracted by having two products. So there was a big argument against introducing something new. But the big argument for it was really, this is our best-case future and fits well with what we ought to do. And by the way, it's more productive. It's actually better for developers.

Then, the second part was, should we open-source it or not? Because, although Google's done a lot with open source, including tons of work on Linux, Android, TensorFlow now-- a recent example, there hadn't been much evidence of open source in core systems things-- things that are kind of the heart of what we want to do.

It's always like-- a good example is Bigtable. Bigtable, we wrote the paper-- not that I was involved-- but we didn't release the system. That actually came back to haunt us in the sense that people had to build HBase as a separate code base with the same goals, and then later, when Google Cloud said, oh, we should make Bigtable into a product, guess what?

We couldn't use the Bigtable API. We had to use the HBase API. So HBase at Google was actually powered by the actual Bigtable under the covers. But we had to build a shim adapter to make it fit what the outside world had built, because we didn't participate in the outside world in that conversation-- other than the paper.

And so there were two arguments for doing Kubernetes open source. One was-- I think the one that carried the day is really, this is too big a shift for the industry for us to do by ourselves. As big as Google is, this has to be a community effort, or it isn't going to work. And by the way, we don't have people to do it, either. But even if we had enough people, we don't-- people won't buy in unless they feel like they can influence the future and they can make it work for them. And so that was the main argument for doing it.

But the other reason was also that Docker was very popular in places besides the cloud. And I kind of felt like, you're going to leverage Docker and try and build something goes with it, you need to be equally available and open-- or more open.

CRAIG BOX: But at this point, we haven't even published a paper on Borg. We sort of hinted at work we were doing with custom management and Omega, the paper we published on that. So there was commercial sensitivity, perhaps. I always heard the story about, we're able to offer Gmail with just a tiny ad next to it because we're able to pack so much into our infrastructure, because we're able to run things at higher utilization than everybody else.

So while the argument of open-sourcing something is obviously true as we become a cloud vendor, there was some push-back on, should we give away the secret sauce? Should we save this thing? How did we overcome that argument?

ERIC BREWER: Well, first, it's something we were able to try. So we didn't have to bet the whole farm on it, in some sense. You could just put it out there and see how it goes. But really, the big change was when we agreed to give away the trademark. So this is something else I was involved in.

This is the Cloud Native Computing Foundation. Its purpose, from my perspective, at the time, is we need a place to put the trademark that is fair and not Google so that, if you use Kubernetes, you know we can't take the trademark away from you.

We want you to be able to invest in it and know that it's something you can use even if Google were to disappear. And so that's a different level of open source than just releasing the source code. It's saying, we are open to ideas. We are open to contributions that are real. And we are going to share the trademark because, otherwise, this isn't really going to take off the way we need it to.

CRAIG BOX: So with your involvement in those two strategic decisions, have you been surprised at the growth over the last five years in this community?

ERIC BREWER: Absolutely. Absolutely. I mean, you really couldn't ask for a better community or a faster rate of change. In fact, I've occasionally said the rate of change is too high. It's very difficult even to keep up, quarter-to-quarter, with all of the stuff going on. And that's a little risky. It scares some people away because it's like-- you really have to be engaged just to keep up. And so that I think that's happened now.

And the good news is, the core pieces are now relatively stable. So I kind of feel like it basically works. And there are still tons of commits going on, but they're a little bit more on the fringe, a little bit more about adding capabilities. The core piece is much more stable than it used to be. But the first few years was pretty wild. It was hard with so many changes going on that were actually so critical to the foundation.

CRAIG BOX: Now, you wrote a white paper not long ago with Jennifer Lynn about what we then called CSP, we're now calling Anthos.

ERIC BREWER: Yes.

CRAIG BOX: What exactly, in your mind, is Anthos? And why is it a big bet for Google going forward.

ERIC BREWER: Right. I'm a big fan of Anthos, and definitely help started this project, also. It has a few things I like about it that are important. The first one is that, even though we believe most workloads will move to the cloud, it's still the case that 80%, 90% of them are on-premises. And some of them are hard to move. And before this, roughly speaking, what was happening was that you kind of-- the requests we were getting basically was, I want to modernize. I want to do containers and service and APIs.

And our original answer was, well, move to GKE, and we will help you do that. And then the question was always, well, we can't move this workload at this time. Or we can't do this right now. We have data center we paid for the next five years, so we're going to use it. Can we just modernize where we are, right? And so the real purpose, the first purpose of Anthos is, say, let's just separate moving to the cloud from modernizing your infrastructure. Make those two different things. And you can mix and match them.

That actually helps a lot of different problems. And again, Kubernetes is open source, the Istio is open source. The pieces are there. You can, in fact, run them on-prem. And we saw people running on-prem, just not very well. Because, again, it's complicated. It's changing all the time. It's hard to run on-prem unless you have a pretty decent, dedicated team.

CRAIG BOX: By applause, who's running Kubernetes on-premises today?

[SPARSE APPLAUSE]

ERIC BREWER: A few people.

CRAIG BOX: All right. That was very muted applause. Let's just note that.

ERIC BREWER: Yes. It's not that easy to do. Not as easy as one would like. So first driver is the need for modernization on-prem is not going away. And if you look at it from the strategic viewpoint, if we can engage with customers while they're still on-prem, then when they move to the cloud later, I think Google will do well, right? We're biasing their choice of future cloud for those workloads.

Second driver is-- again, pretty fundamental-- is many customers who feel like they can't depend on any single cloud vendor, doesn't matter how good the tech is. That's putting all your eggs in one basket. And in fact, Google is the same way. We have to have two makers of disk drives. We have to have multiple suppliers of servers and multiple suppliers of networking equipment, right? And that's because we don't want to put all your eggs in one basket.

So multicloud is here to stay, but it's very hard to do, right? I don't know if you've done any multicloud stuff. But basically, you need two different teams, and it's a big pain because all the tooling is different. The APIs are not the same. It's hard to even-- to move stuff back and forth is not trivial.

So another advantage Kubernetes brings to the table is basically that it gives you an environment that is pretty neutral relative to the underlying infrastructure. So if you're using Kubernetes APIs, then you can run on multiple clouds, and you can run on-prem, right? And so, as a basis of a multicloud story it is by far the best basis. And in fact, I don't know of any serious multicloud attempt that isn't using Kubernetes as its base.

And the reason for this is pretty fundamental. The cloud space is moving quickly. So if you wanted to do a multicloud thing through traditional standardization, you couldn't do it. You can't do it. Even if you could standardize on-- these are the APIs for this year for our multicloud effort, in two quarters, they're already old.

So when something is moving this quickly, the only way to get to a standard is actually by sharing code, right? But the standard is in the code, even if you don't know exactly what the standard is, because it's implicit in the code versus explicit in a kind of standard document-- doesn't matter. It's, at least, the most likely way to get some consistency across environments. So that's the second big driving factor.

And the third one is that, it turns out, we're really good at running Kubernetes clusters. I feel like Google can honestly claim we're better at it than anyone else. We've run lots and lots of clusters. We auto-upgrade, we auto-scale them, we have lots of experience.

CRAIG BOX: And they're all very good-looking clusters, too, because we learned from Inktomi.

ERIC BREWER: They're better-looking, actually, now-- not due to me. But I do feel like it's important. We had real customer asks for-- can you just manage these clusters for us the way-- we love GKE, but we need to run this workload on-prem. Can you just take care of that part? Because running clusters is not core to our business, right?

It's a tool that we need to do whatever else. If we're doing pharmaceuticals, we're doing oil and gas, we're doing financials. So we don't want to spend our time managing clusters. If you can do it for us, that would be great. And it turns out, it's not that hard to do given Kubernetes as a base.

CRAIG BOX: Is that enough? Is Kubernetes the abstraction level that the average enterprise developer should be interacting with day to day?

ERIC BREWER: It's not enough. In fact, I tell customers, bigger customers, especially, that they shouldn't view Kubernetes as the thing their developers use. They should make their own little, almost a PaaS on top of it. I think Kubernetes' long-term value as a platform for platforms. It's a little too general purpose and too wild to use just on its own, I think, for most enterprises.

But it's very amenable to a little bit of layering on top and, say, this is the Faberge oil and gas version of Kubernetes. And we want you guys to use this because it's preconfigured the various things that make it a bit more domain-specific and much easier to manage. So I see a lot of things like that for people are customizing a platform for their own use. And that makes a lot of sense.

And there are other things like Knative and functions. You need a serverless offering. But you need one that's actually portable, hybrid, and multicloud. And Knative it's perfect for that, but it's a specialization of Kubernetes, basically, is one way to think about it. It's one domain-specific view of Kubernetes. And the other ones were TensorFlow, Kubeflow. And other ones, I think, are certainly coming. There are 10 or 20 startups doing functions on Kubernetes, which tells you that that is a useful thing to do.

CRAIG BOX: You can never have too many startups. Now, at this point, I'd like to invite anyone from the audience who has a question for Eric to please step up to the microphones. So if there's anything you'd like to know, he has had a fantastic career, continues to be at the cutting edge of distributed systems research. You've met Bill Clinton.

ERIC BREWER: I have met Bill Clinton.

CRAIG BOX: Someone at least needs to get up and ask about the story about Eric meeting Bill Clinton.

ERIC BREWER: I can tell a little Bill Clinton story in the meantime, but we can take questions. So mentioning, this is the first kind of split-person podcast that's with a live audience. It turns out, the usa.gov site, which is the federal portal for US, which I worked on with Bill Clinton, was the subject of the first-ever presidential webcast. And it was based on a fireside chat, actually. So traditionally, since World War II, the president does a fireside chat production, which is essentially a radio thing. So it's audio-only, and much like a podcast.

The webcast was done by the same team, but with a webcam added to it. And so we're setting up to do the webcasts. And Bill Clinton is getting ready to go. And it turns out, because it's their first-ever webcast that they didn't have a teleprompter. Because normally, he just reads the script for the audio. But now he's on camera. He's like, I can't read this thing on camera, that looks stupid.

CRAIG BOX: Surely Bill Clinton had been on TV before at this point.

ERIC BREWER: A few times. So-- but this is the cool part. Much to my amazement, in about 30 to 45 seconds, he memorized the speech while I was sitting with him. And then he gave it without a teleprompter and without the notes. And he got it incredibly accurate. I was, like, shocked.

CRAIG BOX: That is fantastic.

ERIC BREWER: Surprised everybody in the room.

CRAIG BOX: All right. So sir, if you'd be so kind as to step to that microphone. Feel free to introduce yourself if you wish.

AUDIENCE: Yeah, sure. I'm Zoltán from Ergon in Switzerland-- Zurich, Switzerland. So we are doing application development, and so far, we have not been using so much these large databases like Cloud Spanner, right. So we've been-- we had some Oracle database, or Postgres, or whatever. And now we were wondering-- so one problem that are asking the companies-- so if you're now moving to the cloud, and you move to a large, global database-- because maybe you have large data sets or so-- what is the typical-- I mean, the database is distributed, right?

And you have some overhead because of that. So what is a typical range of overhead that you could experience due to the large distributed database compared to running performance-- well-performing one-node system like Oracle, or Postgres?

ERIC BREWER: Right. So there's a couple of things in that question. First, you can move to cloud and decide to run locally or globally with your database. And we have customers that do both. So we have definitely had customers that run Postgres or MySQL grouping in one zone. And they'll have it replicated to another zone, maybe in a different region, maybe in the same region.

And then you have kind of all the same problems you'd have on-prem with that, which is, it is mostly consistent. It can have some delay, but it does generally work, and it is pretty clear what your overhead is. You have, really, the extra replica copy, and that's all overhead. You can also use more automated replication. Or you can go to the-- in some sense, the high end, which is Spanner, where the replications can be global if you want, or regional. regional Spanner is a thing, in addition to global Spanner.

And I would say, I guess the first answer is, I don't see it being a big deal in practice. I think there's a lot of discussion about it. But it's such a productive database, Spanner, because it kind of feels like-- if you want to use Cloud Run, which is a new serverless thing, or Kubernetes clusters that are mostly stateless-- and they're just using all the state management via Cloud Spanner and it's magically consistent in all the zones you're in, it just saves you so much productivity time and so much operational time.

So I think there is more overhead if you're doing these global things that can affect your reads and writes, although true time for Spanner is made to give you low-latency reads, in particular. The writes can still be slow if they're being globally replicated, for the obvious reasons.

But I think, in practice, the overhead in terms of, at least, the finances, are not a big determinant of success. Because you have to-- if you're going to do that comparison, you have to include the time you would be spending managing the databases yourself, which is a very large number and the amount of trouble you have because of things like partitions and other problems, even upgrades, that you don't have to do. And the productivity that people get out of it is mixed-- in particular, of stateless tiers on top of very solid storage-- that just makes everybody more productive. That cost trumps everything else, really.

CRAIG BOX: All right. We have time for one more question.

AUDIENCE: My name is Greg. I'm from Colorado. I love the podcast, and I really like the sense of humor that you guys throw in. It's fantastic.

CRAIG BOX: Thank you very much. This is totally not a planted question.

AUDIENCE: It's a privilege to ask you a question. So my question is, do you think there's a real need for going nodeless? Like, I don't see the point of managing nodes, or even dealing with nodes in Kubernetes. If we could have a nodeless service, that would be amazing. If my pod can specify the type of hardware I need to be using, or SSDs or something, but never, ever need to provisioning nodes in my cluster.

ERIC BREWER: That's a great question. There are a few different answers on that. So first of all, I just would like to point out that serverless has been an option from Google Cloud since 2008. That is what App Engine is. App Engine is a serverless computing platform. You do not know about nodes in it. But it's actually not just functions. You can run functions in long-lived apps and stateless tiers, and they can call into storage, and that's still all serverless.

Then, moving forward a bit, if you just look at a few other systems, it turns out things like Cloud Dataflow and BigQuery are also serverless. People don't talk about them as serverless because they're not functions. But you don't actually know that you have nodes in those systems. The number of nodes goes up and down automatically, based on the amount of data you have and the queries you're doing; or, in the case of Dataflow, the flow that you're using. So I would say we're big fans of serverless and nodeless things, for sure, and have long been so in that space.

To move on to the next part of the question, though, should your Kubernetes cluster be serverless or nodeless? That's a little bit more complicated. Because I think, if you're a large enterprise, probably no. And I'll tell you why in a minute. But if you're a developer, probably yes. And in fact, the thing we've mentioned today, Cloud Run, actually is a container runner.

Although, at the moment it's really for things that have an HTTP interface, which is lots of things-- all APIs, typically. So that gives you one version of that that's very simple. And in fact, we could-- and I've thought about giving people a nodeless Kubernetes, but it's not a real Kubernetes.

CRAIG BOX: We've spoken about this ourselves together, in that, the abstractions in Kubernetes describe things which are true about the hardware. There are ways to place things where you say, I have two workloads, and I need them not to be on the same machine. Because if I trip over the power cable for that machine, I'm losing them.

If you lose the abstraction of a node, you lose the ability to describe that. So until we extend Kubernetes in some fashion, the work that's being done by people like the Virtual Kubelet team, for example, to say, this is just a magic pool of things that doesn't yet have a way of describing properties which are true of the underlying hardware. And I think to what you were saying about enterprise, and that's a real concern for them.

ERIC BREWER: Yeah, that's a good example. So the way I think about it for the large enterprises, the operations team ought to be using real Kubernetes and thinking about nodes. But the teams that they're building a platform for, their developers, probably don't need to think about nodes.

It's really two different audiences based on how much leverage they're getting from the platform. But I feel like the core team that's managing the platform that most developers use, they need to be aware of nodes for availability reasons and actually other reasons too. There's security concerns. There's licenses. Licenses are tied to nodes. You have to have a node-- this license is only good on this node.

And in fact, we can pack stuff better when we know that that kind of constraint exists. Sometimes you want fake sharing, where I want things on the same node so that they go down together. So there's a lot of subtlety there for actually achieving a high-quality infrastructure that is the purpose of Kubernetes. If you go and say, I want to be a developer, and I don't want to write nodes, I actually agree with that statement also. But that's why there is a platforms team in the middle that's going to, in your domain-specific way, bridge that gap.

CRAIG BOX: And with that, I would like you all to join me in thanking Eric Brewer.

[APPLAUSE]

You can find Eric on Twitter @eric_brewer.

[MUSIC PLAYING]

ADAM GLICK: We hope you enjoyed something a little different in today's show. We'd love your feedback. Please send it to us @kubernetespod on Twitter, or by email at kubernetespodcast@google.com.

CRAIG BOX: You can also check out our website at kubernetespodcast.com and find show notes and transcripts. Until next time, take care.

ADAM GLICK: Catch you next week.

[MUSIC PLAYING]