ADAM GLICK: Hi, and welcome to the Kubernetes podcast from Google. I'm Adam Glick.

CRAIG BOX: And I'm Craig Box.

[MUSIC PLAYING]

ADAM GLICK: Welcome back, and happy New Year, Craig.

CRAIG BOX: Happy New Year, Adam. How are you this year?

ADAM GLICK: I'm doing fantastic. Thanks for asking, yourself?

CRAIG BOX: It's not a bad start to the year. I must say that I promised to do some spring cleaning over the Christmas break. And I've probably got a little bit more of that to finish off with. But it was great to catch up with some friends and have a bit of time to relax, call a lot of family around the world over the Christmas period.

ADAM GLICK: Lovely.

CRAIG BOX: I hear you were away in the middle of peaceful nowhere.

ADAM GLICK: Yeah, yeah. Every year, we take the opportunity to go and disconnect a little bit, to go off the grid, reconnect with family and nature. And it was just fantastic to have that respite, have that break, spend a little time finally winning Evoland 2, you know, the important things.

CRAIG BOX: I thought you were off the grid.

ADAM GLICK: Well, you know, it turns out that the tablet does come with a battery. So as long as you can finish--

CRAIG BOX: You can bring the grid with you.

ADAM GLICK: Exactly. Or at least, for a short period of time. Got any New Year's resolutions?

CRAIG BOX: I don't really do those.

ADAM GLICK: Good for you. I've endeavored to see if I can up my cooking game a little bit. We tried hot and sour soup last night, which, if anyone wants the recipe, it was quite delicious, actually.

CRAIG BOX: We'll put it in the show notes.

ADAM GLICK: Fantastic. Shall we get to the news?

CRAIG BOX: Let's get to the news.

[MUSIC PLAYING]

ADAM GLICK: Platform9 has released a survey of 500 KubeCon attendees that provides some interesting insights into the evolution of Kubernetes deployments. In particular, the research calls out that the of KubeCon is evolving, with 27% of respondents identifying as operators and 24% identifying as developers.

Of course, the biggest group identified is still the group doing both at 41%. The top three concerns that were identified were managing multiple cloud environments at 34%, running stateful applications at 33%, and a tie for third place, between the slow pace of migrating apps and operational complexity with Kubernetes, both at 31%.

Finally, Istio topped the list of technologies connected to Kubernetes that attendees are interested in, with 52% of respondents mentioning that they are currently evaluating Istio for use.

CRAIG BOX: The year opens more security reports in Kubernetes, though, this time of much lower severity. If you're using the open source Kubernetes dashboard, a vulnerability has been discovered which will leak its TLS secrets.

This isn't usually a problem, as the certificates are self-signed and stored in memory by default. And you still need to authenticate to the dashboard with the browser token. However, if you've uploaded custom certificates, they may have been compromised, and they should be revoked. A new version of the dashboard has been published and can be upgraded too. And that new version will be the default in the next patch release of 1.13.

There's also a bug in the API server proxying functionality that could allow users to hit arbitrary internal network IP addresses if you manually configured that address as the pod IP of a pod's metadata. That field can only be written by cluster administrators. And so it's probably only of concern to people who run the API server with multiple administrators in a complex local network environment.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

ADAM GLICK: Dan Kohn is executive director of the Cloud Native Computing Foundation, the host of Kubernetes, as well as over 30 other projects. He previously served as CTO of several start-ups, including Spreemo and Shopbeam. Earlier in his career, he was a general partner at Skymoon Ventures, a seed-stage venture capital firm. Welcome to the show, Dan.

DAN KOHN: Thanks, I'm thrilled to be here.

CRAIG BOX: This is actually your second time working for the Linux Foundation. How did you get involved the first time? And how did it lead to where you are now?

DAN KOHN: It was a very different organization then. I was a colleague of Jim Zemlin's when he was running this tiny predecessor organization called the Free Standards Group. And I was helping him with business development for that back in 2006 when we had the opportunity to merge with the larger and better-known Open Source Development Lab, which, among other claims to fame, employed Linus Torvalds as a fellow, the creator of Linux.

And so this is all ancient history now. But we helped combine those organizations together, come up with the name Linux Foundation, and somewhat of the current mission. And then I was the Chief Operating Officer through a pretty challenging four or five years there of growing it in revenue to about double the size it was when we started, which is still actually much smaller than CNCF is today.

And so I then left and went and did these two start-ups, one on my own that didn't succeed. Another one that had a small exit. And during that time, the Linux Foundation changed to become more of a foundation of foundations.

And so if you look at some of the sister projects of CNCF, Let's Encrypt provides most of the world's security certificates. Hyperledger is one of the leading options in blockchain. Hopefully, all your listeners are familiar with Node.js, the leading application framework. And there's multiple other ones.

And so as the organization has grown, Jim has pulled in different leaders to work on these different areas. And I had the opportunity come in and be the executive director of the Cloud Native Computing Foundation, which, 2 and 1/2 years ago, when I joined, was really quite small and nascent and had some challenges. And then has just had a truly extraordinary amount of growth in that time.

ADAM GLICK: What were the early days at the CNCF like?

DAN KOHN: The idea of what we were doing was really exactly the same as what we are doing now. The governance structure, the charter, et cetera were all the same. It's just we would have a marketing call with 10 people on it instead of 100 people.

And I definitely think there was a lot less agreement about what cloud native was, what our relationship with Docker should be, what the chances were that Amazon or Microsoft or Alibaba or Pivotal or VMware, all these other companies, would decide to align with us as opposed to be against us.

And so all of those things were the promise in the future. But fundamentally, the mission of CNCF, and really all the Linux Foundation open source projects, have been pretty consistent where we're trying to provide a good home for projects. We try and provide a way for companies and organizations to contribute to those projects and provide some marketing help and organization and such.

We've always been focused on technical decisions that the projects are making separate from the business ones. But obviously, things have now increased pretty dramatically in scale, where we are the largest and fastest growing open source organization ever. And so it has been a pretty wild ride.

ADAM GLICK: It sounds like there was a number of challenges to overcome. How did you overcome some of those pieces of the growth and the scaling that you're talking about as well as separating the technical from the nontechnical challenges for projects?

DAN KOHN: Well, the first part, the technical and the nontechnical was really built into the charter and the whole DNA of how the Linux Foundation operates. And conveniently, Kubernetes and the other projects that we host have very strong feelings on these subjects, the leaders of those projects.

And so it just wouldn't have been the case that I, or anyone from our governing board, or someone else would have been able to come in and tell them, oh, you need to do x, or need to do y, or, jokingly, oh, except this patch, or such.

So that was absolutely never in the cards. But I think it was more, the bigger challenge, was just trying to communicate the idea of a software foundation and saying that we were going to provide a home for these foundations, a legal home, protect the trademark, and then that we were going to provide a set of services to the projects.

Probably, conceptually, I think the biggest challenge early on was just trying to convince a lot of the members of our community that these interactions were not a zero-sum game. So we run this event, KubeCon + CloudNativeCon that was 500 people before it was contributed to us, and then 1,000 for the first one we ran, and 8,000 in Seattle in December. And we've gotten feedback along the way of, oh, well, you can't just have one event. Or, CloudNativeCon is smothering KubeCon and a lot of zero-sum thinking.

CRAIG BOX: I think Ice Cube-Con smothered both of them.

DAN KOHN: Fair enough. But I think maybe the biggest takeaway-- and it shouldn't come as a surprise, because it is the underlying premise of open source software development is that many interactions, and hopefully most of them, are positive-sum where not just can developers come together, but actually companies that are competing can come together and cooperate and actually both be better off than if they went their separate ways or decided not to cooperate. So I'd say that was probably the most important theme that we've tried to communicate and echo and focus our strategy around the last three years and going forward.

CRAIG BOX: It was easy, looking at the Linux Foundation in the past, to say what they did. Well, they employ Linux. They deal with the fact that there needs to be a community that exists around the technology that's separate from any vendor.

When we start looking at the CNCF, which is largely a foundation started in order to bring a lot of these vendors together, it doesn't employ the contributors in the way that the Linux Foundation employs some of the Linux lieutenants. How do you describe the differences between those two approaches?

DAN KOHN: Oh, I don't think they're that different. The Linux Foundation has two fellows, Linus Torvalds and the stable kernel maintainer Greg Kroah-Hartman. And I don't have the numbers in front of me, but the two of them together provide well less than 1% of all commits to the Linux kernel.

CRAIG BOX: But they're still jolly good fellows, for the record.

DAN KOHN: They are. And I actually would recommend that you chat with them at some point in the future. They really are both fantastic people to work with. And I've just learned a ton getting to see their perspectives on open source development and also over that process of 20 years.

And Jim Zemlin, from the Linux Foundation, I think coined the phrase that Kubernetes is the Linux of the cloud. And there's definitely some meaningful similarities there that Linux acts as this universal abstraction that can run on any machine. And then your software can run on top of it. And Kubernetes, maybe it's not quite there yet. But it at least aspires to provide that same level of abstraction on top of any public, private, or hybrid cloud where your cluster application can then deploy on top of it and abstract away those details below it.

And just as Linux has kind of become a little boring where you just assume it's built into every consumer electronics device, and every supercomputer, and every kind of server out there, the Kubernetes community, especially, at least aspires to that level of boringness, where it's just a kind of default option. It's very stable. And then a lot of the innovation can occur above it.

CRAIG BOX: One of the things I like to say about the CNCF is there's no other organization that I know of that has all of the cloud vendors as top-tier members.

DAN KOHN: Yeah. And I would say also quite interesting that, it's not just the public cloud vendors, but also the major private cloud vendors. So we have all of the biggest enterprise software companies and then a number of the companies that are providing hybrid cloud solutions. So I do think there's some meaningful value right there in just getting all of those folks together and, ideally, to some degree, seeing eye to eye on things.

CRAIG BOX: Are there people that you have to separate at the governing board meetings?

DAN KOHN: There was a moment in the past where we got an argument between the primary and the alternate representative from the same company in the governing board. So we try and avoid those sorts of conflicts. But given the fact that-- how much work has gone into all of the projects in CNCF, and the fact that there are literally billions of dollars at stake, I think the governing board and the other leadership groups in CNCF have done a remarkably good job of seeing eye to eye and allowing things to move forward.

CRAIG BOX: Does it get more challenging when the more members you add, the more you are adding to your governing structures?

DAN KOHN: It's definitely challenging in a different way. I think I would draw the analogy to a start-up where, when I came on 2 and 1/2 years ago, it was relatively unclear that we really had product market fit. Well, both of Kubernetes, which, of course, was incredibly promising from the get-go, but still had far lower market share on things like Google Trends, than, say, Mesos, or Docker, or OpenStack.

But also, just on the whole concept of an open source software foundation, to say, is there enough value being generated here? Are the dues at the right level? Should we even hold a conference of our own? Do all of these things make sense?

And again, this concept of, oh, the different aspects that CNCF works on can actually build on each other as opposed to having to do this to the exclusion of something else, is definitely an area that we've had to communicate and demonstrate over time.

And then, like a start-up, we do have scaling challenges now. And so we've needed to just automate a ton of different processes as part of-- we had 28 members when we launched three years ago. And we just signed on our 350th member this week.

And so it's a very different process to onboard those, to communicate with them, to engage with them. Three years ago we had 40 sponsors for KubeCon Seattle, actually, just two years ago. And in December, we had 187-- and so the amount of effort that it takes to bring all of those folks on and get their artwork and communicate all the processes and such.

And then, really, the biggest area has been on the projects, where we have created this concept of a service desk. And this was inspired by what Mayor Bloomberg did in New York City of creating a 311 telephone number where every service available in the city was available through a single number.

And so we have a service desk that the maintainers of each of our 32 projects can send a request and say, hey, I don't like my logo anymore. I want to use this CI system. Can you please help me with this? Pager alert kind of system-- I want to redo my website. I want a security audit-- all of these different services that CNCF provides. And then we track that, and engage with them and, hopefully, say yes to them and then have a way of measuring how long it takes us to respond. And are they ultimately happy with the resolution we put together?

ADAM GLICK: Can you share what you tell companies who are hesitant to embrace or contribute to open source that can help change their perspective?

DAN KOHN: Yeah. I think that is one of the areas where CNCF and the Linux Foundation can do a lot more in the next couple years. I feel like 2018 was the year that Kubernetes crossed the chasm and is now the accepted choice for the early majority, if you know the crossing the chasm terminology.

But that late majority and laggards, which is more than 50% of all companies out there, I think are only just beginning to contemplate cloud native and also open source development and the whole concept of engaging with the community and not trying to own everything in-house or not just working with one vendor.

And there's certainly a cost-saving aspect to it. And you can talk about bin packing, and efficiencies, and other kinds of things. But by far, the message that I think resonates the most with these folks is about increasing their internal velocity of their own software development.

And so of course, you have the Andressen cliche, that software is eating the world, that every company is becoming a software company. And it's not just second hand. Within the Linux Foundation, we've had challenges of rolling out a Salesforce system or deploying new software ourselves. So it's not the case, just because Linus Torvalds works for us, that every piece of software we touch just seamlessly works.

CRAIG BOX: Or is open source, for that matter.

DAN KOHN: Exactly. And we don't have the philosophy that 100% of software we use has to be open source. Certainly, all of our projects and everything [they] use is open source. But when you look at that process of just how hard it is to develop software and how important, I think almost every company out there now realizes that and is genuinely scared of being out-competed, that their competitors are also looking at ways to improve software development.

And so I'd say I hear a huge amount of openness to the idea, that national mindset of, oh, I'm going to write a waterfall specification. And I'm going to have a QA team. And every three months we're going to do a new release. They're realizing that that really doesn't work.

And they're looking to companies, certainly, like Google, but really, almost any company in the last decade that's tried to reach an internet scale. Folks like Yelp, and Twitter, and Spotify, and dozens of others have almost independently developed a lot of the key ideas behind cloud native.

And I'd now say that one of the key concepts of CNCF is that those ideas are being encoded into software, that Kubernetes is certainly the leading part there. But if you look at the other 31 projects of CNCF, they're also all trying to encode and represent best practices on how to support this newer philosophy, newer style of software development. And most companies I talk to are really eager to begin their path down that new direction.

CRAIG BOX: Let's have a look then at these projects. You have projects that are in various phases. We have effectively the alpha, and the beta, and the GA phase for projects in the CNCF.

DAN KOHN: That's a good way of thinking about it.

CRAIG BOX: And then you have projects of different sizes and that address different needs. You also have the Cloud Native Landscape, which draws a picture, not only of projects that are run by the CNCF, but that address this kind of space. So how do you think of parts of the cloud native stack that are not currently part of the CNCF?

DAN KOHN: There's one more document that's worth looking at, which is the Cloud Native Trail Map. And if you go to L, for landscape.cncf.io, there's a link right at the top to see the Trail Map, as a PNG or PDF. And this is our recommended path through this complicated landscape of more than 600 projects, both open source projects and closed source products, that we track.

And the way we think about it is that CNCF projects have been endorsed by this independent neutral technical body or technical oversight committee. They have a lot of momentum behind them. They have a lot of companies working on them.

But that said, I don't believe that there are any enterprises in existence that are using all 20 of the CNCF graduated and incubating projects and no non-CNCF projects. And so in the real world, I think most companies are working with some of our projects.

And then they might be using a third-party logging service or monitoring or, often for bigger companies, they have their own internal projects that they're either big fans of and feel like add a lot of value, or that they're just stuck with and are looking at years of effort to migrate away from it.

And so we do recommend that Trail Map as our recommended path through this somewhat complicated landscape. But we also maintain that landscape document because we do think it's worthwhile for folks to be able to look at all of the alternatives out there and understand that you can be cloud native and that most Enterprises have a solution they're very happy with without being solely on the CNCF stack.

CRAIG BOX: Does the Trail Map only highlight projects that are members of the CNCF?

DAN KOHN: It does.

CRAIG BOX: So how would you deal in the situation where there is a clear winner as an open source project or, perhaps, a hard dependency to a CNCF project, but that project, for whatever reason, is not a member of the CNCF?

DAN KOHN: I don't know that I would particularly see it as a problem. If you look at Kubernetes, for example, it's dependent on literally hundreds of different open source projects, particularly Go modules. And obviously, it also has just a hard dependency on Linux underneath it. And so I don't think that we've ever aspired to saying that 100% of the software in enterprise runs should be hosted by CNCF.

ADAM GLICK: Given that there's so much of that out there, and given the popularity and growth of many of the CNCF projects, Kubernetes among them, how do you avoid forking and fragmentation?

DAN KOHN: I think this is a huge question of open source in general. There's been an interesting area of debate about different kinds of license, the copyleft licenses, like GPL that the Linux kernel uses and then the more permissive licenses where all CNCF-hosted projects are licensed under Apache 2.0.

And there's been a debate for a while of, oh, well, if you don't require people to publish their changes on distribution, which copyleft requires and permissive doesn't, then it'll just encourage people to fork. And they won't contribute their changes back.

And conveniently, I think the CNCF, and particularly Kubernetes, has been somewhat of an existence proof that that concern is not valid. That, instead of thinking of it as software as being a one-point-in-time release where, yes, any vendor could go along and fork that and make changes to it and say, OK, here's our commercial implementation of it. But instead, you really should think of software as being a flow, as being like a river.

And there are so many changes that are constantly going on to all of our projects that trying to stop with a point in time and fork it would just leave you off of a huge number of improvements and bug fixes and security fixes and everything else.

And so there's just a natural momentum built into these projects where, if you've done a customization to Envoy or Linkered or any of these projects, to work in your environment, rather than having to maintain that fork yourself, and maintain that patch and continue to move it onto new upstream versions as they come out, you have a huge economic incentive to upstream that patch, to get it accepted upstream, where the whole community then maintains those changes going forward instead of you having to maintain them yourself.

So I find the whole forking and history of it and such fascinating. And it is something that I've read about a bunch and written on. But just the fact that we don't have any forks of our projects today, I think, is a pretty strong indication on the value that both enterprises and then, particularly their end users, to see in staying basically conforming with upstream.

ADAM GLICK: Do you think that's something that has organically grown out of the community in this set of projects, and as people realize, the best way is to work with open source? Or is that something that's been an active process that folks like you at the CNCF have helped to guide?

DAN KOHN: It is definitely the latter. And it is unfortunate that a lot of folks in the community don't get to see all of the conversations that Chris Aniszczyk, CNCF's Chief Operating Officer, who focuses particularly on the projects and myself have, with our members, where I do consider a big part of our task just trying to talk people out of bad ideas and, in a very friendly way, explaining, hey, we've seen this movie before.

We know how it ends. You really don't want to go down that path. And just to be clear, there's no stick that we're holding. It's all carrot. It's all just advice that we're offering. But I do feel like we're able to offer that, particularly, to some of the companies that maybe don't have the same history or track record of open source that Google does.

ADAM GLICK: You mentioned licenses. And recently, there's been a fair amount of talk about open source licenses and their impact on the community, how people are thinking about those. The CNCF currently recommends the Apache 2.0 license. And I was wondering if you foresee that changing?

DAN KOHN: Oh, I definitely don't foresee it changing. I'll go ahead and put my marker in the ground, which is that I am not a fan of the shared source efforts that-- I guess it's Redis and a couple other start-ups have--

CRAIG BOX: MongoDB.

DAN KOHN: Yeah-- have gone forward on. I do want to make just a fundamental point, where the company or the developer who creates the software owns the copyright of the software and absolutely has the right to change the license. And so I have no concern about what they're doing there.

But I also have no concern about the cloud companies. And I don't agree with the accusation that they're leeching, or taking advantage of, or anything else. Because they're all, as far as I can see, using the licenses in exactly the way that they were designed to be used.

And so my philosophy is that, the huge win for open source software is adoption, that the more people you have using the software, the more likely they are to pay for support or pay for services surrounding it, or for a commercial version of it, or anything else and then, in particular, to start making contributions back where you do get this magic of all of us being smarter than any of us and anything that you do that creates friction, that reduces that adoption, makes that process much more difficult.

And so a shared source license that says, well, this only impacts 1% of our users. 99% won't be impacted at all-- I think is actually not a correct way of viewing the situation. Because I deal with a lot of the attorneys from many of our end-user organizations and many of our vendor organizations.

And it is so incredibly helpful to just use a license that everybody is already familiar with. And Apache 2.0 being one of the most common and most familiar licenses out there. Although, I would actually put MIT, or BSD, and GPL v2, in the same category.

But just that mental overhead that's involved in trying to explain, even what a new license is, let alone a license that I think is now, widely agreed is not an open source license, that shared source. It's not open source. So my expectation for CNCF is that we will continue to require Apache 2.0 licensing for years to come and that that will be a real positive for our projects.

ADAM GLICK: Gotcha. And you mentioned a number of the open source licenses that we see that are fairly popular, like BSD, and MIT. How did you decide on Apache 2.0 as the preferred license?

DAN KOHN: The goal of CNCF from the get-go has been that, for the projects that we host, that we would create an intellectual property no-fly zone, meaning that all of these companies, arch competitors, and end users, and hobbyists, and anyone else, could all come together and work on things and be confident that no one was going to come back later and sue someone else.

And so the MIT and BSD licenses are great for giving you protection on copyright. But unfortunately, they're silent on the question of patents. Where the Apache 2.0 has the additional language in there to say that it's not feasible for me to join CNCF and put in some code to Kubernetes, and then file a summary and patent that reads on that code, and come back later and sue the users of the code. The Apache 2.0 license forbids that scenario. We can link to an essay that I wrote on the subject. But I do think that Apache 2.0 is a particularly good license to use.

ADAM GLICK: One of the other things that I've heard people talk about is a concern that, as many of these projects have become more popular, that large organizations may enforce outsized control or effectively take over the projects as they join and take leadership roles in the organizations. How do you address that?

DAN KOHN: It's an interesting philosophical question. First of all, I'll say what CNCF doesn't do, which is, if you look at Apache for example, there's the Apache way of how projects are governed. And when you join Apache, you sign on. And there's an incubating process. And you learn that way.

And interestingly, there is no CNCF way of project governance. Instead, we require that the governance be neutral, which means that it not be biased toward one company or against one company or person or country or such. And we require that it be documented. So in order for a project to graduate, it has to have a governance.md document, or the equivalent, that lays out the process there.

But the Kubernetes governance process is very different from that of small projects, Prometheus and Envoy, of the three that have graduated so far and different again than containerd and some of our others that are moving towards graduation.

And so I think the biggest guarantee against having one company dominate or wield too much control is for the other members of that project community to engage and ensure that that doesn't happen. But in general, in most open source projects, it's the people doing the work who get to call the shots.

And I think that tends to be the case as well for most CNCF-hosted projects. And conveniently, for something like Kubernetes, which originally came out of Google and Google has been the largest contributor to it and continues to be the largest. But its total contributions as a percentage of the whole have actually been going down because so many other organizations have been getting involved as well.

CRAIG BOX: We like to call that growing the pie because we like pie.

DAN KOHN: Exactly. And that is our aspiration for all of our projects.

CRAIG BOX: What projects that are not currently in the CNCF would you most like to see join it?

DAN KOHN: I would love to see Google contribute Istio and Knative to CNCF. I think both of those rely on Kubernetes and Envoy, which are currently hosted in CNCF and would be a natural fit. But I will point out that CNCF doesn't ultimately have any leverage over any of our member companies on this.

It's really the choice of those companies and, in many cases, of the developers within the companies, on whether they feel like the marketing push and the additional spotlight and services that we can provide are worthwhile and then, in particular, when they're worthwhile. There's no magic moment 90 days after a project becomes open sourced when they have to come into CNCF or not.

CRAIG BOX: What happens if the project loses momentum? Is there a process by which a project would leave the CNCF or not graduate from one of its phases?

DAN KOHN: There's no requirement for a project to move from incubating to graduation. And there is likely going to be one or more projects that move out of CNCF and into some kind of attic or deprecated status. And so the goal where CNCF is essentially building a stack and trying to send a signal to enterprises out there about the kinds of technologies we encourage them to adopt, means that there is a valuable role for us to play as well when a technology doesn't live up to its original goal.

ADAM GLICK: The cloud native ecosystem is growing really quickly. And in some cases, there are multiple projects do very similar things. How do you think about the overlap that can exist with both projects and visions?

DAN KOHN: Yeah. The Technical Oversight Committee within CNCF, which sets our technical direction, wrote a really nice document on this that is available on the CNCF website, and we can link to it. But it's their set of principles.

And one of the key concepts there was that they did not want to have a single project-per-box philosophy. And I think that fundamentally comes from a position of intellectual modesty of realizing that none of us are truly smart enough to predict the winning technologies and certainly not universally predicted.

I think Kubernetes is a relatively safe bet at this point. But within service mesh or container run times or registries or other kinds of areas, there are often are more than one winning solution. You could even go to, say, wireless, where both Bluetooth and Wi-Fi have won, in some sense, with slightly different niches.

And so I think the TOC has expressed a strong willingness to have more than one project in a given, say, landscape category if they feel like each of the options are credible and well-run and promising.

ADAM GLICK: Under your leadership, you launched a certification program for administrators and developers with Kubernetes. How's that project going?

DAN KOHN: It has really been fantastic. So we launched that about 18 months ago. And we now have almost 5,000 people who've gone through that certified Kubernetes administrator exam. And this really is something that we're quite proud of.

CNCF staff didn't develop the curriculum. We brought together experts from our member companies, again from competitors, to agree on what an intermediate-level administrator should know. They first put together that curriculum and a set of questions on it.

But then what's quite neat is that the exam itself-- it's not multiple choice. It actually spins up seven different clusters during the course of a three-hour exam and requires you to install Kubernetes, and debug installations, and demonstrate your expertise.

And it's a little stressful. It happens while you're watched over a webcam by a proctor, who you hold up your driver's license to show that it's really you. But the initial version we did of it, a year and a quarter ago, we got a little constructive feedback, oh, this wasn't right. These things were not optimal.

But what's been great is to then be able to iterate on it. And every three months, we come out with a new version in line with the new Kubernetes version, generally, about a month or six weeks behind it. And we've now gotten quite good feedback that the exam is a good test of skills.

And then we feel like that CKA process is a quite nice hiring signal right now, as so many different enterprises are interested in staffing up and building up their own internal expertise.

CRAIG BOX: Are you a certified Kubernetes administrator?

DAN KOHN: I am not. And it is a 2019 aspiration of mine that I would like to pass that test. Maybe I can come back on after failing it a couple of times and say, oh, no, this exam is way too tough. But a lot of people I respect have passed it. And so I would love to join their ranks.

ADAM GLICK: Will you be expanding that out to other CNCF technologies, like Prometheus or Envoy?

DAN KOHN: Yeah. We just launched a training course for Prometheus that would be the natural first step for that. And so we definitely are evaluating how we would offer that, possibly, like an add-on exam. Or in Scuba, you take this base level. And then there's a set of expert-level things you can do like ice diving and deep diving and photography and such. And so we are looking at how we would roll those out.

ADAM GLICK: Kubernetes offers a certification for the distributions. Can you talk a little about that?

DAN KOHN: Sure. Of all the efforts that CNCF has done to support Kubernetes and our other projects over the last three years, and the conferences and services we provide and such, I think the certified Kubernetes initiative is maybe the one that I'm most proud of and I think has had the biggest impact on the ecosystem.

And so this is a way, not just the distributions, but also hosted platforms, like GKE or Amazon's EKS, can make changes to Kubernetes and adapt them for different environments and improve them, but show that their version of Kubernetes remains compatible with upstream.

And what's pretty neat about this process is that it is something that we've collaborated on extremely closely with the Kubernetes community led by SIG Architecture. And they're the ones who ultimately define what it means to be conformant with CNCF and our governing board, which owns the Kubernetes trademark and has the responsibility for that.

And so we have this open source test suite that's built into the Kubernetes project and continues to evolve with it. And then any CNCF member can run this test suite against their hosted platform or their distribution or their installer and demonstrate that none of the changes that they've made to Kubernetes have in any way impacted compatibility.

And so when we launched the program just about a year ago, we thought it had a lot of promise and were hoping it would be successful. But it's really succeeded beyond our aspirations where we now have 80 different certified Kubernetes partners representing the entire industry. And I would say it's actually one of the most successful certification programs anywhere.

CRAIG BOX: It felt to me that the aim of this project was to define what it means to speak the Kubernetes API so that projects could use a different backend and still speak Kubernetes out the front. Like for example, Google's Cloud Bigtable uses the HBase API. So you only have to use the same clients for things that you're familiar with, and you have a different backend service.

However, the way people actually choosing to implement it is just running the open source binary. So all of these sort of distributions, as far as I know, are providing the API simply by running the Kubernetes binary that we all built together. Was that your idea from the beginning?

DAN KOHN: Oh, I don't think that's actually correct. So a number of the distributions out there are making meaningful changes to their version of Kubernetes. And so it's actually-- the aspiration of the program has always been that people should be able to change Kubernetes a lot. And in particular, they should be able to put in a lot of other software on top of it-- that, I think there's a wide consensus that Kubernetes alone is not normally the complete solution.

In fact, it's in some ways, the whole philosophy behind CNCF is to have a constellation of projects that support or add value to it. But that, while making those changes, the core Kubernetes APIs need to remain compliant and to perform the same tasks that they're supposed to. And so that's what the conformance suite is testing. And that's why we're thrilled to see this entire community sign on to that process.

CRAIG BOX: 2018 was the first year that the CNCF has run an event in China. What was the impetus to do that?

DAN KOHN: Yeah. And I was really glad that you guys were able to come over for it. China has been a huge focus of CNCF's and of mine. When we launched CNCF three years ago, we had 28 members. One of those was Chinese, our founding platinum member, Huawei.

And today we have 39 Chinese members, including three platinums, Huawei, Alibaba, and JD.com-- those are the first and second biggest retailers in China-- and then three gold members-- Tencent, Baidu, and ZTE, and a number of smaller ones.

So we've seen just a huge uptake of these technologies and interest in what CNCF is doing. And a number of our Kubernetes certified service providers and our certified Kubernetes offerings are Chinese companies.

So I think what probably finally drove it was going to an event that I attended in October 2016, which was an end-user meetup. And there were 700 people there. And I wasn't sure I could get 700 people at a end-user meetup in the Bay Area.

CRAIG BOX: To get 700 people at a meetup in the Bay Area, you'd have to bring a lot of pizza.

DAN KOHN: Exactly. And just absolutely fascinating end users, like a hotel company that's not targeted at Westerners-- it's more like a two- or three-star hotel. But they were in second- and third-tier cities across China, a million hotel rooms, and all of their backend operations ran on Kubernetes-- and just seeing more and more stories like that.

So we made the decision to have-- KubeCon's always been in North America and a Europe-based conference, KubeCon + CloudNativeCon-- to launch the third one in China. And I really wouldn't reduce the level of complexity that we took on with that.

I mean, there is a huge language barrier. And I think most Chinese developers speak technical English, meaning that they're fine reading the Kubernetes documentation, and using an online translator, and can often get by via email and other kinds of asynchronous communication. But--

CRAIG BOX: They've got to learn Greek as well as English.

DAN KOHN: Yes. But to offer the simultaneous interpretation for, not just the keynotes but all the sessions, was a significant expense. But just in general, working in Asia is very challenging. North American people are happy to travel between US states and Canada. And Europeans are generally happy to travel around. But you just don't see a lot of Koreans and Japanese traveling to China or vise versa. They tend to be very different markets, partly, because of the language issues.

CRAIG BOX: Will you be looking to run events in Japan and Korea?

DAN KOHN: We might at some point. We participate in the Linux Foundation conference, the Open Source Summit, which takes place in Japan. And we're looking at how we might increase that participation. And we're also looking at holding a new kind of conference called Kubernetes Day, which is a single-day, single-track event. The first one will be in Bangalore on March 23rd. And we're quite excited to roll that out into India. But we would love to do that in Korea as well, as an example.

CRAIG BOX: KubeCon and CloudNativeCon are merging with the Open Source Summit in Shanghai this year. What was the decision like to merge those events?

DAN KOHN: Right. So we launched this event in Shanghai in November. And the conference was really, by all appearances, quite successful. So we had over 2,500 people attend. We had over 50 sponsors for it. We had a number of, not just speakers but attendees, travel from the US and Europe. About half the speakers were Chinese and half Western. And looking at how we could build on that start and grow it from there, the Linux Foundation has had a separate event in China that's focused on their other technologies-- so artificial intelligence, blockchain, and networking, and such.

And so we made the decision, for 2019, that we're merging that Open Source Summit in China into KubeCon + Cloud Native Con. And what's great is we're still going to have both the CFP tracks that people have come to love and the maintainer tracks of giving people a chance to engage with our projects and with the project leaders.

And then we'll be offering an additional six tracks on these other Open Source Summit topics. But many of which of those technologies run on top of or next to Kubernetes and the other cloud-native technologies. So there's actually a pretty strong level of alignment. And then things like Linux, of course, are running underneath Kubernetes. And interestingly, we're only waiting seven months. So that next conference is going to be at the end of June 2019, back in Shanghai again.

ADAM GLICK: There were 8,000 attendees at KubeCon US in December of 2018. That's about twice the size of the event the year before. That's got to be heading for, if not the largest, open source conference that, at least, I'm aware of. How big do you think that KubeCon US can get?

DAN KOHN: I don't really know, to be honest. The huge concern that I had was that-- and I did speak to a number of people who'd been to all four KubeCons so far-- San Francisco, Seattle, Austin, and Seattle again. And the huge concern I had for the December event was people were going to say, oh, it's too big now. It's not KubeCon anymore. I couldn't have any useful hallway conversations. I wasn't able to see the people that I wanted to.

And thankfully, that was not the case. We really did get great feedback from a lot of the longtime folks that they still found it an immensely useful and fun event. Also, overwhelming-- and so we do take that feedback seriously. And we are looking at some ways to try and reduce some of that tension or the amount of effort involved in interacting with all those people.

But it really was incredibly heartening that, both our longtime community, and then also speaking to a number of folks who, this was their first KubeCon, had very positive reactions to it as well. So my answer is that we're really focused on making it a useful conference at every level for the core maintainers who, particularly, I think come to the conference more to give back and to share their feedback with other folks-- all the developers from all the vendors, and all the end users who want to interact with those core folks, the sponsors who often want to be trying to sell things to those end-user developers, and everywhere in between.

And as long as we feel like we can keep doing a good job, then I don't think we're going to be capping the size. But right now, it's a little hard for me to predict forward too far.

ADAM GLICK: If it continues to grow, as I think it probably will, how do you foresee it changing?

DAN KOHN: I think the key part just has to be to have tracks and to have really clear signaling. And we do some of this. But I think we can do an even better job of just saying, this is the one-on-one track. Here's our recommended path if you're new to this space and trying to get a foothold in it.

Here's the recommended path if you're really pretty comfortable with it and want to level-up your skills. And then, if you are a top expert here, here's the path we recommend to improve your expertise or maybe to move into some of these side areas that you wouldn't be as familiar with.

CRAIG BOX: Do all those three things need to be the same event?

DAN KOHN: It's possible that we could split it off. But it is challenging for a lot of folks to get more than a week off to engage in this kind of thing. So again, from that earlier thought that this isn't a zero-sum interaction, I think there are some huge positive sum interactions from having them at the same event.

Another area that we are very actively investing in is this Partner Summit or Pre-Day. And somewhat insanely in Seattle, we had 27 different Partner Summits. And so that allowed a number of vendors to reach out to their customer base and have a chance to interact with them.

But it also allowed new things like EnvoyCon, which sold out and seemed to have just a great reception and a very positive review and Observability Summit and several others. And then, within all of that, it's really one of our highest priorities that the Contributor Summit be the first among equals.

And so we are looking in San Diego on how they can have more of their own space and their own conference feel and get all the services and resources they need to make that both successful and lower stress for the ContribEx folks that are managing it. But I am pleased to say that the reports from the Contributor Summit that happened in December were quite positive, that people still seemed to get a lot out of those interactions.

CRAIG BOX: All right, Dan, thank you very much for joining us today.

DAN KOHN: Well, I really appreciate the work both of you do on sharing updates and connecting with different people in the community. I really enjoy listening to the show.

CRAIG BOX: Thank you, we very much enjoy doing it. You can find Dan at dankohn.com, or on Twitter, @dankohn1.

[MUSIC PLAYING]

ADAM GLICK: Thanks for listening. As always, if you've enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at Kubernetes Pod, or reach us by email at kubernetespodcast@google.com.

CRAIG BOX: You can find our show notes and transcriptions on our website at kubernetespodcast.com. Until next week, take care.

ADAM GLICK: Catch you next week.

[MUSIC PLAYING]