CRAIG BOX: Hi, and welcome to a Kubernetes Podcast from Google Special. I'm Craig Box, with my graduating guest host, Jimmy Moore.

[MUSIC PLAYING]

CRAIG BOX: Yes, it is sad but true. Jimmy, your time with us is coming to an end. You've been working with us for 58 episodes, since June of 2020, and you've been my guest host for the last 14. So first of all, my heartfelt thanks to you.

JIMMY MOORE: Thanks, Craig. Actually, I've loved it. I've loved co-hosting this, and I promise you that it was never on my vision board that I would one day co-host the Kubernetes Podcast from Google, but here I am. And my advice to all you kids out there is don't make plans. There's better things coming your way.

CRAIG BOX: You come from an event background. You've been great at the organizational piece. Obviously, Kubernetes was new to you when you joined, so I do want to do a little pop quiz and see how much you've picked up during your time with us.

JIMMY MOORE: Oh, great, because I'm sure I know absolutely nothing about Kubernetes, but we'll get to that.

CRAIG BOX: First question, how many "Kubernete"s are there?

JIMMY MOORE: Hmm. Well, I know that some people say K8s, so maybe eight. But also, there's like Release 1.23, so maybe there's like 1 and 1/2. I'm not sure.

CRAIG BOX: We'll accept that. It's more just talking about the fact that it's a plural.

JIMMY MOORE: That's right-- Kubernetes.

CRAIG BOX: Now, I've got a Scrabble bag here and I'm going to draw out seven tiles. They spell K-U-B-E-C-T-L. How do you pronounce that?

JIMMY MOORE: Well, for the first six months of my time here, it was "Kubecuddle". Kubecuddle, like it was like an action. I thought it was a verb. It's time to Kubecuddle. But I've learned since that it's Kube-C-T-L.

CRAIG BOX: Correct.

JIMMY MOORE: Ding, ding, ding. Is that right?

CRAIG BOX: It depends who you ask.

JIMMY MOORE: I still prefer Kubecuddle.

CRAIG BOX: Now, this is a dialect question, of course. Are you saying cuddle as in hug, or cuttle, as in the fish?

JIMMY MOORE: Yes, I am.

CRAIG BOX: Both are wrong.

JIMMY MOORE: [LAUGHS] That's right.

CRAIG BOX: Tabs or spaces?

JIMMY MOORE: Oh, I actually know this is about, and I'm going to say tabs all the way.

CRAIG BOX: Show your working.

JIMMY MOORE: So one of my jobs here is to do this blog. And when people format their blogs with spaces, it drives me insane. You can't translate spaces over to other software and other platforms. You have to use tabs. Those are universally understood as far as I know.

CRAIG BOX: Good answer.

JIMMY MOORE: Is that right?

CRAIG BOX: If you like.

JIMMY MOORE: All right.

CRAIG BOX: YAML or JSON, Jimmy?

JIMMY MOORE: YAML or JSON. I'm going to say, "What is sharding?" for $1,000, Craig.

CRAIG BOX: I like it. Keeping the Jeopardy theme going that we've talked about in the past, and calling back to your favorite piece of the Cloud Native ecosystem, as we spoke about on our very first episode together.

JIMMY MOORE: That's right. Sharding and resharding. Let's not forget resharding. It's very important.

CRAIG BOX: And finally, in front of you, I have placed the CNCF Landscape. I want your first thoughts.

JIMMY MOORE: At first, I was like, "ah, this makes sense. This is why I don't understand what's happening here". It reminds me of the menu at like a Jack in the Box, or when you go to a Cheesecake Factory, and it's just like pages and pages of menu, and it's just overwhelming.

At least now, like McDonald's and Burger King have settled down their menu. There's just one page of things to look at. You get a combo, you move on.

This is overwhelming. But I support all these people because we've interviewed almost all of them.

CRAIG BOX: I do like In-N-Out Burger. You go there and they've got three or four things on the menu. But apparently, there's this entire secret world and you can just ask for things and they make it for you.

JIMMY MOORE: Yeah, a secret menu.

CRAIG BOX: Perhaps there's an analogy to draw there.

JIMMY MOORE: Yes, I'm sure it's on one of the abstraction layers of their menu. Ha, ha, look at me.

CRAIG BOX: Now, the good thing about an episode not long after a KubeCon is that all the news has already happened. There's very little going on. So I don't feel bad in this Knative 1.0 Special skipping straight to the interview with our fantastic guest. So it just remains for me to say again, thank you, Jimmy, and all the best for the future.

JIMMY MOORE: Thank you so much. But does this mean I don't get to say, let's get to the news? How about this? Let's get to the interview.

[MUSIC PLAYING]

CRAIG BOX: Ville Aikas is a member of the Knative Steering Committee, a Bootstrap Member of the Knative TOC, named in the stone tablets that describe the earliest versions of Kubernetes, and now a co-founder of Chainguard. Welcome to the show, Ville.

VILLE AIKAS: Thank you so much for having me. It's good to be here.

CRAIG BOX: You're a famously humble person, but I'm going to have to start by saying that if Linux is the most active open source community of all time, and by some measures Kubernetes is number two, then that might make you the second most important programmer to come out of Finland.

VILLE AIKAS: [CHUCKLES] That's a very tall order.

CRAIG BOX: Three, perhaps, if you count the guy who created MySQL.

VILLE AIKAS: Yeah, brilliant programmers come out of it, that's for sure.

CRAIG BOX: But I do hear it took you a couple of attempts to get the programming bug.

VILLE AIKAS: Yeah. I had a bunch of friends, very, very good programmers doing assembly stuff and so forth. And for me, it was simply not interesting at the time. So I would just be playing video games and told my parents that I need a computer to help me study when, in fact, it was just playing The Way of the Exploding Fist, and all kinds of games like that.

But then when I came to the States as an exchange student, I got really interested in robotics. Then I was back to learning assembly in 6502, which is basically the Commodore 64. Then I originally was like, God, this is not interesting at all.

CRAIG BOX: You mentioned that you were an exchange student. You moved to the US when you were 16. Are you familiar with the story of Paddington Bear?

VILLE AIKAS: Yes, I am familiar with the Paddington Bear. [LAUGHS]

CRAIG BOX: Do you feel any similarities?

VILLE AIKAS: [LAUGHING] Yeah, a little bit. There are definitely days.

CRAIG BOX: You moved over there and you didn't know where you were going to be?

VILLE AIKAS: At the time, I think there was five or six of my friends — some close, some that I knew of — that were going to go for an exchange student. Some were going to Australia, some were going to the States. And I was told that I'm going to the States. But everybody else knew where they were going, and they were able to go and exchange letters. This is a long time ago, before the interwebs.

And I was like, I don't know where I'm going. So they basically told me, well, take this plane to Seattle, and then we'll sort it out. And I was like, all right, let's go.

I came to Seattle, got off the plane, and there was a sign, actually properly spelled my name, so I was like, that must be me. I stayed there for a couple of days and they were trying to figure out where to put me. And that family’s daughter’s family took me. They said, well you can stay with us.

They had three younger boys. So they were like, ah, you look like a good role model, which obviously was a mistake in hindsight, I'm sure.

CRAIG BOX: That being said, that's quite a number of years ago when you've effectively never left Seattle.

VILLE AIKAS: I had to go back for a couple of months after the exchange student year to go ahead and get a proper visa. So I went back there and came back as soon as I could.

CRAIG BOX: You studied at the University of Washington. I hear there's a lot of distributed systems people come out of that school?

VILLE AIKAS: Yeah. I first went to South Seattle Community College to get a robotics degree, which is where I got really interested in the engineering and then I went to UW. Yeah, there's a lot of brilliant people there, and that department is fantastic. I learned so much there and got a chance to participate in some research projects. Amazing place.

CRAIG BOX: Were there any particular technical interests or specialties that you picked up around that time?

VILLE AIKAS: Two things — one of them was networking. So while I was there, I was working in the NOC, Network Operations Center, as a job. The other one was operating systems and security-related work there.

So we were doing lots of low level bit twiddling and binary rewriting. And some of those things later on became things like NaCLand whatever else. And I forget what it is in Linux. Fun times.

CRAIG BOX: Probably not in 6502 Assembly at this point?

VILLE AIKAS: No. Yeah, this was x86, which made it extremely hard, because just being able to understand where the code is in an x86 binary, it's a bit tricky because it's one key code.

CRAIG BOX: As we will learn later on, there is a very strong tech community in Seattle. And I understand that your path to Google with some friends who ended up working there.

VILLE AIKAS: Yeah. Some of the research that I just mentioned when we spun off a startup out of that. So it was Brian Bershad and Brad Chen, both who were later on at Google. I think Brad is still at Google.

So we started the company, ran that for several, several years. Then I needed a break from the startup world, went back to UW. I thought I needed a break from startup world.

Then a couple of months later, I started another company with a friend of mine, AJ. All the while, ticktock, time goes by. And many of the friends from that startup had gone over to Google — Brian and Brad and a few other folks. And they were like, oh, you should come to Google. It's such a fun place.

And I'm like, well, the timing is not right, because now I have this other startup I want to see through. So again, I worked that for a couple of years. Once we sold the company, I was like, OK, now I'm ready for just one job.

At the time, I had two jobs. I was working for UW and the startup. So I went to Google and was fortunate enough to pass the interviews and got in there.

CRAIG BOX: You were one of the first 20 Googlers in the Seattle office?

VILLE AIKAS: About that, yeah. It was a tiny, tiny, tiny place at the time. It is kind of a funny story, because at the time I was thinking about moving to Europe, and I wanted to move to Zurich, because that's where Google had an office. Then it turned out I couldn't move there for personal reasons.

And Google said, OK, that's fine. You can work at Kirkland. And I said, well, I'm willing to move to Zurich, but I'm not willing to move to Kirkland.

CRAIG BOX: Oh, completely the wrong side of town.

VILLE AIKAS: Yes. So they were like, well, we are opening a Seattle office. So then when I started there, there was 10, 15, 20 people there, so tiny, tiny.

CRAIG BOX: Explain a little bit to our audience how projects worked at Google at the time in terms of defragging and moving them between offices and so on, and where you choose to work influenced what projects you would end up working on.

VILLE AIKAS: So my personal experience was basically this, that I was working in Google Voice, that was the first project that I was put on because it seemed interesting. How does it work? So when I came in there, they were like, well, there's all these things, go do something.

It was a bit overwhelming at first because you're like, I could do anything? They’re like, yep, you could do anything you want to do. And I was talking to David Herb at the time, who was running the Google Voice project in Seattle because we had just acquired GrandCentral a while ago, so they needed to go and hoist that into Google Infra.

And at the time, GoogleInfra only did HTP, so all of the VoIP stuff is in UDP. So that was a fun challenge. So then when the project got chipped, AKA, it runs in Google, that moved to Mountain View, the project. So I had to go down there, train the people, the Play Books, yada, yada, yada.

Then I got back to Seattle and I'm like, OK, well, now what? They're like, well, we don't know. So we kind of started chit chatting. And there was a few of us, Joe Beda was one of them, for example, who also had — this is one version of the chat, Buzz, I think it was at the time — that also was moved somewhere else.

So a bunch of us were kind of trying to figure out like maybe we could get together and build something fun. And we started this mailing list called Plan B. It still might exist there, because it was all of our Plan B because we were like, well, all of our projects just kind of went away. So we decided that we're going to go ahead and do Cloudy stuff.

There was me and Joe and Michael Sheldon, for example, there's a few other people. And we started chatting about who wants to do what. And like, well, in order to do Cloud, you need this and this.

One of those bits was storage. We need to have storage. And I hadn't really done much with storage. I was kind of like, I don't want to go unpack the earlier research. I don't want to do low level machine work, like been there, done that.

And I kind of would like to do something different. I'll do storage. Started banging out, at the time, it was called Big Store. But then when it launched, it was cleverly named Google Cloud Storage for Developers. That rolls right off the tongue.

CRAIG BOX: Two too many words, maybe. Drop the last two.

VILLE AIKAS: Yeah. You can definitely go ahead and start projects there if you want to. You can do 20% projects. It's an amazing place, Google, for being able to go and do those kinds of things.

CRAIG BOX: Google is famous for exposing its internal systems to the outside. When you're starting a project like Google Cloud Storage for Developers, is that just a case of writing a translation layer that makes the internal system available to the outside?

VILLE AIKAS: Yes and no. I mean, the nice thing was that there were those three already that existed at the time. So at least we wanted to go and maintain compatibility there. Some of the semantics, though, I didn't really like, namely, like you can read your writes. So if you were to go and put something, you might not be able to get it back.

CRAIG BOX: One of the most important things for a storage system to do.

VILLE AIKAS: That definitely seemed like it would have been important. So I decided to do that. So I originally wrote it in C++ and then I rewrote it in Java, because at the time, I needed to get strong consistency so that I could provide the read-via-writes.

So at the time, the only thing that provided it in Java was this, way before things like Spanner and everything else. So I had to use some internal stuff. And that was only available in Java.

CRAIG BOX: But in terms of the thing that backs this, you don't have to go and throw servers and directs and put hard disks in. There's some Google system available to you that you can use.

VILLE AIKAS: Yes, there was things that you could always build on. Very rarely do you have to go and really build all the everythings. Although, in the case of the Google Voice, we had to do a bunch of stuff there because the infra didn't really support what we wanted to do.

CRAIG BOX: So, in those days, you've got your storage system, Big Store. You've obviously got BigTable, which was exposed a few years later. There's BigQuery. Google Compute Engine was first named BigCluster?

VILLE AIKAS: Yes. Everything was BigStar. There was BigQuery, BigCluster, BigStore, because the point was it's in the cloud, and everything is big in the cloud. You named BigTable there, so yeah, the first version I'd write using the internal BigTable as the backend store, but the eventual consistency was not something that I did want to expose to customers.

CRAIG BOX: Once all those underlying pieces are built and available to people, then the team starts thinking about high level abstractions. The founding saga of Kubernetes has been told many times in many different versions, but the one that I'm going to choose to tell today and to ask you about suggests that Kubernetes was born at the Unified Compute Management API Summit in November 2013.

VILLE AIKAS: I think it's going to be a little tricky to put dates on some of these, because a lot of the thoughts and the ideas were discussed in the hallways and so forth. And I think maybe that's where it got enough momentum or formal noddings, that seems good and so forth. But if there's a date that you want to put on, as in, this is when all the things started, I think that's just as good as any.

CRAIG BOX: One of the things that dictates whether or not someone gets credit for an idea is whether their name is on the document that created it. The initial product requirement document for what was at the time called Project 7 has four names on it. So first of all, Ville, why do people claim Kubernetes has three founders?

VILLE AIKAS: What does it mean to found something? If you come up with the idea, and you decide to do that, and then the next day or a week later, somebody comes in and starts working with you on that one, the founder is a funny title.

CRAIG BOX: Very odd to use in a Google corporate context.

VILLE AIKAS: Yeah, when a project starts, you start talking about them, and then you get people coming in, and then you get stuff done. The idea of who is the founder — it seems funny in a role.

Like if you start a company, there's a very clear, like we are the founders, or you know, yada, yada, yada. For the project, especially in many settings, you have a, ‘would it be bad?’ moment where you start asking those kinds of questions.

And then somebody goes like, that's terrible, but let me listen. And then you talk about it for a while, and then you kind of riff on each other. Then you get into the, ‘OK, actually, let's do it.’ I don't know. That is a good question.

CRAIG BOX: I bring this up, of course, because your name is clearly the fourth name in this document. Does that make you the fifth Beatle of Kubernetes?

VILLE AIKAS: That is actually what Greg says, or the fourth Beatle. I don't know how many Beatles there are. I think there's four. But yeah, he has made the reference to the Beatles in the past as well.

CRAIG BOX: What was your actual involvement back in those early days?

VILLE AIKAS: Writing a lot of code. At the time, Brendan had basically put together this Java version of the proof-of-concept just so he could show things. So it had a bunch of moving pieces, like salt stacks, and yada, yada yada.

But it was basically like, look, here's an end-to-end thingy that might be like this. And we started tuning that. And one of the things we wanted to do was write it in Go, because none of us had ever used to Go before.

We wanted to learn Go, so we decided that the best way to go and learn language is to write a semi-big project. At the time, obviously, we didn't realize how big this might be. But it was meaty enough where we had to go ahead and start writing these bits in anger, and that's how we learned, and cut off many limbs in the process. And for everybody who still struggles with the directory structures and things like that, mea culpa. We had many mistakes.

CRAIG BOX: So how do you feel about people saying, well, we have to write projects in Go the same way that Kubernetes does, because it's the Canonical example of what to do when it was written by these people who were simply learning the language at the time?

VILLE AIKAS: It makes me chuckle. If people think it is the right thing to do because it makes sense, then great. But as far as the whole like, well, you should do it like this because Kubernetes does it, yeah, that's not right.

CRAIG BOX: As more and more people came on board, and then as the project got opened up to external customers, and then eventually open sourced, what was your involvement in growing the community outside of the initial team who were working on it?

VILLE AIKAS: I guess I would call it, at the time, evangelism. There was actually — kind of going back to defrag things and everything else — very shortly after Kubernetes was open sourced, I was asked to go fix a couple of teams internally at Google, and it was my manager at the time. And I was like I don't want to.

CRAIG BOX: I've got this cool thing going on here.

VILLE AIKAS: Yeah, I got this great thing going on. This is so much fun. I'm having so much fun writing code here, and doing things, and yada, yada, yada. He's like, no, I really want you to do this thing. Do it for me.

And I was like, I really don't want to, but I said fine, I'll do it for you. So I had to kind of start splitting my time going in and fixing some of these GCP products. So I had to go fix those teams up. So my involvement started kind of dropping down a bit for a while there, unfortunately.

CRAIG BOX: We now have an open source sensation, if you will allow me, in Kubernetes. And Google starts looking at what they can do to build the next pieces on top when it comes to service management, and when it comes to developer experience, and so on. What was the next project that you took on after that?

VILLE AIKAS: I'm going to go ahead and answer that in two bits just because what I also want to go ahead and say is that I think one of the things that we did with Kubernetes that is super amazing was, basically, we built — I like to think of it as cloud native assembly language, which is it's necessary, but not sufficient.

So just like writing in assembly in 6502 is fun. Most people do not write that. Kubernetes is just too damn hard for most folks, and it's unnecessarily hard to do some things that you just want to do as a developer, like launch my stuff and so forth.

Some of those things were really kind of troubling me, and I wanted to make it more approachable for the common developer. The next dot com person who wants to come in and live the dream.

I guess now it's .dev, everything is dot dev. Back then, it was dot com. So Chen started this Knative project that became known as Knative. And I kind of think about, the only reason why I brought up the cloud native assembly is because I kind of started to think about Kubernetes as that would be like libc, or standard lib, or something like that on top of it that makes it much easier for you to go and achieve many of the common things. We also wanted to make sure that you can always drop down to the next layer so that it looks and feels the same, because there's a lot of things that the API surface got right.

CRAIG BOX: When you were thinking about this particular problem, were you thinking from a developer and saying, what do I need to bridge downwards to get to Kubernetes? Or were you thinking about Kubernetes and thinking, what do I need to build up on top of it?

VILLE AIKAS: It was actually both. The first and foremost goal was how do we make this more approachable for developers, so from top-down. If you come into Kubernetes, and you're like, hey, I just want to go and run this — let's call it service — the cognitive load, the things you have to understand to even get it up and running, is many. It's not just one, like here's my code, run it.

With the Knative, the idea was, how do we go ahead and reduce that complexity so when you come in, and you're like, well, here's a piece of software, now run it for me. If you kind of think about the App Engine days, the App Engine model, it's like here's my code, run it. Don't screw it up. You have one job.

That model is quite nice, theoretically. But then the problem with the App Engine that I always felt was that once you get out of the sandbox, the answer is, cool, step one, rewrite everything, and port it to VMs or something else.

So we wanted to avoid that from the Knative. And this is the building it up, which is, we wanted to go and use best practices and abstractions that would look that if you are a Kubernetes, OG, hardcore person, and you start looking at it from the bottom-up, and you're like what did those Knative people do? They don't know what they're doing.

They're like, ah, OK well, you know I probably would have done it like this, and this, and this. Because we are using those same constructs instead of going like, oh, we create our own things and everything else. Because one of the things we also felt very early on, that I think there's many things that these things might be upstreamed in the Kubernetes at some point in some form or whatever. So that was kind of the next level on top of it.

CRAIG BOX: I did want to ask about that. When you're building out Kubernetes especially, where was the decision to stop at the level that it was stopped, and not become full service, not include the service mesh functionality, not include the build and serving functionality that eventually went into Knative?

VILLE AIKAS: There were a bunch of PaaS's out there already at the time and everything else. One of the things we also wanted to make sure at the time was we did not want to just go and be a product or a service, because we wanted to go and make sure that you could pick and choose.

So at the time, for example — I think you said service mesh — we know we need something like that, but we want to make sure that it's pluggable. Because we also understood that for adoption, it's very difficult to go to some companies and start mandating, for example, I'm just going to pick an example from the event team, to go ahead and say, you must use Kafka if it's a rabbit shop.

And at that point, you basically come in. So that whole philosophy has been baked into the Knative assumptions, which is roles and responsibilities. So each one of these stacks, if you will — so serving, eventing, and build — has one job. They should be able to be used independently.

And even within each one of those stacks, there are some choices that you can make that, our bet was, that that's going to allow for bigger adoption, because there's plenty of ingresses right now, or service meshes, or so forth that you didn't want to be in the business of choosing one, because people want to use what they're familiar with and what they have in-house expertise. Who knows what comes later.

CRAIG BOX: You mentioned there, the three primary components or stacks that were in Knative when it was started — Serving, Eventing, and Build — were those the three main problems that you saw that the developers had?

VILLE AIKAS: The only one that was there from the beginning was really the serving. So the serving started as the first one, because it was sort of tackling the idea that I first started with, just to run my code and provide an endpoint to it.

So a very common pattern with the microservices land and so forth. So that was what we kind of started. And as we were playing around with things, you always learn stuff. And then we're like, yeah, that's great, but we also need a build system.

We need a story, we need full, ‘how do you go from a source to a container?’. And we also felt that there, we needed to go ahead and at least provide a solution in Knative, because we felt that, again, for the adoption, if you don't have one, you can use what we had.

But if you already have, because many people have their CI/CD pipelines and everything else. So again, we didn't want for you to go ahead and have to use ours. But we felt like it'd be important for people that don't necessarily have some.

So it's just like, oh, I heard of this Knative. It sounds neat. How do I do it? And it's like, again, step one, go figure out how to build these containers and everything else. It felt like it wasn't good. And as we started of identifying it, then eventually Build was deemed to be a meaty enough problem in a sense that it was spun off into Tekton, and that was a good bet.

CRAIG BOX: Most people who will have come across Knative will have thought about it in the sense of it being a serverless system. The idea of something like Lambda or Google Cloud functions, which takes code and puts out running infrastructure. Is that a fair description of Knative? Or is there more to it than that?

VILLE AIKAS: That is definitely a good mental model for Knative serving stack, which is, you basically give it a container, and Knative serving stack will run it for you. It's got one job — run it, scale it, when nobody's using it, bring it down to zero.

CRAIG BOX: That sounds like three jobs.

[LAUGHS]

VILLE AIKAS: It makes sure that it runs. The one thing that Knative spent a lot of time working early on the API model was to make sure that it wasn't just the easy to get started system.

So it will actually go ahead and be a production system for you. So it will grow with you from day 0 to hopefully forever, because many systems are easy to get started. Look, in five minutes, I can get a hello world, and then you come back to it later, and you're like, I want to upgrade, how do I upgrade? It's just like, oh, well, yeah, you can't. So a lot of those things are baked into Knative serving that allow you to safely go ahead and manage the life cycle of your code.

CRAIG BOX: Eventing isn't just about horses. I understand that the Knative stack, obviously, can take HTTP requests and serve up the common serverless use case, if you will. But what other things might you want to connect up to your containers?

VILLE AIKAS: Very early on, it became obvious that this is a very natural fit for being able to go and react to events. So you don't just build microservices or web services for the users, if you will, or other machines, but you want to react to events.

So for example, a very common use case is to go ahead and drop things into, say, cloud storage, and to be able to go and then do something with them. It starts a workflow. GCS can go ahead and fire off an event, and say, new things showed up.

In those cases, the eventing framework allows you to go ahead and go ahead and configure it in such a way that you can react to those events. But the one thing about that eventing that's nice for you is that when you use things like other systems that are imperative or whatnot, you basically end up with, sometimes, a mess, because you start adding new little things.

And then you have a system that behaves this way, but you don't know really how it should be behaving. Whereas with the eventing system, you can go ahead and declare your intent, and you can understand it. I sometimes call that the emergent behavior, which is like, yeah, well, that's what it does. It emerged because Bob, and Linda, and everybody is adding their functions and things like that, but nobody knows what really is happening. So with the eventing, the idea was that it's declarative, and you can go ahead and reason about the system as in how it should be behaving.

CRAIG BOX: Ahmet Alp Balkan has said that he believes that the serving abstraction in Knative should be everyone's first go-to in terms of how to run applications on top of Kubernetes. In large part, people don't think of it that way because it is part of this PaaS, and the same was perhaps true of the build system before it was split out.

Do you think that the initial positioning of this as being tools required to run a PaaS was, in some way, detrimental to further adoption of those things? That if they were perhaps built into Kubernetes, more people would have had access to them?

VILLE AIKAS: I can see that argument. I don't know is the answer. Like many things, I don't know. If you ask me, I don't know. I have opinions.

CRAIG BOX: You have hindsight?

VILLE AIKAS: I think there's a lot of things that people run on Kubernetes that they install after the fact. Very rarely do people just run bare Kubernetes clusters. Pick cert-manager, for example. Many people install that as the very first thing that they do.

Right, OK, cool. I just want to deal with certs. I kind of viewed that as for the Knative as well, where people will be doing that. Now again, going back to the earlier thing that I said, that we had been hoping that many of these things could be upstreamed or brought into the CNCF ecosystem and so forth, because there's a lot of tools there that people install.

So I'm not sure that, ideally, if these would have been first-level Kubernetes abstractions, then maybe people would have used it more. But I do agree with Ahmed, that, for most cases, the Knative abstractions from serving is what people should be using.

CRAIG BOX: You've done a lot of work on something called duck typing. I'm familiar with the idea of explaining something to a rubber duck. Can you treat me as if I'm a rubber duck here, and explain to me that work?

VILLE AIKAS: That was another would-it-be-bad-if question. The particular problem we were trying to solve at the time was we wanted to go ahead and keep ourselves honest. So when we say like serving and eventing can be used independently, part of the reason they are split in the different repos is that there is no backchannels.

One of the things that we wanted to do in the eventing — at the time, I was spending a lot of time in the eventing land — and what I wanted to do was to be able to go ahead and say, there might be a Kubernetes service that I want to send events to, or it might be Knative service, or something else, whatever it is.

And what we wanted to do there was, option one is that you hardcode all those things in there into the eventing land. That seemed goofy. So I went to Matt [Moore], and I said, hey, Matt, would it be bad if we started basically treating all of these resources as, we just care about the little bit of YAML? I know where the location is, I don't care what that resource looks like. So as long as it looks like this particular piece of data, then quacks like a duck, walks like a duck, then it must be a duck.

That's where the duck typing work originally came from. That was the need for it. It really was a terrible idea at the time as far as, because we were just kind of doing things.

But it turned out that it wasn't a terrible idea, because it gets a lot of mileage out of it. And actually, as a matter of fact, we are trying to go in and upstream that into Kubernetes right now.

CRAIG BOX: So the idea broadly is you just interrogate the object that's been given to you, and see if it has particular fields, and you can tell what it is?

VILLE AIKAS: Yeah, exactly. The funny bit was that, at the time, there was — and even today, there is — some, they call them subresources in Kubernetes. So for example, you have slash status, and you have slash scale. So you can have some resources that have a scale in them.

So what we wanted to do was, for the CRDs, we wanted to be able to go in and create custom subresources. And we spent some time chatting with the Kubernetes folks at the time, and they were like, yeah, that's not going to happen right now, at least.

If we would have had the ability to go ahead and create like slash address, or slash what am I, or something like that, then we probably would have used that. But it turns out we don't need to because we can look at it. Now to your question about the querying things, where things get a little bit tricky is that you don't know that until you actually get the resource back. And now you can go and do basically the traversing of, whatever, the JSON object. And kind of go like, well, do you have this field here, yes or no?

CRAIG BOX: Isn't there a nice API version field you can look at which just tells you what it is?

VILLE AIKAS: You can get the API version, but that won't tell you if it conforms to a particular duck type. That's some work that we have done also in the Knative land, is being able to discover what are some of the, it's actually called cleverly discovery in the Knative repo, where we can go ahead and do and discover duck types, and what do they conform to. Because you can use, for example, one object might implement multiple duck types.

There's some work that we are doing there. We'll see. The hacky thing is you basically do things like labels, or annotations, and you say like, I'm a duck, XYZ. And then you have multiple of those, so then you can ask right up front, or you can filter based on those, and say just give me these duck types.

CRAIG BOX: Is there ever a trust problem where you'd have someone claiming that they're a certain type of duck, and that may not be true?

VILLE AIKAS: Yeah, although, the trust problem in the sense that it's just going to fail. You basically get back this JSON. It's like, oh, don't have that. Sorry, better luck next time.

And if you then go right then to do the inverse, where it's just like you're writing into a duck, because at that point, it's just a patch. So if you do a patch, and you try to go ahead and put a field that doesn't exist or anything else, then the nice thing is that it's the resource implementer's responsibility to make sure that the object misbehaves. Just like I can't just kubectl apply yolo, whatever.

And so in that sense, there's not really a trust problem in the sense anything bad is going to happen. Or I'm sure there's a way to do that, but I can't think of anything. It doesn't seem any worse than being able to kubectl it.

CRAIG BOX: With the experience that the team who built Knative had, it, like many open source projects from Google, came out of the gate relatively complete in that it didn't necessarily have everything it would eventually have, but it had the main components — the Serving, Eventing, and Build, and so on — and it worked, and it could be used as-is. What has been added, or built, or brought on by the community between that point and today? What are some of the major changes that Knative has been through?

VILLE AIKAS: Well, the obvious answer is always just make it better and faster. Especially with the serverless side of the house, it's very important to have low cold start. Meaning if I scale down to zero, and then a request comes in, I want to spin that thing up fast.

That always is a little tricky of a problem. And our hopes and dreams, especially in the early on, was, this is something that we as a community should be solving so that not every vendor or every project has to go out and figure it out on their own. That we could go ahead and solve it as a community.

So a lot of work has gone into that, solving the cold start latency problem. That is great. The API has gotten super robust. We have things like multi-container support.

I think there's really neat magic stuff that happens with the DNS stuff, that, you just kind of create it, and it gets you a TLS terminated cluster. That is just fantastic. On the eventing side of the house, there's multiple backing technologies for eventing persistence.

So a lot of work has gone into like Kafkas, and rabbits, and whatever Google has done with the pub/subs and so forth. I feel like for the last many, many months now, a lot of the focus has been stabilizing the system — when I say the system, I mean the project — so that it's really ready for 1.0. There is nothing that prevents people calling a project 1.0. I can go ahead and check in a README and call it a 1.0 if I want to.

CRAIG BOX: More people should do that.

VILLE AIKAS: [LAUGHS] Yeah, so we have been holding a fairly high bar for the 1.0. And also — I'm going to say wonky, but it's not wonky — but the one difference between Knative and, say, something like Kubernetes is that Knative is in one way a reference implementation so that other people like Cloud Run can go ahead and build their version of it.

So being able to go ahead and do conformance tests has been something that the community has spent a lot time over the last seven months doing. Ensuring that we feel confident with, like, Red Hat provides their version of Knative, that people then can have a high degree of confidence that it actually is Knative-compliant. So a lot of work has gone into that.

CRAIG BOX: I think a big difference between Kubernetes compliance and Knative as you describe it, most people are compliant with the Kubernetes spec simply by running Kubernetes. They download the binaries, and they may embellish them, or add a couple of things — in Red Hat's case, to make OpenShift — but on the whole, it is Kubernetes. And so it's easy to say it passes all the tests. Are people who are running against the Knative spec actually writing their own implementation in many cases?

VILLE AIKAS: In some cases, probably more on the eventing side because there's pluggable bits, like the brokers, and the triggers, and whatnots. And because they have different backing technologies, yes, they are writing those.

In the serving side, we have at least two. We have Cloud Run, which is Google. Then I think [IBM] Code Engine, for example, uses it. My understanding, again, is that they are at least running most of the Knative bits as they are, but I don't work there. That's my understanding. I could be wrong.

CRAIG BOX: Another part of the project that you've been involved with is the setting up of the community and the governance structure. You've had the ability to look at other projects like Kubernetes, and see what works and what hasn't. And you've served in both the technical and the steering roles. Tell me a little bit about how that got set up, and how you've personally been on that journey.

VILLE AIKAS: Very early on, we wanted to make sure — I think DeWitt Clinton used the term a big tent. We wanted a big tent.

CRAIG BOX: It's a Seattle Google guy, it's got to be BigSomething.

VILLE AIKAS: Yes, yes, yes. We wanted to set up a big tent for our community members to go, and we wanted a lot of vendor participation because we did not want to do it alone. Again, we wanted to solve these as a community, because all these problems are something that should be solved once. And eventually, again, to solve them upstream if we can.

While we were setting all these things up, we wanted to, again, look at what are some successful projects, and what have they done right, and so forth. And things like TOC came out of that, and steering, and so forth.

We looked at things like Istio. What had Istio done? Because, steal it. You know what I mean? It's just like, great ideas, you should steal them.

CRAIG BOX: Of course.

VILLE AIKAS: Yes. So we set those up.

CRAIG BOX: I stole that idea from Steve Jobs.

VILLE AIKAS: [LAUGHS] Yeah. So all we wanted to do was basically set those up. And at the time, it was just because of our involvement in the project early on, the TOC was basically me, Matt, and Evan [Anderson]. And the steering was some folks from Google side who had also been involved in governing the project a little bit.

It was a bit ad hoc at the time and so forth. But the technical side was a bit easier, because there were really meaty technical problems that we were all trying to solve. And so we all wanted to talk about them and so forth.

So we just wanted to have a good forum where we could have these conversations, and keep track of things, and so forth. So we just kind of got bootstrapped by that. And then as the community grew, grew, grew, grew, there was a lot of folks that were like, well, how do people move into those positions?

Ville, we've seen you for far too long, get out of here. We worked very hard as a community to go ahead and establish elections. We started with the TOC. By this point, me, Matt, and Evan, and Scottie [Scott Nichols] had all gone to VMware, and we were working over there.

And one of the things we wanted to do is that the TOC should not have a single major majority in the TOC. Well, me, Matt, and Evan were at VMware, so one of us had to step down. So I was just like, enjoy.

CRAIG BOX: Kind of drawing the short straw, but kind of saying, yeah, I've done my time.

VILLE AIKAS: Yeah, I don't view it as drawing a short straw. I knew it was left in great hands with them. I had been there long enough, and we got great new folks in the TOC, so everything was fantastic.

CRAIG BOX: You mentioned there moving to VMware with a number of members, Matt Moore, Evan Anderson. That's something I've noticed about the tech scene in Seattle given that there is big representation from three of the major cloud vendors up there. A lot of people move around, and a lot of people do it in groups. They kind of like the people they work with, and they go work on something with the same people somewhere else.

VILLE AIKAS: Yeah, the whole thing about starting Knative is Matt and I had done some work in the past together. I was in cloud storage. I think Matt was doing container registry, just as one example. So our paths had crossed before.

So when I started Knative, or building the bits for it, I was like, hey, man, come take a look. That definitely — to me, personally — is very, very, very important is the people you work with is really important. And especially if you're starting something new, obviously, it's great to have folks that you know with, and you understand how they work, so that you can kind of, I guess, short-circuit the process of building that whole—

CRAIG BOX: Trust.

VILLE AIKAS: —trust and how people work. And it takes time.

CRAIG BOX: You, and Matt, and a number of other people that you have worked with in the past have now done exactly that, and founded a new company. Tell me about Chainguard.

VILLE AIKAS: Chainguard. One of the things that has been fairly recent, but it's something that several of us have been thinking about for a long while is how do people know what is running, where is it running, what is it doing, and so forth. Especially as you mentioned earlier on, and we talked about it, how popular open source is, and how most of the software companies today are relying so much on open source, but there's very little understanding of where that software came from, and how it's being used.

CRAIG BOX: It all came from Finland.

[LAUGHS]

VILLE AIKAS: Well, all the more reason to worry. So Matt, Scott, and I obviously have worked together for a long time. Dan [Lorenc] and Kim [Lewandowski]—

CRAIG BOX: Previous podcast guests.

VILLE AIKAS: —also have crossed paths in the Google days and so forth. And there's a mesh, like the five of us knew each other quite well. And we were very excited about tackling this problem, because it's a huge problem, and we feel like we should have solved this a long time ago.

So luckily, folks have started building those bits already in the open source. And we want to go and participate in those communities, and make the world a better place. By doing this company, we are hoping to be able to go ahead and solve real customer problems in the software supply chain security.

CRAIG BOX: You mentioned before looking at what a developer needed to do on top of the platform that was Kubernetes. If you were looking today to build something like Knative, would supply chain security be an integral part of it?

VILLE AIKAS: I think it probably should be. The tricky bit is that there are so many moving pieces there. Again, I think this is something where the way that some of the design choices in the Knative were made, is going to, for example, make it easier, because it has to plugabilities in there, and it has the ability to go ahead and being able to extend it. So I think it fits in quite naturally there.

And of course, Tekton has a lot of the features built in that allow folks to go ahead and start securing their software supply chains, whether it's chains and so forth. So the pieces are starting to get there.

CRAIG BOX: So do you think that the tools like Salsa and Sigstore, and so on, and the things that you build in your new venture, will they be possible to integrate because of that work that was done in the past to make all these things pluggable?

VILLE AIKAS: I sure hope so. And if not, then I'll blame Matt.

CRAIG BOX: Fair enough.

[LAUGHS]

He's not here to defend himself, but—

VILLE AIKAS: Exactly.

CRAIG BOX: —it sounds like you two have a history.

VILLE AIKAS: [LAUGHS] Yep.

CRAIG BOX: Now if people wanted to learn more about Knative and the new features in 1.0, I understand you are going to be presenting at a meetup in a couple of weeks time.

VILLE AIKAS: Yeah. So we are going to have a community event celebrating the 1.0. We'll talk about that, so if you're interested, please come.

The other thing that we just talked about actually in the steering committee was we should have a party to celebrate the 1.0, and especially for the folks that have been with us on this very long journey to get to 1.0. So we are going to have a couple of parties, because what's better than one party is two.

So we'll have one in the EU friendly timezone, and one with North America. Stay tuned for that, as well, because we want to go ahead and see those people that have been in the project, and maybe left, done something else, and so forth. So it would be great to go and have you all join us for that.

CRAIG BOX: I'm fairly certain that I gave the world's first meetup talk on Knative, the week after it was announced. I went to Boulder, Colorado, and did one of the demos that was done at Next. Does that make me the fifth Beatle of Knative?

VILLE AIKAS: Absolutely.

[LAUGHS]

CRAIG BOX: I guess I should turn up to the party.

[LAUGHS]

VILLE AIKAS: You should.

CRAIG BOX: Knative, why doesn't it have a Greek name?

VILLE AIKAS: Because we felt like it was getting a bit silly to go and have everything named Greek.

CRAIG BOX: Couldn't you have had this thought maybe six years earlier?

[LAUGHS]

VILLE AIKAS: Yeah, although I think it was a stroke of marketing genius to go and name it Kubernetes instead of, say, Project 7, because you can imagine how much fun it'd be to look for search engine keywords for "seven."

CRAIG BOX: I always thought that Knative should be pronounced KUH-NATIVE like Donald KUH-NUTH.

[LAUGHS]

VILLE AIKAS: Yeah. No, there was a lot of discussions about how should it be pronounced, and KUH-NATIVE was definitely there. And it was like, nah, that just no, no, no, no, no. But whatever. And there's always Mark who calls it CANT-IVE.

[CHUCKLES]

CRAIG BOX: Finally, last week, I understand that you managed to get out and see a little bit of sport?

VILLE AIKAS: Yes. Finally, after many, many, many years of waiting, Seattle has a hockey team, a proper NHL team. So last week, we basically had Kraken's first ever home game.

It was amazing. 17,000 people showed up. Atmosphere amazing. We basically had four games last week in Seattle, so I was able to go and get my long-awaited hockey fix.

CRAIG BOX: Is hockey something that is instilled in you as a Finnish youth, or is it something you picked up in the North of North America?

VILLE AIKAS: That's in Finland. So in PE classes, we would play hockey. Not with the proper gear, but with the tennis ball and whatnot, but with the sticks.

CRAIG BOX: Are there a lot of Finnish players making it to the NHL?

VILLE AIKAS: Yeah, there's quite a few there.

CRAIG BOX: I'm sure there's dozens of them.

[LAUGHS]

VILLE AIKAS: Dozens of them! Yes. No, there's quite a few of them. Even Kraken has one Finn in there. The token Finn.

CRAIG BOX: Excellent. Well, thank you very much for joining us, Ville.

VILLE AIKAS: Thank you so much for having me. It's always great to chat with you.

CRAIG BOX: You can find Ville on Twitter at @aikasville, you can find Knative at knative.dev, and you can find Chainguard at chainguard.dev.

[MUSIC PLAYING]

CRAIG BOX: Jimmy, thank you so much for your help with the show not just today, but for the last 58 episodes. It's been an absolute pleasure.

JIMMY MOORE: Thanks, Craig. I've had a great time. And if you've enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at @kubernetespod, or reach us by email at kubernetespodcast@google.com.

CRAIG BOX: You can also check out our website at kubernetespodcast.com where you'll find transcripts and show notes, as well as links to subscribe. We'll be back with another special in a few weeks, so until then, thanks for listening.

JIMMY MOORE: Bye now.

[MUSIC PLAYING]