CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google. I'm Craig Box with my very special guest host, Jimmy Moore.

[MUSIC PLAYING]

As a rule, we don't try and talk too much about current events, especially the unpleasant ones, but Jimmy was actually present in Manhattan 20 years ago this week, when 9/11 happened. And I asked him to tell a little bit of his story of that day.

JIMMY MOORE: Yeah, hey. Thanks, Craig. First of all, I can't believe it's been 20 years, as I'm sure everyone says, because it happened when I was 21 and I'm only 31 now. So I'm not sure how the math is going.

CRAIG BOX: That is pretty magic math there.

JIMMY MOORE: Yeah. And of course, I don't mean to glamorize it at all. But I feel like it's a story where people have a lot of questions. And they're curious about what actually happened beyond what we see on TV, in the same images year after year, or all the films that have been made. So I figured I'd share a little bit about what happened to me that day. And obviously I was really fortunate.

I lived in New York. I was an aspiring actor, just out of college. And I lived in Queens. I was in lower Manhattan with my two roommates. And we all had the same job. We were temping at a magazine, doing surveys for a magazine.

We came out of the subway that day, and we actually heard the first plane hit. And came around the corner and saw this building with a giant hole in it, thinking, wow, I can't believe a plane just accidentally hit that building. And I said out loud, how are they going to fix that hole? And I thought about how disorganized all that paperwork was going to be.

And how do you even fix a moment like that? No idea what was happening. Just these strange — wow, that's going to be a hard fix. So we go on our way to work. We don't want to be late. We don't want to lose our jobs, so we go on our way to work.

At this magazine, we can see the two towers from the window. And we're all looking out this window at this one tower on fire. And all of a sudden, the second tower explodes out towards us. Not knowing it was a plane, but we saw it explode out. Debris hit our windows.

I looked at my roommates and I said, well, we're going to leave. And the woman who was supervising us — mind us, we were temps — so I think it was an intern that was supervising us. And she's like, well, I don't know if we're closing yet. And I was like, OK, well, I quit. So we're going to go.

So I quit our job on the spot. I brought my two roommates to the ATM. And I said, we need to get some cash, because the financial center of the universe is falling apart. And as I was getting my cash out, after the two of them started to rumble. And the connection was lost.

So we started running uptown, criss-crossing the city, avoiding the Empire State Building and trying to avoid the United Nations, hearing all sorts of news. My one roommate Michele was crying because her feet were hurting because we were running and she was in heels. And my other roommate was panicking that the gas lines were going to explode. So I stopped and bought her some slippers at Duane Reade, bought him some chocolate, and I decided I would start smoking again that day.

CRAIG BOX: A somewhat sensible decision, given the circumstances.

JIMMY MOORE: Given the moment, right?

CRAIG BOX: Hopefully a temporary one.

JIMMY MOORE: Yeah, definitely a temporary one. I'm happy to have that silver lining. We got up to the Queensboro Bridge after several hours, which is about halfway up Manhattan. And by this point, emergency vehicles were coming into town and people were walking out of town on the lower level. So we — pretty much in silence, with thousands of people — were walking across this bridge.

And I've never been so happy to be in Queens, never been so happy to get to a McDonald's and just eat and watch what was happening on the news. It took us a few more hours to get back to our apartment in Queens that day. And I realized I was a resident assistant in college. And I think that training really helped me a lot. They prepared us for emergencies — probably never for one of that scale.

My final takeaway that I usually share with people is that what they don't show in action movies is the action star falling apart in the bathtub at the end of a big day, because that sure would be a lot more realistic. And it was for me.

CRAIG BOX: Well, thank you very much for sharing that story. I don't really want to end on that note, so there's a link in the show notes to something from my hometown back in New Zealand, where a gentleman was trying to record a voicemail message to say that the movie theater was closed because the country is in lockdown. And he didn't quite have the same editors that we have on the show, so he had a couple of attempts at it. And unfortunately, the entire message, warts and all, went out on the voicemail. It's quite something.

JIMMY MOORE: It's incredible. I listened to this and I laughed. I shared it with my roommates. I sent it to some friends. It is fantastic. It's definitely great for a laugh.

CRAIG BOX: And not just for the funny accent.

JIMMY MOORE: Oh, no. Not just for the accent. This man was having a hard time. And he just let the whole world see it.

CRAIG BOX: All right, well let's get to the news.

[MUSIC PLAYING]

CRAIG BOX: Last year, at AWS Reinvent, Amazon pre-announced an EKS Anywhere service for running Kubernetes clusters on premises. That service is GA this week, with support for running clusters on VMware. Bare metal is now promised for 2022.

Additionally, an EKS connector has been launched in preview, allowing you to connect any Kubernetes cluster to the EKS console. Congratulations to our friends at the bookshop. and we trust that the Anthos team is going to send you a cake.

JIMMY MOORE: The Container Network Interface spec has reached 1.0. There are only a couple of minor changes to the spec, though the authors say that there are "way more tests." Instead, 1.0 — or actually, 1.0.1 — is all about the declaration of API stability. Congratulations to the maintainers.

CRAIG BOX: Red Kubes is a Dutch startup, building Otomi, a distribution which includes add-ons to Kubernetes in the manner a Linux distribution is bundled with applications that run on the kernel. This week, Otomi's developer self-services features were moved from their for-pay Enterprise Edition into their open-source Community Edition. Red Kubes also introduced a Helm chart for easier deployment of Otomi.

JIMMY MOORE: Two new features have been added to Microsoft's AKS this week. Scale-down mode lets you stop instances upon scaling down instead of deleting them. Disks and their cached images remain available for you when you scale up again. Custom Azure policy lets you create and assign custom policy definitions and constraint templates to your AKS clusters.

CRAIG BOX: In the world of Helm and Operators, it may not be clear to you when you should use one over the other. The K8ssandra community has walked this path recently, with Jeff Carpenter and John Sander from DataStax writing up their findings. K8ssandra version 1.x was installed with a combination of Helm and Operators. Future improvements — for example, adding multi-cluster support — made a Helm install too complicated and the charts too hard to debug. They are now working on a 2.x series built on the Operator SDK. But that Operator will still be installed by — you guessed it — Helm.

JIMMY MOORE: Episode 113 guest, David Ashpole, talked about instrumentation of containers and Kubernetes, and teased the arrival of distributed tracing in the API server. It arrived in alpha in Kubernetes 1.22, and David has written a blog post introducing the feature. By adding OpenTelemetry support to the API server and utilizing etcd's tracing support, you can get a nice graph of what's happening when a request is processed.

CRAIG BOX: Finally, it's not quite a book by Walter Isaacson, but Scott Carey from InfoWorld has written the somewhat inside story of what happened at Docker. Carey tells the story of Docker's early days as dotCloud — see episode 156 — up until its split in two and the acquisition of its enterprise business by Mirantis, which you heard about in episode 110. There's some opinion, but there's also some valuable insight from those that were there, which mostly boils down to, they missed Kubernetes.

JIMMY MOORE: And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Alejandro de Brito Fontes is a software developer from Argentina living in Santiago, Chile. He's currently an engineer at Gitpod. Ricardo Katz is a CNCF Ambassador living in Sao Paulo, in Brazil. He's currently an engineer at VMware. Welcome to the show, Alejandro.

ALEJANDRO DE BRITO FONTES: Thank you, Craig. Glad to be here.

CRAIG BOX: And welcome to the show, Ricardo.

RICARDO KATZ: Thank you, Craig. I'm glad to be here.

CRAIG BOX: I think all three of us started working professionally with Linux in the early 2000s. Do you think everyone from that era is working with Kubernetes now?

ALEJANDRO DE BRITO FONTES: I think that we were preparing for now. We were using Linux for 20 years just for Kubernetes. It's a lot easier to expose features to end users now.

RICARDO KATZ: Yeah, I agree with that. You are not the old sysadmin anymore. You can just say, hey everything just spins around Kubernetes, right?

CRAIG BOX: Well, I'm glad that you say that, because I used to identify as a sysadmin. We hadn't invented DevOps yet. So I didn't feel bad about not doing it.

RICARDO KATZ: Yeah.

ALEJANDRO DE BRITO FONTES: Same here. I started as a sysadmin and then I switched to software developer.

CRAIG BOX: You both grew up in South America — Alejandro in Argentina, Ricardo in Brazil. What was computing like for you growing up?

ALEJANDRO DE BRITO FONTES: It started with an XT model. We didn't have Macs or anything like that. And it was really hard, because I still remember the modems and the connection to the internet in the early days. So yeah, it was fun for sure.

RICARDO KATZ: Yeah.

ALEJANDRO DE BRITO FONTES: Downloading updates, it was really painful at the time.

RICARDO KATZ: My mother is actually a software developer as well. I had a computer at my home since I was a child. It was a Dell 486. But I wasn't supposed to spend too much time on the computer, because it was the work, too, of my mother.

We've got, as Alejandro said, those slow connections. By that time, I didn't actually have a connection. So I used computers mostly to play pinball. The pinball in Windows '95, I guess.

CRAIG BOX: Windows '95.

RICARDO KATZ: Yeah. The pinball one was really cool — or '98, actually.

CRAIG BOX: Many hundreds of hours must have been lost to that pinball game.

RICARDO KATZ: Yeah. And playing the flight simulator in Microsoft Excel — the Easter egg one. Do you remember that one?

CRAIG BOX: Mm-hm.

RICARDO KATZ: It was mostly about that. We didn't get computers in school, for example. So it wasn't something that everybody got access to early in South America. To be honest, I think that this is changing just right now. I got the privilege of having contact with computers at an early age.

CRAIG BOX: When you both started out in computing, was all the information available to you in English? Or was any of it translated and available in Spanish or Portuguese?

ALEJANDRO DE BRITO FONTES: I learned English thanks to this, because my first book was the Slackware Bible. The first release, a long time ago.

RICARDO KATZ: Same to me. We’ve got, in the beginning of Linux, there was a cool guide in Portuguese called Foca Linux, which was a group that translated a lot of things of Linux from English to Portuguese. So we've got a bunch of materials in Portuguese and we have a strong open source community here. We've got a lot of folks translating but the majority of materials, they were in English as well.

CRAIG BOX: What were your first contributions to open source?

RICARDO KATZ: My real first contribution to open source was a documentation on some plugins from Elasticsearch. My development contribution, like doing some code, was for Harbor, the Docker registry from the company that was donated to CNCF.

ALEJANDRO DE BRITO FONTES: For me, it was fixed for Apache POI, a long time ago that was never merged. It was rejected, actually. So that was my first contact with open source.

CRAIG BOX: Not all contributions get accepted, unfortunately. But it doesn't seem like it put you off.

ALEJANDRO DE BRITO FONTES: You could say that.

CRAIG BOX: You're both long-time maintainers and — in Alejandro's case — the originator of the ingress-nginx controller for Kubernetes. Tell me a little bit about the history of the project. When we take our time machine back, we will remember that Ingress didn't exist in Kubernetes 1.0. And when it was introduced in 1.1, there was support for Google Cloud's load balancer, but really nothing else.

ALEJANDRO DE BRITO FONTES: We need to keep in mind that working on the project for the last six years — I started in April of 2015. And at the time, I was using something called CoreOS Fleet. It was almost perfect at the time. It was working also with the project called [Deis. It's an open source path.

At some point, I reached a limit with CoreOS Fleet. It was something like 300 services. That was the reason that I started to research Kubernetes. Like you just said, at the time, there was no Ingress yet. You only have the core services — Pods, Services, and replica sets.

CRAIG BOX: ReplicationControllers.

ALEJANDRO DE BRITO FONTES: Yes, exactly. But at the time, there was a project called a contrib in the Kubernetes organization, where you can find a sub-project called service-load- balancer that basically is just a primitive controller. If you compare it with what we have now with kubebuilder, that basically set up haproxy with templates. Some annotations on services allowed you to expose some services in a centralized way.

Of course, at the time, we had no client-go yet, so the bugs and all the stuff, it was interesting to find - in production, of course. But a couple of months after I started using it, I started to contribute in the project. And by November, some members of the Google network team started to grow it – what you would now call a KEP to something called Ingress. And the goal was to release it before 1.0. Like you just said, we have it there, but only with one implementation — the ingress-gce controller.

When I saw that, I started to think about the ingress-nginx. That was the origin of the project. The original idea was to use something called ThirdPartyResources. It's the father of the CRDs that we have now. That never was merged in the project. It was a really cool idea, but the API machinery was not there yet. And that's how I started the project. Eventually, someone asked me if I want to upstream the project, because this was in a personal repository.

CRAIG BOX: The design work, I remember, largely being done by Tim Hockin and Prashanth Balasubramanian, better known as "beeps".

ALEJANDRO DE BRITO FONTES: Exactly. He was the person that asked me about the contribution to the project.

CRAIG BOX: Just to give an explanation of how the system worked, you have an Ingress object, which you basically define what it is that you want in terms of routing rules. For example, to map external paths to service backends. And then you need some sort of controller that's running. If you have a load balancer available in your cloud, like the GCE implementation, it would take those rules and then program the load balancer.

For someone who is running in bare metal or an environment that didn't have that layer 7 load balancer, they would have to deploy one or more instances of some software that would handle that. You mentioned the service-load-balancer. And there was also an haproxy-router at the time. What was it in particular about NGINX that made that the software that got adopted as the de facto standard?

ALEJANDRO DE BRITO FONTES: I was more familiar with NGINX. That's the only reason. It wasn't a technical decision, actually. It's as simple as that. And also, I was doing a lot of microservices at the time. And I needed more of a mix between applications and microservices. And NGINX was easier than haproxy, for me, at least.

CRAIG BOX: Was this something that was commercially inspired? Was this a problem that you had at the company you were working at, at the time?

ALEJANDRO DE BRITO FONTES: Actually, no. I built this for bare metal, because I was running bare metal and I could not use the GCP cloud for political reasons. And that's why I started ingress-nginx, actually. I was the first user of this. I built this for me, actually. I never planned for ingress-nginx, as you know it right now.

CRAIG BOX: I'll put a link in the show notes to a picture of that bare metal cluster. It seems to be running in a suitcase!

ALEJANDRO DE BRITO FONTES: Yes. That was an interesting experiment, because I built a case with four Intel NUCs. A switch there. I even maintain this right now — in its third iteration, it's a cool thing to do if you want to learn Kubernetes without breaking anything.

CRAIG BOX: And so then, Ricardo, when did you get involved with the project?

RICARDO KATZ: My past job was at a government company here in Brazil, and we decided to use Kubernetes for all of the services. As Alejandro explained, we started with service-load-balancer, which was pretty cool, because it was simple. It was a really simple, nice approach of adding annotations into services and it routing stuff for you. But it didn't scale well.

At that time, Ingress didn't exist as well. So we couldn't use GCP as well, because Brazilian government company — by that time, they weren't supposed to use any public cloud. I don't know how this is nowadays. So Ingress was released. And I told my team, hey, look at this project. Let's try and see how it happens, because this service-load-balancer is going to be deprecated. And they have already marketed it as deprecated.

So we started to use the ingress-nginx by that time. There was a folk in my team, which is João. And we have a fun story about that later. But João started doing some research into the ingress-nginx. And I was doing some research with him.

And we decided to migrate from service-load-balancer to ingress-nginx by that time to have some scalability and some features that you didn't get in the service-load-balancer. And also because we were already on the bleeding edge, I guess, using Kubernetes on a government company. Because government, usually they say, oh, everything is so old. You have mainframes. You have all of those old machines. They are getting yellow inside the data center.

So Kubernetes was already the bleeding edge. And we said, hey, we are already on the bleeding edge. So let's just use this Ingress and see how it breaks stuff for us.

CRAIG BOX: Would you describe Brazilian government infrastructure as more XT or more 486?

RICARDO KATZ: I would say actually that we got a real modern architecture here. People usually think that government stuff, they are old, right? We actually have old processes. I can say we've got them already there.

But we've got a lot of good employees trying to make things work in a modern way. So integrating things that always run on a mainframe with something modern, like Kubernetes. And this got a lot of fun for me, running those tests in bare metal.

CRAIG BOX: How does the controller know how many instances of NGINX it needs to spin up to handle the traffic coming into the cluster?

ALEJANDRO DE BRITO FONTES: It doesn't. By default, there's no Horizontal Pod Autoscaler enabled, but it expects that you set the number of replicas that you want.

CRAIG BOX: I imagine when you started the project there was no Horizontal Pod Autoscaler.

ALEJANDRO DE BRITO FONTES: Exactly, yes.

CRAIG BOX: So it's just down to the sysadmin, if you will, to decide how many instances they need to create.

ALEJANDRO DE BRITO FONTES: Yes. Basically, at the time, it was easier, because it was just a DaemonSet or the precursor of that — just one instance per node.

RICARDO KATZ: A PetSet.

ALEJANDRO DE BRITO FONTES: Exactly. [LAUGHING]

CRAIG BOX: This podcast is a fun journey through deprecated memory lane.

You've created this project to scratch your own itch, to help run your suitcase of NUCs as it were. When did you notice that people started using it?

ALEJANDRO DE BRITO FONTES: That's easy, because when you start receiving feature requests to add new things and you start hearing about new use cases that you never thought about. To be clear, I never expected to expose all the features from NGINX. You can hear now that people complain about the number of annotations. And this is just a side effect of what we had at the time — only annotations. That's when I started to hear about the traction of the project.

CRAIG BOX: What features did you expect would be in the project?

ALEJANDRO DE BRITO FONTES: None. Just expose a service and the easy use cases.

CRAIG BOX: OK.

ALEJANDRO DE BRITO FONTES: Not doing things like SSL passthrough or using it as a cache, for instance.

CRAIG BOX: And how much did the things that were possible in NGINX influence how the Ingress capabilities in Kubernetes evolved?

ALEJANDRO DE BRITO FONTES: This was an experiment at the time, because we didn't know exactly what users expect from something like Ingress. Keep in mind that when Ingress was released, there was no RBAC at the time.

CRAIG BOX: No.

ALEJANDRO DE BRITO FONTES: So you can still hear people complaining "why do I need to define the secrets in the same namespace that the Ingress is in"— that means that I need to expose the secret to someone that maybe doesn't have the permissions. And that's just a side effect of how old Ingress is.

CRAIG BOX: Ricardo, do you remember what features worked for you and what things needed to be added when you came across this as an early adopter?

RICARDO KATZ: Yeah, I do. Because it was something like, we shouldn't be doing this in a Cloud Native world, but we are. The first feature that I am taking a look is actually about TLS mutual authentication feature by itself, because we've got a lot of services that needed mutual authentication and user's credentials. And Ingress didn't support that. So that was the first one.

But that one is good because security is good, right? The worst one — I know that it's still being used nowadays — is about session affinity. Because we've got a lot of Java with JSESSIONIDs, the old Java Server Faces stuff running inside Kubernetes.

And when Ingress balances to one or other pod, you break the user workflow. So those were the first features that we've implemented into that. The original one, they were just fine. It dealt with a lot of traffic by design without those features.

CRAIG BOX: Alejandro, you're maintaining this project in your evenings, and it's starting to become a very very important part of Kubernetes. How did you handle the growing number of feature requests and the dependency that the community had on this project?

ALEJANDRO DE BRITO FONTES: I didn't realize this until three years ago. It was really fun at the time. I never saw this as a job, but I understood the importance of this project for some people. Eventually, that was the reason why I put so much effort into maintaining it. I always tried to be responsive in the issues, releasing at a constant rate, like five or six weeks, at least.

CRAIG BOX: Were you able to bring on other maintainers?

ALEJANDRO DE BRITO FONTES: Yes. We need to understand in Kubernetes, it's really interesting to see that people can arrive to a project just to fix something that they need or they want to add. And they leave. It's hard to get maintainers for long periods of time. It was interesting, you can see in the start of the project that there's almost 700 contributors to the project.

And for instance, there's companies like Shopify that are using ingress-nginx in production. And they have one member of the engineering team doing ingress-nginx. So that's right.

CRAIG BOX: Were people at those companies, or maybe even at NGINX Inc. reaching out to you and offering to help?

ALEJANDRO DE BRITO FONTES: No. We tried to reach NGINX before the F5 acquisition to merge both projects– the one from NGINX, the company, and the one from the community. And they said that they were not interested in doing that at the time, because they wanted to pursue the commercial portion of NGINX, not the open source one.

CRAIG BOX: And so, Ricardo, you started contributing around 2017, when you were using it. And you had a team of people working there. How did you become a maintainer of the project?

RICARDO KATZ: Back by that time, I was one of those persons that came, did something. And I didn't actually leave, but I wasn't a reviewer or an issue triager the same way that we have in Kubernetes community. I kept in contact with Alejandro for some time. And sometimes he asked me, hey, I'm implementing this feature. So you as a user, what do you think about that? Do you use that? And I kept this sort of contact.

I wasn't quite a maintainer. But as soon as Alejandro decided to step down, I felt that I had some sort of depth with the project, because I've learned to code Go because of the project. So I got a lot of opportunities by that. And Antonio Ojea, he asked me and said, hey, I know that you contribute, so why not– no one is getting this? Because Ingress API v1 is going to be GA, and v1beta1 is going to be deprecated. And I said, yeah, maybe I can try to implement that thing. And I reached Alejandro, and he told me, OK, if you try to implement this thing, probably people are going to think that you are going to be the maintainer.

We got actually a CVE back in that time as well. It was a security problem. And I said, hey, I know that you are not with this anymore, but I think that we should fix this for the user. So can you teach me how to make the release? And he told me, as soon as you make one release, people are going to think that you are the maintainer. So be aware of that.

And I said, yeah, I think that I can live with that. But I won't spend my daytime or everything. So people need to be aware that I won't put the same effort that you've been putting, because I have not this time to do that.

CRAIG BOX: Let's step back a little bit from that. You gave a talk at a Rejekts conference once upon a time, talking about Ingress and implementations. What was the story with that talk?

RICARDO KATZ: Yeah, so that one was fun. I was at Rejekts in Barcelona, 2019. And I gave a lightning talk because my talk was rejected in KubeCon, so I decided to give that in Rejekts as a lightning talk. They said, hey, anybody that wants to say about something, just come here and say about something.

My talk was about scaling Kubernetes in a huge bare metal environment. So we've got machines with 768 gigabytes of memory in my previous job running something like 500 to 600 pods per node. It's not recommended. Please don't do that. But anyway, we've got something like that.

Then I decided to tell people what were the problems that we faced with network policies and iptables reconciliation. And one of the parts was, hey, we've got a lot of users, and they have access to Ingress. And they can create their own Ingress. And when they do something wrong, you get into a real problem.

Because Ingress became an infinite loop trying to reconcile objects and to reconcile stuff. And you get a lot of CPU and memory. So I started venting, not about ingress-nginx by itself, but saying, yeah, you might have a problem with your users trying to do that and having a lot of CPU usage. So be aware of this problem. I know that the ingress-nginx folks, they are working on dynamic reloads.

Alejandro told about Shopify folks, Elvin. Anyway, just be aware that this might be a problem. When the talk was over, I went into my place and someone came to me and said, hey, so you are Ricardo? And I said, yeah. So hello, I am Alejandro, the maintainer of ingress-nginx. Thanks for the feedback on the Rejects talk.

ALEJANDRO DE BRITO FONTES: [LAUGHING]

RICARDO KATZ: And I say, yeah, at least I didn't say that ingress-nginx is bad. But this infinite loop and the reconciliation is bad. And he told me, yeah. We need help. I need your feedback. I need your help. So just feel free to keep doing that stuff.

That was the way that we met, personally, each other. And after that, we went into a contributor summit, drank some beers. João Morales, who is actually the haproxy ingress maintainer, was from my team. So we've got a meeting from the Ingress implementatiol maintainers having some beers at that KubeCon.

CRAIG BOX: Is that how you remember the story too, Alejandro?

ALEJANDRO DE BRITO FONTES: Yes. It's funny how things happen. I never expected to meet Ricardo in Barcelona, but it was fun.

CRAIG BOX: A long way for the both of you to fly to meet one another.

ALEJANDRO DE BRITO FONTES: [CHUCKLES] Yes, exactly.

RICARDO KATZ: Exactly.

CRAIG BOX: You decided to stand down as the maintainer of the project earlier this year. I guess there are two questions. Why did you decide that was the right time? And on the flip side, why did you do it for so long?

ALEJANDRO DE BRITO FONTES: I decided to step down because for two and a half years, I tried to find sponsors for my time in the project without much success. Now that I'm here, I need to say that in the last few months, Shopify sponsored part of my time. So thank you, Shopify, for that. But it's pretty hard to get the sponsors in open source.

CRAIG BOX: Yes.

ALEJANDRO DE BRITO FONTES: Also, if a project is present in the Kubernetes organization, it doesn't mean that it's being supported financially. Maybe people in the project are just doing it for free. So please support those projects. That was the main reason, actually.

CRAIG BOX: And that's not necessarily only a Kubernetes problem. After some of the SSL bugs a few years back, there was a real attention brought to the fact that there were these key pieces of Unix and internet infrastructure that were essentially maintained by one person in their spare time, or in some cases, not maintained by anybody at all. And there was a fund put together, I guess, to help sponsor some of those things. Do you think it's time that we extended that to Kubernetes projects as well?

ALEJANDRO DE BRITO FONTES: I don't know. I think that this is a known problem. Everyone knows about this. This is not new. But someone needs to take action. Maybe the CNCF, I don't know.

CRAIG BOX: Ricardo, you took action.

RICARDO KATZ: Yeah. When Alejandro told me about this sponsorship, I actually asked him why he didn't open up GitHub Sponsors on GitHub. But by that time, Chile wasn't on the list.

ALEJANDRO DE BRITO FONTES: I'm still on the waiting list after three years.

RICARDO KATZ: See? Yeah, so that one was a first approach. And I think that from a user perspective, I can say that I sponsored some folks on GitHub. I know that it's pretty hard for those folks to have a hobby project that becomes like a pillar of the internet.

There is a fun XKCD on the infrastructure of internet, and a small piece supporting everything. I don't know if you've seen that already, but I can tell you. I saw that and I faced it, like Alejandro position and the curl maintainer as well on that one, because it's not your day job. It's not something that pays your bills, right? It's just fun. And it became important for people.

So the action that I took was, actually, I think that for Alejandro’s legacy as well, that project shouldn't die. And my reaction was, OK, I'm going to step forward to maintain that. And try at least to form a community of people that want to help.

I think that my biggest achievement in ingress-nginx right now wasn't releasing version 1. Actually, it was forming and getting some folks that really want to help and really want to review. So we have Elvin from Shopify, who Alejandro told us about. And we have James and Jintao and Long and other folks that saw the opportunity to become a contributor to something from Kubernetes, inside Kubernetes Organization, but not actually the Kubernetes upstream code — being issue reviewer and et cetera.

And this was actually the action that I think that I took, was trying to form [a team of] folks that really want to maintain. Because I don't think that I'm going to be in this project forever. The same thing that happened to Alejandro — I have other stuff to do as well. And I don't want to disappoint people waiting for some feature requests or for some pull requests or for something like that.

CRAIG BOX: NGINX Inc. had a conference last week where they actually announced a greater commitment to open source and that they will have a full-time person working in the upstream ingress-nginx community.

RICARDO KATZ: This is actually fun, because they have reached me and James. Before they announced that, they said, hey, we want to help you with that. So how can we help you? And by that time, we had decided to make some biweekly meetings of the ingress-nginx as a SIG-Network subproject. And they have started to join.

Some problematic issues, like a core dump that's happening or ways to improve NGINX, they actually are participating. They are helping with some issue triage and some code review. The good thing about that is that the project still keeps neutral. So that's not an NGINX Inc project. It's still the Kubernetes community project.

But my thought is that they knew that that was going to be bad for them if no one maintains that. It's still like the NGINX word on the NGINX Ingress name, which is the most used project. So I think that once that project became unsupported, people would start migrating to other projects that wouldn't be NGINX but would be something else, like HAProxy, Contour, or anything else.

So it was cool from their part to reach us and say, hey, OK, we want to help you with that. And we are going to put some effort into that. And they are really helping us during the meetings and doing some issue triaging and helping with other code. I just hope that sometime they decide to make hot reloads open source.

ALEJANDRO DE BRITO FONTES: [LAUGHING]

RICARDO KATZ: I'm just kidding.

CRAIG BOX: The other thing that Kubernetes SIG Network is working on in the Ingress space is effectively the V2 implementation in the Gateway API. Part of the announcement from NGINX Inc was that they would be putting some engineers to work on that project as well, whether it be for the community version or for their commercial versions of NGINX. How do you see the transition between Ingress V1 to the new Gateway API?

RICARDO KATZ: I'm going to talk about that, but Alejandro was also part of that in the past. So I think that he can complement that. As I wasn't expecting to work on ingress-nginx or any other Ingress implementation by the beginning of this year, I wasn't following the Gateway API work that was being done by the community on that. So it's not something new to me, because it was actually announced on KubeCon Barcelona 2019. owei, and Tim and other folks had shown their idea of Gateway API. But I wasn't following that.

I think it's going to be good having NGINX folks working on that. On the other hand, I need to give a shout-out to Alejandro, because the way that he implemented ingress-nginx — it's going to make my life easy. I wouldn't say too easy, because it's hard to migrate to Gateway API. But it's going to be easy to convert this structure expected from Gateway API to the structure expected from ingress-nginx.

We are already planning the Gateway API implementation in ingress-nginx as well, as soon as we can get rid of all of the version 1 bugs and the problems with IngressClass that we were not expecting. As soon as we get rid of that, the community is talking about how we can implement that, and getting feedback from the community as well.

ALEJANDRO DE BRITO FONTES: Gateway API is a huge step forward to what we have done with Ingress. And there's more common ground to try now against the different implementations. It's going to take some time to reach a stable API and see exactly what it means between different implementations. That is also one of the main complaints with Ingress, why the implementations are so different between each other, and how we can also expose the same core features.

CRAIG BOX: Now, both of you have taken on new roles in the Kubernetes ecosystem this year. Alejandro, you are now working for Gitpod. And Ricardo, you are now working for VMware. I imagine the experience that you had working on ingress-nginx may have been helpful during the job search process?

ALEJANDRO DE BRITO FONTES: Yes. Of course, it's a lot easier now that you can show the work in the Kubernetes organization. So yeah, it was easier than expected, you could say.

RICARDO KATZ: Yeah, me too. This is my first role, actually, as a software developer. I've been always a sysadmin, SRE — you can call it whatever you want, depending on your age. Yeah, it was really helpful.

CRAIG BOX: Is working on ingress-nginx now part of either of your roles going forward?

RICARDO KATZ: Not on mine. I do this mostly weekends or during the night. Mostly on weekends.

CRAIG BOX: So we still need to get that GitHub sponsors page up for you is what I'm hearing.

[ALEJANDRO/RICARDO LAUGHING]

CRAIG BOX: If we've learned anything from Alex Ellis, it's that people just don't sponsor open source.

RICARDO KATZ: Yeah. Alex got a really strong opinion about that. And I agree with him, because he does a lot of cool stuff. He does OpenFaaS, he does inlets. He does a lot of stuff, but still he decides to do those things as jobs. But those things don't pay his bills. It's hard.

And I really invite people to — if you have like $5 or $10 or whatever — you decide to sponsor that project that you like. I am not asking for ingress-nginx, because I do that as fun when I can. This must be something that people really should know — that maintainers, they work when they can and not when you need. So if you need something, just pay for a license or sponsor someone to do that if it's part of your job. Alex has a really good point on that. And I think that if you want to help a maintainer, just go and sponsor him or her with whatever money you can.

CRAIG BOX: The project has just released version 1.0. So first of all, congratulations to both of you.

ALEJANDRO DE BRITO FONTES: I had nothing to do with it. Just thank you, Ricardo, for 1.0 [CHUCKLING]

CRAIG BOX: Yeah. In fairness, I'm sure you have a little more than nothing to do with it.

RICARDO KATZ: 95% of the code is yours. I just rewrote it. [ALEJANDRO LAUGHS] It was launched this year. The biggest change of 1.0 is actually supporting the v1 Ingress API. This is the major change. There is nothing more — like a feature, like hot reload or a new web application firewall or something like that.

Talking with the community, we decided that as soon as Ingress reaches v1 API, which was a complaint about a lot of cloud providers as well, because I remember about that discussion. "Hey, we use a v1beta1 API, so when you decide to change that, it's going to break everything." So yeah, when SIGnetwork decided to change from v1beta1 to v1, a lot of things changed and a lot of things broke.

And I keep making some jokes about that with Rob Scott, like saying, hey, it wasn't that easy to change that. [ALEJANDRO LAUGHS] But anyway, we decided to make a GA with the API getting GA. So the biggest change was the v1 API and the IngressClass implementation. And IngressClass implementation broke a lot of stuff.

So we are still thinking about adding some rollback or some safeguard for users that want to migrate to v1, because the other one is going to be deprecated. Also because the v1beta1 API is getting deprecated from the Ingress object. And yeah, that was the biggest change.

CRAIG BOX: Something that I think people care a lot about is the magic of a 1.0 version number. Whether or not that means something is generally available, people say, well, hey, at zero-point-whatever, I'm scared of that. I don't want to run that thing. And then the mystique of having gone to 1.0. But it sounds like it wasn't so much a "we've finally tested it", it's that there was a breaking change and we are again properly applying semantic versioning.

RICARDO KATZ: Yeah. That was the decision, actually, was saying something like, hey, we can say that now we have semantic version. We are going to follow the semantic version. We are going to have some bug fix releases or some feature releases. We are now happy that we have this stable API as well. So we can say that we have finally reached stable because the API reached stable on Kubernetes API. So now we may have a deprecation plan as well.

CRAIG BOX: Another way to look at this is, Alejandro, you ran this project for what five and a half years? [ALEJANDRO LAUGHS] And you couldn't get it out of zero-point-something. Ricardo takes it for six weeks and look, 1.0.

ALEJANDRO DE BRITO FONTES: Yeah.

RICARDO KATZ: Hey, yeah.

ALEJANDRO DE BRITO FONTES: Actually, I never wanted to reach 1.0.

CRAIG BOX: You wanted to just asymptotically approach it — 0.9999999.

ALEJANDRO DE BRITO FONTES: Yeah, but the reason for that is what you expect from that. You say, 1.0. OK, there's going to be a 1.01, a 1.1. You are committing to multiple versions and multiple branches. And that was not the reality in the project. That's it.

RICARDO KATZ: It became a problem, because we now have two branches — the legacy one and the new one.

CRAIG BOX: Yes. You've mentioned a couple of things. Hot reloads, obviously, web application, firewall features, and so on. What things would you like to see develop in the project over time?

RICARDO KATZ: First of all, I want to see some old problems getting stabilized. So we have some old problems that we've seen people complaining about. Even the image size, because of mod-security, which is an old problem. Or problems dealing with reconciliation that I want to see solved. So I am really focused with the community on bug fixes.

And for the next feature versions, I want to start working with Curifense as an alternative to mod-security. I don't want to drop mod-security. Alejandro introduced me to the mod-security maintainer, which is a Brazilian, and I didn't know about that.

So it's really cool, because we got to talk as well. But I want to at least try to decouple things and leave for users the opportunity to choose which web application firewall they want. And those choices don't impact in the building process of ingress-nginx.

Also I want to see as a feature Gateway APIs implemented, because I think it's weird that we have all of the other community implementations with Gateway APIs doing their implementations. So we have in HAProxy. We have in Contour. We have in Kong, I guess. And we don't have in ingress-nginx.

So I think we are sending a message here that we are also keeping up with development if we implement Gateway API, we support Gateway API, as well. So this is the feature that I want to see for the next few months — some web application firewall, something focused on security, and something on the Gateway API as well.

CRAIG BOX: Would you say that becomes ingress-nginx 2.x? Or are we now talking about renaming or forking the project and having gateway-api-nginx?

RICARDO KATZ: No, it's going to be probably ingress-nginx version 1.2.0 or something like that, like "ingress-nginx" with lasers. I don't know if we are going to rename actually the project, or some of the projects are going to be renamed as well, because this is something that's generating confusion between the ingress-nginx community and the NGINX Ingress from the NGINX corporation.

So this is something that we are actually discussing with NGINX and we're going to bring to the steering committee as well to say, hey, how can we get this confusion away? Because people go to Google search and search for ingress-nginx something. And they usually are facing problems with NGINX Inc.. And they came to us or vice versa.

CRAIG BOX: That sounds like something that the new person from NGINX could clear up for you.

[ALEJANDRO LAUGHING]

RICARDO KATZ: Yeah, we actually are asking them, just to — hey, just solve that for us.

CRAIG BOX: Yeah, merge the two projects or rename the company or something. What are you going to do?

RICARDO KATZ: Yeah. Sounds like a good approach. I'm not the one to decide, but I think they should do that.

CRAIG BOX: And Alejandro, from talking to you throughout this, it feels to me like you believe the project is in good hands.

ALEJANDRO DE BRITO FONTES: Yes, of course. Thanks to Ricardo, that he stepped in on the project to help. If it wasn't for you, this was not in the current state. Now that you've created the community around the project itself. So thank you again.

CRAIG BOX: And thank you very much, both of you, for joining us today.

RICARDO KATZ: Thank you again, Craig.

ALEJANDRO DE BRITO FONTES: Thank you.

CRAIG BOX: You can find Ricardo on Twitter, @rpkatz, you can find Alejandro on Twitter @aledbf, and you can find both of them on the Kubernetes Slack. You can find ingress-nginx at kubernetes.github.io/ingress-nginx.

[MUSIC PLAYING]

JIMMY MOORE: Thanks for listening. As always, if you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter, @kubernetespod, or reach us by email at kubernetespodcast@google.com.

CRAIG BOX: You can also check out our website at kubernetespodcast.com, where you will find transcripts and show notes, as well as links to subscribe. Until next time, take care.

JIMMY MOORE: See you later.

[MUSIC PLAYING]