CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google. I'm your host, Craig Box

[MUSIC PLAYING]

CRAIG BOX: Daylight saving time started in the US this week, which means for reasons of meetings, I now have to get up an hour earlier in New Zealand. I'm not sure that's how it was supposed to work. The American Senate has voted unanimously to go to permanent daylight saving, not paying attention to the fact that it fades the curtains and confuses the cows. I think it's time now to turn the legislators' attention to my concept of seasonal zones where we make Christmas make more sense by celebrating it in June in the Southern hemisphere when it's actually cold.

And now the container platform news. Almost one year after the Ever Given container ship ran aground in the Suez Canal, a sister ship has run aground off the coast of Baltimore. The unfortunately named Ever Forward is, at the time of recording, stuck in the Chesapeake Bay. Perhaps it left an hour too early before that let enough water into the channel. Let's get to the news.

[MUSIC PLAYING]

CRAIG BOX: Two sandbox projects have been promoted to incubation in the CNCF this week. Backstage is a platform for building developer portals. It was built at Spotify, and we talked about it with its developers in episode 136. In-toto is a supply chain security framework built by the Secure Systems Lab of NYU's Tandon School of Engineering. It is a providence format, the importance of which was touched on in episode 155.

Solo.io had their annual Solo Con last week. The updates announced were primarily focused on Gloo Mesh, their managed Istio product, including support for multi tenant workspaces, a unified API for East-,West, and North-South traffic management, a new UI observability, and improved VM support. Solo also announced their support for GraphQL is now in beta. Meanwhile Buoyant announced an operator allowing cross-cluster failover for Linkerd, automating the automation of traffic routing when a service is no longer available.

Security news of the week, if you use the cri-o container runtime, make sure you patch because a vulnerability means that anyone with rights to deploy a pod can abuse a kernel parameter to escape the container and perform arbitrary code execution as root. The vulnerability has been named cr8escape (with an eight) by its discoverers at CrowdStrike.

If you're running GKE Autopilot, the good news is you don't have to do any patching at all. A number of vulnerabilities were reported to Google by Unit 42 at Palo Alto Networks and as all are safely patched, t hey've released a write up on how they were exploitable. The post shows the fine balance between providing a higher level of abstraction and leaving places where an attacker could hide things that are not exposed to the end user.

If it's your job to secure clusters, you may have seen hardening guidance published last August by the NSA and CSIA. Version 1.1 of that guidance has been published this week taking feedback from industry and offering thanks and acknowledgments but unfortunately, no changelog.

As KubeCon EU approaches, a hybrid event, albeit with the promise of a full in-person experience, the conference schedule has now been published. 148 sessions were selected from 1,187 submissions with 72 project maintainer sessions ranging from getting started to technical deep dives as well. A behind the scenes post tells us the talks were submitted by 480 companies, and the acceptance rate for the event was a mere 12%. I look forward to seeing you at the Kubernetes Podcast booth.

A micro-survey published by the CNCF and their Observability Technical Advisory Group suggests 86% of people are running Prometheus with OpenTelemetry and Fluentd the next most popular tools. Challenges faced include complexity, lack of documentation, and the fear an open source project might become inactive.

Finally, machine learning vendor, run:AI has announced a $75 million funding round.un:AI runs on top of Kubernetes and indeed first made news for the ability to share GPUs and fractions between multiple pods. The round was led by Tiger Global Management and Insight Partners.

And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Jesse Suen is one of the creators of, and a lead for, the Argo project. He is the co-founder and CTO of Akuity with a K, which provides cloud native application delivery powered by Argo. Prior to founding Akuity, Jesse was a principal software engineer at Intuit and the technical lead for the Argo team. Welcome to the show, Jesse.

JESSE SUEN: Thanks for having me. Glad to be here.

CRAIG BOX: Your involvement with Argo goes back through three different companies. So of course, my first question has to be who's the big Ben Affleck fan?

JESSE SUEN: [LAUGH] I'm a little embarrassed to say. I haven't actually yet watched the "Argo" movie. Personally, I'm more of a Matt Damon fan myself. But I did like them both in "Good Will Hunting."

CRAIG BOX: Yeah, I feel you really do have to come down on one side of that debate.

JESSE SUEN: Yeah, I'm very curious on why the movie was called "Argo." So maybe someone can fill me in on that.

CRAIG BOX: I know it was based on a book. I, too have not watched it. I did hear it won Best Picture at the Oscars at least. So it has some reverence. Perhaps then, you have a different story as to why the project is so named?.

JESSE SUEN: Yeah, so I have to give first credit to my former Applatix colleague, Rahul and also Pratik, the CEO, who came up with the name. But the first reason we decided to name it Argo was we wanted to stick with the whole Kubernetes Greek theme. Argo was the name of a ship in Greek mythology which transported this Greek hero named Jason on his quest for the Golden Fleece. Jason's crew were called the Argonauts, named after the ship.

CRAIG BOX: I have heard of them before.

JESSE SUEN: Yeah, probably back in high school. Some scientists also decided to name a species of octopuses as Argonauts. That made our mascot become the octopus because it was related to the Argo name named after the sailors. The octopus kind of fit with our theme because our first product was a workflow engine, which is juggling many tasks at the same time just like an octopus might do with its eight tentacles.

CRAIG BOX: Tell me then a little bit about what Applatix was and why you were building out what became a CI/CD project.

JESSE SUEN: We had a really ambitious goal. We envisioned our product to be this all encompassing DevOps container-based cloud solution, which included not only your CI pipeline, your build pipeline, so you wouldn't need Jenkins anymore. But it also included your infrastructure platform for scheduling and running your workloads. So we abstracted away the cloud and even Kubernetes away from the user. It included application management and monitoring.

So we had our own DSL for defining applications and running them. And there are other things like cost analytics so we could monitor how much is this container costing you to run. The Applatix product was actually this big monolithic product built on top of cloud, AWS, Cassandra, Kafka –oh, and also the first version of it wasn't even based on Kubernetes. It was actually based on Mesos.

CRAIG BOX: What was the background that you had that caused you to want to solve this problem?

JESSE SUEN: Pratik, who founded the company, I actually worked with him with several companies before that. Pratik ended up leading engineering teams, multiple organizations. Every company he visited, they ended up rebuilding the whole CI process and deployment solution. After a couple rounds of this, he realized this is a need that the organizations need to do over and over again. We also recognized the advent of containerization as well as the cloud. And so this was kind of an opportunity where we can merge and combine all those technologies into something really forward thinking.

CRAIG BOX: Would this have been around 2014, 2015?

JESSE SUEN: 2015, 2016.

CRAIG BOX: So you've followed your colleague to his startup to build out the CI/CD project. You've obviously seen the growth of containers and Mesos, Kubernetes around that time. What was it that you ended up building?

JESSE SUEN: It was actually a whole platform that customers would have to install in their AWS account. Like I mentioned, there was a lot of different components to it. It was three separate ones. There was a back end UI piece to it. And then there was we call the DevOps piece, which was build pipeline and the workflow engine. And then there was a low level platform component to it, which was abstracting away the cloud infrastructure and Kubernetes from the user.

And so the users, they knew it was running on the cloud. But they didn't have to think about VMs or containers. And they would write their application in our DSL. It was actually a hard sell to companies because it was a lot to take in at once. Back then, companies were still just looking in and starting to move to virtual machines in the cloud let alone containers. Also at that time, the container orchestration wars were still happening. Maybe by 2016, the writing was on the wall that Kubernetes was going to start to take over.

CRAIG BOX: Would you describe the thing that you'd get when you install this product as being a PaaS?

JESSE SUEN: Yeah, that's probably the best way to describe it.

CRAIG BOX: Was that part of your pitch at the time? Was that something that people understood the value of?

JESSE SUEN: They did, but it was a bit ahead of its time. Like I said, it was a lot to take in at once. Because you have companies who already have maybe the CI piece of that. And they're used to that. They're not ready to replace both their delivery mechanism as well as their build pipelines and entrust this startup with everything from end to end.

CRAIG BOX: Was that part of the reason that Argo is now many separate projects?

JESSE SUEN: Right, so after realizing this, we made a couple of choices. One, we rewrote the architecture and actually switched from Mesos to Kubernetes because it was clear at that point that Kubernetes would end up being the standard. Second, we also decided to make it an open source project. Kubernetes is heavily open source influenced. And practitioners of Kubernetes tend to gravitate towards open source projects versus like proprietary closed source. And so that was the second decision that happened.

Similarly, we decided to strip down the Applatix product into individual parts and base it on the Kubernetes controller architecture. And I think at that time, it wasn't even called CRD. So it was called Third Party Resources. They were just starting to deprecate that in place of CRDs

CRAIG BOX: It still was probably quite a thing to trust your business to Kubernetes at that point.

JESSE SUEN: It was an easier ask to have them maybe accept something like the CI piece of it, which was our workflow engine and so that was the first piece that we rewrote into the control architecture. That was our first open source project. Argo actually referred to Argo Workflows initially.

CRAIG BOX: Was this released by Applatix, or was this after your acquisition by Intuit?

JESSE SUEN: This was right before the acquisition. It wasn't actually until we open sourced this and got a booth at KubeCon before companies like Intuit actually noticed us and started gaining interest.

CRAIG BOX: How did that process go from a company perspective?

JESSE SUEN: It was the 2017 KubeCon in Austin that Applatix got a booth in. I didn't attend it, but we were demoing the workflow engine to whoever stopped by, I guess. I remember hearing Joe Beda actually spent a lot of time at our booth. And Intuit also ended up spending a lot of time. I think the week after they got back, Intuit approached us with acquisition interest. And literally a month after that, we were Intuit employees.

CRAIG BOX: What does a software company building financial and tax software want with a CI engine?

JESSE SUEN: That was the same question I had when Intuit expressed interest in us. I think really, Intuit recognized our expertise in Kubernetes. Intuit's been around for a long time. They're almost a 40-year-old company, and they predate cloud. Traditionally, Intuit has been running on private data centers. If you think about their business, they have a model that actually suits a cloud model better. They have one big event a year, which is the US April tax season. And so the weeks leading up to tax filing, that's when Intuit needs its most compute. And the rest of the year is a lot slower. So it makes perfect sense for a business like Intuit to leverage the elasticity of cloud compute.

By 2017, I think Intuit was already on this multi-year journey to move to the cloud and AWS. But their approach was actually still with virtual machines. And they were taking this lift and shift approach to migration. It was actually Marianna Tessel, who's now the Intuit CTO. At the time she was leading QuickBooks. And she came from Docker and understood like the value of and potential of containers over virtual machines.

They still weren't sure what would orchestrate it if it would be Kubernetes or Docker Swarm, ECS. But at least they were convinced it would be some form of containerization. Intuit, I think, basically went to KubeCon to go shopping for startups.

CRAIG BOX: They picked up the talent of the people building out the software. But they also picked up the open source project that you had released. Was it their intention to be an open source vendor? Or did it just happen to be a side project that they were willing to continue investing in?

JESSE SUEN: Yeah, it's more of the latter. So when we arrived at Intuit, we were tasked to do two things. So the original Applatix team actually split into two with two different directives. Part of the Applatix team, myself and Alex Matyushentsev we were tasked to build out the Kubernetes CD solution for Inuit. The other half of the team would actually build out the Kubernetes distribution that Intuit would run as well as the management portal for provisioning clusters and upgrading and doing namespace provisioning and all that stuff. When we got to Intuit, Alex and myself started the Argo CD project. And then that is what we built next.

CRAIG BOX: You had the CI project before. You have the CD project after. Why are those two things different projects? A lot of people might see that as being part of a single journey. And a lot of people just say CI/CD as if it's one thing. What was the idea behind them being two distinct projects?

JESSE SUEN: I actually see it differently. To me, it's more obvious that CI and CD be different things. They definitely should be integrated. And there should be some tight coupling and automation that happens between the two. But I actually feel like the act of managing a production SaaS solution, you're going to want to do operations to that environment that are not always necessarily related to code changes that you might make in your application source repo.

To me, it was important in the design for CD that it actually remained agnostic to the CI process. And so we focused a lot on making sure that integration was possible by having API, CLIs and things to make it easy to integrate. And also, at the time Intuit wasn't sure about like which CI system it would end up choosing, they ended up choosing Jenkins

CRAIG BOX: As opposed to the one that they just bought from you.

JESSE SUEN: Yeah, exactly. To be honest, Workflows, I always describe Workflows as kind of a building block. Although it had originally a CI-centric approach, there's a lot of gaps missing in it for it to be a CI product. So that's why I call it a workflow engine but not a complete CI solution. In fact, one of the projects that we had started right after open sourcing the workflow engine was something called Argo CI. It might still be in an archived project in the Argo project umbrella.

The idea was that, OK, we now have this generic workflow controller that can be used for any kind of batch processing. But let's create a application-specific use case that was built on top of Workflows. We ended up never finishing that or continuing that after the acquisition. But that was the idea.

CRAIG BOX: When you're looking now to build your continuous delivery product, what was the state of the ecosystem at the time. Was GitOps becoming a thing?

JESSE SUEN: It was not very well known. I give a lot of credit to Weaveworks for actually really pushing this new concept. And we believed in its principles. With Kubernetes, you have a huge amount of configuration that you need to manage because of its declarative state. And so Git just naturally makes sense to manage your Kubernetes configuration.

Second, because Intuit is a fintech company, it has a lot of financial and compliance requirements. The use of Git as change management for production infrastructure like the auditing that you get for free, the approvals that you get through merging, using something like Git also fit naturally with the requirements that we had for building a CD solution for Intuit. Some of the other features were just kind of icing on the cake, like ease of rollbacks because you have Git as your history is easy, that simple to go back to a previous SHA and do rollbacks.

And so GitOps helped us facilitate the velocity of developers because they can move with confidence going from version to version always knowing they can quickly go back to that previous SHA.

CRAIG BOX: Is continuous delivery simply a workflow? And how much of the work that you've done on the Workflow product were you able to bring to the continuous delivery product?

JESSE SUEN: At Intuit, we didn't end up using Workflows for that piece of things. Workflows at Intuit ended up being a lot more on infrastructure automation, also data science teams ended up using it. Like I mentioned before, people are always building a layer on top of it because of the lack of a specific use case that it is trying to solve. And Workflows has a great UI. And I think you can really kind of visualize and see what your complex steps are doing. But it is a bit low level for a lot of people and people always end up building something on top.

CRAIG BOX: But when you were building out your own CD product, were you able to either use the Workflow engine or perhaps some of the code that you'd put into that to bootstrap yourself?

JESSE SUEN: We actually thought of ways where we could incorporate Workflows. One of the ways we originally tried to use Workflows as part of CD was to facilitate blue green and canary deployments. Because if you think about the nature of a blue green, or canary deployment, it's imperative in nature. You have a sequence of steps that you need to go through. Maybe increase the wait. Wait a little bit, run some tests, continue on to the next one. And our original plan was actually to power that using Argo Workflows.

There was an early attempt to use Workflows for the purposes of blue green and canary. We quickly realized that we wanted a more declarative approach to that problem. And so Argo rollouts replaced that effort to be a declarative way of doing progressive delivery.

CRAIG BOX: I guess that opens up to the same question we had before – if CI and CD have two different processes. We want to have two different tools for that. Continuous delivery and progressive delivery or rollouts, are they two different processes?

JESSE SUEN: Some people describe progressive delivery as the next step or evolution of continuous delivery. So the idea behind progressive delivery is that it's one thing to move quick with continuous delivery. But it's another thing to actually move safe. And that's where progressive delivery comes in. So it's easier to automate the deployment and rollout of your new version of your application. But then it becomes a lot harder to automatically detect when something is not going according to plan. And for you to automatically roll back based on things like metrics or KPIs that you're monitoring.

CRAIG BOX: Right.

JESSE SUEN: Rollouts was our way to fill that gap because Kubernetes rolling update, it just wants to get from point A to point B as fast as possible. There's no guardrails around that other than, say like readiness.

CRAIG BOX: There's no validation that point B is actually a place that you want to be at.

JESSE SUEN: Precisely, yeah. The other thing was that with rollouts, people were actually coming from Spinnaker, right, to do VM deployments. And Spinnaker actually has a pretty sophisticated canarying process. And I would say they pioneered a lot of ideas about canarying that we actually kind of adopt or incorporate it into Rollouts. But we had teams who were using Spinnaker deploying. And they were doing things like statistical analysis on a canary versus baseline.

After they moved to Kubernetes, they lost that ability because the only thing they had left was rolling update. And so we wanted to give them back that capability by building them a canarying solution.

CRAIG BOX: A lot of people now use Argo CD to do delivery through GitOps. Another project in this space is FluxCD from the team at Weaveworks, who as you mentioned before, coined the term. There have been different levels of collaboration between those projects in the past, and there was even some talk of merging them at some stage. What happened there?

JESSE SUEN: It started with a joint kind of conversation I think from the executive teams. I think there was some folks on Amazon and on the Weave side and also on the Intuit side of like, hey, we're all doing GitOps. And it makes sense to collaborate and kind of come up with maybe some standardized way of doing this. We had quite different beginnings if you think about the genesis of both Flux and Argo CD. My understanding is Flux was actually a tool that Weave used themselves to help ensure that they could recover from like a disaster recovery scenario,ecause they committed all their infrastructure to Git.

The genesis of Argo CD was that we needed to build a delivery tool for developer teams. And we heavily focused on things like the user experience and the UI. And GitOps happened to be the mechanism that we chose to do the delivery aspect of it.

CRAIG BOX: Right.

JESSE SUEN: Sometimes, I describe that Flux had more of this operator centric approach. And then Argo CD had a little bit more of a developer experience and centric approach. Then you can kind of see some of the results of that in the different products. So yeah, it started with some early conversations. Pratik told the team we're going to go fly to London. And the teams would meet to see if is there any kind of common ground that maybe we have that we could collaborate on some merging of the project.

So we spent like half a week in London meeting with the Flux team. I would say the first day was a little contentious. But on the second day, we actually came to some agreements on what we could do. And so the outcome of that was that we would split the core part of Argo CD into a library, which we called the GitOps Engine. And this would just be a generic piece of software. It was heavily based on callbacks. You can integrate with anything there's no CRDs in this thing. It was purely a library. But it had conveniences on being able to watch your resources in the cluster and calculate diffs. And so it's useful library, actually.

Alex Matyushentsev have actually spent months splitting out the GitOps engine into its product. And he even did a POC with the Flex V1 product at the time built on top of the GitOps engine just to prove that it's something that was viable. At the end of the day, we've decided that they wanted to go into a different direction. And that's what became the GitOps Toolkit, which is now called Flux v2.

CRAIG BOX: As someone coming to the community space today, would you still say there is that same distinction that if you're looking maybe for more developer-centric viewpoint, you might opt for Argo versus an infrastructure team might opt for Flux?

JESSE SUEN: To be honest, I haven't totally kept up what Flux is doing. I will just say we do place the user experience that ended with UI is part and parcel of the Argo CD experience. I understand that Flux also has a secondary component I think, a UI for Flux. But I would say if that's something, for example, that's important to you.

There is also the multi-cluster use case that Intuit originally had a need with,nd I do think Flux V2 had addressed the multi-cluster use case. But when we were developing Intuit, we knew we were going to run many, many medium-sized clusters, like hundreds of medium-sized clusters. We wanted to avoid the management headache of installing all these deployment tools and making sure their config is up to date in hundreds of these workload clusters. And so we had designed Argo CD to be a remote based access versus Flux is installed in every single of your clusters, and it's pulling. A lot of people love Flux. They do have a simpler model than Argo CD does.

Although, I will say that Argo CD, we have been making strides to what we call Argo CD Core. You can actually install Argo CD in a cluster and remove the UI components, the SSO component. At that point is purely just a reconciler of your Git repos. We find that some of our users end up running Argo CD in this way at the edge. If they have many edge clusters or something, they can run Argo CD Core in that manner.

CRAIG BOX: We've talked about three of the projects under the Argo umbrella. There is a fourth one. How does Argo Events relate?

JESSE SUEN: Doesn't get as much attention. And the reason for that is you tend to only use events in conjunction with Workflows. In other words, Workflows by itself is great. You have a massively parallel workflow that can run your steps in a sequence that you designate. What's missing from that is a way to trigger that based on some event. For example, I dropped something in S3. And as soon as I drop that thing in S3, I want some workflow to process it.

And so Events was donated to the Argo project by BlackRock. BlackRock was using Argo Workflows for some data science reasons. I think the original name was called Axis if I'm mistaken. We saw the need to trigger workflows. And we actually started a project briefly called Argo Events and then were approached by BlackRock to say hey, we're building something. And they presented it at the community. And then the original Argo Events, we archived or deleted. And then Axis became Argo Events.

CRAIG BOX: That is open source working as intended.

JESSE SUEN: Exactly, yeah.

CRAIG BOX: In July of 2020, the Argo project was proposed to the CNCF and became an incubation project there. What joined? Was it the individual components? Was it just one of them? Was it the whole thing?

JESSE SUEN: Yeah, that was a bit of a unique situation. So the answer is like the whole project joined. And as you know, most CNCF projects are typically just one software project. And Argo is comprised of four independent sub-projects. All of them are focused on application delivery. But they can be used independently from each other. So CNCF actually did have a concern that Argo could unilaterally decide to add a new sub-project and bypass the whole CNCF process.

In the end, they actually did end up permitting us to join as a whole. But there is an understanding between CNCF and Argo that we're not going to introduce any more core Argo projects and sneak them into the CNCF.

CRAIG BOX: No one approached you about Knative renaming itself Argo Serverless sneaking in that way?

JESSE SUEN: [LAUGH] No nothing like that. It does create some unique corner cases with our attempts to graduate CNCF. Because as we're trying to graduate, we end up trying to do four times the amount of work that a typical project does. The security reviews and analysis and stuff, we end up having to repeat that

CRAIG BOX: Had you considered the idea of taking the four projects and effectively breaking them down into individual CNCF projects that could have different stages of maturity themselves?

JESSE SUEN: There might have been discussions about that with higher ups. It's interesting because we chose to use the Argo branded house versus a brand of houses approach to naming the projects. When we first started Argo CD, there was a debate whether or not we should name it something other than Argo. Because Argo at that time meant Workflows. I bet that if we had gone the direction to name Argo CD into a different--

CRAIG BOX: Golden Fleece.

JESSE SUEN: Yeah, Golden Fleece for example, it might end up a different story with regards to what was actually accepted by the CNCF.

CRAIG BOX: Yes, naming things is hard, famously so.

JESSE SUEN: Yeah. It's the first hard problem of computer science I've been told.

CRAIG BOX: Talking about naming things, you are now the co-founder of a startup working in the Argo space called Akuity, with a K. Tell me about the journey from Intuit to Akuity.

JESSE SUEN: Yeah, I should give credit to my wife who came up with the name. I gave her a couple of parameters to come up with a name. One, we wanted it to start with the letter A. This is actually a tip I learned from Pratik that if your company name starts with an A, up being towards the top of most of the lists.

CRAIG BOX: That is why there are a lot of motels named the Aaron Court.

JESSE SUEN: Oh Yeah?

CRAIG BOX: They used to show up first in the book back when people looked at books.

JESSE SUEN: That's funny. OK, something that starts with the letter A, second, because we're a Kubernetes related company, you have the freedom to replace the letter C or incorporate the letter K into the name. Literally overnight, the next morning she said, "OK, well how about Akuity?" To me, like that was the perfect name.

CRAIG BOX: What is the goal of Akuity? Are you just working on the software? Do you have a SaaS version that you're hosting for people?

JESSE SUEN: Yeah, cloud is a big part of our strategy. Why we started Akuity was first we observed the enormous growth of the Argo project over the years. And we just noticed a lot of users would just keep coming to the Slack and ask about things like vendor support or commercial support to help them use and run the product. And we would help them out, jump on Zooms with the community members. But it was always kind of on this best effort basis.

And as you imagine, a lot of enterprises are uneasy about adopting some piece of technology which they don't have the backing of some expertise behind it, especially if it's something as critical as delivering your services.

CRAIG BOX: Perhaps that's something that Intuit could have done for the other 11 months of the year.

JESSE SUEN: Yeah, a lot of people ask like is Intuit going to commercialize Argo? The answer is they have zero interest in making that part of their business. They want to do accounting and tax software for consumers. This is our second go around with Argo. And one of the things that I am personally excited about with Akuity is being able to listen and focus and understand what the community is asking for and seeing how that can shape the future direction of the project.

If I'm being honest, a lot of the ideas in innovation for the project, these days, they're coming from the community. Because with Intuit, we were pretty much future complete for all the things we set out to do a year ago. And they have already this established usage pattern for Argo, which they're not going to change. They're going to have their way of running Argo. There's a lot of proposals and ideas that have been put forward by the community, which now we can pay closer attention to as well as build out some more ecosystem around the Argo projects.

CRAIG BOX: How do you see that ecosystem going forward? Are you getting customers saying to you, "Hey, my use case is different than what Intuit were doing." And that's something that they're able to contribute or you're able to build.

JESSE SUEN: Yeah, like a perfect example of one thing we didn't anticipate, with Argo CD, we always anticipated people wanting an automated and imperative approach for creating applications. And so Argo CD was focused on providing the APIs and the CLIs to help facilitate the automation. It shouldn't have been a surprise. But people want a more declarative approach to defining their applications. That usage pattern is not actually being used in Intuit. Everything is done more through integrations and automation.

But the community was the one who kind of pushed for declarative management of applications. And so for example, we have something called the "app of apps" pattern where you create one parent application. And that is just nothing but a bunch of application specs. Personally, I never liked that idea. But I threw it out there just to provide some kind of form of solution to the community. What I hope will replace the app of apps pattern is something called ApplicationSets.

And this is an ecosystem project it just actually got merged into our latest 2.3 release. So it's actually included by default. You can disable it if you don't use it but that actually was more of a community-centric use case versus the Intuit one. Because our community was asking for a declarative approach to managing their application specs.

CRAIG BOX: Are there any other use cases that are surprising you that people were Argo for that weren't things that you thought of or that were relevant to Intuit?

JESSE SUEN: I would say the workflow piece always is a pleasant surprise to see what people are doing with Workflows. I love seeing the workflow community presentations because we get organizations like CERN presenting data from the Large Hadron Collider doing massively parallel workflows to process that data. With workflows, I'm always surprised about all the use cases that it's able to solve.

On CD, CD is a bit more straightforward. I tend to describe Argo CD as a glorified kubectl apply or like kubectlapply on the cron job. That's essentially like what GitOps is at the essence of it. CD, I think you have a lot more intricate use cases on how organizations want to deliver their apps. I guess maybe some examples like how people wanted to promote from environment to environment, how people wanted to ephemeral environments.

And so I do see a lot of interesting ideas in the community on how they are able to achieve those use cases with Argo CD. And it's always nice to see new ways people are utilizing Argo CD for that purpose.

CRAIG BOX: If someone is using kubectl apply to deploy the software today, how do you recommend that they get from there to the full Argo experience?

JESSE SUEN: One of the things that was always important to us even with Argo CD is that we wanted people to have an escape hatch in the event that Argo CD was down or for whatever reason they didn't want to use it. So we always stuck with kubectl apply because that's the most basic form of delivering a resource to Kubernetes. You can always go back. If you decide not to use Argo CD tomorrow, you're in good shape. Because the way we've been deploying your stuff is via kubectl apply.

You might miss things like pruning or diffing or health checks and stuff. But delivering the software at the end of the day is just apply. If you're looking to move from that model to Argo CD, really your biggest barrier is adopting, I think, a practice of GitOps. And that's probably the biggest thing you have to do to move to Argo CD. And the biggest challenge that users end up having to do is like, how do I make that new image get into the cluster using a GitOps-centric approach?

There's a lot of ways you can do that. What they're not used to is making a Git commit and have that be an automated process to update an image check. And the tooling doesn't always make it easy. For example, Helm, I think if you just want to change the image in the values file, I guess you have to resort to sed replacing that image tag. But other tools like kustomize kind of anticipated that need. And they provide a nice, convenient command like kustomize edit set image.

And now, it becomes dead simple to just make a small change to update the image as part of your CI and then commit and push that to Git as an automated process.

CRAIG BOX: Once people have adopted Argo CD, how would you like them to get involved with the community?

JESSE SUEN: A couple of ways I would recommend, the first is join our CNCF Slack channel. If you need help, there's tons of people who can help answer your questions on the CNCF Slack. We have also very regular meetings every week, almost too many meetings. So there's two tracks, I would say. There's a Workflows and Events track and also a CD and Rollouts. Check out our community calendar. Every week, we have a meeting on the workflow track and also on the CD.

There's also UI focused meeting that happens and also less on the project, but more of a maintainers meeting that also happens.

CRAIG BOX: Thank you very much for joining us today, Jesse.

JESSE SUEN: Thank you for having me. It was a pleasure to be here.

CRAIG BOX: You can find Jesse on Twitter at @JesseSuen, or on the CNCF Slack, and you can find Argo and Akuity at argoproj.io and akuity.io respectively.

[MUSIC PLAYING]

CRAIG BOX: That's the end of another great show. If you've enjoyed this episode, please help us spread the word and tell a friend. If you have any feedback, you can find us on Twitter @kubernetespod or reach us by email at kubernetespodcast@google.com. You can also check out the website at kubernetespodcast.com where you will find show notes, transcripts, and links to subscribe.

Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll be back next week.

[MUSIC PLAYING]