Kubernetes Podcast from Google: Episode 220 - Falco Graduation, with Mike Coleman

KASLIN FIELDS: In this episode, I spoke with Mike Coleman. Mike is a developer advocate at Sysdig, focused on open source software. And he spends a lot of his time working on the Falco project. We'll explore how Falco enables runtime security and celebrate its recent graduation. But first, let's get to the news.

[MUSIC PLAYING]

ABDEL SGHIOUAR: The CNCF has announced the graduation of Falco. We'll be talking about that today, so stay tuned for more.

KASLIN FIELDS: Google released its new family of open models called Gemma. Gemma is a family of lightweight, state of the art open models built from the same research and technology used to create the Gemini models. The model weights are available in two sizes, Gemma 2B and Gemma 7B. Each size is released with pretrained and instruction-tuned variants. The Gemma models achieve best in class performance for their sizes compared to other open models while being capable of running directly on a developer laptop or desktop computer.

ABDEL SGHIOUAR: The Linux Foundation together with the CNCF launched a new GitOps associate certification. The new credential is meant for dev ops teams, platform and software engineers, and anybody working with internal development platforms with GitOps practices. You can take the exam today. Check the links in the show notes.

KASLIN FIELDS: Linkerd has announced that with version 2.15, it will no longer be publishing stable open source release artifacts. The code of Linkerd will remain open source under the Apache V2 license, meaning that users could build their own release artifacts from source. This change has sparked some discussion and criticism from the community. Check out the links in the show notes for more information.

ABDEL SGHIOUAR: Crossplane introduced version 1.15 with many improvements and features, one of which is introducing a Python SDK for writing compositions. Crossplane is developed by Upbound and is an extension to the Kubernetes control plane that allows operators to declare cloud resources using the Kubernetes resource model.

KASLIN FIELDS: The schedule for Open Source Summit North America is out now. Open Source Summit North America will take place April 16 through 18 in Seattle, Washington and includes 16 micro-conferences. Cloud Native Security Con will also take place in Seattle on June 26 and 27. The call for papers for Cloud Native Security Con is open until April 3. And that's the news.

[MUSIC PLAYING]

Hello and welcome, Mike. I'm really excited to have you on today. We'll dive into our history here in a minute. But first off, Mike is a developer advocate at Sysdig who is focused on open source software, and he spends a lot of time working on the Falco project. He's been in tech for way too long, in his own words, having worked at Microsoft, VMware, Docker, AWS, and Google before coming to Sysdig about a year ago. So welcome to the show, Mike.

MIKE COLEMAN: Hello. Thanks for having me. It's good to speak with you.

KASLIN FIELDS: Yeah. So Mike was-- there's a long history here, honestly. Before I was at Google, I was at NetApp and I started getting involved in the meetup scene in the Bellevue area because I wanted to learn about various technologies in the cloud native space and all that kind of stuff. And I went to this Docker meetup where you spoke, and that was one of the first times I was like, I want to be someone who does this kind of thing. So that was my first encounter with you.

MIKE COLEMAN: Statements like that-- that's what I love about our job-- developer advocates-- is having people come up to you and be like, I saw you talk. I was at Civo Navigate earlier this week and an individual-- I won't call them out because maybe they'd be embarrassed. But they are now a Civo Navigate Ambassador, CNCF Ambassador, Docker Captain kind of person. And he was like I saw you talk at DockerCon and talk about Docker for system administrators, and that inspired me. Because I was a sysadmin and I didn't feel like I had a place, and now I am deeply embedded in this community. And I'm like, that is the best thing anybody has said to me in a long time. And stories like yours, they just get me so jazzed.

KASLIN FIELDS: Yeah. And after that meetup, I also saw you at Docker cons where you had your analogy about the house and the VMs and containers. And that also was like-- yeah, so you've been a big influence in my career.

MIKE COLEMAN: Well, like I said, it's humbling. And if I could help anybody in any way, then I'm super stoked about it.

KASLIN FIELDS: And to kind of kick things off here, you shared something for us to talk about as an icebreaker, which is a fantastic story that I've heard before. But I think everyone needs to hear it because it's great. There's this story about you demoing on stage with Bill Gates.

MIKE COLEMAN: Yeah. So I was demoing with Bill Gates. And I can't remember the year. 2019? Or, no. I don't know. It was late '90s, early 2000s whenever it happened. We were on stage at an education conference in Seattle, and all of a sudden I felt the floor rumble. And I'm like, man, that must have been a big truck that just went by. Because sometimes trucks can shake the house or whatever, right? And then I thought to myself, we're on the fifth floor. And then all heck broke loose.

It was a full-on earthquake. It was the Nisqually earthquake, and I was literally on stage with Bill Gates. And a light canister fell to the ground and I looked at Bill and I'm like, Bill's got a lot of money and I don't, and so I've got to take care of my kids. So I left the stage and Bill stood up there for a good couple of beats, and then he quietly walked off. So there is on YouTube-- if you search for me and Bill Gates in an earthquake, there is footage of me and Bill Gates in an earthquake. I got my 15 minutes of fame out of that.

KASLIN FIELDS: I also at some point along the way heard someone else tell this same story from yet another person's perspective who was at this event that day. I think they were maybe on stage at a different time, but they were also there and they ended up hiding in a door frame with Bill Gates or something.

MIKE COLEMAN: Yeah. The coolest thing about it, after it was over and it was safe when we were trying to decide if we were going to continue the demo, which we did, Bill went out into the lobby and he just stood there. It was a teachers conference, and I know that you've got teaching in your family. And he was just talking to the teachers and just rapping with them. And I've got to imagine that just made their day.

KASLIN FIELDS: Yeah, that's got to be awesome. Very cool. And one other thing that we were talking about before we started this, which I'm going to bring up even though it's not Falco related, is another really cool thing that you do is you do a Christmas light display which-- you program it yourself and everything, right?

MIKE COLEMAN: Yeah. I mean, well, I kind of cheat. I build it all myself, and then I kind of what they call map it. Like, other people program it. But you were my guest star two years ago, remember? You recorded that intro piece for me. So--

KASLIN FIELDS: If you would like to hear me voice a tree--

MIKE COLEMAN: You were my-- I was the other tree. So yeah, we have about 22,000 lights that are individually addressable LEDs. We synchronize them to music. And I'm telling you, watching the kids out there in front of the house dancing at Christmastime to whatever it ends up being-- Taylor Swift, "Christmas Tree Farm" was a big hit, obviously. So yeah, it's great. I love it. Yeah, awesome.

KASLIN FIELDS: I know a lot of folks in our community love doing at-home tech projects like that, so it's super cool. I don't think you involve Kubernetes with it in any way, but--

MIKE COLEMAN: No, but all the software is open source.

KASLIN FIELDS: Nice.

MIKE COLEMAN: The sequencing software is open source and the playback software is open source.

KASLIN FIELDS: And we are going to talk about open source today. We're going to be talking about Falco, right?

MIKE COLEMAN: Yeah.

KASLIN FIELDS: And I was saying before we started that it's been a while since I've gotten a demo of Falco or really dived into it myself, so I'm looking forward to getting an update on how things are going with Falco. But to start off, could you give everyone a little overview of what Falco is?

MIKE COLEMAN: Yeah. So Falco is a project that's aimed at enhancing the runtime security for cloud and Kubernetes and container workloads. And essentially what it does at the most basic level is it listens to events coming through the kernel. It passes those events up into an agent that's running in user space. And that agent has a set of rules, and it's looking at the system calls and it's saying, OK, did somebody open a file they shouldn't have opened? Or did somebody change a file name, or did somebody run a package manager or shell into a running container? And then from there, we can issue alerts.

So the idea is, if you think about it, all the great stuff that gets done up until the app gets put into production. So we do vulnerability scanning, you do network policies. You may even have, like, admission controllers. And you're doing all this stuff to protect your environment, but what you put into production yesterday may not be what's running today because somebody might have gone in and done something. And so what Falco does is acts like a security camera for your environment to give you real-time monitoring about what people are doing and what's going on in that environment.

KASLIN FIELDS: And this is why I love your content-- because I just learned something new. So I hear a lot from the security folks in the Kubernetes open source project about attacks-- cases where you could get into a Kubernetes cluster and insert code into already existing workloads that are running in that Kubernetes cluster. And one thing I always think about with these use cases whenever they come up is, how would you figure that out if you're someone who runs these workloads?

Because if it's a workload that's just already running there and somebody just messed with it-- but you might not necessarily see that in any of your logs or anything. How would you figure out that that happened? And so doing it this way through the actual kernel calls that are happening-- that makes a lot of sense to me.

MIKE COLEMAN: Yeah. And it's actually a little bit deeper than that, too, because we pull from the container runtime as well as the Kubernetes API. And we do that for a couple of reasons. One, it lets us get a lot of metadata. So when we give you output messages, we can give you things like what's the container ID? What pod did it come from? What namespace was it running in? So you can get all of this metadata.

But to your example specifically, we have rules. There's one rule, application executing, that was not part of the base image. So any code that wasn't in the base image we can fire an alert on that because we will know-- people that are familiar with containers know that anything that gets written to a container gets written to the topmost layer. So if an application is being executed off the topmost layer, the read-write layer, then we know that it wasn't there originally. So we can track that, what layer was executed in, and then we will know that that's not part of the base image.

KASLIN FIELDS: I think there might be a really interesting opportunity to play around with this and get hands-on with it. There's a Kubernetes security workshop that they ran at KubeCon in 2019, and it was like Tabitha Sable and Brad Geesaman and a couple of the other folks from SIG security worked on this workshop, and it's available online. I'll put a link in the show notes. It's like secure.gates.io or something like that. It's a very clear URL when I can remember what it is.

But it's one of these cases where you are an attacker and you're given a URL that is going to get you into a Kubernetes cluster, and you just have to mess around with it and see what you have access to. And then your task as the attacker is to insert a crypto miner, of course, into a workload. And so adding Falco into this would be a really interesting way to see how all of this works. Both be the attacker and--

MIKE COLEMAN: Well, we actually have a great blog post on how to detect crypto miners using Falco. That's a use case for us.

KASLIN FIELDS: Do you have, like, a workshop to do it? Because--

MIKE COLEMAN: We might. Let me look at it. And if we don't, I'll get my colleague Nigel and I to cook one up. It's a pretty core use case-- that whole idea of running code that isn't supposed to be there. So if someone, for instance, installed Netcat into a running container, you'd probably want to know about that, right?

KASLIN FIELDS: Yeah.

MIKE COLEMAN: So yeah, all that kind of stuff is kind of covered in that regard.

KASLIN FIELDS: Interesting. Cool. So if you find any useful links for hands-on workshops and content, we'll make sure to have all of that in the show notes of course.

MIKE COLEMAN: And Falco is part of the certified Kubernetes Security Specialist exam, so any materials about the CKS are going to get you playing with Falco a little bit and testing it out.

KASLIN FIELDS: That's a good point. I haven't actually taken any courses on that. I should try that out. There's probably some good practice tests where you could do some cool stuff. And the reason that we're on here talking about Falco today of all times is that you're approaching CNCF graduation, or have already graduated, depending on when this comes out.

MIKE COLEMAN: When we're recording this and when we drop it, there's going to be-- yeah, so let me just talk about the whole history of the project, I guess, a little bit to kind of back up. So the founder of Sysdig, Loris-- he wrote Falco about seven years ago. So 2016-ish he came out. And I think Kubernetes 1.0 was 2015. No, 2015 or 2014? Yeah, 2015, right? I don't know.

KASLIN FIELDS: Yeah, it's 14. Because it's 10-year anniversary this year. June 2014.

MIKE COLEMAN: Yes. Right after that he was like, how are we going-- that question that you asked. How are we going to know what's going on in these environments? And so he wrote Falco. And in 2018 we donated it to the CNCF-- a sandbox project. It was the first runtime security project in the portfolio for the CNCF. 2018, we were donated. 2020, we went to incubation status. And then in 2022, we applied for graduation-- at the very end of 2022.

So we've been on that path for a little over a year now, and yeah, now we are a freshly graduated CNCF project and we couldn't be more proud. More importantly, we couldn't be more grateful for everybody who has either used the project, filed an issue, contributed, all that stuff. Because nobody gets to this point without a really involved community of really great people helping you all along the way.

KASLIN FIELDS: And I have been learning a lot about the CNCF maturity levels lately at KubeCon. In 2023, Bob Killen-- I think he had a co-speaker as well whose name I can't remember right now. But he gave a talk.

MIKE COLEMAN: I think it was my boss, Aizhamal.

KASLIN FIELDS: Was it? It might have been. That sounds right, actually.

MIKE COLEMAN: I think it was. Her and Bob Killen did a talk on the maturity levels of projects and how you move through the CNCF. And yeah, that was my boss Aizhamal, who--

KASLIN FIELDS: Fantastic talk.

MIKE COLEMAN: Yeah. She's fabulous. I love

KASLIN FIELDS: Thank you very much to Mike's boss for that talk. And so I learned a lot about the maturity levels from that, and it's really hard to move between the maturity levels in the CNCF. So is there anything in particular you want to call out that Falco is excited about that they achieved through getting to graduation, or that they're looking forward to doing past graduation?

MIKE COLEMAN: Yeah, a couple of different things. Like, what does it mean, I guess? It's a validation, right? When you look at a project and you say, OK, Falco-- it's not in the same level of adoption, but it's the same level of maturity as, say, Kubernetes. That's another graduated project. And so really, to get there, you have to really make sure you have good governance, you've got documentation that's up to snuff, you've got good processes, you've got good solid code, and a variety of maintainers and contributors.

So the idea of this culmination of all that work by hundreds and hundreds of people-- it's a sign of dependability and maturity, and it's a big recognition. I don't know how else to say it. But it's a big deal, I guess. Right? It's a validation.

KASLIN FIELDS: Yeah, definitely.

MIKE COLEMAN: It's also a validation that runtime security is something that people need. It's a project that's been raised up to say, look, this is important stuff. So it's good. I love it.

KASLIN FIELDS: Yeah. And I think one requirement of graduation, which I'm not sure if you mentioned it or not, is end users. You have to have people using it in production in order to make it to graduated status. So at this point it's like, it's not just something to toy around with. It's not just a concept. It's something that people are using in production-- companies are using in production today.

MIKE COLEMAN: Absolutely.

KASLIN FIELDS: So you talked a little bit about the history of the project, and the concept here really is-- runtime security is the terminology that we would use of doing these checks in the kernel and everything. So can you dive a little bit deeper into the concept of runtime security that Falco approaches here?

MIKE COLEMAN: Yeah. So you know I love a good analogy, so let's use--

KASLIN FIELDS: You do love a good analogy.

MIKE COLEMAN: I love a good analogy. let's use an analogy for this. So if you think about, you have a building and you're trying to secure it. One thing you might do is you might put a fence around it. And that's sort of like what we used to do for security. It was firewalls, right? And if somebody cut a hole in your fence or got through your firewall, then you knew you had a problem. But now we operate in this world where there aren't these fences anymore, right? Everything's in the cloud is happening, and--

KASLIN FIELDS: It's hard to build a fence around it. There's so much.

MIKE COLEMAN: Right. Throw a VM Google Compute instance or a EC2 instance or whatever Azure has-- I'm trying to be equal opportunity. Throw that up on the internet and see how long it takes for somebody to start pounding on it. They're always looking. Like, did you put the Kubernetes API servers? Is it exposed? Let me see if I can find it.

And so it's not really about keeping people out. It's about how do you monitor people. And if you think about the complexity of building security, OK, well, we had the fence, but now we've given you a badge. And the badge says, I'm Kaslin and I'm going to go ahead and be able to go into the building, and I can check into the research lab but not the data center, for instance, right? You have authentication and authorization. That's the badge.

And those things are so important across not just your compute environment, but also your GitHub access and how you set up repositories and Okta for single sign-on. We're not just using a cloud provider. We're using a set of cloud services across this vast, massive spectrum, right? So the idea of runtime is like, if somebody in your building and they find a key on the ground-- an attacker who's come in and breached the perimeter-- and they're walking around and they're just plugging this key into every lock that they can find, how would you know that as the building owner?

You would have a security camera. And so this idea of runtime security is basically like a security camera for your workloads that's looking for the person walking around doing something they shouldn't be doing, whether that is nefarious or whether it is an overworked sysadmin who's taken a shortcut. Maybe someone like myself who has pushed their AWS secret keys up onto GitHub or who has published a blog post where they said, hey, just give it S3 full access. Nobody cares. And then somebody says, but we do care.

And so it's that idea of, people make mistakes. People are out there doing nefarious things. But in that action of being able to see what they're doing, that's what runtime security is, and that's that visibility. And it's not a standalone thing. So you still need vulnerability scanning, you still need network policies, you still need to run the benchmark against your Kubernetes clusters, right? You still need to make sure that you're taking all of these other things. But you want to be able to say, OK, all that was good. But what happened the next day? And that's the essence of runtime.

KASLIN FIELDS: And that's an interesting area that I want to dive more into as well. So Falco is an open source project that's graduated in the CNCF, obviously. So what is it like to actually use it? How does it fit into a practitioner's environment? Who are the practitioners that are using Falco?

MIKE COLEMAN: I think it can vary. I mean, it's obviously going to be used in concert with the security team, but it's also going to be implemented by the platforms team, typically. When I talk about Falco, I generally find a couple of types of people. I find security professionals who maybe aren't as well versed about Kubernetes and just general cloud security, and I find people who are practitioners and know a lot about-- like, I know how to build a cluster. I know how to administrate cluster. I want to know how to lock it down. I want to know how to secure it.

So I think it's a combination of both those groups, and I can't really say percentage-wise, but definitely it's going to be-- like all things security, it's going to be a group effort between the platform team, the application development team, and then the security team as well.

KASLIN FIELDS: Yeah. One thing I've noticed, especially in recent years in Kubernetes, is that more and more of my audience is application developers who are being tasked with running their own Kubernetes clusters in addition to writing the code and actually making the applications. Are you running into many application developers in Falco land?

MIKE COLEMAN: In Falco land-- so what I have seen and what we have been talking about is in both development and build environments, people running their code against Falco. Having Falco installed in those environments to see if it's triggering any rules before it gets to production. It's just another version of shift left, shift left being the idea that let's get security as early as we possibly can into the development process, right?

And so if you're running Falco on your development cluster and you see that your application is accessing something that Falco is going to alert on, you can let the security team know and they can update the rule. So one of the things about-- so let's take the file esty.shadow. Your hashed passwords in Linux. We don't want untrusted programs accessing that. Like, a program typically shouldn't have to access to that file, so there's a rule and it's like untrusted program, dah, dah, dah.

So you write a new application and you for whatever reason need to do that-- that's where you go to your security team and you're like-- or whoever's managing the rules and you say, we need to update the rule. We need to add our application into the permitted program so that we don't trigger that. So by testing that earlier in the development process, you can eliminate false positives when you get to production.

KASLIN FIELDS: This is a use case that I hadn't thought about earlier. My brain went straight to that, like, someone maliciously inserting code into running applications use case. But with rules in place, this can also be like a policy kind of tool as well. Not really like a policy tool, but it kind of has elements of that. You know what I'm getting at?

MIKE COLEMAN: Yeah, no. I do-- exactly what you're getting at. So you remember Mathieu Benoit, our old colleague who was a DRE up in Canada? He and I are supposed to be-- and I've been falling behind on this. We were supposed to be working on a blog about the intersection of runtime security and policy implementation.

KASLIN FIELDS: Oh, there you go.

MIKE COLEMAN: And how they overlap and how they're similar and how they're different. So it is-- it sort of is like runtime policy, right? If you think about Gatekeeper and OPA, that kind of stuff generally works at stopping things from happening. And we are letting you know when something happens.

KASLIN FIELDS: So it's a different perspective, but it has a hand in that space as well.

MIKE COLEMAN: Exactly. And that's why when I thought about the blog, I'm like, this seems really like people learning about Falco might immediately equate it to OPA and those type of things. Gatekeeper.

KASLIN FIELDS: But it's definitely at a different level.

MIKE COLEMAN: Yes. It's not an admission controller, right? The other thing is, we're not going to block anything. So you've got Gatekeeper there that will stop things from happening. But with Falco, we detect. We don't respond. So there's no jelly for our peanut butter in that particular project.

KASLIN FIELDS: In your analogy, it's kind of like OPA-- Open Policy Agent and those types of tools are building the fences and putting up the signs that say, here's what you're supposed to do. I did an illustration once, describing a similar concept in security of park security of-- there are fences around and there's rules. You have to be this tall to go on this ride. These kinds of things. So Open Policy Agent is kind of setting those things in place, and then Falco is, like you said, the cameras and the on-site security. Just because you have fences and rules doesn't mean that people are going to follow them, so you still need to have security on site.

MIKE COLEMAN: Yeah, exactly. Once you've let them into the park-- so yeah, in your analogy, all of that stuff is about who can get into the park or who can be in. And then we're not supposed to be blowing up fireworks in the park, but now somebody is. Well, who's going to catch that, right? Well, with fireworks, maybe the security officers would because they'd be blowing up. But you get my point.

KASLIN FIELDS: But with the ability to set rules and to learn from these things that you're picking up from these tools, I feel like the security guards are kind of consulting on the security of the park as well is the role that it plays in the policy bit.

MIKE COLEMAN: So if you go back to the security camera analogy. What Falco is doing is it's throwing that image up in the command center where the security guards are looking at all the screens, and they're, like go over here and check this out. Something is over here that doesn't look right. You know, we talk about preventing things from happening. But when an attack happens, it's usually a set of steps, right?

You got a warning that someone shelled into a running container. OK, that's an operational anti-pattern and they shouldn't be doing it, but that's not necessarily indicative of an attack. OK, well now someone just downloaded Netcat. OK, wait. Hold on. And now what they've done as they've renamed it to one of the applications that was running on that machine. Well now, hold on, right. It's that increasing pattern. That pattern of the different steps that sort of manifest themselves, and the ability to go and put those things together and correlate them.

KASLIN FIELDS: Yeah. I feel like I have a better view of what Falco is right now through all of this. So we have this tool that can monitor what your workloads are doing to the underlying machines, I guess is a way to explain it, and tell you about what's happening. You can set rules on it. There are clearly some interesting use cases here like we've mentioned, like with the malicious insertion of code or just accidental access of things that you really should never access that this can help you catch. What other kinds of use cases or what other kinds of features is Falco looking to release?

MIKE COLEMAN: So there's a couple ways to talk about that. One is the future, which we can talk about next. But even in today's world-- so I mentioned GitHub and pushing your credentials up. So Falco has customization options on both the input end and the output end. So we have the ability to write plugins that can monitor basically any event stream. Matter of fact, I just co-authored a blog with Michelle over at Google about using it to monitor a bunch of services on Google Cloud.

We can take the inputs, we can take Google Cloud logging output, run that into Falco, and we can write rules against that output to like, someone just deleted a Pub/Sub topic, and we can alert you on that. Or we have a plugin for GitHub. Someone just pushed their credentials up, right? Or we have a plugin for Okta someone just tried multiple times to log in and they failed multiple times.

So I did a talk at a conference earlier this year called "Nearly Universal Threat Monitoring-- or Threat Detection-- excuse me-- with Falco." Because with the plugin architecture, if you've got a known stream of events you can map those events into Falco's JSON essentially and then write rules against it. So we can take the system calls, but we can also take logging input.

And so then on the output-- by default we write things to standard out logs, gRPC, endpoints and HTTPS endpoints, but we have another project called Falco Sidekick which has over 60 endpoints that you can then send those alerts to. So you can send an alert to Slack or you can send the alert to a lambda function on AWS and have that lambda function write a little code to take action for you.

So that's like the jelly to the peanut butter. Because I said originally, we don't have that sort of capability built into Falco, but we have the ability for you to build it if you want to build it. So stem to stern, you really can build a remediation pipelines, and you can cover a wide swath of the tech stack you use. So that's kind of how you integrate it in other environments and that kind of stuff.

KASLIN FIELDS: When I've talked to customers and end users recently, one of the biggest problem areas that they've pointed out is the noise of logging and how they want the right alerts to come to the right places so that they can see them in places where they actually watch for them. So it sounds like this Falco Sidekick is a tool to help address that problem, right?

MIKE COLEMAN: Yeah. So it can be, because you can then send that into like Prometheus or Elastic or whatever and then do aggregation in there. The other thing that we've done recently with Falco-- because that noise is probably the number one piece of feedback we get from new users in our Slack channel. So we are #Falco on the Kubernetes Slack. They're like, hey, I just installed Falco. Man, I'm getting a lot of information.

So we originally shipped with like 80 default rules in our rules file, but we've recently gone into this-- we've kind of adapted the CNCF model and we've split our rules files into three files. And we have about 20 to 30 rules in each file to reduce the noise. So in the default rules, we just really said, this is the core of what you need to help reduce noise. But you still probably want something on the back end to do that as well.

KASLIN FIELDS: Yeah. I mean, it's definitely not a problem that any one tool can solve because what folks are trying to get to is the right alerts in the right places where-- a lot of folks use like Slack or something where they want the alerts to come into Slack. But you don't want all of the alerts there, because then it would just be a never-ending, eternally scrolling Slack channel.

MIKE COLEMAN: Yeah, exactly.

ABDEL SGHIOUAR: So it's never going to be a one-tool fix for that problem, but--

MIKE COLEMAN: Now. And what I do like about the Sidekick approach is that as a practitioner, I'm not having to rip everything out, right? If I'm already using certain tools, I can plug Sidekick into those tools and integrate Falco with those tools. If I'm already a Prometheus shop, I'm already running Slack or I'm already running PagerDuty or whatever else it ends up being, Sidekick can just integrate with those existing systems.

KASLIN FIELDS: Cool. So the Sidekick project is a relatively recent development that is ongoing? Where are we at?

MIKE COLEMAN: I don't know. It's been around for a little bit. I don't know the exact date. It's been around. It's not brand new. It's been around for a while because we have those 60 integrations. But I can't tell you the exact day it started. But it's in the Falco security repository. So if you go to Falco Security/Falco Sidekick, you can find it and find all the integrations with it.

KASLIN FIELDS: Awesome. And so this plugin model that you were talking about as well, that's something that already exists in Falco with the Sidekick piece. Is it kind of in between the two, or is it--

MIKE COLEMAN: It's actually-- if you draw it left to right, it's sort of like syscalls and plugins feed into the Falco engine. The Falco engine sends the output out to Sidekick. So plugins are inputs, and Sidekick are outputs.

KASLIN FIELDS: I see. So there's plugins on both sides.

MIKE COLEMAN: Yeah. Expansion opportunity everywhere.

KASLIN FIELDS: Cool. So what would you say-- you mentioned what's next for Falco. What would you say are the next things that will be coming for Falco?

MIKE COLEMAN: We're working on a couple of things. You know, at the end of the day we believe that runtime security is critical on the project. That's why we do it. And we see with Falco now being a graduated project, that's a way for us to be the mantle, if you will-- the engine powering that runtime security. As we continue to move into the cloud era, what we want to do is we want to work on just more detections, better signals, lower noise, better performance, just like that-- in the US, we would say motherhood and Apple pie, right? The stuff that people need to just make their jobs easier.

But we also want to make sure that we're increasing where people can pull data. So we will continue to work on plugins. Plugins of course are open source and they can be contributed by anybody in the community, but we'll continue to work on them as a project in and of ourselves.

KASLIN FIELDS: The benefit of open source.

MIKE COLEMAN: Exactly, right? And same on the Sidekick side. We've also just recently released-- this is a very nascent project, but it's called Talon and it is designed to set up very rudimentary remediation actions. So for basic stuff you could use Talon. For more advanced stuff, you could use Sidekick. So that's an area we're going to go.

And then we made some recent changes, like changing the way our driver gets loaded. And we've used eBPF for almost the entire length of the project as a way to talk to the kernel, but to continue to make loading the drivers and getting all that set up just much easier for users-- that's always going to be a priority for us as well.

KASLIN FIELDS: Interesting. So we've got Falco, which is the core runtime security tool. And then we've got Sidekick, which adds in the ability to kind of manage some of the alerts and logging in a sense. And then you've got Talon, which also adds in the remediation piece of all of this. Interesting.

MIKE COLEMAN: Yeah, it would be like a peer to Sidekick.

KASLIN FIELDS: Gotcha. All right, well I've certainly enjoyed learning about Falco today. I hope that everyone else out there has.

MIKE COLEMAN: Right on. I'm happy to hear it.

KASLIN FIELDS: Like I said earlier, I'll make sure that we have links to some of those hands-on tools and learning tools that we mentioned during the episode. Are there any other resources that you want to call out? Though CKS was a good call-out earlier, too.

MIKE COLEMAN: Yeah. So Falco is part of the CKS, so you can find resources there. There is an O'Reilly book. If you go to KubeCon, we're probably going to be giving it away. We give it away at almost every conference we're at. If you hit me up on Twitter or LinkedIn-- Mike G. Coleman everywhere-- I could probably get you a copy with a QR code kind of thing. But we've got that book. There's also the falco.org website. We're on LinkedIn and Twitter, obviously. Or, excuse me, X. Whatever we're calling it today, we're there.

And then we do our weekly community meetings on Wednesday at about 4:00 PM GMT. So anybody is welcome to join. And then if you want to just hang out with us and chat, I mentioned this briefly, but #Falco, #Falco on Kubernetes Slack. We have our channel there. And there's about 2,700 people in the channel, so it's a pretty good community of folks. And all the maintainers are in there every day. I'm in there most days. So I think we're really responsive on getting questions answered and helping people out. So yeah, website-- falco.org. Falco security up on GitHub. Pound Falco or hash Falco on Kubernetes Slack. That would get you covered.

KASLIN FIELDS: I love that you called out those things, because it's not just go and try Falco out because it sounds cool to try out. It's, get involved with the community. It's open source. You can hang out with us and ask questions and learn together in public.

MIKE COLEMAN: Yeah, and that's the thing-- the learn in public piece. Even if you're not going to deploy it, even if you just want to come in and ask questions about it-- and we're always looking for contributors. Do you want to write a plugin? Do you want to write an extension for Falco? Do you want to help with the documentation? All of that stuff, man. And we're trying to make all that easier for folks all the time.

KASLIN FIELDS: Plug for all open source on the behalf of all open source projects. Project management skills are also very much needed in open source, so don't limit yourself.

MIKE COLEMAN: OK, yes. I mean, I think that's the thing. I mean, I am a hardcore non-code contributor, right? That's my space, and--

KASLIN FIELDS: Got to give it a shout-out.

MIKE COLEMAN: Right. And I think folks like yourself and myself who do a lot of the work around that-- and it's valuable, it's viable, and it's necessary. And I think that sometimes people feel intimidated. Oh, I can't write Go. I can't sling Golang like a master. It's like, well, can you write? Can you design graphics? Can you educate? Can you do any of these things? All of that stuff's needed.

KASLIN FIELDS: Yeah. So if you just want to join a cool community and learn about a cool project, make sure you check out Falco on Slack and join their meetings sometime. And, Mike, are there any last communication venues that you would like to share with folks who might want to get in touch with you?

MIKE COLEMAN: Yeah. To get in touch with me, I'm just Mike G. Coleman on pretty much everywhere. GitHub, Twitter, on LinkedIn even, I think. If you've got a question for me or you want to get connected, just reach out. I love helping people. That's kind of my jam.

KASLIN FIELDS: Seems like Slack would probably be the easiest way to get a hold of you?

MIKE COLEMAN: Actually, probably Twitter. But Slack works, too. I'm more responsive on Twitter than I am on Slack, but I am in both places. And on Slack I'm in the Kubernetes Slack. I'm in the CNCF Slack as well.

KASLIN FIELDS: That is the tricky thing with Slacks-- is that I know that I am always flipping between the CNCF Slack and the Kubernetes Slack. So that can slow things down.

MIKE COLEMAN: And I actually have logged into the CNCF like with two different email addresses, so there's two Mike Coleman's. So--

KASLIN FIELDS: A lot of people have that.

MIKE COLEMAN: --you want to look for the one-- I'm the one with me on the motorcycle is the one you want to talk to.

KASLIN FIELDS: Or just ping in the Falco Slack and Mike will just appear.

MIKE COLEMAN: I will, like a genie in a bottle.

KASLIN FIELDS: Cool. Thank you so much for being on today and teaching me about Falco.

MIKE COLEMAN: A thousand thanks for having us. And thank you for this effort you do with the community to keep the podcast running, you and Abdel. It's a great resource and we're all grateful and lucky to have it.

KASLIN FIELDS: I'm so glad to hear that. And we'll see you at KubeCon.

MIKE COLEMAN: Adios. Or should I say, au revoir?

KASLIN FIELDS: Au revoir.

[MUSIC PLAYING]

ABDEL SGHIOUAR: Well thank you, Kaslin, for that interview. It's always great to hear Mike's voice.

KASLIN FIELDS: It is. He's such a prominent member of the community. He's done so many conferences. And like we talked about in the interview, he's done so many things. Lots of people kind of know him around.

ABDEL SGHIOUAR: Yeah, we only crossed paths for a very short period of time at Google, but one of the things that I would remember is he was the person who got me into all things open, which was one of my favorites. Because he was there, I think, 2022 and he talked about it, and that's how I knew about it. I submitted. And it's still one of my favorite conferences in 2023.

KASLIN FIELDS: Nice. Lots of great connections there. He knows good stuff.

ABDEL SGHIOUAR: As he mentioned, one of the cool things about DevRel is it's a small community and everybody knows everybody. So it's pretty cool. Nice. So Falco graduated by the time this episode will be out?

KASLIN FIELDS: Yeah. I had to break the fourth wall there. It was like, it's not technically out yet when we were recording, but--

ABDEL SGHIOUAR: By the time the episode is out, right? So that's--

KASLIN FIELDS: The thing with graduations is that it's a series of votes that happen with the CNCF Technical Oversight Committee. And so they do all of these discussions about, like is the project ready for graduation? I mentioned in the episode as well that Bob Killen and one of Mike's co-workers, I learned, did a really great talk at KubeCon about what it takes to move to the different levels in the CNCF. So there's a whole bunch of things that happen. We were like, right there.

ABDEL SGHIOUAR: Yeah, exactly. So we were discussing, lining up an episode with a project's graduating with the right guest is super difficult to do. So we were hoping nothing will go wrong. So that everything lines up in the right order, and hopefully it will. So we'll see. All right. So there was a bunch of discussions you've been having there. I had to learn how Falco works when I was taking my CKS exam, and I did actually forget most of it. So when I was listening to the episode, I had to go back and look at the documentation. It is a pretty cool tool.

KASLIN FIELDS: I do want to talk to you more sometime about the CKS. You've been taking all of them, right, because of the changes to the certification process with the CNCF? Well, the change to how long you get the certification for when you get the certification.

ABDEL SGHIOUAR: Yeah. The CKS is the only one that didn't change, by the way. It was two years, and it was still two years. But for those who doesn't know, starting April 1, most certifications will go from two years validity to one year. From three to two, sorry. Three years to two years.

KASLIN FIELDS: OK. But CKS was already two?

ABDEL SGHIOUAR: True. But I am taking it again because it's expiring in March. So I have to do it again.

KASLIN FIELDS: Oh, interesting.

ABDEL SGHIOUAR: I have to do it regardless. It's expiring, so I have to do it at some point.

KASLIN FIELDS: Yeah. I've helped review some of the content for the certifications, but I haven't actually gotten any of them myself yet. I really would like to do that sometime, so I'll be looking for your guidance.

ABDEL SGHIOUAR: The CKS-- I remember very well that it was one of the most difficult ones. You have to be administrator certified before you can take the CKS, right?

KASLIN FIELDS: Yeah. You have to have the CKA before you can take the CKS.

ABDEL SGHIOUAR: Yeah, correct. And then it was very difficult. I think I was right on time. You have, like, 120 minutes and I used all of it.

KASLIN FIELDS: Yeah, the time limits are tough on those exams.

ABDEL SGHIOUAR: Yes, they are. Yes, yes, yes, yes. They are.

KASLIN FIELDS: It's amazing, because you're like, oh, I'll have plenty of time when you first start looking at it. And then you're like, no. No, that covers a lot.

ABDEL SGHIOUAR: Yeah, exactly. And they are hard, like a hands-on, right? So you get an environment and you have to get things done.

KASLIN FIELDS: Also, one more thing on this CKS is that the Kubernetes Book Club actually just did a book about the CKS. So if you're interested, you might--

ABDEL SGHIOUAR: Oh, yes. Carlos' club. The club that Carlos is running. Yeah, there was an article about it on the CNCF blog. We should probably find the link and add it to the show notes.

KASLIN FIELDS: Yeah. I can do that.

ABDEL SGHIOUAR: Cool. Yeah, so I remember Falco validation. You run it, it looks into a bunch of events. When I was looking at the website, it says eBPF because everybody is using eBPF these days.

KASLIN FIELDS: Interesting.

ABDEL SGHIOUAR: But it's a pretty cool tool. It can tell you when it notices things are wrong in your runtime.

KASLIN FIELDS: Yeah. That is very unique, I think, for a lot of security tools. You don't find many I think that operate at that level, though with our conversation about eBPF with Bill, maybe that is becoming more common to operate at that level. But that's one thing that we didn't really talk about that much with Mike, is the connection between Falco and eBPF. Does Falco use eBPF, or--

ABDEL SGHIOUAR: On the website according to the documentation, I think it uses eBPF as a source for events.

KASLIN FIELDS: That would make sense.

ABDEL SGHIOUAR: Because, like Mike was saying, it's an input, right? It has to get a bunch of kernel events. So eBPF is like the way to do that these days.

KASLIN FIELDS: Yeah. And Falco has existed for several years, but eBPF has existed longer than people think, too. So I don't know. Now I need to look into the history of that.

ABDEL SGHIOUAR: Yeah, it's quite interesting. And I think that a lot of times people confuse probably Falco with the Open Policy Agent-- OPA.

KASLIN FIELDS: Do you think? Yeah?

ABDEL SGHIOUAR: I mean, if you don't understand what they do, you might confuse them, right?

KASLIN FIELDS: Yeah. I noticed in the episode that there are some overlapping use cases, but I had never thought of them as being similar before we talked through the use cases in that conversation.

ABDEL SGHIOUAR: Yeah. I think that Mike explained it properly by saying that Falco doesn't block anything. It just notifies. So OPA is technically-- it's a rule engine, so it would block certain things from happening if they don't meet whatever rules you input. But I think probably the confusion is also in Falco world, we talk about rules.

KASLIN FIELDS: Yeah. Overloaded terms. Common problems with new technology.

ABDEL SGHIOUAR: Exactly. But I think the thing I learned is Falco Sidekick.

KASLIN FIELDS: Yeah. I've definitely seen that around a couple places, but I've never looked into it. So I didn't know what it was until we talked about it.

ABDEL SGHIOUAR: Yeah, so do you want to talk about it a little bit? Do you want to remind the audience what is it?

KASLIN FIELDS: Well, I was hoping that you would remind us by saying what you're interested in about it, but--

ABDEL SGHIOUAR: I mean, my understanding is it's a way to take all these alerts and then send them to different places like Slack or whatever, right?

KASLIN FIELDS: Right, yeah. So it's kind of the extensibility component of Falco-- is how I would explain it in a sentence.

ABDEL SGHIOUAR: Yeah. And it does support quite a lot of destinations, but-- and this is in case Mike listens to this-- it doesn't support Security Command Center in Google Cloud.

KASLIN FIELDS: We'll have to get him on that.

ABDEL SGHIOUAR: It's also a security tool. Yes. So that's our way of shaming Mike. It doesn't support Security Command Center.

KASLIN FIELDS: Also, now that I've said that sentence, I don't know that that's quite the right sentence. It's extensibility in terms of where things go. But there's also extensibility on the other side, which we talked about with the other project. So it's a little bit different.

ABDEL SGHIOUAR: Yes. Yes, yes, yes.

KASLIN FIELDS: But where are the alerts go and tools for helping to manage the alerts. So definitely look into Sidekick, because like I said in the episode, I've talked to a lot of folks lately and it's been a problem for a very long time-- is that alerts are just overwhelming. And so tools to help manage them and get them into the places where you want to see them are really useful.

ABDEL SGHIOUAR: Yeah. And we also talked about, even if you use something like Sidekick, it used to be that people complained about getting too many events or too many alerts, right?

KASLIN FIELDS: I mean, definitely still is like that. Everybody always complaining about getting way too many alerts because, I mean, everything is just so chatty these days. We want to make sure that everything, but when there are too many things, you can't know anything.

ABDEL SGHIOUAR: Exactly. I remember we built a tool that does a little bit what Falco does for GKE with some of my people from PSO. But it sounds like we should probably go back and try to look into, did we actually just reinvent the wheel, and could we have just used Falco for that? Which is very possible.

KASLIN FIELDS: Yeah. Falco incorporates a lot of interesting use cases these days, sounds like.

ABDEL SGHIOUAR: Yeah, exactly. Well, thank you very much. That was really interesting to listen to.

KASLIN FIELDS: Yeah. And we mentioned a whole bunch of resources in this, so make sure you check out the show notes for links to all of them. And stop by the Sysdig booth if you're at KubeCon EU, because they'll have stuff there.

ABDEL SGHIOUAR: Yeah, and you might get a book.

KASLIN FIELDS: Yeah. I actually have that book, and I could use some book club accountability here if anybody wants to read it with me.

ABDEL SGHIOUAR: And speaking of KubeCon, I don't know how many episodes are we going to put out between now and KubeCon, but if you are at KubeCon, come see us. We have stickers for the podcast.

KASLIN FIELDS: KubeCon is three weeks away. We're planning to do one the week of KubeCon, so if we stick to our schedule-- I'm saying this out loud so that you all can keep us accountable for this-- we should have a couple more between now and the end of KubeCon.

ABDEL SGHIOUAR: Yeah, but then if we put an episode during KubeCon, I don't think people will have time to listen to it. So this is the right time to say, we are going to be at KubeCon. Come talk to us.

KASLIN FIELDS: We're at KubeCon. Come talk to us.

ABDEL SGHIOUAR: Exactly. Good. That's what I was trying to get into. All right. Awesome. Thank you very much, Kaslin.

KASLIN FIELDS: Yeah. Thanks, Abdel. That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @kubernetespod or reach us by email <kubernetespodcast@google.com>. You can also check out the website at kubernetespodcast.com where you'll find transcripts, show notes, and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]

View More Episodes