KASLIN FIELDS: Hello, and welcome to the "Kubernetes Podcast" from Google. I'm your host Kaslin Fields.

ABDEL SGHIOUAR: And I am Abdel Sghiouar.

[MUSIC PLAYING]

KASLIN FIELDS: This week, we chatted with Jesper Larsson. Jesper is a freelance pen tester. He focuses on hacking Kubernetes and cloud-based infrastructure. In his own words, he breaks things and shows how broken they are. But first, let's get to the news.

[MUSIC PLAYING]

ABDEL SGHIOUAR: The removals, deprecations, and major changes blog for Kubernetes 1.29 is out now. The 1.29 release is scheduled to launch on December 5th. We are planning to do our regular interview with the release lead, so watch out for that in the next weeks.

KASLIN FIELDS: A blog is also up now on Kubernetes.io about sig-etcd. Etcd is a critical component of Kubernetes, which acts as the source of truth for the cluster. While etcd used to be a graduated project in the CNCF on its own, it recently joined Kubernetes as a special interest group. We dove into etcd in its new SIG status on a recent episode. If you enjoyed that episode and want to learn more, check out the blog post. A link will be in the show notes.

ABDEL SGHIOUAR: Red Hat announced the availability in developer preview of WASM on OpenShift. With this new features, users can run WASM binaries directly inside OpenShift using the crun WASM runtime.

KASLIN FIELDS: The Linux Foundation has a good dozen events still to come in 2023 around the world. Remaining events include Open Source Summit Japan, KubeDay India, KubeDay Singapore, and AI.dev-- Open Source Gen AI and ML Summit in North America.

ABDEL SGHIOUAR: Leadership transitions are integral to the health of open-source projects, especially when they are as big as Kubernetes. SIG contributor experience in open-source Kubernetes has a recent pull request around their leadership transition. The new co-chairs are Nabarun Pal and Kaslin Fields. And the new tech leads are Priyanka Saggu and Madhav Jivarajani. Congratulations to the emeritus leads, Bob Killen, Josh Berkus, Nikhita Raghunath, and Christoph Blecker.

KASLIN FIELDS: And that's the news.

[MUSIC PLAYING]

ABDEL SGHIOUAR: Today, I'm very happy to talk to Jesper Larsson. Jesper is a freelance pen tester, works with a hacker community called Cure53. He co-organized a Security Fest in Gothenburg, or Goteborg in Swedish, and hosts-- that's the part you will have to help me with-- "Sakerhets Podcast?"

JESPER LARSSON: Yeah, that's pretty good. "Sakerhetspodcasten." And that's true cool.

ABDEL SGHIOUAR: Which is literally stands for the Security Podcast. I had to Google that one, by the way.

JESPER LARSSON: Yeah, yeah.

ABDEL SGHIOUAR: You were actually a star on a Swedish TV series called "Hackad," which was about hacking.

JESPER LARSSON: Yeah, yeah, star-- it's a far stretch, isn't it? But, yeah, I was one of the hackers on the TV series that did a lot of shenanigans.

ABDEL SGHIOUAR: Well, your name was on the cast on IMDb.

JESPER LARSSON: Yeah, it's true. I'm one of the hackers in the world that have my own IMDb page. That's pretty cool.

ABDEL SGHIOUAR: Awesome. So just for the purpose of the podcast, let's just say you are a star, so we can get up the rating. And in your own words, I actually watched one of your videos for preparing for this episode, you said, "I don't fix anything, I just break them and leave. #Ijustbreakstuff.

JESPER LARSSON: Yeah, yeah, that is actually my work in a gist. Yeah, I tend to work with offensive security only. So yeah, penetration testing of different kind of companies. Not by bounty, I always get paid. So a normal day for me is that I get assigned a project, and I get maybe five to nine days at a specific work package where I only try to break things.

Mainly, I don't do any black box testing anymore. We only do code-assisted or configuration-assisted penetration tests. But we tend to only break things, show how we broke it. It could be we write a proof of concept or we do actually a code example, like here is the sink. This is how we sort of exploited the specific sink, and then we write a report about it.

ABDEL SGHIOUAR: Nice.

JESPER LARSSON: It's very rare that we do fix verification during a test. We have some clients that are able to do that, but then they have to make a patch during the actual testing period, which leaves a lot for the client, basically.

ABDEL SGHIOUAR: Yes. So we're going to talk a little bit about all of that in details. I actually got you on the podcast because we met back in June in a meetup in Goteborg. None of us knew either of us is going to be there.

JESPER LARSSON: No, no, that's true. That's true.

ABDEL SGHIOUAR: And then I watched your talk, and it was awesome. You were talking about some war stories with Kubernetes pen testing and hacking. So I was like, yeah, let's get you on the show. So welcome to the show, Jesper.

JESPER LARSSON: Yeah, I'm really happy to be here. I'm really glad you asked. It's really funny.

ABDEL SGHIOUAR: Awesome. So for the people, if anybody comes after me saying why this guy's name starts with a J and they're pronouncing it "Yesper," that's how you pronounce it in Swedish. A J in Swedish is a "Ye."

JESPER LARSSON: Yeah, I'm a lot of "Jasper" when I work. I only-- the majority of my clients are international. So "Jasper" is like sort of my middle name. Actually, I have an emoji in our Slack when someone calls me "Jasper." So I'm fine with either.

ABDEL SGHIOUAR: Nice, but I live in Sweden, so I'm going to pronounce it right.

JESPER LARSSON: Good. Good.

ABDEL SGHIOUAR: All right, so I think you already introduced yourself. We're going to get to know you a little bit more. But let's just get into, like, what got you into hacking? And what got you into pen testing specifically? What's your background?

JESPER LARSSON: I started-- I think this is-- the legal thing is already over, I guess. But my first gig as a teenager was to do modifications of PlayStations and Xboxes.

ABDEL SGHIOUAR: Oh.

JESPER LARSSON: So I added a lot of homebrew stuff in hardware actually. So I started in hardware basically tinkering with stuff that you should not do anymore. I'm not doing that anymore, for anyone that would like to pursue me or get me. But yeah, so I started with that whole shenanigans, modding Xbox and PlayStations basically.

And I was really bad at video games. But I was a lot better on cheating on video games. And that's sort of how I got into it basically.

ABDEL SGHIOUAR: Nice.

JESPER LARSSON: And then for my career, I actually started as a sys administrator. But this is in the before, this is like 20 years ago. So then I was a sysadmin, but a lazy one. So I started with automation very early. I don't want to do monotonous things, and that sort of got me one foot into scripting or programming, you could say, but task programming basically.

ABDEL SGHIOUAR: Yeah, and that was sysadmin before it was called DevOps, right?

JESPER LARSSON: Yeah, exactly. Yeah, yeah, basically. And I had no idea that you could actually make money breaking things. That came years later when I got introduced to consultancy as basically a network administrator of sorts, one could say.

And then there was a security group that actually did penetration testing. But this is like 15 years ago. So no one knew what penetration testing was. So you sort of got us for free when you did QA testing. It has evolved a little bit since then. But, yeah, that's basically I got started to this.

And then in the beginning, it was everything from application security and infrastructure security. It was a little bit about everything. The whole security field was sort of meshed into one thing. So you could do penetration test one day, and then look at governance and risks within a company. So it was really broad back then.

ABDEL SGHIOUAR: Nice, nice. And so just for the sake of my own sanity or my own knowledge, when people say penetration testing, what does that mean? And how is it different from other offensive security methods?

JESPER LARSSON: Yeah, you test a lot of Bic pens, and ball pens, and-- no, that's not true. Yeah, so penetration testing in my definition of it is that you try to be the antagonist. You're trying to be the hacker that you don't want in your system. So you pay a group, or a person, or a company to try to break into your company, basically, through the digital space. So you try to find weaknesses, flaws, misconfigurations in code basically.

ABDEL SGHIOUAR: I see. So when you get a project or a gig with a customer, does that come with a pre-knowledge of how their system looks like or is it a black box for you? You don't know how it looks like.

JESPER LARSSON: I don't know in detail, but I always get privileged access basically. So let's say I'm going to-- like a typical penetration test gig for me is me and maybe two other colleagues. We're sort of in charge for different work packages. One could be the front-end application, one could be back-end, and one could be orchestration back-end basically.

And then in terms of Kubernetes or infrastructure as code, or modern topology of an application, we get repository access to-- it could be Terraform or any infrastructure as code repo. And when I do Google GCP-specific stuff, or AWS, or Azure, I actually have a user within the organization that I'm looking at basically just read list and get permissions, so that I can query everything but not change anything basically.

ABDEL SGHIOUAR: OK, cool. Cool. I have another follow-up question on this, but we'll get into that. When I asked you to come on the show, I was like, hey, just come tell us stories, because that's what you did basically in the meetup. I thought it was very cool. So yeah, let's go for it. Tell us something that you had to do and it was cool and surprising, I guess.

JESPER LARSSON: Yeah, we can give the audience some background story about this. So we met at a Cloud Native Computing Foundation thing, where I was sort of tasked in order to shed light on what modern architecture or orchestration is all about in the terms of hacking it and the security aspects of it.

And then, I just more or less went with one bug that I found, and then I gave the audience the sort of unfiltered version on getting from an application vulnerability in order to take over an entire organization within a cloud. And that is basically a home run for me. That's always nice.

But, yeah, so when the cloud was a thing, or became a thing, it opened a whole like plethora of nice things for us hackers, because everything is connected basically. Everything is sort of in the same bucket of sorts.

So very common is that web vulnerabilities, web application vulnerability will always be a thing, right? When we write custom code, we will write bad code. That's just how it is, because we're not very good at it. Even though ChatGPT or the large language models maybe help us a bit, but it's sort of based on statistics of internet or training and that will become shitty at some point anyway.

ABDEL SGHIOUAR: It's like a never-ending circle. It's an ML model that is training itself on shitty code. And it just keeps going in circles, basically.

JESPER LARSSON: Yeah, and I've done this for like 15 years now, which is insane. But basically the same thing that we found back in the day is true now, but in a little bit longer attack chains basically. So the whole injection vectors are still there, and they will be the initial point of compromise. And then we added so much layering on top of our services that makes it really convenient to hack. So we can-- let's just do an example.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: I did a banking-- software as a service is a new thing, right? So every cool company nowadays needs to have a software as a service product basically. And the one that we got tasked to hack was a software as a service authentication shim for banks. So what they do basically is that they provide you with all the fancy authentication schemas out there. It could be really cool SAML federated access, or OAuth, or you could log in through Facebook, or Google, or whatnot, the whole shebang.

And what they do is they do sort of standardize how you do that, which helps the development team a lot in order to create templates, basically, on how to do authentication. That's basically their product. And then they sort of jack themselves into your ecosystem by running either an image, it could be like a container, it could be a pod, it could be whatever. But that's basically what they do. So you sign up, and you get a dashboard basically where you can build really cool authentication flows pretty fast.

And what people do is that they install this image, because it's a signed image. Everything looks good. The application lives inside of that image, and you build your logic flows to that authentication endpoint basically. But then again, everything is linked and connected to each other.

So in this case, we found a really nice vulnerability where you can extend the configuration on the user side of things where you could create a shell, a very rudimentary one liner in Bash. And how did we find that? Everyone when I tell this story, because this is so insane, is that we have full repo access. So we had all the code. We knew that there is a binary that is running in Python that will evaluate input from the user. And that was very easily escaped.

And then, a lot of people-- distroless containers are nice, but a lot of people don't use distroless containers. So there was a full Alpine in there, which led us to the nice box of everything you need in order to get persistent.

So in this case, we just piped it to a really simple reverse shell, and we get a backtick, and we have a shell within this workload where we don't really know where we are. But we know that when you get back and you sort of do ID or whoami or whatnot, we see that we are root, of course. Because root containers are nice.

ABDEL SGHIOUAR: Of course.

JESPER LARSSON: And that gives us the idea that, yeah, what if we can reach the internet? And of course we can reach the internet, because egress traffic filtering is hard. Because, yeah, I don't know. And then, we're able to use cURL, because cURL is, of course, installed in the Alpine distribution.

ABDEL SGHIOUAR: Of course.

JESPER LARSSON: And then we could just cURL-- maybe we could cURL the Kube control binary? And yes, we can. And then in this case, they had appended access to the default service token. And that is auto mounted by default, right? And they had-- they'd sort of given us access to not all of the secrets, but enough secrets for us to find a service account within the cloud.

And that was not really privileged, but we were able to read things. And through that, we were able to read tfstate file, the Terraform state file. And there you can see all the deployment of a service. And that was a pretty good jackpot, because those were containing a lot of nice things, and they were not purged.

And one thing led to another. We were able to escalate privileges throughout the cluster in order to take over the entire infrastructure from one application vulnerability that the client really didn't have any idea about. So the client in this case was this software as a service provider. But by proxy, we would be able to hack all their clients, which is all of the largest banks in the world, basically. A lot of large banks use this product. So that was like, whoa, this is insane. I'm not going to give you any more specifics on this, because then it will be very easy to Google the specific company.

So I did a talk about this basically, and then I added some of the sources in my talk. And then I just googled it, and it was like the third-- the API requests were really specific. So it was like the third result, so I was like, nope, I have to remove this altogether. But yeah, and I would say this is pretty common that you don't really-- I don't think you have the understanding on how a modern infrastructure works basically.

ABDEL SGHIOUAR: Because it's so complex.

JESPER LARSSON: Yeah, yeah. So if you get access within a workload these days, or a pod, or container, or call it whatever you want to, then we have to have an idea on what's next.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: Because that's a thing that I see a lot. Once you get an initial access, there is very, very rare cases where people are defended, or even know that someone is in there.

ABDEL SGHIOUAR: Yeah. So to recap, so it started with an unfiltered user input in an interface, which then you input a shell, which got evaluated and executed on a Linux pod.

JESPER LARSSON: Yeah, living inside of the client's cluster.

ABDEL SGHIOUAR: Yeah, which was not distroless. And, of course, had root.

JESPER LARSSON: Yes.

ABDEL SGHIOUAR: And, of course, egress to the internet was not filtered.

JESPER LARSSON: Yes.

ABDEL SGHIOUAR: So that you could download kubectl, and then use the default service account in the namespace to just call the API and say, give me all the secrets, right?

JESPER LARSSON: Yeah, basically. We were not able to get all the secrets. We were getting enough secrets in order for us to pivot to a storage object. I won't name any cloud providers. It was not Google, this case, actually, but it wasn't--

ABDEL SGHIOUAR: You don't have to mention one. But from the way you're describing it, I would assume if such a workload would be running on GKE, it would be exploitable the same way.

JESPER LARSSON: Yeah.

ABDEL SGHIOUAR: Because a lot of these defaults you're talking about, everybody has them because that's just how Kubernetes works.

JESPER LARSSON: Yeah.

ABDEL SGHIOUAR: Right.

JESPER LARSSON: Yeah, basically. And it basically looks the same. I would say the outlier here would be Azure. They have a flavor that I'm not really into just yet. But AWS and GCP is basically the same.

But Azure is a little bit different, and not for the better thing. These are really complex products. And I think an educational standpoint here would be beneficial. I don't favor anyone. But I think that AWS and Google is the ones that are most mature in the realm of Kubernetes now in my opinion. This is just my opinion. You can shoot me later.

ABDEL SGHIOUAR: I'm not saying anything.

JESPER LARSSON: But, yeah, the problem is that it is hard to do it correctly.

ABDEL SGHIOUAR: Yes.

JESPER LARSSON: I can give you an example that I thought about the other day when I was configuring my UniFi stuff here. I have Ubiquiti's ecosystem. And I'm pretty good at networking, low level networking. I know my things.

Normal protocols and shenanigans that goes on in the network stream are called something completely different within the UniFi space. They sort of invented their own names, which makes it problematic. And that is basically the same within the orchestration space, because we name things weird stuff and we don't explain what it is. So people tend to not use it or don't understand what they're doing. And that is problematic. So misconfiguration and insecure defaults are basically what I make my money on.

ABDEL SGHIOUAR: [CHUCKLES] Awesome.

JESPER LARSSON: To be honest like, OK, I find a bunch of stuff, but it's like, basically that is the gist of it.

ABDEL SGHIOUAR: Yeah. So then, I have a question related to that. When people talk about security, one of the things that comes very often is this concept of zero trust. It's inside a distributed system like Kubernetes.

Zero trust usually means do not allow things to just talk to each other randomly, have maybe some network policies, or firewall rules, or maybe go further and have authorization and authentication between microservices in the same cluster, or environment, or whatever. So the question is, realistically, from your experience how much people do that? Is it a thing that people do or--

JESPER LARSSON: No one, I would say. I have a bunch of clients that are good, like really good, but there are very few on-- I would say like 1% or 2%.

ABDEL SGHIOUAR: OK, so basically people build like thick walls around their houses. But inside, it's just like one gigantic open space.

JESPER LARSSON: Basically that would be the case. I think it's better now, like namespace isolation and there are concepts with like service meshes that are out there that sort of visualizes stuff, like ingresses. There is a bunch of service meshes and there is a bunch of ingresses now that you could use that helps you visualize access and how data can flow. In terms of zero trust, I think that there are people trying to create a concept of zero trust, but it's hard. And if you're able to do that, I think you are very mature overall as a vendor, or as a service, or as a company basically.

And it is a hard thing to solve. I don't think people really do their risk assessment on what they're actually exposing and what parts of the infrastructure will be critical if they leak or not, if that makes sense.

ABDEL SGHIOUAR: I see, yeah.

JESPER LARSSON: So separation would come a long way. And I tend to say to people, is it reasonable that a web application, one, cURLs down kube control, two, connect back to a server in Sweden? Nope. That is not a normal behavior. No, no.

So that should be flagged. Monitoring, we should really look at monitoring. We should really look at anomaly. Is it reasonable to call out-- do we need egress traffic? Does it really need to connect to the internet? And the sad story there in the websphere right now is, yes, because everything is decentralized. There is like you host your node application, and then you have eight CDNs and a database back-end somewhere else. So you have to have a really, really decentralized infrastructure, basically. And that makes it a lot harder, but a lot easier for an attacker to get an initial foothold.

ABDEL SGHIOUAR: Yeah, yeah, it also sometimes boils down to kind of developer friction, right? So let's take the example you were talking about, which is the egress traffic. I worked on projects where customers go like, we're going to put a proxy and we're going to only whitelist certain URLs. And then, you have a developer coming in deploying something.

And maybe the developer doesn't even know what URLs their own app talks to. So they run their thing, it crashes, and then they go like, well, I don't know what's going on. And then you have to go and debug like, OK, where is the problem? Well, this URL is not on the list, right?

JESPER LARSSON: Yeah, I think that boils down to it. Everyone wants convenience and are a bit lazy, I guess. So distroless containers is good. Workload identity in GCP, for example, is nice, because then you have an-- that's another topic. We can come into that later. But that's a different story.

But in terms of usability, that goes both ways. So it's not only the native application, it's also usability for a developer to ship fast, and be able to deploy instantly, and have access to debug, and whatnot. And there are a bunch of solutions for this, of course. You could have QA testing, dev, and then go to prod. And prod should be hardened basically. So then you could have a lax security setting during like dev and staging basically. And then on production, you should really have like multi builds for your container workloads and whatnot, so you drop privileges and whatnot.

But that is hard. And I don't think we have good enough common practice. Not best practices, that's not a word I want to use. Good enough common practice to make that happen. And we can see that in Terraform, which is immensely popular now, that the templating that Terraform provides is basic. It's just the generic configuration examples that you need in order to get it up and running. And that is a problem as well, because it doesn't dictate the beginning and the end of the actual thing that you're deploying with that template, if that makes sense.

ABDEL SGHIOUAR: Yeah, it makes a lot of sense. Yeah.

JESPER LARSSON: But that's one thing that I see that people-- yeah, the "just get it to work" phase is always-- that's the norm now. Not in depth what's going on.

ABDEL SGHIOUAR: I think we can add to that also what I call the stack overflow effect, which is just go copy random code from Stack Overflow and run it.

JESPER LARSSON: Exactly.

ABDEL SGHIOUAR: And as long as it works, you are happy, and then you move on with your day. You don't have to even understand how it works.

JESPER LARSSON: Yeah, I think that is a thing. Maybe it's because I'm getting old now, but it's like, that's a thing as well. A fresh developer now, everyone is a full stack developer from the start nowadays. So that's a weird sort of thing. And it's like, network layer, that's just a package you add, and then you have networking, of course.

ABDEL SGHIOUAR: Yes.

JESPER LARSSON: And for us that's been around for a bit, it's like, yeah, that's true, but there is a lot of stuff going down in that stack as well. So it's like-- yeah, I'm not sure we're progressing to the better ends of security.

ABDEL SGHIOUAR: The good thing here is that you will always have work, right? Because as long as people keep screwing things up--

JESPER LARSSON: Yeah, yeah.

ABDEL SGHIOUAR: --then there will be a need for Jespers to come fix them, or to come break them.

JESPER LARSSON: Yeah, exactly. Exactly. But the thing is it's ever-moving and shifting very fast, as well. There is new features pushed out every day basically and the services change. We branch out certain features, and then we discontinue those, and then we add new ones.

And so it is a space that is-- I've seen a lot of development throughout the years, which is both good and bad. I think the whole open source thing is actually a good thing in my opinion. In the before time, I tend to always use licensed software basically. And now, I shifted completely around, because I think open source is a lot better in terms of, there's a lot more eyes on the actual code, which makes it progress.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: That's why Cloud Native Computing Foundation is a good thing, because we will help those projects get traction and help financially in order to evolve and make the internet to a better place in my opinion, which I think is good. But then again, if we're changing and moving very fast, it's hard to keep up.

ABDEL SGHIOUAR: Yeah, of course. Yeah, yeah, that's a very valid point. Everybody is playing a catch-up game. And I think that-- not to also point at developers as the major cause of all these problems, everybody is also under lots of stress to just release stuff very quickly, because it's a very competitive market and you have to be out yesterday.

JESPER LARSSON: Yeah, exactly. Exactly. So that's what I tend to do in the-- when I have a few clients that are not run through the Cure53 space, which is my normal day-to-day work. They sort of do 80% of my revenue comes from the Cure53 crew. But the other stuff is, I have a bunch of Swedish clients that I do regular [? reaudits, ?] basically in the cloud realm. And then I have a bunch of big companies that hire me every year to do an audit basically.

And what we do with the results of the audit is that we workshop around them. So in this case, it is in the cloud space, of course, and Kubernetes specifically, because that's sort of my thing. But then we take all the application teams, the cloud teams, the network teams, and everyone sits in, and we talk about the findings and how they correlate through all of the teams, so everyone gets an understanding what's going on. How can this be a problem, and why.

So you don't create a space where you have this whole blame game. It's like, oh, it's the front-end guy's fault, or it's them. So everyone takes ownership and understand, like, oh, this is how everything is connected. And then you sort of learn a lot more than just reading a report, and then just patch the vulnerability. That's something I think is really good, because then you get more of an educational thing going.

I think that is a good thing-- learn by creating an environment where it's OK to be-- not know what's going on and learn from each other. I think that's good.

ABDEL SGHIOUAR: Yeah, that's a very good point. Besides the stuff we talked about, like the root, and distroless and some of the other stuff, any other kind of common mistakes that you see out there?

JESPER LARSSON: Yeah, common mistakes. One thing that still amazes me is ConfigMaps, or just read the manual for ConfigMaps. It says like in the third line that this is not a store for secrets.

ABDEL SGHIOUAR: Oh, yeah?

JESPER LARSSON: Yet again, everyone deploys stuff to the ConfigMaps that-- and the weird thing with that is that it's readable by everyone, because it's a configuration map. So it's like, don't do that. That's really bad. That is really, really, really common in my opinion.

That has to end. Even though you don't use the ConfigMaps, but then you sort of added a service account to your cloud there some-- yeah, some day or some time in the past, you added it. Remove it.

Don't let it be there, and remove the revision of it as well. So it should not be there. It's like pushing secrets with git. It's really bad.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: That is really common, I would say.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: But then the thing that I see now that is trending is the whole misconception of IAM concepts, basically. And especially when we have-- we've sort of created, we bridged the gap between the Kubernetes cluster and the IAM roles within our native cloud space, that creates a lot of confusion, and a lot of privilege escalation vectors sort of arising from that.

ABDEL SGHIOUAR: I see.

JESPER LARSSON: Not only through Kubernetes, but a lot-- different services. Since if you are an adopter of cloud, you know that you ship everything to the cloud, and then there is a certain lock in factor, because you want to have it as efficient as-- as low cost as possible, if that makes sense. So then you tend to, OK, we had this computing instance, but that's really expensive now, so we're going to move it to Cloud Functions, or to Lambdas, or to some kind of serverless idea, and then we port it to that.

But then, we need to have connection and backwards compatibility, or a back connect to this service and whatnot. And then we start to create this IAM policy, or groups, or whatnot, that we don't really know how it works. And then we sort of end up using wild cards. And this is a tip for everyone who's listening, if the solution is to use a wild card to get it to work, you're doing it wrong.

ABDEL SGHIOUAR: [LAUGHS]

JESPER LARSSON: So if a star is there, it's wrong. It's like regular expressions. Don't do it. And that is really common nowadays, as well. And it is a little bit of a black magic with cluster binding roles, and cluster roles, and resource roles. It is a little bit of a weird thing.

ABDEL SGHIOUAR: Yeah, RBAC is very complex in Kubernetes, yeah.

JESPER LARSSON: Exactly, exactly. But then you could use-- yeah, the Rapid7 tool, the RBAC analyzer, where it sort of gives you a graphical interface of how every group and action is connected, which is really helpful, which everyone can install. It's in Krew, the package manager for Kube control, and then you can install something called-- I think it's RBAC visualizer, or RBAC-- yeah, just search for RBAC, and you will find it.

ABDEL SGHIOUAR: You will find a name for it, yeah.

JESPER LARSSON: And that's really helpful, because then you will see everything throughout the eyes of the cluster. So you can see how the IAM policies are delegated down through the Kubernetes topology, and see what access are mapped and what not. And how it correlates with service accounts and whatnot in the cluster, which is pretty helpful. Because that is a hard thing to figure out, I guess.

ABDEL SGHIOUAR: Yeah. I mean, that's a very valid point. The whole IAM story is-- I know that Google has added something called IAM Recommender, where they would analyze the permissions behind and say like, hey, this is an overprivileged account or whatever. But it's very difficult, right?

JESPER LARSSON: Yeah, yeah.

ABDEL SGHIOUAR: Being able to provide recommendations in an efficient way means also being able to snoop on the traffic to see what that account does. And then you start getting into the realm of, oh, is this OK privacy? Are you looking at my API traffic?

JESPER LARSSON: That's a valid point.

ABDEL SGHIOUAR: Yeah. So this whole concept of absolute security doesn't exist. And there is always this fine line between security and privacy that you have to balance, right?

JESPER LARSSON: Yeah, especially as a cloud provider. It's hard.

ABDEL SGHIOUAR: Yes.

JESPER LARSSON: Yeah, and I don't want to shift seats with the cloud providers ever, because it is a hard problem to solve. Yeah.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: It's not an easy fix.

ABDEL SGHIOUAR: Yeah. Cool. So what else can you recommend, I guess, the audience? How can they do things better in general?

JESPER LARSSON: That's a big question. But in general, I would say you have to know your product, know what it does at the back end. Just know what path your application use network-wise and how it lingers around in your Kubernetes topology.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: Do you have to have access rights to all the namespaces? Does it have to call the internet that we talked about before? And then I would say to have multi builds with your container loads. If you're going to build something and ship it to production, I think it would make sense to build it as root and then drop the capabilities when it's in production so you don't need a bunch of high privileges, or you can't even elevate your permissions within the container or the pod or whatever. I think that makes a lot of sense.

And then there is a bunch of nice projects like Falco and Sysdig, which are open source, that you could use in order to visualize what's going on within the cluster and scan your images and whatnot.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: And there is a really cool project that I have looked at a little bit called Wolfi from Chainguard, which is not distroless container, but it is secure context, basically, for containers. So they have a bunch of secure configurations in the form of YAML files that you can use in order to validate what you're actually running. And they have a bunch of those for things that developers tend to use, like Nginx or Node or Java or whatnot.

That is something I really believe in. So they give you a hardened base plate that you could use. And then you have SBOM features and whatnot in that one so you can see what you're using and not using.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: So I think visibility is a good tip here. Try to audit what's actually going on within the cluster.

ABDEL SGHIOUAR: Yeah. And if I may add to that, the kubectl does have a debug method. So you don't actually need to run a distro-full container. You can just attach a container to debug your app and then detach it after you're finishing debugging, right? So--

JESPER LARSSON: That is a valid point. That's a really good point as well.

ABDEL SGHIOUAR: Because that's what we talked about is the convenience. People do things with a distro because it's convenient to just kubectl exec into the container to try to figure out what's going on, right? But then, yeah, you're shooting yourself in the foot, especially if you add root to that.

JESPER LARSSON: Yeah, yeah. And, yeah, people are lazy. So that will happen. But the debug flag is actually good and useful.

ABDEL SGHIOUAR: Yeah, yeah, yeah. Cool, well, Jesper, this was really cool. Thank you for sharing the story with us. As you were talking, I just remembered our time back in Guteborg when I was listening to it live. I had more details.

JESPER LARSSON: Yeah, a lot more details. But, yeah, they'll have to come another time when we meet.

ABDEL SGHIOUAR: Yeah.

JESPER LARSSON: But thank you so much for having me. It was really a pleasure to be here.

ABDEL SGHIOUAR: Awesome. Where can people find you?

JESPER LARSSON: Well, you can find me the platform formerly known as Twitter. Is it Xwitter, or what is it?

ABDEL SGHIOUAR: X.

JESPER LARSSON: I have no idea. No one knows.

ABDEL SGHIOUAR: We're trying to figure out a name for it.

JESPER LARSSON: Yeah, but there I'm @HerrJesper. So it's H-E-R-R Jesper, which is "Gentleman Jesper" in Swedish.

ABDEL SGHIOUAR: Yeah, "herr" is gentleman in Swedish, yeah.

JESPER LARSSON: Yeah. Cool. And you can find me ox4A.se-- that's my home page-- and yeah.

ABDEL SGHIOUAR: We will have all the links to your social media on the show notes for this show.

JESPER LARSSON: Oh, nice.

ABDEL SGHIOUAR: Thank you very much for being with us, and I hope you have a good day.

JESPER LARSSON: You too. Thank you for having me.

[MUSIC PLAYING]

KASLIN FIELDS: Thank you very much, Abdel, for that conversation. You certainly picked an interesting guest this time. I don't know of many folks in the tech community who I know have an IMDb page, especially for a TV show related to their specialty in tech.

ABDEL SGHIOUAR: Yeah, it's very interesting. I watched, actually, the TV show. It's in Swedish, but you can get subtitles in English.

It's really, really cool. It's a mixture of TV reality meeting people. It's funny because you know the joke about movies is generally speaking, whenever Hollywood wants to betray a hacker, it's usually green text on top of a black window, right? But this is actually real hackers, people doing that as their daily job.

KASLIN FIELDS: Yeah. That would be fascinating. I feel like the US has never attempted a reality TV show about actual hackers. And if they did, I feel like they would be sorely disappointed at the amount of drama in it.

ABDEL SGHIOUAR: Yeah.

KASLIN FIELDS: Which should be the joke, but I don't know if people would do that right. If anyone has done that, please let us know.

ABDEL SGHIOUAR: Yes, please.

So I met Jesper in a meetup in Gothenburg, Goteborg, which is the city on the west coast of Sweden. We were both invited to a meetup, but we didn't know each other. It was basically the organizers just invited both of us.

So I went first, I did my talk. Then he went after me, he did his talk. And I was like, I have to get you on the podcast.

KASLIN FIELDS: And this is one reason to go to a meetups.

ABDEL SGHIOUAR: Oh, yes.

KASLIN FIELDS: You meet so many interesting people.

ABDEL SGHIOUAR: Yes, yes. That's actually the main reason why I go to meetups is networking.

KASLIN FIELDS: Yeah, fantastic networking. I remember the days. Maybe one day I will get back into the meetup scene in Seattle.

ABDEL SGHIOUAR: Yeah. I mean, you should.

KASLIN FIELDS: Yeah. So there are a number of things that you all talked about that are very interesting. And one thing that's always funny with security conversations, especially for someone who is a professional attacker, is you can only say so much about the situations that you're talking about that you want to teach people about, which definitely played in here. But still, the story about the bank software that he worked with was very interesting.

ABDEL SGHIOUAR: Oh, yes. What's becoming very interesting with security these days is that basically, attackers are targeting third-party software companies, because they know once they hack them, then whoever uses that software will become vulnerable, right?

KASLIN FIELDS: Yeah. That kind of occurred to me as he was talking. But hearing you say it out loud, yeah, that makes a lot of sense.

ABDEL SGHIOUAR: I mean, you remember the Continental Pipeline in the US last year, that whole company that was-- it's the company called Continental Pipelines. They supply the East Coast of the US with oil.

KASLIN FIELDS: Oh.

ABDEL SGHIOUAR: And they were actually hacked with a crypto heist, locked hard drives, like, pay us Bitcoin so we can unlock your hard drives.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: And that was due to them using a third-party software that was actually hacked. So it was not them. It was somebody else's software they used that had a vulnerability in it. And stuff happens. Things are connected to the internet, so.

KASLIN FIELDS: We should actually look at these stories more often. They always have something interesting in them.

ABDEL SGHIOUAR: Yeah. We will try to leave a link in the show notes. There is a very, very super long article about what, in my opinion, was the first evidence of these sort of attacks happening. So there is a company in Denmark called Maersk. Maersk is a freight company. They move containers around the world. They are responsible for 1/5 of the world freight.

And back in 20-- I think, '17 or '18, it was, they got hacked. And the way they got hacked is that somebody hacked a software that people use to declare their taxes. So somebody injected a backdoor in the software. Somebody in Maersk downloaded the software, installed it on their computer, and it's a worm. So it spread inside the entire network and shut down all their IT.

KASLIN FIELDS: Gosh.

ABDEL SGHIOUAR: And the article is on "Wired." It's super detailed because the journalist went and actually did interviews, anonymous interviews with the people who were there when that happened. So they were shut down for two weeks.

KASLIN FIELDS: Wow.

ABDEL SGHIOUAR: And what was interesting is that because they were shut down, apparently, in ports these days, the trucks they have automated license plate scanners. So there's a camera that reads the license plate and then tells the driver where to park to leave the container, right?

KASLIN FIELDS: Mm-hmm.

ABDEL SGHIOUAR: That was down for two weeks. So there was lines of trucks in ports around the world--

KASLIN FIELDS: Oh, wow.

ABDEL SGHIOUAR: --because they couldn't check them in, right?

KASLIN FIELDS: Huh.

ABDEL SGHIOUAR: Yeah. It was very, very interesting. I remember I read it on a flight from, I think, Stockholm. It was to Poland for two hours and a half. And I was dived into the article, because there was so many details in it.

KASLIN FIELDS: Wow. Talk about ways that software affects the real world.

ABDEL SGHIOUAR: Oh, yes. Oh, yeah. And we're only going to see more of those going forward.

KASLIN FIELDS: And also a continued slight departure from our main conversation here about your conversation. But I saw something recently about an attack where an anonymous attacker group, I think-- not Anonymous, but an anonymous attacker group-- attacked a company. And then the threat involved with it is that they told, like, a government agency that they attacked this company and that they hadn't told the government agency yet. And there's laws around when you need to--

ABDEL SGHIOUAR: Disclose.

KASLIN FIELDS: --announce that there was a breach.

ABDEL SGHIOUAR: Yeah, yeah.

KASLIN FIELDS: And so they're getting the government involved to punish the people that they attacked for the attack. So that's a very interesting thing.

ABDEL SGHIOUAR: We're living in a very interesting space in terms of security. It's going to be only interesting going forward.

KASLIN FIELDS: And I loved how you all brought the community in at the end there to talk about how this affects the technical community here. Because there are so many folks these days, like you all said, everyone's kind of expected to be a full-stack engineer these days.

ABDEL SGHIOUAR: Oh, yeah.

KASLIN FIELDS: People have so many responsibilities and so much pressure on them to move so fast. Everyone can't understand all of this stuff when they need to understand it.

ABDEL SGHIOUAR: Yeah, yeah.

KASLIN FIELDS: And so all of these, like Jesper was saying, the same misconfiguration issues and the same mistakes happen over and over again because, I mean, everyone can't know everything.

ABDEL SGHIOUAR: Yeah. And also, cloud is only becoming more complex, right?

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: Not only Kubernetes itself, cloud in general. To do one thing, you need like six services. You need to wire them together properly, and you have to have IAM and firewall rules and all this stuff. So yeah, we live in an interesting space.

KASLIN FIELDS: And one issue that you all kind of alluded to but you didn't fully dive into is that because folks are so overwhelmed, and there's so much that they need to know and don't know, and they have to move so fast anyway, a lot of the time, folks end up using example code provided by, like, Terraform.

ABDEL SGHIOUAR: Oh, yes.

KASLIN FIELDS: You all talked about the Terraform examples, or provided by the cloud providers. Wherever they find examples, they take those examples and they just build on them without considering the limitations of those examples.

And that is a huge problem across the industry where folks are using these tools, which inherently use security issues, and then they don't go back and fix them, because they don't understand them, because they were just built into the templates that they are building off of.

ABDEL SGHIOUAR: Yeah. I call that the "Copy from Stack Overflow Syndrome."

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: Just copy code. Put it in your editor. Run it. If it runs, don't even touch it again.

KASLIN FIELDS: And it's a really hard problem to solve.

ABDEL SGHIOUAR: It is.

KASLIN FIELDS: Jesper was saying, the templates are very basic, and they need to incorporate more-- I like the term "common practices" instead of "best practices," what should be common practices.

ABDEL SGHIOUAR: Oh, yeah.

KASLIN FIELDS: And our team at Google Cloud did a project this past year, the Jumpstart Solutions, in an attempt to kind of do that. And it's just so hard to incorporate what should be common practice for security reasons or best configuration efficiency reasons. Whatever the reasons may be, incorporating that stuff into samples is really difficult because a lot of it is very contextual. It depends on the application that you're trying to deploy.

And so the examples get more and more specific and less and less usable by a large number of people. And so then you need more samples, and then you've got to maintain all of those samples. And it's just a self-perpetuating problem. It's really, really hard to solve.

ABDEL SGHIOUAR: It's a real chicken and egg problem. You know, when you were talking, you reminded me of the interview that I did which has not aired yet, but it's coming. It's NAV, the Norwegian Welfare people.

KASLIN FIELDS: I'm so excited about that one. It'll probably be close to the holidays this year, we'll release that.

ABDEL SGHIOUAR: Yeah. So one of the questions I asked-- I don't want to spoil the episode. But basically, they are building a platform that runs on top of Kubernetes, and they allow their developers to use it, kind of like a platform engineering type thing. And one of the questions I asked them was, so you work with, like, 800 teams. So what if some teams have a feature, have a request for a specific thing?

The question was, what's the threshold of how many times people ask for things that will prompt you to go and build it instead of having the individual team figure it out? And the honest answer I got from one of the guests was, like, well, you know what? Like, 99% of our people just need the same thing. They need a database and a place to run their app, and that's it.

They don't care-- which was a very interesting answer because they are basically on the platform side. They're sitting on the receiving side of all this, like, oh, we need this. We need that. We need this. And basically the answer was like, yeah, no one really-- no one is unique. Everybody is basically doing the same thing.

KASLIN FIELDS: Yeah. We talk about that all the time in the tech space, the 80% case.

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: Make sure that you're serving 80% of users, and those other 20% need so much more work to get the little specific details that they want.

ABDEL SGHIOUAR: Exactly. I think we talked about this also with Nick in our platform engineering interview. It's super interesting how everybody thinks that their use case is super unique.

KASLIN FIELDS: And in some ways it is.

ABDEL SGHIOUAR: But at the end of the day, everybody is copying same code from Stack Overflow.

KASLIN FIELDS: Yeah, but in the end, it's probably going to be the same as a lot of other stuff.

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: But just getting to that point of figuring that out is so difficult, so.

ABDEL SGHIOUAR: Oh, of course. Yeah, yeah. And now what's going to-- I mean, what's interesting as well is that we live in this space of AI where--

KASLIN FIELDS: Oh, yeah.

ABDEL SGHIOUAR: --people don't even bother checking Stack Overflow. They just ask a chat bot, like, generate code for me.

KASLIN FIELDS: Very curious what kinds of new or old problems this is going to introduce into code bases. Because in my efforts playing around with AI so far, it's fantastic at giving me what I need to get started. It'll give me a really nice template, like for a YAML file, for a resource that I need to spin up in Kubernetes, or something like that. But its purpose, really, at this stage is for AI to give you something that looks right as an answer.

ABDEL SGHIOUAR: Oh, yeah.

KASLIN FIELDS: That's about all it can do right now. It's not really meant to give you exactly the perfect answer every time. It's meant to give you something that looks right. And so it'll give me the right format, and it's a great thing to jump off of. But it'll-- we say "hallucinate." It'll just kind of make up the parts that it doesn't know so that it looks right in the end.

So for example, I tried making a Kubernetes resource definition, and it put in just a random container image. It looked like a container image call, but it was completely made up. It's not a real image at all. [LAUGHS]

ABDEL SGHIOUAR: So it showed you what it thinks it should look like.

KASLIN FIELDS: What it should look like. And it was absolutely right. That is what it should look like. But it was not going to work if I ran it.

[LAUGHTER]

ABDEL SGHIOUAR: Yeah, that's very interesting. And yeah, so as you said, basically, what's going to happen is the same thing that is happening with people copying other people's code is just people are just going to blindly trust an AI generating code, and then it's a self-perpetuating problem, right?

KASLIN FIELDS: Yeah. It can be such a fantastic learning tool. But folks have to understand the limitations and that it's just giving you something that looks right as an answer. And you're going to have to go and look up some more stuff, still, to actually make it work, yeah.

ABDEL SGHIOUAR: You'll have to double-check it. Yeah.

KASLIN FIELDS: So I don't want to go too long on this. Thank you, everyone who has listened this far. But I do want to go over the common mistakes that Jesper mentioned one more time for everyone.

ABDEL SGHIOUAR: Yes.

KASLIN FIELDS: ConfigMaps-- I was really honestly kind of excited to hear him mention ConfigMaps as the first thing, like, everybody keeps doing this. Stop putting your secrets in ConfigMaps.

ABDEL SGHIOUAR: Oh, yes. You'll be surprised.

KASLIN FIELDS: Yeah. I have done some content, of course, around ConfigMaps. And the first thing we always say is do not use ConfigMaps for secrets. Maybe don't even use Secrets for secrets because that's--

ABDEL SGHIOUAR: Yeah.

KASLIN FIELDS: --still dubious.

ABDEL SGHIOUAR: Probably. If you can avoid it, that would be nice, yes.

KASLIN FIELDS: Yeah. So I thought that was really funny that that was the first thing that he mentioned. And then another one is also something that I've done videos on, was IAM concept misunderstandings.

ABDEL SGHIOUAR: Oh, yes.

KASLIN FIELDS: And specifically he mentioned the difference between Kubernetes role-based access controls in IAM and cloud role-based access controls in IAM, which is really interesting to try and understand. In Google Cloud, we have a concept that the name of-- I'm blanking on right now-- but the concept where we tie your Google Cloud IAM to your Kubernetes.

ABDEL SGHIOUAR: Ah, Workload Identity?

KASLIN FIELDS: There you go. Thank you.

ABDEL SGHIOUAR: Yes, yes, Workload Identity. Yeah.

KASLIN FIELDS: Yeah. So we have that concept, which was really hard for me to wrap my head around at first. But it's in this space.

ABDEL SGHIOUAR: Yeah. It's interesting that it doesn't seem like any cloud provider, including Google Cloud, have solved this mapping of IAM to IAM inside Kubernetes. So most of what cloud providers do is that they give you some predefined roles in IAM, which map to some RBAC roles that already exist in Kubernetes.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: But they cannot map everything because RBAC is huge, right?

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: So it's actually so complex that there are companies that have built products just to solve the RBAC issue inside Kubernetes itself, right?

KASLIN FIELDS: So hard. I'm really hoping that AI can actually help with that to some degree.

ABDEL SGHIOUAR: Oh, yes. That's probably something where AI can help, yes.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: I think, by the way, I don't know if you know this, but on Google Cloud, we have this thing called the IAM Recommender.

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: That's actually ML-based, by the way. That's machine learning-based.

KASLIN FIELDS: I haven't actually tried it out myself yet. But I'm very excited about the concept if you could explain.

ABDEL SGHIOUAR: Yeah, it's simple. It basically analyzes what a certain identity does, especially service accounts and users. And then, based on the audit logs, it will tell you if a certain role or a certain user has an overprivileged account, right? So it basically looks at what roles do you have and then analyzes what kind of actions do you actually do in your project. And then if it sees that you are only viewing logs, for example, but you have an editor role, it will tell you, OK, well, this account does not need an editor role, because all you do is just view logs, right?

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: And so it's a combination of audit logs and then also the ML-based model that makes these recommendations.

KASLIN FIELDS: Awesome. So we have that as an attempt to address this problem for Google Cloud users. And I'm sure that all of the other cloud providers are also trying to find ways to make this a little bit easier for folks to understand, because it is really hard.

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: And then the last piece of big advice that Jesper mentioned was know your product, which I feel like is such simple advice for such a difficult thing.

ABDEL SGHIOUAR: Yes. I mean, you have that "I'm going to use this thing just because all the big fan companies are using it."

KASLIN FIELDS: There's a lot of that.

ABDEL SGHIOUAR: And that's actually a real problem with Kubernetes specifically.

KASLIN FIELDS: Mm-hmm.

ABDEL SGHIOUAR: So, I mean, sometimes people don't like to take, like, two seconds to just stop and ask themselves, do I actually need this, right?

KASLIN FIELDS: Yeah.

ABDEL SGHIOUAR: They will just use it because they read it in some blog post somewhere or some tech influencer have told them that this is what you need.

KASLIN FIELDS: And it's another one of those victims of, everybody can't know everything at the same time.

ABDEL SGHIOUAR: Exactly. We can't all be full-stack engineers.

KASLIN FIELDS: Yeah. As much as we try, there's a lot of stuff that falls through the cracks. And we try our best to fix it.

ABDEL SGHIOUAR: Exactly.

KASLIN FIELDS: That's what Jesper does. He comes in, and he tells you what's broken.

ABDEL SGHIOUAR: Yeah, exactly. I like that quote he said. It's like "I don't fix anything. I just tell you how broken they are."

KASLIN FIELDS: Yeah. So thank you so much again, Abdel, for that conversation. We'll see folks on the next episode.

ABDEL SGHIOUAR: Thank you.

KASLIN FIELDS: Maybe we won't actually see people, because it's an audio-only podcast.

ABDEL SGHIOUAR: That's fine.

[LAUGHTER]

[MUSIC PLAYING]

KASLIN FIELDS: That brings us to the end of another episode. If you enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media at @KubernetesPod or reach us by email at <kubernetespodcast@google.com>. You can also check out the website at kubernetespodcast.com, where you'll find transcripts and show nodes and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

[MUSIC PLAYING]