CRAIG BOX: Hi and welcome to the Kubernetes podcast from Google. I'm Craig Box.

ADAM GLICK: And I'm Adam Glick.

[MUSIC PLAYING]

CRAIG BOX: Hey, Adam. What's new?

ADAM GLICK: I just got back from Velocity in San Jose last week, which is a combined O'Reilly event between their Fluent event, which is kind of a front-end event and Velocity, which is more of a back-end event. And it was a great chance to chat with some folks about Kubernetes and what's going on in the cloud, see a lot of folks in the midst of the transition, and run into some friends from KubeCon, which I hadn't seen since then.

CRAIG BOX: Any presentations stand out?

ADAM GLICK: There were some nice presentations, actually, on security that I saw that I really enjoyed. And that's probably the stuff that stood out most for me of a lot of the things that were happening around the security space. How about you, Craig?

CRAIG BOX: Not a lot. After my big international trip, I spent a well-deserved week at home just getting ready for chatting with you.

ADAM GLICK: Wonderful. I'm sure that took lots of preparation. Prepare for it all week, right?

CRAIG BOX: Absolutely.

ADAM GLICK: Awesome. Why don't we get into the news?

CRAIG BOX: After being pipped at the post by Amazon, Microsoft last week announced the general availability of their Azure Kubernetes Service, or AKS. New features include role-based access control and support for integrating an Azure Active Directory instance. Along with this, HashiCorp would like to remind you that they have support for deploying AKS clusters with Terraform along with Amazon's EKS and Google's GKE.

ADAM GLICK: IBM announced multi-zone clusters this week for the IBM Cloud Kubernetes Service along with announcing that their cloud would now offer the region and availability zones model required to host them as opposed to their individual locations offered previously. IBM Cloud Kubernetes clusters now have a feature called a worker pool, which is like a GKE node pool and is a collection of worker nodes with the same template. IBM additionally claimed that their Watson services are now running on top of Kubernetes. No word yet on a repeat "Jeopardy" performance.

CRAIG BOX: Docker hosted the American half of their biannual conference series DockerCon last week. They continue to move in the direction of big E Enterprise software vendor, with their main announcement being that federated, or multi-environment, application management is coming to Docker Enterprise Edition. This tooling can be used to manage applications and security policy for containerized applications across on-premises clusters and cloud-hosted environments. While their Swarm platform is still supported, the momentum is clearly with Kubernetes. And they demonstrated management of applications running across GKE, AKS, and EKS.

Secondly, Docker previewed Enterprise Edition running Kubernetes on Windows Server. Docker has been working with Microsoft to build container support into Windows, which first launched in 2016. Their initial work was directly with the Docker daemon, and they're working on migrating to containerd. Upstream Kubernetes support for Windows is still in beta as of Kubernetes 1.10 with the control plane still required to run on Linux.

Finally, Docker Desktop added a new graphical user interface, templates, and workflows for deploying Docker files and Compose, with a C, templates to allow developers to build containers for commonly used software, like Nginx or MySQL. All three features preannounced at the conference are promised to us in public beta in the second half of 2018.

ADAM GLICK: Cisco last week detailed more of its move into the Kubernetes ecosystem. As part of their partnership with Google, Cisco talked at their annual conference, Cisco Live, about their Cisco container platform and its integration with Google Kubernetes Engine. Cisco's CCP runs on top of upstream Kubernetes and ensures the applications running in on-premises environments and Cisco HyperFlex hardware will be able to easily migrate into Kubernetes Engine.

CRAIG BOX: Spotify, the music-streaming company, are moving from cloud-hosted to cloud native by adopting Google Kubernetes Engine. Historians among you may recall that Spotify launched a container management platform called HeliOS at the very same DockerCon that Kubernetes was announced. They are now in the process of migrating workloads running on HeliOS to Kubernetes.

At a talk at DevFest DC last week, James Wen, a site reliability engineer at Spotify, explained that in the migration process, they set up testing clusters with permissions and quotas and wrote developer documentation to communicate between the three teams involved. They built custom admission controllers for their Kubernetes environment and integrated with the Spinnaker continuous delivery platform.

After proving out the model, they are currently beta-testing self-service migration. And teams that are ready to move to Kubernetes can follow the documentation built in the first stage. Wen reports that some services are now fully off HeliOS and only running on Kubernetes. When the new platform has been deemed generally available, any new container service at Spotify will be deployed only on Kubernetes with tools for developers, such as a one-click migration from HeliOS and both vertical and custom, metric order scaling.

ADAM GLICK: And that's the news.

[MUSIC PLAYING]

Today's guest is Google Cloud product manager Maya Kaczorowski. Maya found her love of cryptography in her undergraduate number theory class, then went on to study a masters in applicable mathematics, focusing on cryptography and game theory at the London School of Economics. She worked as a management consultant on security strategy before moving to Google's encryption team and then focusing on container security. Welcome to the show, Maya.

MAYA KACZOROWSKI: Thanks for having me.

ADAM GLICK: What attracted you to the container security space?

MAYA KACZOROWSKI: Yeah, so I was working on encryption for the last couple of years, which is obviously a topic that I find very interesting. And I knew that I wanted to do something else in security. And I was looking at the industry and saying, what do I think is going to 10X in the next five years? I want to work on something that's going to have a lot of impact. And you look at this thing, containers, and it's just flying off the shelves. People can't get enough Kubernetes. They're our fanboys.

And so that seemed like an area that was potentially very interesting. And the more I investigated, the more I saw there was kind of interesting new problems that we hadn't solved yet and there's an opportunity to be a leader in that space. And that was exciting. And so I decided to start working in container security.

ADAM GLICK: What was the first thing that people should think about, from a security perspective, when they decide they want to run Kubernetes?

MAYA KACZOROWSKI: Maybe you should tell your security team that you're going to run Kubernetes. That's, like, a nice thing to do probably. Often I end up talking to the security team, who has been told that somebody else in the org is going to run Kubernetes. They weren't part of the decision-making process.

And so they're asking me-- they're saying, how do I even secure this from the get-go? What do I need to know? And there's a couple of fundamental differences between containers and VMs, and that's worth going over. So things like you don't have the same isolation for containers that you had for VMs, although they're in new projects, like gVisor and Kata containers. They don't have that traditional hypervisor, so you can't isolate workloads in the same way you would have done them with VMs.

Your threat model is slightly different in that because you don't have so much isolation, you can have multi-tenant concerns that you wouldn't have had with virtual machines.

And another major difference between VMs and containers is that containers are supposed to run a smaller host OS. They're supposed to have a minimal host OS that has a smaller attack surface than what you would find on a VM. And so you can potentially have a much more secure environment that way without having as many dependencies in your environment.

So talking through some of the biggest differences-- like, another major difference between containers and VMs or how this changes your thought model-- containers are meant to be immutable and they're meant to be short-lived. So if you want to change something in your environment, you don't just change a line of code. You actually redeploy the whole container. And what this means for security people is something actually really, really powerful. It means that you can have a true software supply chain of everything that ends up in your environment. You can actually verify everything for known vulnerabilities and for provenance before you deploy into your environment.

So the first thing I would have a conversation with the security person is, is this a different model? Some of the concepts are the same. Like, identities or slightly complicated, like networking, and secret management, and that kind of thing. But some concepts are completely different, like immutability of containers. And then in terms of practicality of what you would do first, in some sense, go look at what's built into Kubernetes. A lot of this functionality's already there-- things like role-based access control, network policies, pod security policies. A lot of it's already included as part of the orchestration platform.

ADAM GLICK: What are things that are important but people often forget about?

MAYA KACZOROWSKI: People forget that the Kubernetes is running on top of virtual machines. [LAUGHS] It's not enough to just secure your containers. You still have to secure the VMs. Or in other cases, you have to secure the application running on top of Kubernetes. So having a perfectly fine-tuned pod security policy doesn't stop you if you have some really basic OAuth top 10 vulnerability in your app. You can't think of this completely independently of everything else in your environment.

ADAM GLICK: So that brings up a good point about traditional infrastructure security versus Kubernetes security. How do you think about the differences and the similarities between what people do to secure their VMs and traditional infrastructure that may be sitting underneath Kubernetes and then what they need to do in securing their Kubernetes infrastructure?

MAYA KACZOROWSKI: I don't think it's actually that different at the end of the day. I mean, some of the pieces, like I mentioned, around isolation or minimal OSes are different. It's more powerful in how you can, like I said, verify the software supply chain. But if you had an identity management system before, you still need an identity management system. If you had an IDS/IPS before, you still need an IDS/IPS. If you had a firewall before, you still need one.

I think the frustration that I'm hearing from the industry is that not all the vendors have got there yet. Not everybody else has understood that model yet. So right now, we have customers who are asking me, what's my IDS/IPS solution? Because nobody's giving me one that I like, or I have to pay for a second one but I'm already paying for one. So that's more the thing that's different is that not everyone in the industry has caught up yet that they have to do this, and that containers are the new reality, and we have to secure them the same way that we've secured everything else.

ADAM GLICK: Do you think that the security people should be thinking about this holistically then? Or do you think it's different security people thinking about what's happening in the Kubernetes layer and what they need to do around their containers and Kubernetes versus what's happening with the infrastructure layer?

MAYA KACZOROWSKI: I think right now it's different security people. What I would like to see is that everybody and every piece of the security industry understands that anything that they've built for a VM they also need to build for a container or they need to adapt for a container. Right now I think what we're seeing is a lot of people opportunistically realizing that not everybody gets that message. And focusing on just containers as the differentiator.

ADAM GLICK: Gotcha. This week a security company called Kromtech published an article talking about Docker images with malware and specifically talking about cryptojacking. Can you talk about what happened there?

MAYA KACZOROWSKI: Sure. Kromtech's article was talking about 17 malicious Docker images that they found on Docker Hub. And the images had-- they were called something like Docker123, 321, so they looked kind of like play images. And they had some malware built into them that let you mine cryptocurrency, specifically mining Monero. Kromtech found that looking at the number of downloads and looking at the accounts that were used and whatnot for that. Attackers were able to get about $90,000 worth of Monero in the last year.

Now, in the grand scheme of things, that's actually not a lot of cryptocurrency. So it's kind of an interesting study in, like, people pulling random, untrusted images from Docker Hub more than it is a particularly successful attack.

ADAM GLICK: So you don't consider this particularly a large concern, but the concern sounds more about what's happening in people's container repositories.

MAYA KACZOROWSKI: I think that this particular incident isn't a very large concern. I think the potential of what this could have been could have been significantly worse. So 17 malicious images. Like, it could have been thousands or hundreds of thousands. And the fact that they didn't actually mine that much cryptocurrency. But what you're asking about is just, in general, yes, today if I'm going to deploy a particular application in my environment, if I don't have my own private repository, which is what we would recommend you do as a best practice, you're probably pulling public images from Docker Hub.

You're going to pull an image that somebody else wrote that isn't necessarily a maintainer of that project and that you don't actually know what's on that image. It could have malware. It could have cryptocurrency mining software. It could have-- you don't know what's going to be on it. It could have known vulnerabilities that they've purposely put there, so that they can then come into your environment later.

ADAM GLICK: Yeah. I believe there was a study that was published at an ACM Conference earlier this year. It was talking about 80% of the official images that were out there had at least one security vulnerability and some many, many more than that. Is it safe to use public images?

MAYA KACZOROWSKI: I think in general it's not. Nothing is ever secure. So it's never safe to do anything, really, but we have to do something. The vulnerabilities in public images-- so it really depends on the severity of vulnerabilities. So vulnerabilities are often categorized with CVSS scores in the CVE Database, which is Common Vulnerabilities and Exploits database.

And in that database, you'll have things that are very low vulnerability. And I'm not really concerned, as a user, if I have a low vulnerability in my environment or in my image, because it's very unlikely to be used or it has low impact, et cetera. But I suspect a lot of the vulnerabilities that were found here were low vulnerabilities. If I have a critical vulnerability, like, if I had something like a Heartbleed, then I might be more concerned as to what's in my environment.

It is best practice to scan your images for known vulnerabilities and patch those vulnerabilities where patches exist. That being said, 80% of official images having at least one security vulnerability is pretty terrifying. Either we need to, as a community, start patching these images, or give the users a way of patching these images, or really give people the tools to manage their private registries themselves, and do the patching themselves.

ADAM GLICK: Is there a project out there that helps people kind of do that monitoring and take a look at what's happening in images?

MAYA KACZOROWSKI: There are two open-source projects I can point you out. One is Clair, and one is MicroScanner. And Clair and MicroScanner both let you look at known vulnerabilities in your images quite easily.

ADAM GLICK: Awesome. We'll put a link to that in the show notes. Kubernetes has an object called the secret, and most people know that it's a convenient abstraction but not actually all that secret, at least within the cluster. How do you recommend that people maintain their secrets?

MAYA KACZOROWSKI: Yeah. So Kubernetes secrets were introduced a while ago. And in Kubernetes 1.7 onwards, you can encrypt secrets using AES CBC mode, AES GCM mode, or a secret box. And then in Kubernetes 1.10, we've introduced this concept of KMS plugins. You can actually encrypt secrets locally with a locally managed data encryption key and then encrypt that data encryption key centrally using a key encryption key typically managed in an external KMS, an external key management service.

In terms of managing secrets in Kubernetes, I would say that if you're running mostly on Kubernetes, it makes a lot of sense to use that secret object. I would strongly recommend that you use an encrypted option for that secret object, not just keeping it in the default that was there before 1.7 as a, I think it's called identity, the default. If you're not using Kubernetes as your single source of truth for your application, if you have lots of things running on compute, and server-less options, and whatnot, and this is just one piece of your environment, then it makes a lot of sense for you to have a central point of truth for all of your secrets. And in that case, you probably want to authenticate from Kubernetes to that central point of truth and get those secrets.

One popular option in this case is HashiCorp Vault. And you can authenticate a HashiCorp Vault using a Kubernetes service account.

ADAM GLICK: Great. Would this also be the category of things like an HSM, for instance, which I think of as kind of the on-prem and kind of version of the key management services that are out there?

MAYA KACZOROWSKI: Yeah. An HSM is a Hardware Security Module. It typically provides a stronger requirement, I should say, for how you encrypt data. So it would be the method that you use to encrypt secrets, whether those secrets are in Kubernetes or whether they're stored elsewhere. If you're storing secrets in Kubernetes and you're using the external KMS, if that external KMS is dependent on an HSM, then you can get that requirement. There isn't a direct plug-in and core, at least, from Kubernetes to an HSM. So if you have such a requirement, you're going to need to use an external KMS first.

ADAM GLICK: Gotcha. And when you think about the trends that are going on right now, what are the trends that you're seeing happening in the world of containers and Kubernetes attacks?

MAYA KACZOROWSKI: I think it's honestly a little bit boring, like, in a good way, in the sense that the attacks that we're seeing on Kubernetes and containers are not different than the attacks that we're seeing elsewhere. It's things like privilege escalation, things like crypto mining, things like DDoS attacks. What we aren't seeing is people in the wild trying to escape a container, which is the thing that I often hear users be very worried about-- like, oh, what if somebody comes in and escapes my container? I'm like, what's the likelihood of somebody dropping an 0 day on a random container that they find out in the wild? It's extremely unlikely. If they're going to use that, it's going to be something very, very targeted.

So what we've seen some of the very few public hacks, or bounties, or other things that have been talked about, we were just mentioning earlier the Kromtech article, which was looking at cryptocurrency mining software in images, in vulnerable images. We've had at least two known instances, one of Tesla having their IAM account credentials stolen from the Kubernetes dashboard as well as Weight Watchers just a few days ago had the same issue. I don't believe it was hacked. I believe that was just like a security review. And then Shopify had a bug bounty for having [INAUDIBLE] have access to credentials that they shouldn't have had access to.

So there's starting to be some interest in the security community, but I'd say that the-- and I say interest from-- I mean from hackers, from people who are trying to get into your stuff. They're starting to realize that Kubernetes is interesting and potentially worth attacking. But I think what we're seeing is still a lot of the same classic attacks that we're seeing elsewhere.

ADAM GLICK: Gotcha. And it sounds like a lot of those come from the software that people are putting into Kubernetes versus attacks on the Kubernetes infrastructure itself at this point.

MAYA KACZOROWSKI: I would say it's mostly misconfigurations of Kubernetes more than anything else. It's people who haven't followed all the best practices. And I feel for them, because there's a lot of things that right now we ask users to do in order to be secure in Kubernetes. We give you a laundry list of items you have to go configure yourself. And that's just not going to work in a year from now. If I'm a security person and I have to configure 15 things, that's not helpful. That's not going to be-- that's not usable.

ADAM GLICK: So that brings up a great question of, if people are saying, hey, how do I keep up to date on that, how do I find out about what those things are and what are the best security practices that I should be doing to protect my Kubernetes cluster, where can someone go to learn more about securing their clusters and their containers?

MAYA KACZOROWSKI: Yeah, there are cluster-hardening guides on both the kubernetes.io site as well as the Kubernetes Engine site as well as a bunch of blog posts on the Kubernetes blog post and the GCP blog as to best practices when it comes to container security.

ADAM GLICK: Great. We'll add a link to the most recent one of those in the show notes as well. Thank you so much, Maya. It was great having you on here to talk about the world of Kubernetes security. If folks wanted to find out more about what you're doing, where can they find you?

MAYA KACZOROWSKI: Sure, you can find me on Twitter under Maya Kaczorowski. Good luck spelling that.

[LAUGHTER]

ADAM GLICK: We'll hyperlink that in the show notes as well.

MAYA KACZOROWSKI: Thank you.

ADAM GLICK: Take care.

[MUSIC PLAYING]

CRAIG BOX: We've reached the end of our show for another week. If you want to learn more about Kubernetes security, the best place to start is our show notes at kubernetespodcast.com, which has links to the container security blog post series and all the other resources that Maya mentioned.

ADAM GLICK: Thanks for listening. As always, if you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter at @KubernetesPod or reach us by email-- kubernetespodcast@google.com.

CRAIG BOX: Don't forget. That website again-- kubernetespodcast.com for our show notes. Until next time, take care.

ADAM GLICK: Catch you next week.

[MUSIC PLAYING]