CRAIG BOX: Hi, and welcome to the Kubernetes Podcast from Google. I'm your host, Craig Box.

[MUSIC PLAYING]

CRAIG BOX: I am of the opinion that karaoke should be done on hard mode. If you need to look at a screen, that's cheating and you're clearly not a big enough music fan. Far be it from me to be a gatekeeper-- you do you. But that is to say, I think I might have done OK on that "Don't Forget the Lyrics" game show. People's memory works in strange ways, and mine seems to remember things in a chain. Give me the first line-- I can generally give you the next.

Anyway, a party trick that I don't often mention is that I know all the words to the song "Getting Jiggy With It" by Will Smith. I'm not necessarily proud of this-- well, OK, I am-- but at least I know why those words are permanently etched into my brain. When I was in my second to last year of high school, we set up a low-power FM radio station. In order to test exactly how low power it was, the guy we were renting the gear off put a CD in and played a song on repeat, and a friend and I drove around town listening to where it could be heard.

It probably also acted as a way to build hype for the thing we eventually launched. Why exactly is "Getting Jiggy With It" on repeat on 100.6 FM? But because it was years before social media, we will never really know.

Because it was also years before genius.com, I probably had no idea what half the words I was hearing in my car meant or even were. We didn't have a DKNY or a Prada in the town that I grew up in. Will Smith was on "Top Gear" a few years back, and Jeremy Clarkson pointed out that they never made a convertible 850iS, so Will couldn't even have been the kid in the drop.

The place I'm staying at seems to have been an Airbnb for about 20 years, given the age of the pile of CDs and DVDs in the cabinet, and the fact that it has a CD player in a cabinet. There's a copy of "Big Willie Style" in there. I never actually owned it myself. I thought about playing it when I first saw it, but as of this week, I don't really feel inclined to do so. Maybe I'll go listen to Sister Sledge instead.

Let's get to the news.

[MUSIC PLAYING]

CRAIG BOX: Cisco has introduced new features into its inter-site Kubernetes service. You can now attach clusters running in GCP, Azure, or Amazon to the Intersight control plane. That same pane of glass can be used to manage on-prem or EC2 VMs. Updates were also made to the Cisco HyperFlex line, which can be used to run on-premises workloads using Google's Anthos or Cisco's Intersight.

Red Hat has launched version 4.10 of their Openshift platform, bringing Kubernetes 1.23, Knative 1.0, and many other upgrades. Sandboxed containers, based on the kata container runtime, are GA in this release. There is now support for running on Arm, and version upgrades should require fewer reboots and thus run faster.

Chaos engineering company Chaos Native, sponsors of the CNCF's LitmusChaos Project, have been acquired by DevOps company Harness. Chaos Native was spun out of MayaData, home of OpenEBS, last February, and LitmusChaos moved to the incubation phase of the CNCF this January. The new owners pledged to maintain the community-first values of Litmus while adding new integrations, more fault injections and experiments, and an improved Chaos dashboard. A purchase price was not disclosed.

At last week's game developer conference, Microsoft's Playfair team released Thundernetes. Some jargon unwinding is now required. Playfair was a startup acquired by Microsoft in 2018. PlayFab Multiplayer Servers — abbreviated MPS, go figure — is a service for hosting game servers on Azure, which was internally code-named "Thunderhead." Thundernetes is thus a spin-off project, enabling you to run Linux game servers on a Kubernetes cluster. It is a code name with a GitHub issue to replace it. If your memory stretches back to episode 26, you might be thinking that Agones would be a good name, and you would not be unlike the average Hacker News commenter in drawing this comparison.

The latest release of Weave's GitOps platform adds features from the acquisition of policy-as-code startup Magalix in January. The new trusted delivery feature makes sure that the manifests generated by your getimage are actually valid, and blocks the deployment if not. There is now an easy path to upgrading to Weaveworks' commercial product from their open-source tool Flux, as well as a cheeky adapter for migrating from Argo. Integrations with Terraform and VSCode round out the release.

Storage vendor Qumulo, with a Q, was founded 10 years ago by ex-Isilon employees after that company's acquisition. They aim to manage the world's unstructured data, and they've just noticed that a lot of that is now being generated by apps on Kubernetes. They've now got on the train by releasing a CSI driver, which is now available in preview.

Kubernetes management vendor SpectroCloud has announced a $40-million series B round, bringing its total raised to $67.5 million. Spectro's product is called Palette — in the paint box sense, not in the shipping sense, and I will admit I had to look up the spelling to make sure I had that right. Definitely not in the top of the mouth sense. Recent enhancements include Palette Edge and Palette Bare Metal. The round was led by Stripes, which is not to be confused with Stripe.

Finally, some peeking behind the curtain at how containers are run at scale by some of their largest users. Pinterest wrote a blog post talking about control plane latency, measuring things like how long it took from submitting a batch job to the pod being created. By changing how the controllers handled the work queue, they were able to eliminate spikes.

Combined with a change in how leadership election is handled, they were able to add an extra 9 to the SLO, or in other words, see a 10x improvement.

Uber wrote about CPU throttling in a constrained environment and how CPU pinning allowed them to slightly increase P50 latencies in return for a significant drop in P99 latencies. This resulted in an 11% reduction in compute usage due to less variance in resource requirements. Uber operate their own homegrown system, but they kindly point out where you can configure the same settings in Kubernetes.

And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Santiago Torres-Arias is an Assistant Professor in the Electrical and Computer Engineering Department of Purdue University. He is the team lead for the in-toto Security Framework, as well as a contributor to the Update framework, and a member of the Arch Linux Security Team. Welcome to the show, Santiago.

SANTIAGO TORRES-ARIAS: Thank you. Happy to be here.

CRAIG BOX: You started university in 2007. Did you ever leave?

SANTIAGO TORRES-ARIAS: I have a couple of explorations outside. I worked a little bit as a consultant for the Mexican government in topics of cybersecurity. I guess that's it. The rest was me teaching or learning all throughout.

CRAIG BOX: Projects that we're going to talk about today were founded at NYU, where you did your master's and your PhD. How are Purdue and NYU affiliated?

SANTIAGO TORRES-ARIAS: Technically not affiliated at all. You could almost see it as two companies — say, Google and Microsoft, in a sense. I learnt a lot from NYU and I moved to Purdue to continue working on the space.

CRAIG BOX: Is it OK for you to keep working on things that started there? Does that get really awkward at the basketball games?

SANTIAGO TORRES-ARIAS: It does, although my best chance of survival is to support Purdue from now on. And I think they're better. They have a pretty good basketball team.

CRAIG BOX: Are you required to teach classes?

SANTIAGO TORRES-ARIAS: Yes. I am actually very excited to teach a course that I designed here called "Open Source Software Senior Design Project." I think it's one of the first in which students are required to contribute to open source.

CRAIG BOX: Excellent.

SANTIAGO TORRES-ARIAS: Part of the requirement is a capstone project — pretty much a thesis. So they present proposal to contribute to open source software that they're excited to work on, and I coach them with pretty much everything you'd want us to know, but nobody taught you in school about how to develop software, how to contribute to open source, how to maintain a community, and so on, and so forth.

CRAIG BOX: That sounds like an excellent match between the research goals of universities and the community goals of open source. That's something I could imagine a lot of other schools should pick up as well.

SANTIAGO TORRES-ARIAS: I hope so. Philosophically, this is something that I want to see more of. I believe that open source and academia — we're very connected in the beginning. But at some point, there started to be some divergence of what are the goals of academics — producing knowledge for everybody to see — and the open source community that's trying to build this common set of tools to pretty much carry out anything computing.

CRAIG BOX: Your PhD supervisor at NYU was Justin Cappos, who is the founder of several security projects, a couple of them that you've worked on. Let's talk first of all about PolyPasswordHasher. What was wrong with other password hashers, and why can a parrot do it better?

SANTIAGO TORRES-ARIAS: [LAUGHS] PolyPasswordHasher — I never thought it could be a parrot. But yeah, it is a way to rethink the way that we store passwords. And the issue with passwords generally is that people pick very weak passwords. If you look at the data of the password leaks, then most of the common passwords are "let me in" or "123456." The thought was, how can we use stronger passwords in the database to protect the weaker passwords?

The metaphor that we usually use is, imagine a padlock. If you want to break or guess the combination of a padlock, then you need to guess all of the dials at the same time. Now imagine that we turned a whole database of passwords into dials, and that we link them together so you cannot guess any password unless you get all the dials correct. That makes it so much harder to crack. I still believe that that's probably a way to protect weaker passwords in an ecosystem.

CRAIG BOX: My password is "gandalf." Can you tell me if that's a secure password?

SANTIAGO TORRES-ARIAS: Not anymore. [LAUGHS]

CRAIG BOX: We last talked about secure software supply chain in July of last year, in episode 155 with Priya Wadhwa. It seems the whole world has changed since then, in more ways than one. So as not to give anyone any homework, let's start by asking, what does securing the software supply chain mean to you?

SANTIAGO TORRES-ARIAS: Securing the software supply chain, it is an interesting challenge, mostly because we still don't fully understand how things are made, and we're constantly shifting how things are made. But the fundamental limitation with the way that we make software is that we don't have efficient ways to communicate trust information, or to even make interesting trust assumptions about the things that we consume.

To think about the metaphor again, say, a bottle of juice. We have a very natural, intuitive way to know whether we are going to drink it. If there's a seal cap on it and it's popped up, you know that maybe somebody opened it already and that has a couple of implications. If you don't see an FDA — here in the United States, an FDA approval seal — on the medication, then you go, hold on a second, this may be iffy. Even if the label is printed with a mismatch and it looks funny, you go, hold on, I really don't know if this is something I want to use.

With software, it becomes a little bit more complicated because everything is in the realm of the ethereal, in a sense. We need to develop new ways to communicate this information to users, and for users to also use this information to better protect the way that they consume software.

CRAIG BOX: The price of my groceries has gone up due to constraints in the supply chain. Can I expect the same for my software?

SANTIAGO TORRES-ARIAS: Yes and no. I kind of love this question. Yes because I assume that everything is connected. Physical goods are going to have an impact on how much it costs to develop software goods. But the other highlight is that digital supply chains are harder to secure or intrinsically different from physical supply chains.

If you make a thing, then the thing is made and there's one of that thing. But with software, things are copied over just by — I don't know — typing something on the command line. How do you track all of these changes, and where do they go, and how is it transmitted?

CRAIG BOX: Blockchains.

SANTIAGO TORRES-ARIAS: Exactly, blockchains. You can use a blockchain to save all of that.

CRAIG BOX: Please don't.

SANTIAGO TORRES-ARIAS: [LAUGHS] Well, a transparency log is not a bad idea, but I think we'll get to that in a bit.

CRAIG BOX: The first project that you worked on in the software supply blockchain space at NYU was The Update Framework. I understand you can draw an analogy between that and perhaps the shrink-wrap that you get on some goods that you buy at a supermarket.

SANTIAGO TORRES-ARIAS: The Update Framework — TUF — is pretty much that shrink-wrap that you would put on a product when it goes to the shelf, and that pretty much secures you from any tampering that happens between the vendor as they packaged everything and into your home. That's the fundamental last-mile problem, and that's pretty much what TUF was designed to do, to ensure that nothing happened between the developer, publishing — I don't know — a Docker container or Python package or whatever into your machine.

CRAIG BOX: If I want to install a Debian package, then the step before adding the repository is often adding the public key from that repository to my own keychain, such that it can verify the signature of the application. Is TUF effectively the same thing?

SANTIAGO TORRES-ARIAS: It is effectively the same thing. It has a couple of more package-dependency timeliness guarantees. It has more layers on top of it to make sure that the package is actually correct.

But again, it is not much different from, say, an expiration date on a bottle of juice or the shrink-wrap telling you that it was tampered, or even a way to effectively say, this is a Debian package that came from this developer and not from this other developer that you shouldn't trust to be giving you this package. So it handles things like namespacing, conflict resolution in smart ways.

CRAIG BOX: I don't mean to drag the metaphor on too much, but with my bottle of orange juice, I could say, hey, it's past the use by date but only by a day or two. And I can sniff it, and I can say, hey, do I trust this? Is there an equivalent to software? I imagine software is a bit more binary — yes or no. It says the signature failed, so the system won't let me install it. Are there cases where I might want to say, hey, this is a signal, but it's not necessarily a blocker?

SANTIAGO TORRES-ARIAS: There's definitely cases like this. One of them that comes from the top of my head is actually research. When you're doing a scientific experiment, you sometimes want to trust things that are old or run things that are old. I think part of what I was also saying, in that we need to have a more transparent understanding for users, is to be able to say, well, what are the implications of drinking day-old orange juice versus the implications of drinking year-old orange juice, right?

CRAIG BOX: Is it simply a stamp, though, that says this is the date that it was produced and I can have rough confidence that only the person who made the juice has put the stamp on it?

SANTIAGO TORRES-ARIAS: It should be a determination made by the user. Again, I think the goal is to pigeonhole users into making the safe decisions, and then let more advanced or risky, non-risk averse people, to be able to say, well, let's just try this milk and see what's on it.

CRAIG BOX: So when I get a piece of software that is signed and says it comes from this vendor, I can understand that it came from the person who had made the signature. How do I prove that the person who made the signature is actually who they claim that they are?

SANTIAGO TORRES-ARIAS: That is a good problem. Not a good problem to have, it is a good problem to be solving. There's really many ways to do it. You had Priya, who works on the sigstore project. She probably had some ideas as well.

But overall, identifying people in the broader internet ecosystem is a challenge. I'm an academic, so if that existed, then it is important to say, well, this person that's identifiable on the internet is the one that you should be trusting for this. That's pretty much a tough challenge.

CRAIG BOX: Many, many years ago, I first met Mark Shuttleworth of Ubuntu fame when we signed each other's GPG keys at a ceremony in the Linux conference. Is that still the way that people acknowledge each other's identity, by doing some sort of offline check: a passport, driver's license, web of trust mechanism?

SANTIAGO TORRES-ARIAS: Yes and no. I think it's part of the ecosystem. I definitely — at least, for example, in Arch, there's a little bit of that. You need to get people in the community to acknowledge that you exist. Part of a challenge at a broader level is this works at a community level, right?

But what happens if you get an Arch package, stick in a Docker container, that then is released by — I don't know, name your favorite company — and then it ends up in somebody's cloud. There's a chain of custody problem there. And that's what starts to hint about the in-toto challenge.

CRAIG BOX: So when Priya talked about in-toto, she talked about it in the context of provenance. And there was a particular metaphor that you used, which is farm-to-table provenance. I bought some potatoes recently that had a code on them that I could enter into a website to learn about the farm they were grown on. I didn't do that, but I guess it makes me feel better to know that I could.

SANTIAGO TORRES-ARIAS: That's actually very close to something that, early on, when I was doing my research at NYU, it made me realize how bad we were in terms of provenance for digital goods. I was talking to a guy on the FDA, I think he was. We were talking about E. coli outbreaks on lettuce.

CRAIG BOX: As one does.

SANTIAGO TORRES-ARIAS: As one does, right? [LAUGHS] And he told me that essentially, the FDA had a technology to sequence the DNA of lettuce and say, hey, this lettuce was made on this farm. This was very useful in E. coli outbreaks because then they would say, oh, there was an E. coli outbreak in California. We took this lettuce, we sequenced it, we identified the farm that it came from, and then we told everybody in the United States that had bought from that farm that they probably had E. coli on their lettuce. And that's amazing. That's what we should be striving for in the software supply chain world.

CRAIG BOX: That is really interesting in a utopian or a dystopian sense, depending on which way you choose to look at it. The obvious implication there is to say, well, if every wet market in the world that was potentially selling animals had this, then maybe we'd know where COVID came from.

SANTIAGO TORRES-ARIAS: Yeah. That's an inspirational example, at least. To me it was incredibly illuminating. I would love a world in which we, say, have a Docker container, we stick it somewhere, and it tells you, hey, by the way, this disgruntled employee did something on it, so you probably don't want to run it.

CRAIG BOX: How then do we apply that to software, if we think about the attestation of who are the people who were involved in producing something, which companies did they work for, who did a thing like, it's all very well to say, this person, which we can validate, works for this company made a change to this container, but that company has no business being involved in it. How do we take our artifact that we've received, and run back through all of those steps in a safe and secure way to figure out whether or not we should or should not run that code?

SANTIAGO TORRES-ARIAS: That's pretty much the in-toto goal. Overall, without getting too academic, it is pretty much a way to describe a provenance graph. And then walk through the graph and find notes that shouldn't be there.

CRAIG BOX: in-toto describes itself as "a series of tools that describe how a software development process should be performed, and verify that the processes are followed properly. Giving those tools to administrators and developers, end users can verify the software they're about to install is correct, not tampered with in transit, or that no errors were made during the elaboration with it." So let's say I have some source code, some containers from popular open source projects, and a Kubernetes cluster in or two. How do I use in-toto?

SANTIAGO TORRES-ARIAS: You can think of in-toto as a way to rubber stamp, in a community, everybody's actions. There's two families of tools. There's the developer tools, and then there's the consumer side tools. On the developer tools side, you can integrate in-toto into a bunch of different technologies — say, for example, Tekton Chains that's building your code.

You can integrate it through git hooks, you can rubber stamp your commits, you can integrate it as an admission controller on the cloud. The goal of this is to essentially gather evidence of everything that happened through very, very small pieces of metadata called in-toto links or in-toto attestations. With those, we can build this huge provenance graph that tells you where everything came from, pretty much decide whether that's correct or not.

The in-toto way would be to define a policy, pretty much like, I don't know, like OPA, or a CUE type of policy, or Kyverno to say, well, I actually just trust these people to do whatever they are doing in this particular project and nobody else.

It can get a little bit more interesting when you start thinking about, well, I trust this person when they are working on this computer, or I trust this person if they're using the latest SDK, but nothing else. Even crazier, right? I only trust these two people if they agree with each other on what they did.

CRAIG BOX: A lot of the way that trust is validated in the corporate world and software is through the use of digital certificates. My laptop has a certificate on it that says this was issued to me by Google. I have a password and a security key. And effectively, all those three things have to be present and accounted for, for me to be able to do things. Is that the same as in-toto? Are we able to say, these IDs are required, and we can map them back somehow, kind of like we were talking about before in terms of physical identity?

SANTIAGO TORRES-ARIAS: Pretty much. Imagine that you took those three things, then we're recording a podcast. Imagine that you were concerned about being impersonated. What you could do is you could take those three things and say, oh, this is the laptop that was issued by Google. I typed my password. Here's proof that the password was verified correctly by that laptop, and here's a signature from my security key. Now, here's the recording of the podcast, and it proved that it was me, the one that was doing the recording. Does that make sense?

CRAIG BOX: It does. And that's very interesting because I could ask you then how would we — in terms of recording this podcast — what is the in-toto equivalent to metadata that we would need to publish to show that it was both you and I participating in it?

SANTIAGO TORRES-ARIAS: I don't know. If you pay the pizzas, I think I could get away with it just using in-toto record start and in-toto record stop.

CRAIG BOX: Excellent. Is this something that I should care about? Or do I want my tool vendors to care about it on my behalf?

SANTIAGO TORRES-ARIAS: A phrase that I usually repeat — it's more of a running joke now in the in-toto community — is that it's insecure turtles all the way down. You cannot trust your tools if your tools were built by bad tools, and, well, how do you trust the computer they were made on?

My hope, as a broader ecosystem vision, is that end users are not the ones that should be interested in this that much, but rather the critical pieces of software — the ones that start producing in-toto metadata so that we can verify it — we can have core, trusted supply chain tools to say it some way.

CRAIG BOX: Could in-toto help Ken Thompson sleep better at night?

SANTIAGO TORRES-ARIAS: I believe so. [LAUGHS] I don't know if you got to see the Supply Chain Security Con talk from SolarWinds, but one of the fundamental challenges that they did was how can they solve the Ken Thompson problem using reproducible builds and in-toto to verify that different compilers or different tool chains and different systems agree on the result.

CRAIG BOX: For the younger members of the audience, it might be worth pointing out that Ken Thompson, inventor of Unix, posited a thought experiment, perhaps in the 1980s, where he was talking about the concept of trusting trust, and what would you do if you had a compiler that was designed to turn out bad binaries. And when it were to compile itself, it would continue, effectively, a worm in the compiler, so to speak.

SANTIAGO TORRES-ARIAS: Exactly. In a sense, the SolarWinds compromise was that.

CRAIG BOX: They were then turning out software which was signed by the SolarWinds key but contained things in it that weren't supposed to be there.

SANTIAGO TORRES-ARIAS: Exactly.

CRAIG BOX: Is the idea of attestation through in-toto and generating these provenance documents, is it useful if not everyone does it? If there are six or seven steps in your supply chain, and only steps three through seven use it? Is it still useful? Or is it just so much unknown in the beginning that we can't really validate anything?

SANTIAGO TORRES-ARIAS: It is still useful. Realistically speaking, you cannot expect everybody to just commit to something like this. But the goal is to minimize the points of entry. Something that you can get from using in-toto is that you can say, well, it was OK until we got to three and four, and then we got hacked.

You can say, well, three and four were the problem or three and four onwards. The idea, again, it's shifting left, but also probably starting from the left and forward. So you can remove the unknowns from the left-hand side.

CRAIG BOX: I guess if you have a piece of software stop behaving the way you expect it, there are many different steps along the supply chain process that it could have been interfered with, and being able to pinpoint that and DNA sequence that lettuce, that that would be useful to be able to tell where the exploit had actually happened.

SANTIAGO TORRES-ARIAS: And another thing is sometimes it's easier to put in-toto in some places, and sometimes it's harder. But you can do other things. You can say, well, in-toto doesn't fit here, but we're going to secure the hell out of this thing so that we know that this thing, in particular, doesn't get hacked.

CRAIG BOX: What does the in-toto project have against capital letters?

SANTIAGO TORRES-ARIAS: Personally, I just don't like how it looks. I don't think there's a better answer, I just don't like how it looks.

CRAIG BOX: Fair enough. TUF — T-U-F — for example, that's all in uppercase.

SANTIAGO TORRES-ARIAS: Yeah. I think that that's our quota for capital letters, then we run out of it.

CRAIG BOX: In 2019, Datadog published that they're using TUF and in-toto to secure the distribution of the Datadog agent. How does that work?

SANTIAGO TORRES-ARIAS: It's an example that I love. Pretty much what it does, it uses in-toto for their developers to track how they're developing everything. Then it uses in-toto to track how their CI is building what the developers build. It's all eventually packaged into a single TUF repository.

From the TUF repository, it's almost like, imagine that you've got an IKEA furniture, and the IKEA furniture, you get your box and you know that it's sealed. Nobody did anything from IKEA to your home. That's the TUF part. Then you open it, and then you get this manual.

And the first page will include, this is the amount of planks that went in there. This was made in, I don't know, Vietnam or whatever. There is this amount of screws. There may be other information that I haven't looked at. But this is pretty much what you get from this Datadog integration, in that you get the secure distribution of Datadog code, but you can also trace it all the way back to the Datadog developers. And you can even tell which Datadog developer did what.

CRAIG BOX: It sounds like a very useful pairing to have those two things together. Why are they separate projects?

SANTIAGO TORRES-ARIAS: There was a recognition that not everybody wants to use the TUF model. You, for example, brought up the GPG, the Linux Distro world sort of thing, in which they pretty much figured out their own distribution solution. Trying to force people into adopting both felt both bad for us, but especially bad for in-toto because, again, the more adoption you get the more secure you can claim that the whole ecosystem is.

CRAIG BOX: The project website says, "This research was supported by the US National Science Foundation, by DARPA, and by the Air Force Research Laboratory." How does one go about getting those agencies to sponsor their work?

SANTIAGO TORRES-ARIAS: These agencies usually set up calls for research projects, especially those that are trying to secure either the cyber-physical or cyber-critical infrastructure. Especially for, say, ARPA] which is the Air Force, it is very important that the software that you put in a fighter jet, for example, doesn't have any sort of funny business in there.

CRAIG BOX: On the topic of government, in May last year, President Biden issued an executive order, basically telling government agencies to up their game when it comes to software security. What has changed since then?

SANTIAGO TORRES-ARIAS: As somewhat of a supply chain hipster, everything has changed since then. I started working on the space in 2015. I remember people were skeptical of this even happening. Then SolarWinds got hacked and then everybody was like, oh my God, this is actually a problem.

Since then, I see way more recognition of this as a technical challenge but also as an ecosystem-level challenge. I think that keeps things interesting in terms of having a lot of innovation, but a lot of new developers or new contributors trying to see how they can fit in the picture. And I think that's great.

CRAIG BOX: There was a meeting at the White House about software security in January. Did you go?

SANTIAGO TORRES-ARIAS: I did not go. I was a little salty because if you see how it's described, it says, oh, it's the White House and industry stakeholders.

CRAIG BOX: Not academia?

SANTIAGO TORRES-ARIAS: Not academia. Not that they did anything intentionally. It was just like, wow, let us have a little bit of the pie as well.

CRAIG BOX: Of course. Now talking of the pie, sigstore is the new cloud native security hotness. How does in-toto fit into that?

SANTIAGO TORRES-ARIAS: It is almost sigstore native, I would say, or sigstore in-toto native. I don't know what's the right phrasing for it.

CRAIG BOX: You were their first, claim it yourself. sigstore is in-toto native.

SANTIAGO TORRES-ARIAS: [LAUGHS] OK. sigstore is in-toto native. Part of a question that's not answered is — as I was talking about this graph-building and finding evidence, and then scrutinizing and deciding what to trust — is that, how do you find this evidence to begin with? That's pretty much where sigstore comes in.

It is a way to discover evidence of different formats about supply chain artifacts. The idea is to simplify the way that you can create and publish information about software supply chains. But also how can people query and validate and even share their insights about artifacts that are there.

CRAIG BOX: If we think back to the concept of GPG keys, you have your own [public] keys and signing and so on, but then there was this set of servers that you could upload them to. Is this an equivalent-- that we might have an attestation attached to a piece of software ourselves, but we need to have some way of going and asking about that before we download and run the software?

SANTIAGO TORRES-ARIAS: I would say that's a fair assessment. There's, again, more bells and whistles. And of course, it can do interesting things. It's also way better than a regular key server in that there's like a lot of integrity guarantees about what you get. I don't know if you remember key servers have a bunch of issues, like spamming and forgetting about keys and stuff like that.

CRAIG BOX: No, it's been a long time since I've ever had to touch my GPG key. But I trust that it's still sitting there somewhere if I can remember the passphrase.

SANTIAGO TORRES-ARIAS: [LAUGHS] Oh boy.

CRAIG BOX: in-toto is the second most-used mechanism in sigstore. The first is the standard X.509 certificate. The kind of thing we were talking about before that effectively powers PGP and is used for everything. It doesn't seem to me like these two things are the same?

SANTIAGO TORRES-ARIAS: No. It is similar to the way that in-toto and TUF interact, in that X.509 is pretty good if you are going to sign a binary and, again, close up the IKEA box or put shrink wrap over a bottle of juice or something like that. But it doesn't really tell you anything about how that thing came to be in the first place.

CRAIG BOX: So you really need that little piece of paper inside it that says, this was done by this person on this date?

SANTIAGO TORRES-ARIAS: Exactly.

CRAIG BOX: Some of the more hipster pieces of furniture you can get probably have a picture of the artisan.

SANTIAGO TORRES-ARIAS: Exactly, and a signature from them.

CRAIG BOX: Another project that sounds similar, but not quite the same, SPIFFE is a format for validating identity of workloads. And I see now in-toto has built integration with SPIFFE. How exactly do those two projects interact?

SANTIAGO TORRES-ARIAS: I would even say that three projects interact very well in that in-toto framework to create the evidence, then you can have sigstore to discover it, and then you can have SPIFFE to tell you who's who. That's exactly what you were talking about earlier. It becomes super important when you're thinking about ephemeral workers or containers that live for a couple of seconds to do their job, that they spin up and spin down, and you just need to know that they were the ones that did this thing, and you need to trust them for a small window of time. And that's where SPIFFE comes in. SPIFFE is very good at that. So rather than trying to integrate something different into in-toto, well, let's just start supporting SPIFFE identities.

CRAIG BOX: There are a few projects — Kubernetes, Prometheus, and so on — that form the nucleus of the cloud native serving stack. And it feels to me like there are projects — TUF, in-toto, SPIFFE, like you say — that almost form the nucleus of a security stack. Is there a goal to have one overarching brand that brings all these things together, so that I just say, hey, I'm compliant with whatever this thing is, whether it be sigstore or something else, but then effectively is a tick-box that corporate management are going to look for?

SANTIAGO TORRES-ARIAS: I will say yes and no. At a philosophical level, that's where I want to get at. I want us to get this overarching framework. It doesn't need to be called in-toto. But rather, download a thing, put it on your computer, and forget about it. Another piece of the puzzle that you spoke about with Priya was the SLSA project.

That tells you a little bit more about, well, maybe I don't want to reason about graphs and understand all of these complicated provenance structures. I just want to know if the thing was done according to best practices. I believe that that's another piece of the puzzle. Now, you would have a way to identify the workers with SPIFFE, a way to gather the evidence with in-toto, a way to distribute everything nicely and securely with TUF, and then you would have something like SLSA to tell you, well, what you just got is a SLSA for a piece of software.

CRAIG BOX: And then, of course, you've got the salsa on the shelf that has the little button on the top of the lid, which pops up whether it's been opened. So it effectively has its own TUF-like attestation built into it.

SANTIAGO TORRES-ARIAS: Exactly.

CRAIG BOX: In August 2019, in-toto was accepted as a project in the Cloud Native Computing Foundation. What was the decision at the time about how to progress that project through the foundation?

SANTIAGO TORRES-ARIAS: I think both TUF and in-toto, we come from a different background from your usual CNCF project, in that we are an academic-first development. So our needs are a little bit different. Most of all, we don't, for example, have the muscle to have developers on staff working on this full-time and doing the community building.

CRAIG BOX: But alternatively, you could play the long game, where you set up a university course that requires people to do open source development.

SANTIAGO TORRES-ARIAS: [LAUGHS] Exactly. That was the plan all along. That's kind of why things are a little bit slower, but also it gives us a little bit more freedom to say, well, honestly I don't care about this industry politicking around things — I just want to make people sign their stuff. When we joined the sandbox, we were actually also eager to try new things and to develop a standard that fit well within the CNCF.

CRAIG BOX: It seems like you've done a good job of that because the process has recently moved to the incubation phase. Congratulations on that.

SANTIAGO TORRES-ARIAS: Thank you. Very excited.

CRAIG BOX: in-toto was the first project to go through the security assessment with the CNCF's new Technical Advisory Group Structure and the Security Group. What did that involve?

SANTIAGO TORRES-ARIAS: Well, that's one of the things that we were happy to try out, to help develop things not only going through the regular channels, but even being happy as Guinea pigs. Early on, actually, the process was not very well established.

So I think the in-toto assessment took a little bit longer than usual because we would circle back and say, well, actually was that question enough? Or should we dig a little deeper and reframe things or re-understand things a little? It took a little bit. But in a sense, I'm also happy to see that this was not only useful for us, but also to start setting the tone of future projects within CNCF.

CRAIG BOX: What other benchmarks or achievements did you have to make to move to the incubation phase?

SANTIAGO TORRES-ARIAS: For incubation, speaking mostly as in-toto, it was around adoption and around how mature the project is in terms of its users. Overall, especially after SolarWinds got hacked, then things did take a fever pitch, and adoption started to be pretty widespread.

CRAIG BOX: Is SolarWinds the company involved in any of these attempts to stop a SolarWinds-like attack happening again?

SANTIAGO TORRES-ARIAS: SolarWinds felt like an amazing anime redemption arc type of thing. Because they did get hacked. I think just a couple of weeks after or so, they messaged me, hey, we need to talk. We spoke a little bit. And they were like, hey, we're going to figure out how to solve this. We think in-toto part of the puzzle. And they just started developing a lot of software around in-toto, and in in-toto, to make it fit the way that they thought they could stop this from happening again.

CRAIG BOX: Another security vulnerability or situation that's in the news as we record this is the hacking group, for want of a better term, Lapsus$ or "Lapsus-string", not 100% sure. They are effectively using social engineering attacks to get inside companies. Okta has been in the news as a company that they had some access to for a while, and then perhaps pivoted that to access to their customers. In the past, people would either look to use ransomware or they would look to get strategic access for longer term inside.

But this attacker basically seems to say, well, what can we do in the very short amount of time we get before we're detected? And they're actually going and announcing that they're hacking things in real time. So they're detected a lot quicker than they would be otherwise. Does this kind of change in the way that people are being attacked talk to how we should think about securing things differently?

SANTIAGO TORRES-ARIAS: As everything security, it's pretty much a race between attackers and defenders, at least for supply chain and at least in-toto context. And it's interesting when you step back and you realize that, well, developing software is just a process. How can we use in-toto-style protections to, say, secure an API server?

Or being realistically, there's many software development processes that include API calls, right? Like you go get a vulnerability scan over your software. And how can you get a signed attestation, a proof that this was done by, say, Black Dog on this container at this time with the right security definitions, and so on, and so forth?

And I could see a future in which you could also include, again, in-toto attestations to say, well, this was an Okta two-factor authentication exchange that was used to log this person in, that was used to exfiltrate this repository. Hopefully, it could stop things, but it could also give us more traceability as to how things came to be.

CRAIG BOX: Yeah. In the case where the company determines that there was access between this date and this date, there are a lot of people wanting to say, hey, I really wish I had logs to look at all the things that were using those APIs during that period.

SANTIAGO TORRES-ARIAS: Exactly.

CRAIG BOX: What does success look like for the in-toto project? What does it look like when it's finished?

SANTIAGO TORRES-ARIAS: I would put two benchmarks. One of them would be utopian and the other one would be a little bit more realistic. To me, the utopian one, it's a little bit more obvious, right? Everything is using in-toto or some iteration of in-toto, in which we can know where everything came from, and we can make sure that these supply chain compromises are way harder to carry out because now you need to overcome this defense.

My success would be something like TLS. Imagine that in the same way that we see a success story for encrypted internet communications with a web server, we have this for software. It's easy to sign now that we have Let's Encrypt, and it's easy to verify best practices, and there's understood ways to do it.

I know that there are sysadmins listening. Making a CA back in the day with OpenSSL was a headache. Now you can use CFSSL and it's simpler and it's getting better, right? TLS 1.3 is amazing and everybody feels way more confident about the way things stand. Now, I would hope that we get to a point in which in-toto is almost as seamless as something like TLS.

CRAIG BOX: When we were talking before about the orange juice, that kind of led me to a similar thought when it comes to TLS and web security. If I have a web page that has an expired certificate or the date is wrong and it has information that I find relevant, I might want to go through and read that. But I might make a choice not to send my credit card number to it, for example. So I think that there is a very obvious parallel there between how we consume web software and how we might secure the software supply chain.

SANTIAGO TORRES-ARIAS: Exactly.

CRAIG BOX: Now, your web page says that you like playing math rock. Do you have a favorite time signature?

SANTIAGO TORRES-ARIAS: I think it's more of a combination of time signatures. There's a couple of songs that have very, very nice changes between 7/8 and 11/8 that I really enjoy.

CRAIG BOX: Do they happen to be by the band Tool?

SANTIAGO TORRES-ARIAS: Maybe, although yeah, I haven't played Tool in a while.

CRAIG BOX: I'd say they're the band most famous for the genre. If I had to say who's a band that I would associate with that term "math rock," Tool are the ones that always come to mind.

SANTIAGO TORRES-ARIAS: Yeah, no, they're super fun.

CRAIG BOX: Look at us. We do crazy counting because we can.

SANTIAGO TORRES-ARIAS: With this signature, I know there's a song by a band called Covet called "falkor" or something like that. I may be butchering the pronunciation. But it has very, very cool jumps between those two. There's another one by a band called TTNG. The song's called "Plus 3 Awesomeness Repels Water." It is just amazing to play. It's so much fun.

CRAIG BOX: We'll put links to the videos for those songs in the show notes, so that you can all try and count along and get confused as you try to dance.

SANTIAGO TORRES-ARIAS: Awesome.

CRAIG BOX: On the show sometimes, we like to talk about the New Zealand Bird of the Year Contest, and I understand you have a favorite New Zealand bird.

SANTIAGO TORRES-ARIAS: The kea. I hope I get to see one live once. There's a video on YouTube of one breaking a police car, which I find hysterical because it just doesn't seem to care.

CRAIG BOX: Yes.

SANTIAGO TORRES-ARIAS: I wish I could travel to New Zealand and see a kea just wreaking havoc on something.

CRAIG BOX: We very much hope that that is something that you're able to do very soon. I look forward to wrapping up your travel plans in a nice, sealed bag so that we can tell that they haven't been tampered with along the way. And it just remains for me to thank you for joining us today, Santiago.

SANTIAGO TORRES-ARIAS: My pleasure.

CRAIG BOX: You can find Santiago on Twitter @torresariass or on the web at badhomb.re, with a dot before the "re."

SANTIAGO TORRES-ARIAS: Yeah. It's more of a visual joke, I think. [LAUGHS]

[MUSIC PLAYING]

CRAIG BOX: Thank you for listening to our show. If you enjoyed it, please help us spread the word and tell a friend. If you have any feedback, you can find us on Twitter @kubernetespod or reach us by email at kubernetespodcast@google.com. You can also check out the website at kubernetespodcast.com, where you will find transcripts and show notes, as well as links to subscribe. Thanks for listening, and we'll see you next week.

[MUSIC PLAYING]