KASLIN: Hello, and welcome to the Kubernetes podcast from Google. I'm your host, Kaslin Fields.

ABDEL: And I am Abdel Sghiouar.

[MUSIC PLAYING]

In this episode, we speak to Jonathan Grahl. Jonathan is a Team Lead of Infrastructure at Lovable, where he oversees the platform stack the company runs on. In this episode, we talked about Kubernetes, sandboxes, and chocolate.

KASLIN: But first, let's get to the news.

[MUSIC PLAYING]

ABDEL: OpenTelementry is a CNCF graduated project, cementing its status as the de facto standard for observability. The project reached technical maturity with massive adoption. The JavaScript and Python EPIs both recently surpassed 1.3 billion downloads. OpenTelementry is seeing an increase in interest as the standard layer for observing performance, reliability, and trustworthiness, specifically for AI workloads. Congratulations to of the OpenTelementry team.

KASLIN: The CNCF Technical Advisory Group elections are open. With three former tag leads, Brandt Keller, Mario Fahlhandt, and Mauricio Salatino stepping up into the 2026 Technical Oversight Committee, this opens up new positions within the advisory groups. Nominations are open for TAG infrastructure, operations resiliency, dev experience, workloads foundation, and security. And votes will start soon. Check the link in the description for details and dates.

ABDEL: In events news, KubeCon and CloudNativeCon India is happening in Mumbai on June 18, 19. KCDs in Czechia&Slovak, New York, and Kuala Lumpur are happening on June 4, 10, and 27, respectively.

KASLIN: And that's the news.

[MUSIC PLAYING]

ABDEL: Our guests on the show today is Jonathan. Jonathan is the Team Lead of Infrastructure at Lovable, where he oversees the platform stack the company runs on. We'll be talking Kubernetes, sandboxes, and chocolate. Welcome to the show, Jonathan.

JONATHAN: Thank you so much. So nice to be here.

ABDEL: Awesome. So I think everybody knows who Lovable is. But just in case people don't know, who is Lovable? What do you guys do?

JONATHAN: Yeah, so Lovable is a platform that aims to be your co-founder if you're making, like, a SaaS product or a tech-based product. Well, I guess it's called like vibe coding nomenclature. So we help you build applications, even though you don't know technology at all. So we aim to help the 99% who are not technical to achieve their dreams, basically.

ABDEL: Awesome. So Lovable is clearly one of the biggest players today, and has been for a very long time one of the biggest players in the web coding space, which means that you folks have seen actually a very rapid growth, right? Your platform has been growing very fast. And you run primarily on Kubernetes.

You do also run on other platforms. We're going to talk about that. Have you seen any scaling challenges? Because we all know Kubernetes is not easy to scale.

JONATHAN: Yeah I think all of them, as expected.

[LAUGHTER]

ABDEL: Of course.

KASLIN: So this is why it's very interesting and very timely to talk about sandboxes overall-- or I guess overall, infra as you can imagine, or Lovable has gone like crazy. I can't actually remember how big we were in the summer. But right now, we're building several thousand new projects per day on Lovable.

ABDEL: Wow.

JONATHAN: So it's pretty crazy. I mean, we were not that many when I joined in August. And now we're significantly bigger. So in terms of scaling problems, I think we've hit every type of bug and scaling problem in all our providers. We're kind of at the frontier here.

ABDEL: Yeah. So can you talk specifically about some scaling challenges you've seen? Obviously, I think we all know that scaling envelope that Kubernetes has, which is like a multidimensional scaling problem that, if you pull Kubernetes in one way, you stretch it too much. Can you talk about some very specifics?

JONATHAN: Yeah, so how Kubernetes works-- you're absolutely right. Depending on what you're scaling-- and there's so many different scaling factors, things will start breaking, or you need to optimize for different things. For example, a big one that we wrote about lately on our blog is Cilium, or like the Dataplane V2 breaking.

We had a problem for a couple of weeks where we would just drop packets sometimes during load very uncontrollably. And it was due to us enabling and disabling internode encryption in Dataplane or in Cilium, which meant that the MTU or packet size was different in different places. And this probably wouldn't have happened if we didn't do it under load.

So this is a place where Cilium has been maybe one of the things that is the most problematic in terms of getting-- in our case, our scaling vector is actually pod churn or pod creation rate more than know, how many pods we run. So our pod creation rate can be a few hundred per second in certain times. We try to optimize each cluster to support that, which is, you know, a really hard problem.

ABDEL: Yeah.

JONATHAN: So in this case, Cilium could take up to 30s to get a pod IP to be routable in the cluster, even though the pod is ready.

ABDEL: Yeah, I remember when we met back in March, you were preparing for International Women's Day. I believe you were offering Lovable for free for International Women's Day. So we were performing basically tests to check if Cillium would not be a bottleneck.

JONATHAN: And it wasn't that day. But we did--

[LAUGHS]

That day went great. We we had like a crazy day. I don't know how many multiple times more load we had only on one day. But we had prepared for it since October. Last month, I think we-- or September, we started optimizing for running on open source software, rather than proprietary platforms.

ABDEL: Yeah, and specifically for people who are following us who might not know where this problem comes from, it's basically when you scale up your pods, the time it takes for all IPs propagate across the cluster, right?

JONATHAN: Yes, exactly and In our case, it's actually they need to propagate globally like on public across, in the case all of Google. Because we run in multi-region and we use this native network access from our backend all, the way to the pod IP when within Google. So actually it needs to be available cross-region as well.

ABDEL: Yeah. So maybe just to give an audience a little bit of a deeper understanding, can you explain the reason why you care about the individual pod?

JONATHAN: Yeah, so this is where I think sandboxes are like a new type of workload. It's not new. It's actually the very old. But there's a reason why it's important for us.

Lovable requires you to have the-- let's take it from the product perspective. So you are a user of Lovable. You open lovable, you want to work on your project.

When you get in, then you get what we call a live preview. That's actually like a websocket connection directly to the pod. And if the pod disappears and the agent does stuff, and we have Git, the pod is not allowed to disappear, because we don't know if we lose state or not.

So our pods have to be alive until the user essentially closes their browser, which could be five minutes or 24 hours. So this means that whenever the user clicks Edit on the product, that time until the pod is ready, it's like how long the users need to sit and wait? And that's why it's so important for us.

ABDEL: In a way, you are kind of using Kubernetes in a bit of an unconventional way, because it's not your typical horizontally scalable workload. You are basically assigning effectively one pod per user session or whatever, right?

JONATHAN: Exactly. And it's similar to CSCD in many ways as well. It's interchangeable. In this case, we have like a connection open during the work. But what we're doing is now we start a sandbox.

We do kind of a install, start some services, et cetera. And then it needs to stay around until it's built and ready. And otherwise, you just need to start a new one.

ABDEL: Yeah, so obviously, startup time is important. State management is important. And we're going to talk about this later. But I wanted to pull back a little bit and talk about what the sandbox means.

JONATHAN: Yeah.

[CHUCKLES]

Yeah for those who sit on X and follow all the new benchmarks or sandboxes-- and now the hyperscalers are getting into sandboxes. I think there's different categories of sandboxes that focus on different things. So for example, in our case, how we view sandboxes is that they are stateful, but non-persistent, workloads.

So they need to be around and they have state in them. And whenever the user is working, they need to stay. It will not be able to just be replaced at any time. So it is stateful. But we can refresh them from cold at any point, as long as we've done the Git commit.

ABDEL: Got it.

JONATHAN: So that sandbox is for us. For other people, for example, Cloudflare released this new sandbox thing last week, which is based on V8 isolates, which is just JavaScript code. It runs in a sync or a process without a container or a microVM at all. So they use the V8 like the Chrome virtual machine just for JavaScript. And as well before cloud, there was something called VPSs, which is essentially what sandboxes is today for most people, which is like, give me a machine with some RAM and some CPU, and hold it until I want to delete it.

ABDEL: Yeah.

JONATHAN: That's the most normal one today.

ABDEL: Yeah, and that's interesting. The way you described it, it's stateful, but it's not persistent, in the sense that once the user is done with the work, you don't necessarily have to persist it in its current state, because you can always recover. So that's an interesting way of putting it in the context of Kubernetes, because when people talk about Kubernetes, it's always stateful versus stateless, right?

JONATHAN: Yeah, exactly. So what Kubernetes is really good at is, like, if you describe a workload, it's just like make sure to keep it around always. But it does not ensure-- in our case, for example, if we connect to a sandbox, we need a pod name or the IP to be the same for session of the user.

ABDEL: Yeah.

JONATHAN: So we can't in Kubernetes case, it's like, oh, you have described the pod. I'll just recreate it. And availability is high, but that actually means that it's not the same sandbox anymore for us. So we don't use the things like deployments, stateful sets. We use pods directly because none of the workloads fit us at all.

ABDEL: Yeah, because if you use a deployment or any type of controller, you get unpredictability, because you can get assigned any random pod, essentially.

JONATHAN: Exactly. And deployment, for example-- interesting case. If you use a deployment to, have your pods available, if the pod restarts, you are not able to make the pod not restart if you use a deployment. But in our case, if the pod restarts, it loses all the state. So then we don't need it anymore.

ABDEL: Oh yeah.

JONATHAN: So if a pod restarts, we actually need to proactively delete them to make sure our system doesn't think that a sandbox is still there.

ABDEL: Oh yeah. So you have to deal with, like, risk conditions essentially.

JONATHAN: Yeah, all the time. And this is where we have broken-- we break the replica set controller actively.

[LAUGHTER]

We hack Kubernetes a lot to make it work.

ABDEL: And so I think that this is an interesting use case that you're talking about. I personally think that there will be more and more use of those specific use cases going forward, because I think that Kubernetes is by far moving away from the standard, you know, give me deployment of 10,000 pods and put a load balancer in front of it. It's like the standard back end use cases or whatever. I think that there will be more and more of these particular use cases.

So let me ask you a question. And I guess I'm jumping ahead. But how do you deal with keeping state?

JONATHAN: We keep state outside. Or we try to think of primitives. I think it's a very healthy way of thinking about making stateful system work well.

So in our case, the state of the user is stored in a document database where they keep all their chats and what we call the trajectory, which is the chat messages, and their Git, which is the state of their project. And then we have Lovable cloud, which is, you know, the state of the application, which is not our state. So there's two pieces of state here. But mainly, everything is driven through the project itself, the trajectory, and Git.

ABDEL: So in a way, you have basically two types of states. You have the state of the projects where it's being developed. And then you have the state of the projects where it's running.

JONATHAN: Exactly.

ABDEL: So when the application is deployed essentially.

JONATHAN: Yes.

ABDEL: So from the perspective of the user, if I understand it correctly-- and correct me. If I go too Lovable, I log in. I get assigned a pod. That pod has my workspace, essentially, let's call it, right?

JONATHAN: Sure.

ABDEL: That's the sandbox from your perspective?

JONATHAN: Yes.

ABDEL: I do my chat. I talk to a large language model. The large language model does the code changes for me. I assume I hit Save, or there is autosave?

JONATHAN: Yeah the agent is autonomous here. Of course, Lovable is a bespoke agent or harness that does things a certain way. But yes, every time you chat, if it succeeds, then it's always persisted. That's part of the iteration loop.

ABDEL: But then it's persisted outside the pod. So in case the pod inside the session ends unexpectedly or whatever, you can always go back and reassign it to a new pod, but then reload the existing state.

JONATHAN: Exactly, yes.

ABDEL: OK. If this is a question you can answer, how big are these pods? Like, how in the size, like the image, the container, whatever?

JONATHAN: That's a very nuanced question. So this is why I brought up the CICD or SCI before If you imagine you're running a CI job, you start a new GitHub action or whatever. In the beginning, it's like, oh, I need to compile this before I test it.

So it will spike in CPU immediately to as much as it can in memory, and then go down to basically zero or whatever your workload needs. And this is very much the problem we have where I think our pods are very small. Like, our sandboxes in terms of unit economics at our scale is very important. So we need to manage it very well, so all the overcommitment and this. But I think they can spike to like 15 or 16 cores, or I guess as much as they are allowed to do.

ABDEL: Yeah, of course.

[CHUCKLES]

JONATHAN: And then they go down to less than half our VCPU after. Depending on the size of the project, it's kind of a power law thing where big projects take more time.

ABDEL: Of course.

JONATHAN: So we have to act kind of around the heuristics, basically. We can't really have a fixed amount of resources we give or take away from pods.

ABDEL: Are you hinting to the fact that you are using the dynamic resizing of pods?

JONATHAN: We do not. So in this case, Lovable might be highly-valued company, but we are very new and we are quite small. So we do a lot of not overprovisioning, but warm pooling in this case to work around these kinds of problems. So that's why Kubernetes and how we built it's very nice, because we just built it around native OS Kubernetes primitives. So no resizing and stuff for us right now at least.

ABDEL: Yeah. I mean, the feature itself is open source. But the dynamic resizing is still not there in open source. So you hinted to an important point, which brings me to the next question.

When we met back in March, we talked about the agent sandbox, and then you told me. You were not actually using the agent sandbox today. Or at least back in March, that was not the case, right?

JONATHAN: Yeah.

ABDEL: But you were actively looking into it. So can you talk a little bit about how did you build the system? And where does the sandbox fit into the system?

JONATHAN: Yeah, so how we start the system today is I have a brilliant colleague, Will, who came up with this idea that if you create a deployment in Kubernetes, then it will assure pods always around. But if you remove the selector label and the owner reference from the pod, then the deployment be like, oh, I lost a pod, I need to create a new one. But the pod will continue living on forever.

So this is warm pooling for us. Whenever someone says sandbox, we remove the owner reference and the selector label. And thus, we have warm pooling. So we didn't really have to build a net technology around that.

But as you get to scale-- and this worked really well up until maybe one or two months ago, because we had grown so much that we just hit problems in the replica set controller where it thinks it has pods still. Because we popped them from the deployment, and things kind of break. So we are looking into refining this.

But we didn't know that like the SIG group for Agent Sandbox was coming up when we started working on this. And we have a tight relationship with Google. So when that came in, we're like, oh, this is what we should use. Like, why not move towards a shared understanding of sandbox scheduling and operations? But to be honest, Agent Sandbox as the cgroup today is still not ready for high-throughput sandboxes.

ABDEL: Yeah, clearly.

JONATHAN: It's built for-- not a different purpose. But a good example is that each sandbox with Agent Sandbox creates a Kubernetes service that is headless.

ABDEL: Yes, because you need to talk to that specific pod. Yeah.

JONATHAN: Exactly. We don't really need that. And we can't do that because we would have one service per pod. And when you run 100,000 of them, it ends up being a problem.

ABDEL: Yes.

JONATHAN: So there's all these things that will work great if you're at low scale here, meaning thousands still. But as soon as you get to the edge, the API server starts breaking, you need to reduce queries per second in all your controllers than how the agent sandbox is today. It doesn't work that well.

ABDEL: Yeah. I think we jumped a little bit too far, I think. So for those who don't know, I'm going to leave a link in the episode for the Agent Sandbox itself. But maybe, Jonathan, can you describe the Kubernetes Agent Sandbox, the open source projects briefly? What is it actually?

JONATHAN: Yes, it's a set of custom resources. In this case, it's a sandbox warm pool and a sandbox claim. They all each have a controller that sits and waits for these objects. And what they allow you to do is this thing that I mentioned before.

You create a warm pool. You describe the warm pool. It's like this is the pod template for that warm pool. And it has some small details in how you create pods and if you should recreate them, if there's a new image, things like this.

And when you want to take a pod from the warm pool, you create a sandbox claim. It's quite simple, and I think it's very elegant. But it does not use deployments and replica sets underneath. So it's essentially a replacement for--

ABDEL: It just creates pods.

JONATHAN: Yes. So it's kind of a replacement for deployments or stateful sets in this case.

ABDEL: Yeah. The reason why it's called Agent Sandbox, it was specifically designed for AI agents to be able to run agent-generated code inside a sandbox environment. But I have talked about it in a couple of conferences, including your use case, which, by the way, I saw on X from your CTO. I have seen it being considered in other use cases, including what you're talking about, because what you want is not to just run agent-generated code, but run the entire agent with all its environments inside a sandbox environment, basically.

JONATHAN: Yes, for sure. So this is where things are moving around week from week in the agent, because the agent world, tech world is very fragile or fast-moving.

ABDEL: Very agile I guess.

JONATHAN: Yeah, so like in December, it was like, no, of you run the agent on your laptop. And then like, oh, wait, now open cloud, I want to run it somewhere else. And now there's, like, AIs, Claude, and I guess all the pods OpenAi also like, Oh, why don't we just run the agent all the time at the provider? And this is where, I guess, these kind of agent sandbox-esque solutions come in.

ABDEL: Yeah, and specifically when you look about the design of the agent sandbox projects, literally, what makes it sandbox is the fact that you change the runtime that's in the pod template. That's the only thing that makes it sandbox, right?

JONATHAN: Yeah, exactly.

ABDEL: So speaking about that, in the context of Kubernetes, the sandbox usually refers to either using gVisor, which is the Google thing, or using something like Firecracker from AWS or Kata Containers, right? So how do these tools fit into your particular use case?

JONATHAN: Yeah, so we've used gVisor since we started. Actually, we started with Firecracker, but that was before I joined. It also works well.

But there's some really both operational and security details between this, or performance details between all of them. So you have to know each of the characteristics before you pick one. There's no one solution fits all, really.

But we are really very happy with gVisor because there's no virtual machine in gVisor. So it's maybe a bit deeper tech, but it's like a userspace kernel that manages things for you. What this means is that if a sandbox in our case uses a lot of memory, and then it uses no memory, then like on your normal machine, the machine will just reclaim it.

This is not really possible with other solutions. There are some hacks around it, like memory ballooning and things. But this is the biggest reason why we use gVisor in this case, because then we can give a lot of memory, and just take it back whenever we want.

ABDEL: Yeah, I mean, to go more into the details, in this particular case, the key difference between gVisor and something like Firecracker or Kata Containers is that Kata Containers firecrackers are microVMs, right?

JONATHAN: Yes.

ABDEL: The sandboxing boundary is a microVM. But for gVisor, it's a Linux user space kernel, essentially, right?

JONATHAN: Yes, exactly. It's like a normal process on your machine.

ABDEL: Yeah, and so speaking of that, like being a Linux user space kernel, it means that it doesn't pass through the syscalls to the node. It implements them in that particular kernel.

JONATHAN: Yes.

ABDEL: So have you seen any incompatibilities between your application code and the implementation that gVisor supports?

JONATHAN: Yeah, we actually could not use gVisor when we started, when we started experimenting this like in August. We love a technology called Bun. It's great. It's a replacement for MPM, more these kind of things.

ABDEL: Yeah, Node.js replacement. Yeah.

JONATHAN: But how Bun works, to make it really fast, you have something called hard links. So Bun installs your node modules in a shared space. So if you have a different project, you essentially link from your file that is in your project to the shared storage. So you don't need to install multiple times. But gVisor doesn't like that because it doesn't give you access to the disk and to the system.

So how they implemented hard links is they just write it twice, as far as I understand it. So it's actually quite inefficient. And so most of the things we've ended up having problems with is file system and network. It's just based just pure performance.

ABDEL: Yeah.

JONATHAN: So as you can imagine, a userspace kernel, in some cases, it can be more efficient than the kernel itself. For example, if you do file system things, you can often skip the kernel directly and just go directly to the disk with direct I/O or BPF, for example, for networking. But in our case, it ends up in some cases making our Bun install times take five times longer in the worst case.

ABDEL: Yeah, because it has to duplicate the dependencies on the node essentially.

JONATHAN: Yeah, exactly. And since you're adding more work in the network in the file system path directly, the CS takes longer.

ABDEL: Yeah.

JONATHAN: So it's not really an incompatibility in that sense. It's just slow.

ABDEL: Because I get this question quite a lot when I talk about gVisor from people, the question of, what limitations do you know in gVisor that might impact my application? And my typical answer is, you have to test it. It's the only way to know. Do you have a different-- would you answer this question differently?

JONATHAN: No, I think you're right. We use gVisor, and it works. It works great. And it's the opposite, right? Like what doesn't work in other solutions?

ABDEL: Yeah.

JONATHAN: Like you have to pick the one that has the least amount of flaws that you care about. In this case. So yeah, we have to live with the fact that it's slower just because we have the memory capabilities. This does not mean that it's not fixable in gVisor. So there's commits all the time with the vendors that we work with they use fix file system-specific things in gVisor to make it faster for us.

ABDEL: Yeah, so it's not really incompatible. It's more a performance issue that is solvable essentially.

JONATHAN: Yeah, exactly.

ABDEL: Yeah, and I think that brings us, or brings me at least, to a question, or kind of a logic, that I have in my mind. So whenever we talk to people about both process isolation for containers-- so gVisor, Firecracker, whatever, or the whole concept of, like, fast start of containers, like how fast they get to start pods, blah, blah, blah, there is always this trade-off between performance and cost, essentially. So how much are we willing to pay in performance, and how much are we willing to pay cost? And everybody seems to be wanting the best of both worlds, which is not possible. So in this particular case, the choice between gVisor, Firecracker, or Kata Containers will be how much isolation do you want and how much speed do you want?

JONATHAN: Yeah, there's also nuances to that. Right now, the big fad, I guess, of the last week is to have as fast cold start as possible. But people still look at Firecracker and be like, oh, you can start. In like 70 milliseconds. And then some vendors, like, we can start in 10 milliseconds.

But in the end, a lot of those things really don't matter in reality. This is why-- again, goes down to the use case. Imagine, a user for Lovable will go into their browser. it probably will take, 200, 300, 500 milliseconds to render the browser.

So there's no need really for the sandbox that could run in parallel to start in 20 milliseconds. The user will still have to start writing. It takes a couple of seconds.

So this is where people really need to optimize for difference. But specifically, cold starts have been the recent competitions There's competitions about who has the fastest cold-started sandbox.

ABDEL: So this is actually an interesting discussion, because in the context of Kubernetes, your cold start can be basically caused by literally two things. It's either the node cold start itself, so how fast cluster itself can autoscale, or how fast the pods can autoscale, which then includes things like how fast you can download the image into the node if it's not already cached, how fast your application itself starts, blah, blah, blah. So the way I reason about this all the time is that somebody has to pay for that cost somewhere.

Like, we can't make all of these things fast always, all the time. So you either have to accept to balloon or to have one pool, and then you're paying the cost for that. Or you have to accept to pay the cost for startup.

JONATHAN: Yeah, we take the choice of the warm pool in this case. We optimize for the user experience only. Of course, cost is a thing, and we believe that the cost from being able to reclaim memory offsets the warm pool availability. In our case, we have to fetch a Git repo and do a bun install And something we actually haven't talked about, which mitigates a lot of this is, you know, disk and memory snapshotting.

ABDEL: Yeah, that's an interesting point. So let's talk about it. So what are you using? How are you solving this with snapshotting?

JONATHAN: We have experimented with every type of snapshotting. So, bundling the most common like node modules in the image is kind of snapshotting. You always take the latest image.

We have moved since then to a more efficient way to use Git. But before, we used to take disk snapshot only of one folder, and then mount that folder. Whenever the pod is claimed or the sandbox is claimed from a warm pool, we would mount a file system for the Git repo, so they don't have to fetch it again.

Because there's some limitations, we had to fetch the entire repo, all branches and all commits every time. We don't have that problem anymore. So now disk snapshotting has less impact on our general projects.

But of course, as with anything, you're from Google, there's this power law problem where some customers or the far end is a lot bigger than the general. So you can imagine people create projects that are massive. And then of course, skipping a Bun install could save you minutes. So then again, the cold start of the pod does not matter. All that matters is, you want to do as little amount of work possible until the user can start working.

ABDEL: Yeah. There are always extreme solutions you can take to solve these kinds of problems. So one of the things I was thinking about for your particular use case, maybe you could always get the most expensive network-attached disk. And then literally just dump all of your Git on it, and then mount it to the node when the node starts. But then that's going to be a problem in the sense that somebody has to pay for that disk, right?

JONATHAN: Yeah. And unfortunately, in Kubernetes you cannot add a PVC to a running pod.

ABDEL: Not dynamically.

JONATHAN: Exactly, so we can't really use the warm pool Kubernetes native way to do it. So here are some other ideas like FUSE. There's also a new movement here, like doing FUSE file systems, like userspace file system, to GCS or to S3 to mitigate that problem.

ABDEL: Or maybe the other one would be the new-- I'm not sure if it's possible in Kubernetes yet. I think it is. It's mounting a Docker container as a volume, essentially.

JONATHAN: Exactly.

ABDEL: But that also comes with caveats, because basically, it matters how often that Docker image has to be updated.

JONATHAN: Yeah, we do this. I think we update our-- we have this, I think it's called secondary boot disk, which I believe is rather new, where you can get a boot disk into a node where you have all your images. And then you don't have to stream it.

So here, we can skip, 85%, 90% of the image streaming to each of the nodes. And we update that every week or every day. And it saves at least a few hundred milliseconds in the general case.

ABDEL: So there is somebody working on something. And it's not public yet, so I'm not going to mention the name. I'm not going to mention any of this stuff.

But I'm going to just tell you on a high level, what they're working on. Somebody I know is working on something that is around the lines of a dynamic registry. So what does that mean?

It's just a Docker registry. And you would do a Docker pull on that registry on a specific image. And then you would do a comma separated list of plugins.

In your particular case, you could imagine you do a Docker pull on Bun plus a bunch of bundles. Then the registry would return to you a Docker manifest that is composed on the client side. So the Docker manifest contains the Docker layers.

And for each layer, it contains the link to where that layer is in registry. And it's coming soon. So I think once this becomes available, this could be actually something you could use to solve the dynamic linking of Bun. Because essentially, what you want to do is dynamically bundle Bun installation on the pod, right?

JONATHAN: Yeah, this is very familiar. I know there's another-- I think it's a CNCF product called Dragonfly, if I remember correctly. I'm confused if it's the Redis clone, which is also called something similar, or a sandbox type. It's also called Dragon-something, which allows you to stream OCI layers down in peer to peer and also send them out.

So I've heard what you're working on. It seems to be also propagated in the industry the last six months or since last summer. I've heard it pottering, and people figure out like, how do you hack the OCI? Or not hack, but use the OCI manifest to add new layers from the machine?

ABDEL: Yes, like dynamically inject the layers into the node. That's another one. That particular use case is useful for patching a running container, actually. So if you discover a vulnerability in an application, you could just patch the particular layer containing that that dependency without restarting the application, right?

JONATHAN: Right, so if it's like the read-only layer, for example, for a specific path, you can just replace it as it's running.

ABDEL: Correct, without restarting the container. So this actually brings me to-- I believe it's going to be my last question. A lot of what we talked about for your particular use case is that you are doing things in a "hacky" way. I'm putting "hacky" in codes here because you're trying to optimize Kubernetes for your use case. But do you see Kubernetes moving in a direction where all of these things are not going to be a problem for you?

JONATHAN: I don't believe that's going to be possible, but not from a technical reason. I think you can definitely optimize Kubernetes for our use case. But you would have to be willing to replace core parts of Kubernetes that other people depend on.

Kubernetes is quite pluggable. But a content and topic, which depends on what you talk about is like, oh, we only use etcd for example, and then as like, oh, this is the layer we should work on. And etcd is perhaps not the most scalable database, depending on-- I don't know if they fixed it. But at least during last year, you cannot have more than 16 gigs of storage in etcd That's like a hard cap.

ABDEL: Yeah, max? Hard Cap

JONATHAN: So many providers are like, oh, we have to make something etcd compatible and run it on Spanner, for example, or run it on foundationdb and things. So where our problem is going to go specifically for these high churn, high super-wide horizontal running 500,000, a million sandboxes, and the churn is hundreds of thousands per second, I don't believe Kubernetes as it is, will be right. Maybe it is a nice solution to work with. But it would require such radical changes in the core that it's hard to get it upstream into Kubernetes. So then maybe it's easier to work around it.

I know, for example, people use virtual kubelet for this heavily. So for example, you make your own per node scheduler that you don't have to rely on Kubernetes API or control plane because it can be quite slow. But it's Kubernetes-compatible, so you run your control plane in Kubernetes.

But the node itself is just kubelet compatible. So you can register it as a node. This is, I think, the way I like to think about it.

ABDEL: Yeah, I know exactly what you're talking about. It's something like an external runner where the control plane stays Kubernetes. But where the stuff runs is not the way Kubernetes runs things, right?

JONATHAN: Yeah, exactly. So I do think-- I'm not sure virtual kubelet is-- I'm pretty sure it's a CNCF project.

ABDEL: I think it is. I'm not aware of any provider that has this today.

JONATHAN: I'm quite sure you can run, for example-- like, I don't know if it's ECS or other functions as a service, essentially.

ABDEL: Got it.

JONATHAN: I think that's the common one, though you can also do like CRI, which is like run any type of container through it. So in this way, we can keep the good parts, which is if we work better, this is good, this running control planes where it's like really good. And I know people don't like Kubernetes for stateful workloads. But Vitess is huge and it's great. And if you put the thought into understand how Kubernetes works, it's like best in class for that.

ABDEL: So the more you talk, the more I have questions. How are you guys solving that? Like, are you using Vitess?

JONATHAN: No, we do not.

ABDEL: But do you give people-- like, when they're building an application, how do you handle the databases within their sandbox?

JONATHAN: Yeah, so Lovable cloud is, whenever you ask for a database, we will give you essentially a Postgres database that can scale vertically in this case and a S3-compatible bucket and a few more things.

ABDEL: Running inside Kubernetes.

JONATHAN: No, that runs on a third-party provider today. But this is also a very interesting question-- which is maybe besides here, but where I believe there's a project, I want people to work on it. Maybe I should bring it up.

I would love for people to work on something like Vitess. Vitess is essentially MySQL, but it's made for like massive single databases. YouTube is famous for like one MySQL database, but it's, I don't know, thousands, tens thousands, hundreds thousands of nodes?

ABDEL: A lot of nodes.

[LAUGHTER]

JONATHAN: Massive, where the new world is sandboxes. You could, of course, run the database on the sandbox. But then you need to keep the sandbox always running.

ABDEL: Yes.

JONATHAN: So I believe the world is going towards where-- I guess Neon revolutionized the industry last summer for a different type of use case, which is not production-ready. But it's like, how do we run 100,000 databases?

ABDEL: Yeah, and they were so successful they were acquired essentially.

JONATHAN: Yeah, exactly.

ABDEL: Eventually.

JONATHAN: They did very well. And it's a completely different problem than running a database and keep it always available.

ABDEL: But I do see your point that, in the future where you might need to run hundreds of thousands of agents on the same cluster, you need to solve this. Like, how do you handle hundreds of thousands of databases? Because you want to have isolation between each autonomous agent. I don't want my single open core instance to just be able to talk to all my databases, because agents are agents.

JONATHAN: Yes, exactly. To be honest, it's where you need people with knowledge. It's really hard to install a managed system to do the isolation for you today. So I guess it's the way security works. You can never know really, like truly, that all your stuff is isolated.

Even if you're no network, airgapped, you still don't know. So that's also something that we work on all the time. In our case, it's mostly about finding the anomalies if they happen, and then trying your best to keep them from happening, and pentesting and such.

ABDEL: Yeah, I mean, as we always say, security is as strong as the weakest link within the system, essentially, right?

JONATHAN: Mm-hmm.

ABDEL: Yeah, awesome, cool. Well, I do have one more question, and this is unrelated to Kubernetes. How do you come from a family that makes chocolate to Kubernetes?

[LAUGHTER]

Help me draw a link there.

JONATHAN: Yeah, I had I talked about it with my-- I was out with my dog during lunch. And my neighbor came-- like, my family is a family of entrepreneurs, Essentially everyone-- my grandparents, and all their kids and all cousins.

So chocolate ended up being it for my mom. She has had so many companies, solo entrepreneur her whole life, including my dad. So she's like, I want to make people happy. And she likes to talk and converse.

And she's kind of-- I don't know what you would call it, pastor or herder or whatever you want to call it. So she thought that, chocolate is fun and tasty. And what if I just teach people how to chocolate and we just have fun?

ABDEL: Wow, nice.

JONATHAN: And so I think 15 years ago she became a chocolatier, did the formal training, and started from home. And now she's the local famous person in our municipality, like, oh, the chocolate lady. And she named her company Chocolate for the Soul, that mentality.

ABDEL: Awesome. That sounds awesome. All right, so I think that this is probably going to be the first time we do this on a podcast. We're going to leave a link to your mom's company or store in the show notes.

JONATHAN: Please. Yes, you can find-- 30 minutes drive outside of Stockholm in Sweden.

ABDEL: All right, probably the next time KubeCon comes to Stockholm, we can make sure that your mom talks to the CNCF.

JONATHAN: Yes, she'll do the catering for the desserts.

[LAUGHTER]

ABDEL: Awesome. Thank you so much, Jonathan. It was a great conversation.

JONATHAN: Yeah, likewise. Super nice to be here.

ABDEL: Thank you.

JONATHAN: Yeah thank you so much.

[MUSIC PLAYING]

ABDEL: That brings us to the end of another episode. If you enjoyed this show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @kubernetespod or reach us by email at kubernetespodcast@google.com.

You can also check our website at kubernetespodcast.com, where you will find transcripts and show notes and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thank you for listening, and we'll see you next time.

[MUSIC PLAYING]