CRAIG BOX: Hi, and welcome to the "Kubernetes Podcast" from Google. I'm your host, Craig Box.

[MUSIC PLAYING]

CRAIG BOX: Thank you for allowing me another couple of weeks to have a summer break here in the northern hemisphere. London gig update, I saw Queen, but I am yet to see the Queen. Let me tell you about one of the other highlights of the trip so far.

There's a very pretty area, not far from here, called the Cotswolds, full of rolling hills and the kind of stone buildings that you picture when you think of the English countryside. One of the villages in the Cotswolds is called Bourton-on-the-Water, one word, three hyphens, named for its sitting either side of a river with bridges that date back to the 1650s. Three thousand residents, 300,000 visitors a year. I'm sure you can picture it. In case you can't, it very helpfully contains a 1:9 scale replica of itself in the garden of a pub at the edge of the village. Properly framed, it can make a toddler look like a giant.

It being the English summer, of course it rained cats and docs while we were there. That would have also been very odd scale-wise had it been literal. The interesting thing is that the model village is within the boundaries of the village itself. So the model village has a model village. And that model, being a model of the model village contained, you guessed it, its villages all the way down.

Unwinding the stack, we have a painting of a model village inside a tiny plaster model village in a stone model village in the perspex case in the 1:9 scale model village in the village of Bourton-on-the-Water. I think I have that right. I can't begin to count how many hyphens there would be. Please make your own recursion joke while I get to the news.

[MUSIC PLAYING]

CRAIG BOX: GKE Autopilot, the operation mode which provisions and manages your clusters' nodes, continues to close the gap with its Standard sibling. You can now use Google's Dataplane V2 on Autopilot clusters. The eBPF-based programmable data path offers support for network policy as well as additional logging and monitoring metrics. You can also now use egress NAT, useful if you're using a private internal space like Class E for your pod IPs, but you want them to be able to talk to the internet. Alternatively, if you never want to have to worry about that again and you're happy to run GKE Standard, you can now create dual-stack clusters, which support both IPv4 and IPv6. Other new features in GKE Standard include time sharing for GPUs, and confidential GKE nodes, where all data and memory is encrypted.

Some new open source projects. Paralus is a Kubernetes access management tool built by Rafay Systems. A single login promises authorized users seamless and secure access to all clusters through kubectl. It is derived from Rafay's Zero Trust Access Service and supports integration with SSO providers or identity providers using OIDC. In case you were wondering, the Paralus was a messenger ship of the ancient Athenian Navy.

Furiko with a K is a Kubernetes-native platform for scheduling, executing, and managing ad-hoc and CronJobs. Furiko was able to schedule jobs to be run on a periodic or once off basis and had several features above and beyond the standard job and CronJob controllers. Furiko was built at Singaporean e-commerce vendor Shoppee with two Es and is the Japanese for pendulum.

Some existing open source projects have been accepted into the CNCF sandbox. Please welcome Clusterpedia, for searching for resources across clusters; OpenCost, which we covered in our last episode; Aeraki Mesh, adding the ability to manage any Layer 7 protocol to Istio; Curve, a distributed storage system; OpenFeature, a standard for feature flag management; Kubewarden, a policy engine that's less REGO and more WASM; and DevStream, an open source DevOps toolchain manager.

The team that brought you the Traefik ingress and proxy have this week released Traefik Hub, a centralized platform for publishing services. You set up the Traefik proxy in your clusters to talk to the hosted hub service. And then users can access the applications running in your clusters published over a secure tunnel. Hub manages things like authentication and SSL certificates for you and is pitched to help with collaboration between teams.

Security vendor Cyble, with a C, has found almost a million Kubernetes endpoints publicly available on the internet by searching for simple things like the string "kubernetes", and complex things like the hash of the favicon served. They reported 900,000 discovered endpoints. This is not a concern in and of itself given that most cloud implementations run a public endpoint by default, and Cyble acknowledged most endpoints returned a 403 error when someone tried to access them anonymously. Almost 800 clusters allowed interaction however, with some of them running the venerable Kubernetes Dashboard. Plenty of security products exist to make sure you are not doing this. Please make sure to run one if you're concerned.

Finally, a follow up on the Bitnami Helm chart index truncation we mentioned in the last episode. The CloudFlare-mirrored index remains truncated, restricted to items published within the last six months. But Bitnami has published an FAQ on how you can update your chart to refer to a full index which they have now published on GitHub.

And that's the news.

[MUSIC PLAYING]

CRAIG BOX: Steve Wade is an independent Kubernetes consultant and trainer based in the UK. Welcome to the show, Steve.

STEVE WADE: Thanks, Craig, pleasure to be here.

CRAIG BOX: When someone puts a year in their Twitter handle, it's reasonable to assume that might be their date of birth. I think that probably puts you a little bit too young for the 8-bit micro, but were your formative years on the PC or were you an early starter?

STEVE WADE: That is correct. 1987 is indeed the year of my birth. Yeah, I actually started out playing football more than I did sitting at a computer. But I had an eye on what my father was doing. So my father was in computing for a long time.

He was sat writing random gobbledygook on his laptop. And I was fascinated by what this all meant, because I was learning how to read and write and reading books. And the thing that he was writing on the screen looked absolutely nothing like what I was reading. So I had an inkling and a kind of interest in what he was doing, but I wouldn't say a passion at a very early age. I had other distractions.

CRAIG BOX: Football sounds like it was that distraction for you. I understand that you could have had a very different career path.

STEVE WADE: Yeah, so I played professional football at a well-known Premier League team. Premier League, for anyone that doesn't know, is the top football, or soccer, division here in the United Kingdom. I actually got scouted at the age of four.

CRAIG BOX: Wow.

STEVE WADE: I don't know how that's even possible, to be honest, how you can tell that someone's good at four, but never mind. And played up until I was 18. And I was due to become a professional footballer, sign on, and do that. But one of the things that was important for the club at the time, in the academy, was that playing football wasn't the only option. And you had to have a backup plan.

So I went to school. I went to college. But one of the things that I also did is they also did the concept of football in college and football school. So what you do there is you learn all of the bad things about football. So think about gambling, think about other kinds of addiction.

They also wanted everybody to get to a certain level academically, because they felt that was very important. I spent a lot of my time — whilst I was playing football at football school — also being more interested in coding. So that's where I kind of picked up my love for it, because I saw my father was writing code, he spent all his time on a computer. We go back a couple of years. I was talking about how I was always interested in what he was typing. And then thought to myself, well, he's earning good money, he's making good decisions, maybe I can have that as a backup plan.

CRAIG BOX: Right.

STEVE WADE: Roll forward to when I was 18, bad injury like probably every footballer that's never made it. Bad knee injury caused me to essentially have to stop playing. And what I did is I took all of the time that I was training outside of when I should have been training, so I was doing extra work. I was doing rehabilitation, I was doing stretching, I was going swimming, I was doing ballet, I was doing all of these things outside of the core football training to make sure that I put my body in the best position for longevity.

It turns out that wasn't the best use of my time, but never mind. We move on. But I used all of that time that I was previously doing that and I turned it into using it to write code. At the time, I didn't know what I was doing. So I sat down with my father.

He was talking me through all of the basic concepts, the if statements, the for loops, the while true loops, all of the basics. And I scratched the surface at that point. And then I used a lot of my time to really go hell for leather into learning this way of writing code. And it got to the stage where I was as passionate around coding as I was when I was playing football.

I was watching football videos. I was watching previous matches. I was watching matches of myself. And now I'm reading books about programming languages, and I'm looking at the latest programming languages. And I think that passion that I had has put me in good stead moving forward because the landscape is constantly moving in technology.

And you've got to know what itches you want to scratch. But you've also got to know where you want to put your passion in such a broad landscape around technology. There's so many things that you could do. But you really have to become that T-shaped or Y-shaped person that everybody talks about. You've got to have one main passion that you're really interested in, but have a broad understanding of how that fits into all the remainder of technology.

CRAIG BOX: It sounds like it was very forward thinking of the Premier League to put people through that situation where obviously some of them aren't going to make it through — illness, injury, whatever else — but some people, I guess, will play for a while and come out the other end as well and need a different career. Do you feel that the particular combination of high tech programming and football was quite unique at the time? I wouldn't imagine that growing up through the '80s and '90s that those two things were necessarily paired together in many people.

STEVE WADE: No, I think it was everybody looks up to somebody. And at that time in my career I was looking up to footballers. But outside of my football career, I was looking up to my father.

CRAIG BOX: Of course.

STEVE WADE: And I think that's natural for a boy. I saw what he was doing. And I found it fascinating. It was that interest that I had that allowed me to move forward. A lot of other children and kids, and boys and girls, at the time that were at the academy, weren't interested in this kind of stuff. They wanted to go into the more obvious routes. I think coaching, they wanted punditry, commentary, those kind of things that followed the route of being a footballer and allowed them to continue to do that.

That was something that I was interested in, but I didn't want to put 100% of my time into football, or soccer. I wanted to put 90% and have 10% that was totally different. It's important to be able to switch your brain off from one thing and have a kind of "distraction," in inverted commas, to go and do something completely different.

And writing code was that. It was an interesting story, really. I would sit at the Academy with a very old laptop, typing away. And all the boys would be like, what on Earth are you doing? You've just become like a total nerd overnight. We don't know what you're doing.

CRAIG BOX: Tell them you're playing "Doom."

STEVE WADE: Yeah, if only. Yeah, I'm still in contact with the people that I used to play football with. And we go out for dinners. And we have some very interesting conversations around what I've been doing and what I'm doing currently. And the career trajectory of some of those people are vastly different.

We have people that are actually doing coaching and who are coaching children. We have some people who are also in IT, but in totally different fields. And then we have people that currently work on oil rigs up in Scotland. So there's a very broad range. But when we were at school — at football school — they weren't teaching you necessarily all of these things. But it's just an inquisitive itch that people want to scratch that lead them down that direction and that path.

CRAIG BOX: People do get their experience from all sorts of places. One thing that I found interesting reading through your CV was that you were a meat and poultry team leader at Tesco. I worked at a supermarket when I was at high school, only ever part time. But it really feels like you had a background to gain experience and leadership at a very young age.

STEVE WADE: Yeah, so I was lucky enough that when I was playing football, I had the opportunity to be captain of some teams. Not all of the teams, not throughout the whole time I was playing football, but I was pretty good with people and people management, obviously not in the technical aspect, but in a building relationships and making sure everyone's singing from the same hymn sheet style.

And when I got the role at Tesco's, I actually asked to position myself as a team lead and not just to be somebody that comes in and stacks the shelves. That wasn't really something that I wanted to do. I wanted to continue with my people development skills, because I knew that when I was going to go into technology — if that was the route that I was going to go in when football had finished or come to an actual end — I needed to be able to engage with people at different seniorities, both above me and below me.

And I knew that that was a good place for me to be able to do that. An interesting fact is, to this day, touch wood, of all the time that I was at Tesco's, I never once got on a checkout, no matter how hard they tried. I somehow managed to get away with not going on a checkout. And for any of you, specifically you, Craig, who's worked in the supermarket, that is quite the feat.

CRAIG BOX: I was on the checkout. And I always used to enjoy that. I was very multi-purpose, I guess. I'd go fetch trolleys some of the time. Most of the time was stocking the shelves, but on occasion they'd need me on a checkout. I had my own till. And every week, if I had all my change back in the right amount, within a dollar or two of the correct amount of change, they'd give you a chocolate fish.

STEVE WADE: Nice.

CRAIG BOX: Think of all the chocolate fish you missed out on by never being on the checkout.

STEVE WADE: I know. I'm thinking now maybe I should have actually gone on the checkout after all.

CRAIG BOX: That did set you up for a consulting gig where you do have to come and work with management, you have to work with engineers. But the form of work that you do in contracting is somewhat unique to the UK, especially around the London area. Could we start by digging into what contracting is, hopefully without using the term IR35?

STEVE WADE: Sure. So for me, contracting or consulting is where you come on site and help the engineering team or multiple engineering teams facilitate change. And that change may have been asked by the permanent members of staff for many months, many weeks, many years, but they can't get it through senior management.

And for some reason, when a consultant comes online or a contractor comes into the organization and starts making statements, they for some reason get listened to more than permanent members of staff. Which, to some degree, blows my mind to be perfectly honest with you. But in other ways allows me to help to build relationships with those permanent employees and myself, but also with those permanent employees and senior management.

And normally, the way that you get contracted could be right at the start of the project, because they're a start-up, they've got a lot of money, but they don't want to have permanent employees. Or it could be midway through a project that they need some specialist expertise, and you have that, and they want to leverage that. Or it could be that the project is going in a direction that isn't the way that the management wanted it to go in, and you're kind of parachuted in to help it or save it. I think those are the three different categories that I would put contracting in. You're going into one of those three.

CRAIG BOX: You can get a much higher rate as a contractor. But then you're obviously also self-employed, you have to pay your own taxes, and you don't necessarily have the guarantee that you have a job at the end. They can terminate you when the project finishes. And it could be many months before you get more work.

So there are ups and downs obviously between whether people want to take contracting gigs or whether they want to be the "permy," as they're called by the contractors. You were a "permy" for a couple of years after finishing school. You've mostly been doing contracting with a couple of different companies in there with permanent roles as well. How did you decide at each step whether you wanted to be a permanent employee or a contractor?

STEVE WADE: I think, for me, it was always around the cliche of when you stop learning is the time to move was always what I went with. So if I'd stopped learning or felt like I'd reached an actual point in my career where I needed to make a jump, I would make the jump. And in terms of contractors versus perm, the reason why I went into a contracting role was I was working with contractors, when I was permanent, who told me that I had the skill set to become a contractor.

And I just needed to make that first leap and then everything would just work out for itself, minus the fact, to your point, you have to pay the taxes, you have to get an accountant, you have to set up a business, all of those things. I felt that I couldn't become a contractor straight away. I needed to earn my stripes. I needed to see a lot of different industries and get a good understanding of the landscape first.

I'd played football for so long, I'd done a little bit of technology, I have a degree in computer science, but I hadn't spent a lot of time in industry. So how do I engage or have conversations with a product manager? Well, I'd never done that before.

I'd spoke to a manager or my boss playing football, but I never had all of these different types of people in my team. I'd always had the same kind of people. And I wanted to be able to build up that skill set. And what you'll notice is in my permanent career, I was actually consulting.

They were putting me out on site. I was going to customers. I was kind of doing contracting without doing contracting, if that makes sense. I was having to build the relationships on site with the clients. I'd have to speak to people above and below me in terms of the area that I was working in.

And I just thought to myself that going out on my own would allow me to have a better choice or a wider breadth of options, potentially, of companies that I wanted to join. And yes, the day rate — you can charge a higher day rate like you were saying — but it's a risk and reward. I, at the time, backed myself that if I could make it work while I was permanent and people wanted to keep me at the client site for longer than the initial duration, I knew that I could make it work if I went on my own.

It was just a case of getting in the door and starting building the connections like I was when I was permanent. And in terms of when did I choose to leave companies and when did I choose to move on permanent versus contracting, the permanent stint right in the middle of when I was contracting was a chance to be one of the principal Kubernetes consultants at that company.

At the time, Kubernetes was relatively new. The experience that I had was relatively limited. It was at one company. We just deployed it. It was one of the first versions of Kubernetes. It was before deployments were even a thing.

And to be able to go out to multiple clients around the world, seeing multiple different industries, being in multiple different domains, and experiencing ways that people are leveraging Kubernetes that are vastly different to what I was doing, was just an opportunity that I couldn't miss. It put me back in a permanent position, but allowed me to continue consulting. So I was still doing what I was doing, but they were going out and finding the clients that were leveraging Kubernetes or that wanted to leverage Kubernetes. I didn't have to go and do that work myself.

The resources and network that I built from the time that I was at that company are now invaluable. The Kubernetes community is large but actually is relatively small, and the same people pop up every now and again. And I'm still in contact with the people that I worked with at that company.

I'm still in contact with the people that were my customers whilst I was at that company. And they help me, I help them, it's the classic Kubernetes community. Everyone's there to help each other. It really put me in good stead for the contracting roles that I've had post that permanent stint.

CRAIG BOX: A lot to unpack there. A lot of companies bring in consultants or contractors to deal with new technology until they've been able to train their own staff up. And that means you're expected to be able to come and be up to speed with something immediately. And you're not expected to be learning it on the job. At that contracting gig you had before you joined the permanent Kubernetes role, how did you go about learning Docker and then Kubernetes?

STEVE WADE: I was lucky enough that when I was at university, I did what's called a placement year. And for those of you who are not familiar with placement year, essentially my degree was three years. I did year one and two at university, year three I did in industry, and year four I came back and completed my final year.

And during that third year, it was a US-based company that I was working for, a pharmaceutical company. And I got to spend some time out in the US. And I met some incredibly passionate people in technology that I'm still friends with today.

If we roll forward the clock a couple of years after I finished my university degree, one of them joined Docker. At the time, I'm doing no Linux, I'm doing Windows only, I'm doing .net development. He's banging the drum and was banging the drum when I was on placement year, move to Linux, move to Linux, start learning Bash, start learning PowerShell.

Then he moved to Docker and said, look, this container thing, Steve, is really going to take off. I really strongly believe that. And if I looked at the Linux trajectory when I didn't listen to him, I can't be the same fool twice and not listen to him again.

And at the time, this container pattern really fell into place with what I was trying to achieve on client site. I was working with a team that was redeveloping their website, wanted to do it completely differently, wanted to do continuous delivery. The service had to be Linux, so I was forced to learn Linux.

And I just said, you know what, I'm just going to go out in my evenings, every evening, and I'm going to learn this Docker and container thing. I'm going to get on Hangout calls with the person that I'm talking about and just figure it out. I'm going to create my own lab locally. And I'm just going to figure it out.

And it got to the stage where we were running Docker Swarm in production, but I wouldn't call it actual production, because there were no customers actually using it. But it was production for us internally. Then I hit a stumbling point because I wanted to scale. And I couldn't scale because there was port clashes, et cetera, et cetera.

Then I found this Kubernetes thing, again, tried to figure it out. And I remember explicitly, Kelsey Hightower. I wrote a tweet and I said, I can't figure this thing out. And Kelsey dm'd me and said, let's jump on a Zoom call. And he explained Kubernetes to me as a bowl of fruit for an hour.

I've never even met Kelsey at this point. I didn't really even know who he was. And he explained Kubernetes to me as a bowl of fruit. And the whole thing just clicked. And that was the start of learning Kubernetes.

It was thanks to Kelsey and spending that time with me and explaining it and distilling it into something that I could understand, that was invaluable for me. And that's what I want to do moving forward. And that's why I chose to continue to be a contractor so that I can give back the way that Kelsey gave back to me and helped me in my career. And if I can do that to a number of people, then the world is going to be a better place.

CRAIG BOX: You mentioned that you're a trainer as well as a consultant. How do you bring that into the companies that you work with?

STEVE WADE: I think a lot of consulting is training, depending upon where they are in their journeys. So a prime example for me is, relatively recently, a number of consultancy gigs that I've been on, they were doing Kubernetes because someone told them that they should be doing Kubernetes, but didn't really know what they were doing. They had a couple of deployments.

They were running some commands from a Jenkins machine. And they thought, oh, "we're doing Kubernetes," in inverted commas. But sometimes the reason why you're on site is not just to help, it's also for them to learn from the things that you've seen and the mistakes that you've made previously. So I was training people, at this point, how to build containers, how to build images, how to make sure they're secure.

And then we go into the basic Kubernetes concepts. So how does your deployment work? Don't just take it at the fact that you're throwing this YAML file at some server and something magic is happening behind the scenes, and you can just do kube ctl get pods, and you see the thing that you've deployed. There's a lot behind the scenes that, when you come to managing it and maintaining it and having to support your applications, you actually need to know that stuff.

As we all know, Kubernetes is extremely deep. But the training that I have been providing more recently has been what I would call generic Kubernetes training. And it's mostly skewed towards application developers and platform engineers, and a little bit of security. Most of the time what I see, specifically in the customers that I've been on recently, is the platform engineers kind of get it. They understand how it works.

But one of the things that I always say is, the platform's customers are the developers. The developers don't understand what they're doing. They can't really get the maximum benefit from the platform. So a lot of my time was spent with application developers, explaining to them how they package their applications.

We talk about Helm, how to create Helm charts, where they should be stored, how we can deploy them. Then we move into GitOps as a way of being able to deploy them, why GitOps over "kubectl apply"s or "helm install"s from a centralized Jenkins box. And the training naturally fits around consulting. And then, from there, companies have reached out to me to perform dedicated trainings for them. So that could be very generic Kubernetes trainings, or very specific Kubernetes trainings.

At the permanent gig that you were talking about that was kind of the filling of my sandwich in my career, I got to train, I think it was 16,000 people worldwide on Kubernetes during that stint. So I had enough practice at making a load of mistakes and getting in front of a microphone and talking incorrectly. It's always been a passion of mine — back to the Kelsey story — to give back. And training is a great way of being able to do that. Consulting is also a great way.

There's nothing as fulfilling as seeing you train someone and they just get that eureka moment where everything just clicks. I absolutely love that. And training, to me, is exactly that. If I can get everybody in the classroom to have a eureka moment in the 2 to 3 or 5 days that I'm providing the training, that is invaluable. I go away with a smile on my face. They go away with a smile on their face. Everyone's a winner.

CRAIG BOX: You have been able to work in the platform team at a number of companies who were at various different stages on the Kubernetes journey. Is there a consistency between them? Is this something that they're all trying to do? Or are they all very different and you have to come with a very bespoke approach?

STEVE WADE: I think they're all trying to do the same thing. They're all at a different stage of doing that. So all of them really are trying to provide a self-service platform for developers and allow developers the ease of deployment and management of their applications. And there's a range of different steps that they could be at. But the way to facilitate that platform and the engagement between a platform engineering team and a development team, or a range of development teams, is always something that's interesting to me.

Because it's the cliche that the technology is easy, but the people are hard. And it's extremely true. And these companies that I go into, a lot of the problems are actually people problems. They're not technology problems.

People know how to write infrastructure as code using numerous different tools. They know how to deploy to Kubernetes in multiple different ways. They know what a Helm chart is. But the platform engineering team doesn't talk to the development team and the development team doesn't like the platform engineering team. And it's always like a butting of heads.

I think what Kubernetes is good at doing is it provides a common baseline that facilitates communication across the organization. Once you unravel that and you start to get different members of different teams talking to each other, you start to allow and build a platform that everybody can leverage. And that doesn't just come down to the developers. That comes down to people that are in security.

So how do we secure things? How do we make sure that application A can talk to application B, but application B can't talk to application C? And how do we do that in a formalized pattern that an application developer can create a pull request for but also a security engineer that doesn't necessarily know anything about Kubernetes can also do?

So the value of custom resource definitions and operators. On client site, I wrote a number of different operators. And the custom resource definition is the contract between the platform and the person that's going to leverage it. So if it's a security engineer or a group of security engineers, you work with them very closely on the customer resource definition.

They have to understand that, because that's what they're going to be writing. They're going to be writing YAML in a declarative way that matches that contract. Then they feel like, immediately, they have buy in, right? You've come to them and you've asked them what they would want to use.

Now there's some negotiation, like there always is. And then you build something that allows them to use that contract. It's not a definitive way of doing it. You don't have to do it like that. And I think what I've seen in the past is it's a very dictatorial way of implementing it.

So the platform engineer builds, the operator builds the customer resource definition and then goes to the developers or the security folks or whoever it is and says, this is the thing that you're going to use. This is gospel. If you want us to change it, let us know.

That never really works, because — back to the developers as the platform engineering's customers — if the customers aren't happy, they're never going to use the tool, no matter what it is. If it's a platform, if it's an application that they use on their mobile phone, if the customer's not happy, they're just going to find another way of doing it, or they're just not going to use it at all. And I think this collaboration is extremely important.

CRAIG BOX: Let's ask a very broad question, then. You can have a platform which takes in source code, you can have a platform which requires you to get in with kube ctl. How does a company decide where on that spectrum they should be?

STEVE WADE: I think it depends — classic consultancy answer there — on where they feel most comfortable with. For me, with the companies that I've been working with in the last couple of years, I try and steer them towards a GitOps paradigm, whereby Git is the source of truth, because a lot of companies that I've been working with, they get audited. And trying to show somebody that, yes, this change happened at this time can be somewhat difficult if you have to try and traverse through a load of Jenkins pipelines, or name any other CICD tool that you're looking at. But if we can use it and drive it from Git commits, it becomes very easy.

And also, there's a security aspect to that as well. When a centralized machine or set of machines is deploying to multiple different clusters, that machine or multiple machines has the keys to the kingdom. So in terms of if you're going to get pen tested, that's the first place they're going to try and get access to. And once they've got access to that, they can pretty much get access to everything else within reason.

So I always talk about, when we're building platforms, where are the security boundaries, where do we need to think about from an access perspective and also from, if a pen tester comes in, where is the first place they're going to go in the landscape that we're currently running. What I've seen a lot of is this concept of, what I call "apples and oranges," which is where they have one way of deploying to their downstream environments.

And then when they get to production, it's a totally different way. They're using a different tool or they're using a different script. It could be anything. And I think that only really works for me, and it's the term that I always use with customers, is consistency is key across everything. The more consistent that we can make things, the more boring we can make things, the easier it is for everybody.

CRAIG BOX: Do you think that a deployment team should spend more effort upfront building that out or do you think that they should be implementing it as they grow?

STEVE WADE: I think that you have to set the patterns and paradigms early on because, otherwise, you end up having to retrofit it, number one. And number two, when a new product team gets spun up, if there's no pattern or paradigm, they're just going to do it the way that they've done it before. They may not necessarily do it the way that other product teams are doing.

So one of the things that I try and do when I go on site, specifically, is I talk about the set of patterns and paradigms and we come up with them and we agree on them upfront. And then we try and build and facilitate a platform that meets those patterns. So one of the things that I always talk to companies about — like C-level executives, for example — is we need to make it as easy as we possibly can for a new product team to start, where all of the developers have never worked on the platform before.

And if we can make that as easy as we possibly can, it enables application developers and product teams to have free reign with guardrails to deploy microservices and build value for customers. If we have inconsistency, everybody's doing it slightly differently, when the auditor comes along, the way that each product team gets audited is completely different, and it just becomes a massive headache. And then when there's an incident, you've got four or five different ways of deploying.

I think the ability for simplicity is extremely important for a range of different reasons. And if you take all of the steps that you have from application development to incident management, and kind of all the way in between and all those touchpoints that you have with the clusters, the more boring that process is the better, because it allows anybody to jump in at any point and be able to provide assistance.

So one of the things that we had is, if I take one of the customers, they had, I think it was on the order of magnitude of 700 and something microservices at this time spread across, I think, 10 or 12 development teams, all doing everything slightly differently. But some product teams had services that needed to talk to each other, right? So when there was an incident, you needed both teams to be involved because product team A is doing something totally different to product team B.

So what I wanted to make sure of is that if we can standardize up front, when there's an incident it doesn't really matter, at some level, who is involved because the paradigm is always the same across all of the product teams. When we need someone specific from a product team is when it's down at the code level where something's gone wrong, like we're talking about a bank. If the invoicing is returning back the wrong information, then we're probably going to need someone from the invoicing team. But if there's issues with deployments or something's not scaling up properly, it doesn't really matter who is involved. Everybody can look at it.

And I think there's a lot of the companies that I've worked with in the past, these product teams are all very siloed, but one of the things that I talk about a lot is that a platform is for everybody. Everybody in engineering succeeds and fails together. There's no point in hiding anything, right?

When there's an incident, I or anybody else needs to be able to see everything. If I can't see something, I can't fix it. So I can create a pull request to a product team’s repository if I think that they need to change something. And they can provide a pull request to the platform engineering repository. And that's perfectly fine. And that's the kind of community and open and transparent communication that I try and build within organizations when I go on site. And if the platform works like that, everybody's much happier.

CRAIG BOX: You're currently working with KSOC, the Kubernetes Security Operations Center. We'll talk a little bit about the specifics of that role. But I want to bring security into the conversation here, because those teams are traditional in their role being to say no to everything and say, this thing will make things insecure, we can't have that, in heavily audited environments, especially.

How do you find that the security teams have dealt with the changes, and the fact now that we're talking about teams needing to be able to send pull requests to each other's repositories? How has that change happened in the security world?

STEVE WADE: I think now in this world of platform requiring open collaboration and the way that security engineers engage with a platform, trying to facilitate it in a common pattern. Previously they said no, right? And I think that is a very valid statement. They are the "no team" or previously they were known as the "no team." But putting words in people's mouths that can't defend themselves right now, a lot of the times when they said no is because no was the easier option, because they didn't have enough understanding to be able to say yes, so no was just their default position. Or they didn't want to understand what we were trying to do, so no was always the default position.

And I think with this concept of continuous delivery and the ability to be able to ship features consistently or changes consistently and frequently to end users and customers, they can't keep being the barrier to saying no, so they have to start engaging. And they have to not necessarily say yes to everything, but there needs to be more yeses than nos.

And one of the things that I'm a big fan of is this concept of value stream mapping, which is essentially where you make a one line code change, it could be for anything, and you just plot how long it takes to get that one line code change into production — no matter what it is — by minutes. Sometimes it could be days, sometimes it could be weeks depending upon what organization we're in, but doing that a couple of times and laying those over the top of each other, and then going to senior management, maybe C-level executives, and saying, look, I've performed a value stream mapping exercise and security seemed to be a huge blocker in moving things forward.

So let's engage with security and now ask the questions of why they keep blocking things, how we can give them and provide them with more confidence that we're not doing something silly, how do we put the guardrails in place. And that's where validating web hooks come into play, the ability for us to be able to create operators, both to facilitate things for application developers, but also facilitate things for security engineers, provide visibility, so we've got tools now at our disposal, like open source tools, think Prometheus and Grafana.

We can provide them visualization into how secure or insecure things are. They now have to be more data driven than they ever were before, so that they have to be able to say, no, and here is the reason why I'm saying no, rather than just no. And I think now Kubernetes is providing us a landscape where, unfortunately or fortunately, a large majority of it is either YAML or JSON.

So everybody has to become a YAML engineer or JSON an engineer or both. But we can have controls in place that stop us from making insecure decisions. And we can have them in an automated fashion, so we don't have to keep going to security. We can stop them earlier on. We can stop them in CI. We can stop them at the gate of Kubernetes with admission webhook controllers. And then we can even stop them post-deployment if we have something that's introspecting and looking at the things that are running in the cluster.

CRAIG BOX: Tell me then about what you're working on in KSOC.

STEVE WADE: At KSOC, we are working on a SAS product that empowers development and security teams to leverage Kubernetes securely at scale by continuously remediating security issues through known GitOps workflows. So if we think about the three areas of lifecycle of changes, you've got the CICD stage, you've got the at the door of Kubernetes — think of validating webhook controllers and mutating webhook controllers — and then you've got post deployment.

We are trying to hit all of those stages and providing and facilitating ways of performing real-time compliance, so providing a user interface that tells you when you're being insecure or where there are vulnerabilities, but also providing remediation as code. So one of the things we are doing is people are leveraging GitOps more now, so hooking back into repositories and providing them pull requests that fix the change, because Kubernetes is such a fast moving landscape and developers are leveraging it a lot.

If we go back to the story where I was talking about 700 microservices, the security engineers, or the number of them, are not going to be as large as the number of developers, and they don't just want to see a wall of issues. And if we can implement and fix those issues for them, we allow them to be able to work on things that are more specific to their organization, so securing their perimeter more than just securing the default configuration of Kubernetes or the default configuration of their applications.

We're also, as well, tackling RBAC and the concept of what we talk about RBAC visibility, so what is the RBAC configuration and how does the change that you're looking to make impact the whole entire cluster. And without having visualization of that, it's somewhat difficult, specifically if you're not necessarily a Kubernetes expert. How can we make that accessible and that information and visualization accessible to security engineers that don't necessarily know what a cluster role and a cluster role binding are, but need to know what it does from a Kubernetes standpoint and an access standpoint.

CRAIG BOX: It sounds like you're working on building a product that your teams that you'd previously consulted to would have liked to have. Are you there as sort of the expert who brings the experience from those things? Are you there helping get some connection between this company and potential future companies? Or is this just a different challenge for you in that you wanted to work on a product rather than a customer's thing for a change?

STEVE WADE: It's all three. We are building products that I'd previously built, but worse, on my own or maybe with a couple of engineers, at customer sites. And the reason why I joined is because they're solving real-world problems that I've seen at a number of different clients. And the ability to be able to provide not only just a user interface but a CLI for engineers as well to be able to interact and understand what's going on is extremely valuable.

So I'm bringing the pain points from previous customers to the implementation. I've seen a number of problems previously, how can we fix them? I can bring a number of clients to the front door and say, look, I'm working on this with the team. This is what we were trying to do, remember, when I was on site with you or working remotely with you.

And then finally, in the last couple of roles that we've touched on previously, I was a platform engineer that worked and engaged with developers. And I kind of missed writing a lot more code. I was writing operators but I wasn't writing microservices. So I'm back spending more time now writing microservices, writing APIs, talking about contracts, and going back to doing more of that.

So right back at the beginning, I talked about this Y-shaped person. A lot of my main emphasis and work that I've been doing for the last number of years has been around platform engineering. And I've neglected, to some degree, API development and API design. And I wanted to go back and do some of that as well.

So this role has provided me with the ability to be able to do that and solve problems that I had with previous customers. And when the time comes, and it will happen for me to move on to pastures new, KSOC will be one of the tools in my toolkit that I can take to customers and say, look, we should really deploy this. It's going to solve a lot of problems.

CRAIG BOX: You've just come back from three weeks in India. And you spend a lot of time there, I understand, looking for very specific food. Why were you there? And did you find it?

STEVE WADE: I was there for a break, firstly, to have a vacation. I've been there a number of times. I've been lucky enough to go there and visit. And the reason why I was looking for very specific food is because some of that food that I actually asked for, I've had previously in India. Some of that food that I asked for, I had not had previously in India. But even good Indian restaurants in the UK will never be the true, authentic Indian cuisine.

CRAIG BOX: Of course.

STEVE WADE: So the reason why I was asking for very specific food is two reasons. One, because I wanted to have it again. And two, because I wanted to try it authentically. And one of the interesting things to note, it wasn't on my list of asking people where to get it, but mangoes in India taste absolutely nothing like any mango that you've ever had anywhere else, specifically on the West.

CRAIG BOX: You did give some talks while you were out there. How did those go?

STEVE WADE: I was lucky enough to give a number of talks. I was actually talking about building a self service platform for developers and talking about the journeys that I'd been on with customers. Yeah, it was great. It was the first meet up that this company had ever put on. We had a very good turnout.

And there's the potential that they could become some KSOC customers. Some of them were extremely interested in what I was doing. We actually turned talking about training and consulting and giving back to the community.

Afterwards, I spent an hour with a gentleman talking about what he was doing and how I would recommend him architecting his solution that he was working on. They were relatively new to Kubernetes. They were just kind of onboarding with it. And that was extremely invaluable for me. I like giving back to a range of different people.

And one of the things that I did when I was there is I actually went to three separate orphanages. And it was a very humbling experience to go there and be able to do that and spend some time talking to these people and giving back to a community that is not the technical Kubernetes community. They're just a community of people that are in, unfortunately, a much less privileged position than I or yourself are in. But just sitting down and talking with them is an experience that I'll never, ever forget.

And there was some times during those three days that I was there that will live with me for the rest of my life. Talking to small children, that were talking about football, that brought a tear to my eye because I'm thinking about when I was a child and the position that I was in to be in a fortunate position to even play football, to them wanting to be able to play football and not being able to. I'm now looking at ways that I can continue to help those orphanages, moving forward, in some capacity.

CRAIG BOX: Finally, you mentioned before that you had Kubernetes explained to you with a fruit bowl metaphor. What would the Indian mango represent in that fruit bowl?

STEVE WADE: To me, mango is the best fruit that I ate whilst I was in India. I'm probably going to get slated for that and people are going to come back with a range of different fruits that I should have tried if I haven't tried them previously. So for me, if it's the best fruit, then I would say it's probably the scheduler. Because without the scheduler, the workloads are never going to land on the machines. So yeah, I would go with the scheduler.

CRAIG BOX: All right, well, thank you very much for joining us today, Steve.

STEVE WADE: No problem. Pleasure to be here.

CRAIG BOX: You can find Steve on Twitter @swade1987 or on the web at StevenWade.co.uk.

[MUSIC PLAYING]

CRAIG BOX: Thank you very much for listening. If you've enjoyed the show — and really, who hasn't — please help us spread the word and tell a friend. If you have any feedback for us, you can find us on Twitter @kubernetespod or reach us by email at kubernetespodcast@google.com.

You can also check out the website at KubernetesPodcast.com where you will find transcripts and show notes as well as links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Five stars feels like about the right amount. Thanks for listening, and we'll see you next week.

[MUSIC PLAYING]