← Back to posts

Why (and How) Spinnaker and Kubernetes Work Together Seamlessly

Sep 23, 2016 by Isaac Mosquera

The adoption of Kubernetes is happening at an incredible rate. Naturally, the next step is to look for tooling to support continuous deployments workflows with Kubernetes. In this quick interview with Lars Wander and Matt Duftler from Google, we talk about how Spinnaker and Kubernetes work together to help deploy better software.

Lars Wander

Lars Wander – Google

Lars was responsible for the integration between Spinnaker and Kubernetes:

Matt Duftler
Matt Duftler – Google

Matt wrote the integration support for Google Cloud within Spinnaker:

You can watch the full video below:

Topics discussed:

What is Spinnaker? (timecode 1.34)

How does Spinnaker work with Kubernetes (timecode 2.30)

How does Spinnaker compare to the competition (timecode 4.10)

How mature is Spinnaker support for Kubernetes (timecode 5.10)

Spinnaker deployments versus Kubernetes deployments (timecode 5.50)

What happens with existing Kubernetes deployments (timecode 8.12)

Can Spinnaker be used without the UI (timecode 9.54)

How Spinnaker works with existing CI tools (timecode 11.30)

How can one find support for Spinnaker on Google Cloud (timecode 13.00)

What kind of customers are running Spinnaker (timecode 13.47)

Want To Dig In More?
Read all posts related to Spinnaker + Kubernetes

Here’s the Transcript:

Isaac: This is Isaac Mosquera. Hi, nice to see you. I’m the CTO of Armory. We are working on a product called Spinnaker. And today we’re going to talk about how Spinnaker integrates with Kubernetes. Today I have Lars and Matt, two Google engineers who are also working on the project. Do you guys want to introduce yourselves?

Lars: Sure. I’m Lars. I wrote most of the Spinnaker-Kubernetes integration. I’ve been at Google on Spinnaker team for a little over a year now.

Matt: I’m Matt. I’ve work for Google in New York with Lars, also on the Spinnaker team. I did the bulk of the GC integration and a bunch of stuff and the core components of Spinnaker and have been working with the Netflix guys and some other folks on the Spinnaker project since close to the beginning of the project.

Isaac: All right, cool. We’ve talked to over 90 customers. And a lot of them are moving over to Kubernetes or have already moved to Kubernetes. And obviously, the next step is how do I automate deployments into Kubernetes. And there seem to be a lot of questions around Kubernetes and Spinnaker and how they work together. So would one of you guys tell us a little bit more, just generically, about Spinnaker and what it does and what its project goals are?

Lars: Yeah, sure. Spinnaker is an open-source, multi-cloud, continuous-delivery tool. It was originally developed by Netflix, and I think now it’s servicing over 97 percent of their internal deployments. A couple of years ago, Matt led the integration with Google, meaning that it can now deploy the Google cloud the same way that it can to Amazon. And recently, we’ve been adding a whole lot of other cloud providers to Spinnaker – meaning that you can deploy to Kubernetes, you can deploy to Cloud Foundry, you can deploy to Azure. And the goals of the project really are to allow you to automate and make reliable your deployments and your rollback steps, so you can set up pipelines describing how you want to deploy to your cloud provider of choice and also safely roll back in case anything goes wrong.

Isaac: Let’s dive in specifically around Kubernetes and Spinnaker, which seems to be the question that always comes up. How does Spinnaker fit with Kubernetes and what does that look like?

Lars: It’s a good question. I think they fit together very well for two reasons. The first one is Spinnaker does a great job of giving you oversight into your deployed infrastructure. And a lot of that is showing you what you have running, what’s eating traffic, what’s healthy. And the resource mapping that Spinnaker chose is server groups that are load-balanced behind some sort of a service with security rules. It’s very similar to the model that independently Kubernetes has come up with. So the resource mapping of the Spinnaker abstractions to Kubernetes resources is very straightforward. So that works just as one might expect. So if someone is familiar with Kubernetes or familiar with Spinnaker, that mapping is very natural.

And the other part is that Kubernetes, while it does have this Deployments API, this isn’t always enough to give you a fully automated deploy from code to production. And Spinnaker fills the gap there by allowing you to orchestrate multiple steps by allowing you to define your deployment pipelines and say that maybe I’m going to roll out traffic to this cluster with this [to this mini pause – 03:54], check that everything is okay. If so, keep going. If not, roll back to my prior version. And that’s really where Spinnaker amplifies what you can do with Kubernetes.

Isaac: Right. And I guess why wouldn’t I want to do this myself or what other tools are there for me to do this on my own?

Lars: The biggest tool that people are using right now, at least that we run into, is Jenkins. And Jenkins is great for running arbitrary scripts and passing data between those scripts. But what lacks is a set of first-class constructs for objects in the cloud. So for example, in Spinnaker, you can say, “Take whatever I deployed in the last healthy server group in this region and then redeploy it in a new server group in this region. And these are all first-class concepts in Spinnaker, so it’s really hard to screw up. So you can select from a drop-down and say, “Take this artifact from here and then redeploy it here after it’s been validated.” And in Jenkins you wind up writing quite a bit of glue code to get these things to fit together. And if something goes wrong there, it could be an error in the script, it could be an error in your deployment policy. And Spinnaker makes all this very foolproof.

Isaac: Sounds good. So how mature is support for Kubernetes within Spinnaker as of today?

Lars: The support has been feature-complete, we could say, for about six months now. And over the course of those last six months, we’ve had many people come in and validate it. We know there are customers using it to deploy to production. So I think we can safely say that the support is very mature. There are always tweaks we’re doing and we’re always making improvements and trying to improve the performance or see what else we can support inside Kubernetes, within Spinnaker. But in terms of the core Spinnaker features, that’s been done for about six months.

Isaac: Another question that tends to come up frequently is around the Deployment object inside of Kubernetes. How does that differ from just what Spinnaker is doing? And I know you’ve answered some of that. But just to be very direct around this question, people think it’s a very similar thing.

Lars: Yeah, absolutely. Going back a bit, when we first wrote support for Kubernetes and Spinnaker, the Deployment object wasn’t so much at the forefront. And we decided to model the underlying, replicated set of Pods with the replication controller, which is what at that time Kubernetes was pushing. And recently, now that the Deployment object has become more popular, we’ve migrated the underlying model from the replication controller to the Replica Set, which is what the Kubernetes deployment model depends on. And what that allows you to do is it allows you to orchestrate deployments using Kubernetes’ Deployment API inside Spinnaker the same way you would as you would outside with kubectl rollout or kubectl deploy. So if you want to rely on Kubernetes’ underlying deployment mechanisms inside Spinnaker, it’s totally fine. But Spinnaker also gives you a lot more on top of that.

For example, if you’re doing rollout and you have hundreds of Pods and you want to roll back immediately because you realize something went wrong, it’s very possible that to do that rollback, you now have to re-download non-cached images onto nodes that don’t have the image deployed. You have to wait for the application to start up. And depending on your rollout policy, you may have to wait for all of these Pods to come out one-by-one and be scheduled individually.

In Spinnaker, the way Spinnaker handles these kinds of deployments if you so choose with the Red/Black deployment strategy, you already have these Pods running live and ready to serve traffic on every single node where they’re needed. So you can immediately go back and say, “Switch through the help of Spinnaker the IP table rules allowing these Pods to receive traffic within minutes or seconds of you doing the rollback.” And that’s not possible with just the Kubernetes API alone.

Isaac: Another thing that frequently comes up is around the [… 08:11] deployments and YAML file. So there’s a Kubernetes cluster that has a bunch of YAML definitions already defined. There are already deployments. What happens if I’m already in that particular state and I want to work with Spinnaker? How does Spinnaker handle those configuration files?

Lars: If you have something already deployed in a Kubernetes cluster and you point Spinnaker at it, Spinnaker will go ahead and cache everything in there and represent it as if it had been deployed by Spinnaker in the first place. In fact, Spinnaker really doesn’t add any extra metadata on top of any of your cloud providers or deployment environments, aside for things like the deployment strategies used to roll out infrastructure to those deployment environments. So if I were to point Spinnaker at a cluster, I would see everything running there as if Spinnaker had deployed it. If I deploy stuff with Spinnaker and I kill Spinnaker, flush all its caches, and then spin it back up, everything is going to look identical. So what that means for you is that if you have some Kubernetes YAML files that you want to deploy to a cluster that’s being managed by Spinnaker and you do that deployment, it’ll show up in a cluster as though Spinnaker had done it. And then you can use that deployed object either as a template for future deployments, or you can go ahead and make edits to those YAML files within the Spinnaker UI by effectively cloning a server group and saying, “I want to redeploy this but with these attributes changed.” And you can do that with a deployment strategy as well. You can say, “I want to roll out this change,” or “I want to do a Red/Black deploy to this change.” So basically work seamlessly with any sort of Kubernetes YAML files that you already have for your [… – 09:50] cluster.

Isaac: The next thing that typically comes up is around, okay, so you can edit it with the UI, but there are a lot of folks who are trying to avoid changes that happen through UI and want to manage it maybe through some sort of configuration management or tool. Not configuration management tool. That’s a bad thing. More like Infrastructure as Code, I think, is what people usually say. Am I forced to use the UI? Or what other ways can I modify the YAML file that are there in the pipeline [… – 10:32]?

Lars: The UI that Spinnaker uses is actually written after the core set of Spinnaker features are completed. So it’s really a thin wrapper around Spinnaker. If you want to post pipeline definitions, which are represented as JSON to the Spinnaker API server, and you want to manage all your pipelines like that, that’s totally acceptable. And I’m pretty sure that there are people in Netflix who do this. You’re not stuck to using the UI if you want to use Spinnaker. And at any point in time, when you have a pipeline definition which describes how you make changes to your cloud environment, you can click Use JSON in the pipeline editor and you’ll see exactly what’s going on behind the scenes so to speak. Furthermore, all of the resource definitions that were deployed by Spinnaker are easily captured as YAML using either kube control and Spinnaker services as option too.

Isaac: More generically about Spinnaker, how does Spinnaker treat CI tools like Jenkins and things like CircleCI? How does it work with it? Does it replace it? How does Spinnaker think about these tools?

Lars: Spinnaker really doesn’t want to become a CI solution. Instead, it depends on [… – 11:51] CI outside of Spinnaker for the most part. [It can call out to – 11:54] CI solutions like Jenkins or Travis. But the ideal workflow is really that you have your source code [in some repo – 11:59] you push a change, and that’s picked up by Jenkins or Travis. And then Jenkins or Travis build this into some deployable artifact. In the VM world, that can be a Debian package. For Kubernetes, that’s obviously a Docker container. And once that Docker container has been validated, it’s then pushed to registry. Kubernetes will listen for a trigger on. And that registry can be any [… – 12:20] registry, whether it’s one that you host locally as a Docker [… – 12:24] registry or a hosted solution like with GCR or Quay or Docker Hub proper. And that’s really when Spinnaker takes over.

Isaac: That’s the majority of the questions we see around Kubernetes and Spinnaker, so thank you for that. I’m going to ask Matt a question since he’s been there just quiet. Lars [… – 12:53] thunder man.

Matt: I’m here for mostly emotional support.

Isaac: [Laughs] So how is the support for GCP and GCE in general for Spinnaker?

Matt: We mostly direct focus to the Slack channels. We have like a growing set of solutions docks and tutorials and code labs and things like that. But mostly, we try to do the support out in the open and the community has become pretty active. And as Lars mentioned, there’s a growing set of platform support. And each of those platforms has a pretty sizable team working on that support. So there are a number of folks who are competent and experienced enough to help [… – 13:32].

Isaac: So if I need immediate support, I’ll go to Slack.

Matt: There are people in that chat room in all hours of the day and plenty of examples to share [… – 13:43].

Isaac: Of the customers you see moving to GCE and GCP just in general, what did those customers look like? Is there a consistent profile of people who are adopting those tools?

Matt: Initially, it’s definitely towards the end of the spectrum where folks are really running [at scales – 14:06]. But we’re working steadily to make it easier and cheaper to run Spinnaker itself. And as we make progress on that, it makes it that you don’t need quite as much skill to justify the amount of work and expense to run Spinnaker. It’s just becoming easier and cheaper. There are definitely several users we have that are one- and two-person shops. But most of the initial users are larger outfits.

Isaac: When you say large, I think that is another question people have, just around, okay, works for Netflix at scale. From your experience, without giving out any names, of course, but just maybe giving a descriptive profile in terms of how many nodes or how many cores you see in one of the larger users of Spinnaker within GCP.

Matt: I don’t have numbers but certainly on the order of hundreds of VMs is a reasonable footprint to imagine. You could see even in the chat room. There are other large companies like the Openstack integration [apps – 15:17] that’s targeted Veritas [4K – 15:19]. So you can imagine that Target is a large company. I think most of the larger companies have several folks that are dedicated to running the Spinnaker deployment itself, and any individual development teams have access to that [… – 15:36]. So that seems to be fairly common that there are folks [dedicated to – 15:40] operating it, and then they make that available to the other engineers [… – 15:43].

Isaac: Okay, great. Is there anything else you guys want to add?

Matt: Lars, you’re still unmute. You can tell us here.

Lars: No, nothing from my end.

Isaac: All right, this has been good. Hopefully, this clears up a lot of questions. I’m sure there are a lot of people who are using GK, GCP, who are interested in using Spinnaker and seeing how it fits [… – 16:10]. All right, guys, thanks.

Lars: Thanks, Isaac.

Matt: Thanks, Isaac.

Recently Published Posts

Introducing Pipelines-as-Code Plugin for Open Source Spinnaker

Jul 21, 2023

Easily Scale and Automate with Version Control in Git Developers choose best-of-breed version control systems like GitHub for a reason: they need the ability to collaborate and improve code together.  But a broken Spinnaker deployment pipeline can often be the last thing standing in the way of getting your application to market.  Until now. Armory’s […]

Read more →

What is FedRAMP and Why It Matters

Jun 8, 2023

What’s FedRAMP? Federal Risk and Authorization Management Program (FedRAMP) is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. FedRAMP is important since it’s the gold standard for assessing cloud service providers (CSP) within the government. Under this program, authorized FedRAMP cloud service providers […]

Read more →

New Spinnaker Operator Updates Now available for the Spinnaker Community

Mar 15, 2023

Stay up-to-date with the latest Kubernetes release with Spinnaker. The Armory crew has worked diligently the past several weeks to release a new stable version of OSS Operator (1.3.0). This is the first release in just over 18 months and is now available for the open source community.  What Changed? The Spinnaker Operator is the […]

Read more →