← Back to posts

Comparing Terraform and Spinnaker

Jun 29, 2017 by Armory

Update: We’ve built a powerful new integration between Spinnaker and Terraform. Check it out here!

Much like our GoCD & Spinnaker article, we interviewed two engineers for their objective viewpoints on two popular tools in the DevOps world. We hope to help the reader understand the differences that exist between Terraform and Spinnaker, and the best ways to apply each tool.

Brandon Leach, who manages the Continuous Delivery Team at Lookout, provides a summary of the two tools: “Terraform allows you to define and manage your infrastructure, but Spinnaker allows you to manage your infrastructure from the perspective of code releases and deployment workflows.“

A look at the tools:

Terraform

###Spinnaker

Here is a list of feature comparisons and key takeaways between Terraform and Spinnaker.

Where Terraform Excels Over Spinnaker:

Where Spinnaker Excels Over Terraform:

We wrote a separate blog post on global software deployment best-practices to highlight the areas where Spinnaker really shines.

Essentially, use Terraform to set up your infrastructure and use Spinnaker to manage your deployments. You CAN use Terraform to accomplish Spinnaker tasks but it would require a lot of glue-code that is difficult to copy for other tasks. We see a lot of companies attempting to extend Terraform to do Spinnaker’s tasks when they can simply use Spinnaker.

Additional Perspective: Demetri Mouratis, Senior Staff Site Reliability Engineer at Box:

###Additional Perspective: Andrew Backes, Principal Engineer at Armory

We did a whiteboard session with Andrew Backes, a principal engineer at Armory, to discuss the relationship between Terraform and Spinnaker. We’ve seen customers unsure about how these two technologies can best work together. This video covers:

##So, how does one choose between Terraform and Spinnaker?
Below is a lightly edited Q&A with the two engineers from different companies. The anonymous engineer works at a Fortune 500 company.

Anonymous Engineer from F500 Company:

“It’s hard to compare the two. Spinnaker is a deployment platform while [Terraform] is an Infrastructure as Code tool which can be used for deployments. As such TF is significantly more flexible than Spinnaker and can do much more.“

Brandon Leach from Lookout sums it up as this:

“Terraform and Spinnaker are very different tools which solve different problems. Terraform is good at allowing you to define infrastructure as code. With Terraform I can define an AWS ASGs and associated ELBs and manage their creation and mutation, but in order manage a code deployment across this Terraform managed infrastructure I would need to write a bunch of custom glue code which wraps my Terraform. Spinnaker’s strengths lay in it’s ability to orchestrate deployment workflows with infrastructure. Spinnaker creates and manages the same infrastructure (ELBs and ASGs) within the context of a code deployment.”

For those looking into evaluating the tools, the anonymous engineer suggests to first know what you want to solve: “[We wish we had first clearly identified] internally what problem we were trying to solve. Both tools have things they excel at but due to lack of clarity within the organization it feels like we are attempting to solve too many problems with each tool.”

So what were their organizations attempting to accomplish with each tool?

Anonymous Engineer:

“Our teams are/were trying to extend Terraform’s functionalities into the deployment space while also investigating Spinnaker. Spinnaker accomplishes our desired functionalities out of the box, which makes it an attractive option. My team is at an internal evaluation of Spinnaker, and we have successfully enabled a test application to go from commit to production. Currently we are hoping to get an external customer to a PoC to prove value and determine work required to integrate Spinnaker with other tools.“

Brandon Leach, Lookout:

“At Lookout we built in house custom deployment tooling which utilized Terraform before using Spinnaker. We launched many services and completed our DC to AWS migration with this custom tooling which was heavily dependent on Terraform for infrastructure creation and management. Our internal customers were unhappy with this internal tooling and we kicked of an effort to address these problems. In the end we decided to follow the community and base our revamped service delivery efforts on Spinnaker rather than building further upon our Terraform based solution. We still use Terraform to manage some AWS primitives which are not part of the Spinnaker workflow.”

Which tool excels at what?

Anonymous Engineer:

Brandon Leach, Lookout:

The tools have coding trade-off and cost distribution. How do they differ?

Anonymous Engineer:

“[The] cost of operation is a big discussion point, with something like TF there is little cost to operate but a large cost to integrate/develop solutions around that are shareable. With something like Spinnaker there is a larger operations cost but less start-up cost. The cost to use TF is much larger up-front and requires a lot more “Glue-Code” to be developed prior to presenting value. Spinnaker has a much larger operational cost since it’s a service that has to be maintained compared to a command line tool. It should be noted that even though the core functionality is there for Spinnaker there is/has been a semi-significant amount of work required to get Spinnaker operational including figuring out security settings, packer templates, etc. We would need to solve these in both cases but I don’t want to imply there is no up-front cost to Spinnaker.”

Brandon Leach, Lookout:

“One of the main complaints we had about Terraform at Lookout was the amount of boilerplate code that had to be copied around in order to create new environments. We tried to mitigate this by providing Terraform modules for common infrastructure patterns and were in the process of exploring methods of generating Terraform when we decided to move to Spinnaker. At the end of the day our engineers didn’t want to spend time worry about managing their base infrastructure — they were more interested in defining deployment workflows which would allow them to ship features and services faster. Spinnaker provides them a workflow to manage their infrastructure from the perspective of deploying and testing their code. They can focus on the things that matter to them and let Spinnaker deal with the creation and management of infrastructure.”

Looking towards the future:

The engineers answered the following questions assuming their company had committed to Spinnaker or Terraform for a year.

Anonymous Engineer

For Terraform:

For Spinnaker:

Brandon Leach, Lookout:

Terraform

Spinnaker

Transcript of video with Andrew Backes

DROdio Okay, so I’m DROdio, the CEO of Armory [inaudible – 00:02]. Andrew, our illustrious principal engineer. And we’re seeing a lot of customers having a hard time exactly understanding the relationship between Terraform and Spinnaker. So we thought it might be awesome to do a little bit of a whiteboard session before we can describe that. So we’re [inaudible – 00:20] couple of questions as we have them [inaudible] get started [inaudible].

Andrew I’ll start with just describing these two things, the way I think about it. And then you guys can ask me some questions. The way that I think about Terraform is its role is really infrastructure [inaudible – 00:42]. So you write code that represents the infrastructure you want to create. And then you run Terraform against this and it [inaudible – 00:49]. The code that you write looks similar to something like a shell script. And you describe I want to make a VPC. In that VPC, I want these auto-scaling groups. And then in there I want my launch configuration. And you spell everything out exactly how you want it.

This is opposed to just going straight to the AWS console, and in the AWS console, you [were to – 01:17] create with the UI, you’re going, “Oh, make a new VPC.” And then once it’s generated, then you would go in and you would make your ASG or your launch configuration ASG. The advantages to terraforming that versus just the raw AWS console is the fact that it’s code. It’s repeatable. You can rewrite. So that’s a huge, huge, huge advantage.

You could do all of these things as code just interacting with AWS API. The problem with just using the AWS API is that it’s bulky. It’s kind of hard to use directly. And HashiCorp guys have done a great job of just condensing it only into things that you need so that you can interact with Terraform in that way. So if you wanted to make an auto-scaling group interacting with AWS API, it would be just a large payload, a large JSON payload. But with terraforming, it’s a short payload or a smaller payload and also reusable. So you can templatize it and it’s a lot better situation.

Comparing that to Spinnaker, there’s such a different beast. It’s hard to even compare them. What Spinnaker does is it deploys packages. So it will create images and deploy those images to your infrastructure. So you already have a VPC that exists. You may have used Terraform to create this VPC.

And then once that VPC exists, you want to create… I guess the way to think about it is that the central unit in Spinnaker is an application. So everything is from the point of view of the application. You are deploying that application. You are deploying it using immutable concept so that you’re creating new instances. Every time you deploy, you are mutating. So in that way, you are deploying infrastructure, but it isn’t exactly the same.

DROdio I guess one thing that would be interesting as you describe these two different worlds is to view kind of a just Terraform world here. So I’m a developer and I’ve got some code. And then on the other end, you’ve got the world that’s going to receive that code. This is the Terraform-only world. And then this is the Spinnaker world. So how would you visually represent the path to production and how they’re different in these two different worlds.

Andrew So if you want an automated system to… So starting with Terraform, if we just focus on getting applications out, which Terraform does a lot more than that. It can do any task in AWS basically or any cloud really. So it can kind of do… It’s a jack of all trades sort of thing. It really just wraps your API and allows you to codify what you want to do.

So with Terraform, if I’m over here and I’m writing an application, so I start with something like GitHub. [inaudible – 05:03] then it goes to my build system, say Jenkins. Okay, now the output of this is an artifact. Let’s say it’s a Debian. Now I have to get this Debian out and running. So if you use something like Terraform to help you do this, basically, you’re going to have another Jenkins job or some sort of system that does the same thing. And what this guy is going to do is basically play the role of Spinnaker. So what this guy is going to need to do is make ASGs that basically have in the launch configuration, they have your AMI, and your AMI has your deb. So you’re going to have to, in this step, create a template that does all of that stuff. So you make your template, your .tf file, and you put all those values in there. Your deb is going to be your variable. So the input to the tf is going to be the deb. And then the output is going to be all of this. So that’s pretty good. You get it deployed. It’s all out there.

And the thing that’s running the show here is Jenkins. So Jenkins is utilizing your Terraform. So I guess all of this should’ve been… should’ve done that here, right, right.

DROdio I’ll erase this part for you.

Andrew And then finally, the [inaudible – 07:08]. Okay, so with Spinnaker, there are some similarities in the flow. So you’ll start with a GitHub change and then you’ll go to Jenkins. You’ll make your deb or whatever package. And then from there, now you go into Spinnaker, and then the world will be able to see. And then we’ll describe what happens in here. So with Spinnaker, this sort of activity is a first-class citizen. It isn’t anything that you have to code up. You can specify [inaudible – 07:46]. I said before that Spinnaker treats applications as first-class citizens. And then pipelines are first-class citizens of applications. So pipeline is a an add-on or a sub-concept of an application. So this will be an app that’s been deployed. And now we’re going to go into a pipeline. And what this is going to look like is you’re going to first have this and we’re going to bake an AMI. And all you have to do here [inaudible – 08:28] this pipeline is click the Bake AMI button, and then it’s going to know that you want to make this Debian into it.

After that, you can go to a deploy step. And the input for this is something like a region. And then it’s going to know I’m going to deploy this AMI that I just built. Say this is a staging environment or something. Then after that, you can have it run a shell script that does a bunch of integration tests. And that’s just a button, and then in the integration test step, all you do is you give the path to the script in GitHub. Then over here, you can say, “Okay, that passed. And now I’m going to deploy to product.” Okay, same thing. Now you might want to do like a validate step. And this is just a shell script or something like that. Then after that you’re going to then say, “Okay, I’m good to go.” So check, done. That means a full flip, a full Blue/Green has happened. [inaudible – 09:53]. You have a bunch of deployment strategies. So in here, the one that’s kind of recommended is a Blue/Green or what they call Red/Black. Blue/Green deployment strategy, so you have one cluster up and another cluster up. This is your old cluster. You spin up a new cluster. Once this one is good, you cut traffic over, and then you can destroy this cluster. So if you want to do that, in here you’re going to have to do a lot more bash scripting, I guess.

DROdio When you just show these two worlds, to the casual observer, they don’t look all that different. But here, as a developer or as an ops team or automation team, I’m writing a lot of glue code to extend this functionality to mimic pieces of what Spinnaker can do. But this is a system. It’s a platform where if I wanted to make a different pipeline tomorrow for this application, if I wanted to modify this pipeline, I would literally just be either using the UI to drag and drop stages or changes or I could be doing it programmatically. But I’d love for you to dig a little bit more into the actual level of effort or work required to get to these two.

Andrew Yes. So if you’re using Jenkins to do this sort of thing like the Blue/Green deployment, first you’ll use Terraform to get your ASG up. And once your ASG is out, then you’re going to have to do a bunch of shell scripting here that’s going to do the Blue/Green. So you’re going to have to put all that logic in there. And you’re going to have to put all the logic for how you want to handle failover in there. So what happens if it fails? Well, I want to clean up the mess. So I’m going to put in all this code to clean up my mess. Or maybe I want to send some notifications via Slack or something like that. Or maybe I want to do something more serious. You never know. So all of those kind of things are first-class citizens in Spinnaker that you can just do with a drop-down menu. You just click. You have to program all of that up here.

DROdio So out of the box, Spinnaker will do a Blue/Green. But as the automation is here, as the ops team or whatever group it is, I don’t have to spend a lot of our time manually scripting out that brutal glue code.

Andrew Yeah, so this is brutal for a few reasons. One big reason is APIs change. So the things you’re interacting with is going to change over time. Also, you’re going to start with a small set of features. And then if you want more features, you have to add it. Spinnaker is a super active community. People are adding new stages and new patterns all the time. And API changes, it’s fixed immediately. So you don’t ever have to worry about something like that. If you want [inaudible – 13:05] on something else, you just [inaudible] Spinnaker to the version that supports that. Whereas here, you have to spend the resources to fix the underlying problem.

Also, you’re never going to be able to develop features. Well, you might be able to, but I’d be amazed if you’re able to add features to your deployment platform that you created with Terraform as fast as the Spinnaker community.

DROdio I’d also love for you to talk a little bit about the self-service nature of Spinnaker, allowing application teams or even product managers to hit a button to deploy kind of how Spinnaker can enable that self-serviceness, I guess, and how that compares to a world of just Terraform.

Andrew Yeah, okay. So really, the real comparison when you’re talking about that sort of thing is more of a Jenkins compared to Spinnaker because Terraform does a very limited thing. Getting into something like the Blue/Green deployment, that’s not in the scope of what Terraform is trying to accomplish. Terraform is, like I mentioned before, the infrastructure as code, and it does a phenomenal job at that. But when you need to apply logic on top of deployment infrastructure is where you need another tool. That’s where someone would have to use something like Jenkins or another thing that can do it run-time. And I guess the view from Jenkins is there are pipelines in Jenkins, now with 2.0 being very popular and everyone using the common plugins. But none of the stages are pre-programmed. So you have to program your entire stage.

But with Spinnaker, everything is pre-programmed. You don’t have to do anything. It’s more of just putting the pieces in the right order.

I should also add that with Spinnaker, you don’t have to just use the UI. You can also back it by coding a repo, and then do pipelines as code. So you can do that. That is one thing that it does have in common with Terraform. Because you have all of these pre-configured stages here, if you wanted to do something like wait after these integration tests passed for a user to interact with it, for a user to say yes or no, like a product manager or QA team or something like that, you just pick the stage. You’re just picking, put it in there [inaudible – 15:55] send notifications to [inaudible] say, “Okay, Slack these people, email these people.” And then they’ll come and they’ll press the button. You’d have to program all of that here. And I don’t know the plugin. There might be one that will allow you to stop a stage and wait for user input in Jenkins. But the way I probably do it is divide up the pipelines and make it so, “Okay, run this first job. And then once that’s done, then notify somebody. And then after that, they’ll come and they can run the next Jenkins job.” That’s probably how you would have to do that. There might be a plugin to make it a little bit easier, but it certainly isn’t a first-class citizen.

DROdio We were also talking about [inaudible – 16:42] production. What about when I have multiple apps or multiple application teams that are all [inaudible – 16:50].

Andrew Yeah, that’s true. I guess the way that you would do it while with Spinnaker, you would just go and you’d make a new pipeline and compose your pipeline with the preconfigured stages, however you want. It’s very composable in that way. Here I guess you would probably have to make a new Terraform template that has what you need in it. You’ll probably be able to reuse a lot of what’s in here, so maybe turn this into a module and then use it in that way. But then depending on what you want your flow to be, you’re going to have to program that flow every time. So some themes might not want to do a Blue/Green. They might want to just eliminate all the old stuff and make new stuff. The integration tests that are run here, you’ll have to get that all wired up. To me, it just feels like there’s going to be a lot of copying and pasting [inaudible – 17:50].

DROdio Just to make sure that we’re being clear about this, this is a pipeline for an app in Spinnaker here. And what I have another app, app #2 in Spinnaker, I literally can just define another pipeline just like you did here. And then we can have a conversation about it, and we say, “Oh no, I think it needs [inaudible – 18:17]. We just literally drop that in. And we want to do some canarying. There’s an ability for us to focus on how to optimize that pipeline.” And in this world, we are actually literally coding in each step, each stage per application and then having to maintain that over time.

Andrew Yup, Yup.

DROdio Now what are some of the disadvantages of Spinnaker versus this kind of other approach that people will probably be more familiar with? For example, I’d love for you to talk about how Spinnaker is an opinionated system in the ways that it won’t work if [inaudible – 18:54] for example.

Andrew Yeah. We mentioned before that Spinnaker prefers the opinion of immutable deployment. So what that means is for every commit in GitHub or every package that’s generated from Jenkins, it’s going to deploy a completely new ASG and wire up that ASG. When I say ASG, I mean in auto-scaling grouping, Amazon. It’s going to deploy a whole new one and then wire it up with [inaudible – 19:31] and send traffic over to it. So what that means is your immutable object is really your launch configuration in your auto-scaling group. That won’t change. So when you make this a new Debian, a new launch configuration is going to be created and then a new auto-scaling group from there. If you wanted to do something like have a set of hosts just sitting there (maybe Chef installed on them and then you want to install a new package to convert Chef), that really isn’t going to work very well with this model. You can make Chef work, especially if you’re in transition period, moving from that way of doing it, from immutable world to an immutable world. Spinnaker can help you, but it does take a little bit of work, and it’s not really working with its strengths. You kind of have to force it in. And the way you really do that is on this big stage.

One other thing is I said before that Spinnaker has an app-centric view of the world. So it thinks of everything apps down. So when you go to the main screen on Spinnaker, you see all of your applications, and you click on your applications, and then you see your clusters of that application, and you see the pipelines. So what that kind of means is Spinnaker wants you to have certain types of applications. What it wants you to have is something where you have a load balancer or some sort of service discovery here. And then below that you have your clusters.

And if you have a different type of topology for this application, it would be hard. So some of the other things that are a little more difficult (not impossible) to do with Spinnaker are stateful applications because if there’s state on an instance in here, you have to figure out how to manage that state. So what a lot of people do is they push the state out. They have their databases hosted somewhere else or something like that. But if you’re using [EBSs – 22:07] or something, of Jenkins in this case [inaudible] everything [inaudible] system, you’re going to have to do a little bit of work to have it work in your advantage.

DROdio What else should people know about the differences between the two?

Andrew I just want to reiterate that mainly, what we came to talk, we talked very little really about the differences between Terraform and Spinnaker. They’re just so different. They don’t accomplish the same thing. The way that Spinnaker gets applications out to AWS or to your target, there is something in common there with Terraform. But just that one piece, the real power of Spinnaker is the workflow engine and being able to use these pluggable and composable stages and not having to write your own every single time. People are coming up with these stages all the time. Just the other day, webhook stage was added, and that was incredibly valuable, very, very great stage. And it’s so easy to add to. Anyone can do it. I don’t know why anyone would really choose to do it over here with their shell or Python type way. You can just go and look at the interface in the Spinnaker code base for a stage and see how easy it is. And you could so easily just make it yourself and give it back to the world, and then everyone can maintain it for you. And it’s just so much better to be on the back of something that is so used. It’s so much better in my opinion.

DROdio Just to expand on that point for a little bit, so Spinnaker has a community [inaudible – 24:04] project. Netflix has got a community, a robust community with 30-40 full-time engineers working across many companies, including [inaudible – 24:14] Google, Microsoft, Veritas, Target.

Andrew Of course Netflix.

DROdio Obviously Netflix, yeah. So that’s what you’re referring to, right?

Andrew Exactly.

DROdio The vibrancy of that [inaudible – 24:28].

Andrew Yeah, exactly. And not having [inaudible – 24:31] all by yourself. If you do this, you’re going to have to have… If you put your organization down this path, you’re going to have to commit resources to this, resources that you could be spending on [core – 24:47] competency, something that will add value to your company.

DROdio So I guess in summary, don’t reinvent the wheel. Don’t maintain that reinvented wheel every time. Instead, apply your resources to whatever your company does best. Leverage the power of the open-source community, and leverage the way that Spinnaker has this very [pluggable – 25:06] model and this very app-centric view of the workflow engine that allows for self-service and a lot of safety and velocity for the application teams and a lot of relief for the [inaudible – 25:19] infrastructure teams [inaudible].

Andrew Yes. I also want to say Terraform is an amazing tool. We use it all over the place. We use it to actually bootstrap our first version of Spinnaker into somebody’s infrastructure. It’s a great, great tool. It just does a way different thing. So comparing the two, it’s apples and oranges. The amount of overlap is just so, so, so minimal. I say the strengths of both of these aren’t the same so that you can’t compare them. The strength of Terraform is to be able to have any infrastructure in basically any cloud as code. It’s just amazing. Who wants to manage DNS entries in the console? No, just put it in a GitHub repo and have Terraform do it for you. It’s way, way, way better. Derisks everything. It’s amazing. Terraform is an awesome tool.

Spinnaker doesn’t want to do that. It’s not trying to do that. What Spinnaker is trying to do is help you deploy software. So to me, they’re just so different, not even comparable.

DROdio Yeah, that’s a great summary. Okay, Colin, you’re also here. Do you have any questions for Andrew, things that you’ve seen others asking?

Colin An engineer said that Spinnaker is a deployment platform while Terraform is infrastructure as code tool which could be for deployments. But as such, Terraform is significantly more flexible than Spinnaker and can do much more.

Andrew Yeah, that’s kind of what I was talking about. The role of Spinnaker is to deploy your application. The role of Terraform is to codify your entire infrastructure. For example, from an AWS perspective, what Spinnaker is going to talk to is launch configuration, auto-scaling group, EOB, or an application load balancer, security group. That’s basically it. That’s all that Spinnaker is going to want to interact with. The look compared to Terraform, Terraform does everything, every single thing. So VPC, it does all of this stuff. It interacts with all of those, RDS, elastic cache, elastic surge, every single service, S3, everything. Anything that the Amazon API has, Terraform can do. That’s what that statement means.

Colin So do we recommend using that together? One of the [inaudible – 28:37] they do.

Andrew Yeah, I mean they’re different. Their strengths are just completely different. I guess the recommendation is if you want to deploy software, use a deployment platform. Spinnaker is a deployment platform. If you want to codify your infrastructure, use Terraform. That’s what it is for.

Colin I think the whole thing in a nutshell, [inaudible – 29:09].

DROdio Yeah, that’s great. Thank you so much for spending the time to walk through it. And for anybody who watches this, if you would like to get more of these talks from Andrew, please let us know. We’d be happy to do more of them.

Recently Published Posts

Introducing Pipelines-as-Code Plugin for Open Source Spinnaker

Jul 21, 2023

Easily Scale and Automate with Version Control in Git Developers choose best-of-breed version control systems like GitHub for a reason: they need the ability to collaborate and improve code together.  But a broken Spinnaker deployment pipeline can often be the last thing standing in the way of getting your application to market.  Until now. Armory’s […]

Read more →

What is FedRAMP and Why It Matters

Jun 8, 2023

What’s FedRAMP? Federal Risk and Authorization Management Program (FedRAMP) is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. FedRAMP is important since it’s the gold standard for assessing cloud service providers (CSP) within the government. Under this program, authorized FedRAMP cloud service providers […]

Read more →

New Spinnaker Operator Updates Now available for the Spinnaker Community

Mar 15, 2023

Stay up-to-date with the latest Kubernetes release with Spinnaker. The Armory crew has worked diligently the past several weeks to release a new stable version of OSS Operator (1.3.0). This is the first release in just over 18 months and is now available for the open source community.  What Changed? The Spinnaker Operator is the […]

Read more →