Spinnaker, Accounts, and Kubernetes: How Accounts in Spinnaker Work and How Those Accounts Authenticate with the Kubernetes

I’ve recently had the opportunity to work with many customers who are starting their journey into Kubernetes and using Spinnaker to facilitate that process. One of the most significant points of confusion is around how accounts in Spinnaker work and how those accounts authenticate with the Kubernetes API. In this post, I hope to clear up any confusion around the subject!

I’ve written about this subject in the past so if you’d like to see an example be sure to check out this post on the Spinnaker community blog.

Accounts

The first thing we should cover is the concept of Accounts. Accounts are Spinnaker’s way of defining a set of credentials for a particular cloud provider. Since Spinnaker was initially designed around AWS, let’s use it as an example. For the majority of AWS users, it’s common practice to split your cloud footprint into separate logical AWS accounts which allows you to separate infrastructure between environments. In the most simple use case, you may have an account for Development and an account for Production. More complex organizations, however, may have tens to hundreds of accounts depending on security or organizational requirements.

With this understanding in place, we can start to look at how Accounts relate to Kubernetes. Typically, Kubernetes users split workloads across various clusters. Much like the previous example, splitting workloads across physical clusters provides a logical separation of resources and prevents outages in one cluster to affecting other environments. These physical clusters, then, effectively serve the same purpose as an AWS account.

Authentication and Kubeconfigs

An Account within Spinnaker isn’t solely about separating resources. It’s important to note that an Account represents a set of credentials for a given cloud provider and that your users interact with that cloud provider through the account. In other words, the Account is acting on behalf of your users. Whenever Spinnaker performs an action within your cloud provider, such as deploying a container or creating a load balancer, it uses credentials defined by the Account to do so.

Since an Account effectively acts as a user, we need to consider how we might enable the account to authenticate with our cloud provider. In the Kubernetes world, we use something called a kubeconfig which allows us to define a list of clusters that we can communicate with and the various ways we can authenticate as a particular user. A single user and cluster are then combined to form a context, which is used by Spinnaker to communicate with the Kubernetes API. There are a multitude of ways we can use to authenticate with our Kubernetes clusters so you should choose a method which suits the needs of your organization. For demo or POC purposes, I like to use a Kubernetes ServiceAccount token paired with a cluster-admin ClusterRole but, for more restrictive or explicit permissions, you should use Kubernetes RBAC to define a specific set of actions this user is allowed to perform.

Each Kubernetes account within Spinnaker defines the kubeconfig that it will use when authenticating with the Kubernetes API. For small shops, you may be able to use a single kubeconfig which lists all of the clusters, users, and contexts and use the --context option to specify the context when configuring accounts. For larger shops with many accounts, this approach may prove difficult to manage so you may prefer to use a single kubeconfig file per account.

RBAC

Kubernetes RBAC is a powerful tool for restricting the actions a user is allowed to perform on your cluster. You can read about it in depth here, but I’ll briefly cover some of the resources you can use to ensure proper permissions for your accounts.

Role and ClusterRole

Roles and ClusterRoles are used to define a list of APIs and actions that the holder of this role is allowed to access. A Role is a namespaced kind within Kubernetes which means that the permissions defined in the Role only apply to the namespace where it is created. A ClusterRole is a non-namespaced kind. This means that ClusterRoles can define permissions that span across all namespaces within a cluster. It has the added benefit of being able to access other non-namespaced kinds such as the Namespacekind itself.

ServiceAccount

ServiceAccounts are used within Kubernetes to provide processes a way to access the Kubernetes API. ServiceAccountsdiffer from User accounts in that they are meant to be used by a machine or application instead of a human user which means they are perfect for our use case.

Once created ServiceAccounts provision a token that can then be used to authenticate with the Kubernetes API. Any request using this token will be subject to the policy defined by the Role (or ClusterRole) that it’s bound to.

RoleBinding and ClusterRoleBinding

A RoleBinding is used to tie a Role to a ServiceAccount. Similarly, a ClusterRoleBinding is used to tie a ClusterRole to a ServiceAccount. For example, let’s consider the scenario where you want to give a ServiceAccountpermissions across multiple namespaces, but not all. In this case, you would begin by creating a ServiceAccount in one namespace and then creating a Role and RoleBinding in each namespace you’d like the ServiceAccount to have access to.

By using these Kubernetes objects, you can easily create policies for Spinnaker that allow as much or as little access to the Kubernetes API as you like. For example, if you’ve created a Role that only allows the user to create Deployments and Services, a user within Spinnaker will not be allowed to deploy anything else. If they attempt to deploy a StatefulSet, for example, it will fail.

Spinnaker RBAC

Finally, Spinnaker provides the ability to add RBAC to Accounts using Fiat. While not as rich as Kubernetes RBAC, you are still able to limit read and write access to Accounts and Applications using external systems like LDAP or Okta (SAML). To do this, you should define the groups who are allowed to use a particular Account when configuring them. Users will only be able to select and use the Accounts which match their particular set of groups. For example, the following Account configuration only permits users in the engineering team read/write access but restricts users in the qa team to read-only. This can be useful if you want to allow certain teams to see what’s deployed but disallow them from making any changes.

accounts:- name: dev-environment
  kubeconfigFile: /some/path/to/kubeconfig.yml
  context: dev-environment
  permissions:
    read:
      - engineering
      - qa
    write:
-  engineering

Conclusion

How you configure Accounts is largely dependent on the needs of your organization. By using the tools detailed in this post, you’ll be able to define a set of accounts that should suit your needs. Accounts provide a powerful abstraction that allows you to slice-and-dice your Kubernetes clusters into distinct areas for each team that uses them and adds a layer of multi-tenancy that makes it even easier to expand the use of Kubernetes within an organization. RBAC provides a way to ensure this type of multi-tenancy by defining an explicit set of actions that users are allowed to perform.