Are You Logging Your Deployments Correctly?
May 17, 2017 by Armory
So our CTO, Isaac Mosquera, grabbed me and said, “Hey. We should talk about auditing trails.”
My reaction was: “What? We need to talk about this?!”
It appears that talking about being organized and accountable in your organization is one of those topics that people love to talk about but have difficulty committing to (shipping small diffs is another culprit). As companies migrate to the cloud for all the benefits it comes with, they are forgetting to lay down strong groundwork and infrastructure for gathering information regarding their own activities, a necessary step for responsibility and accountability. This becomes an unfortunate source of management debt with some technical debt that companies accrue due to their desire for velocity, which in turn becomes a strangling bottleneck as the lack of standardization kills efficiency.
Current companies generally rely on some form of ticket tracker (like JIRA) as a form of auditing trail; the thinking may be along the lines of “Well, if everyone’s logging their activities with tickets then we should have no problem, right?” However, this method relies on human diligence, and humans are prone to making mistakes. These mistakes can be anything from incorrectly reporting what was done, forgetting to report what was done, or even worse: compromising security with clandestine activity, something Uber and Waymo is currently in a legal battle over.
“It’s not a good system of record, because JIRA has nothing to do with deployments,” says Isaac.
The additional dependency on 3rd-party tools like JIRA for tracking tickets and changes involved means that your organization’s auditing trail lives and dies on the dependability of these tools (not to mention, the security). At best, this seems to be a fragile, temporary measure that will need to be addressed as management debt accumulates, introducing unnecessary overhead with each line of code that is committed or erased. JIRA tickets treat small and large tasks in an equal manner and further complicates the auditing trail when organizations conduct reviews of their logs.
Occasionally companies do something worse: they cut off developer access to production, and gate the access through specific engineers that are tasked with deploying to production. This makes superficial sense because the responsibility becomes centralized to several people, but makes no sense in the pursuit of speedy deployments and faster iteration because you’ve introduced a bottleneck. Additionally, this method introduces reliance and dependency on those people; if they have a sick day, quit, or go on vacation, deploying to production becomes halted.
But Spinnaker solves the root of these problems right out of the box, with an auditing trail easily set up for concise logging of Who, What, When, Where, and How in JSON format. Spinnaker is a tool that can be used for deploying software to multiple deployment targets already, and so engineers won’t even need to switch UIs or worry about making sure the new process is being logged. By using Spinnaker’s sub-service Echo you can literally set up an auditing trail in seconds in any secure environment your administrator wants. Since the auditing logs are in an easily searchable JSON format and do not contain security-sensitive information like passwords, a company can easily present the logs to auditors for quick and easy auditing. Additionally, if your company is used to using tools like Splunk for auditing, the JSON format can be easily imported without hassle.
Here is an example of the JSON format and Event logging that Spinnaker produces:
{ "details": { "source": "orca", "type": "orca:task:complete", "created": "1495056070817", "organization": "armory-io", "project": null, "application": "armoryspinnaker", "_content_id": null }, "content": { "standalone": true, "context": { "asgName": "armoryspinnaker-prod-v001", "credentials": "prod-aws-account", "deploy.account.name": "prod-aws-account", "deploy.server.groups": {}, "kato.last.task.id": { "id": "10203" }, "kato.task.id": { "id": "10203" }, "kato.tasks": [ { "history": [ ], "id": "10203", "resultObjects": [] } ], "notification.type": "enableasg", "regions": ["us-west-2"], "targetop.asg.enableAsg.name": "armoryspinnaker-prod-v001", "targetop.asg.enableAsg.regions": ["us-west-2"], "user": "[email protected]", "zones": ["us-west-2a", "us-west-2b", "us-west-2c"] }, "executionId": "a7ca5574-4629-119a-c9ac-ga873aa165b2", "taskName": "a72239a7-c57a-308d-9d72-1a77484e050c.enableAsg.monitorAsg.758e6e5-3c37-4599-9e93-g62118adc7c6" } }
Responsibility and Accountability are important no matter how big or small your organization is.
We’re not attempting to introduce the concept of auditing trails but the current evolution of auditing trails as companies move into the cloud – especially with regards to the current ecosystem of companies caught mid-transition with wrangled-together methods of logging their deployments. Everyone talks about having good audit trails, but in the end it’s always a good idea to check within your organization and ensure you’re practicing what you preach.
