Tags: , ,

At the time of this writing the main AWS landing page lists twenty product categories, under which is an overwhelming number of resources to be deployed and configured. All resources can be created and configured via the collection of AWS consoles (the web UIs) and while this is probably the simplest option it does not make resource creation reversible or easily repeatable, nor does it allow for easy tracking of changes to infrastructure. It’s best to have a repeatable, visible and version-able process. This implies that AWS resource creation be accomplished via scripts or configuration source (property files, json, YAML, etc..).

One option is to roll your own process via a combination of home grown and third party libraries such as boto3 https://boto3.amazonaws.com/v1/documentation/api/latest/index.html, the official python AWS API client. Any source for scripts or libraries can be versioned via source control and subject to internal control such as pull requests, code reviews, etc.. This option provides the most control but requires quite a bit of work. There are generally better options.

An option which may seem like the obvious choice is to use AWS CloudFormation https://docs.aws.amazon.com/cloudformation/index.html, a product designed to do exactly what is needed. CloudFormation consists of JSON or YAML templates that combine and are grouped into collections called Stacks. A Stack’s resources can be deployed or destroyed together, think resources that depend on each other and must be deployed together. AWS CloudFormation, being an AWS product, has the full backing and support of Amazon and is constantly being improved. There may however, depending on a team’s requirements, be a better option.

HashiCorp’s Terraform https://www.terraform.io/ is a popular alternative to CloudFormation. Terraform allows you to define your infrastructure in a (mostly) simple declarative style. Terraform detects differences between your declared configuration and the current actual state of your infrastructure. It then can apply and changes necessary to make your infrastructure match your desired state after first presenting the proposed changes in an easy to read fashion. Terraform promotes re-use through its modules concept. Critically and in contrast to CloudFormation Terraform is platform agnostic. AWS is merely one plugin-in provider among many. Like CloudFormation Terraform takes the infrastructure-as-code principal to its logical conclusion allowing all configuration to be versioned and subject to whatever normal processes are desired including review and approval. While Terraform is lacking some killer features, including the ability to reverse engineer existing infrastructure into Terraform config source, it’s difficult to imagine an easier and more transparently way to manage AWS (and other) resources. When migrating a project from CloudFormation to Terraform I have observed that Terraform’s declarative style is simply easier to decipher at a glance, making collaboration such as Terraform code reviews and discussions regarding configuration more efficient.

Terraform’s state management, when setup properly, makes it difficult and unnatural for collaborators to simultaneously update infrastructure. Only one change can be applied at a time and Terraform updates the shared state before every update. It’s possible to bypass this state locking feature but it has to be done deliberately and explicitly. The best use of Terraform is in conjunction with a standard code review process. I have found that using GitHub or GitHub Enterprise with pull requests works very well, forcing infrastructure changes to be reviewed and discussed via the excellent GitHub tools before being merged into the official repository.

Perhaps a better declarative style infrastructure management tool will be created or perhaps AWS will evolve or replace CloudFormation such that it is much improved but for now Terraform seems to be the best/de-facto solution.