Best practices for infrastructure-as-code (IaC) scripts

The current market demands increasingly faster deployments, faster release cycles, faster disaster recoveries, and so on. DevOps practices like infrastructure as code (IaC) promote the management of all infrastructure through reusable scripts, which replaces slow, time-consuming and error-prone processes with fast automated processes that are (almost) guaranteed to work every time.

IaC is a widely adopted practice and consequently a lot has been written about it online. This week’s paper presents the results of a systematic grey literature review on best and bad practices (side note: I skip these in my summary, because they’re too similar to the best practices.) for infrastructure as code. The review includes resources about Ansible, Chef, and Puppet, but I’ll only focus on Ansible here as it’s the only one of these three that I use.

The literature review uncovered ten best practices for infrastructure-as-code programs.

Infrastructure as code programs are no different from regular software in the sense that they need to be readable, understandable, and maintainable.

• Make names consistent, distinctive, and meaningful. Task names should communicate the purpose of the task, while variables can include usage context to help users quickly identify the origin of a variable. Moreover, a single naming style should be used throughout the program.

• Make code style and formatting consistent. For Ansible, this means that you should organise your code into playbooks and inventories, and follow YAML best practices.

• Make parameters, their types, and defaults explicit. Doing this makes it clearer how code works and reduces the likelihood of errors due to incorrect assumptions about the program’s behaviour and parameters.

• Use conditionals properly. Improper use of conditionals can make the code hard to understand.

Care should be taken to reduce code clones and increase the reuse of IaC programs and tools.

• Modularise IaC programs. Decomposition of IaC programs into modular fragments gives you greater control over who has access to which parts of the infrastructure code, and promotes reuse.

• Reuse code instead of rewriting it. All IaC languages provide ways to encapsulate and extend pieces of potentially reusable logic.

• Select the right modules for the job and use it correctly. In Ansible this can be done using roles and modules. When possible, task-specific modules should be preferred over general modules that execute ad hoc OS-level commands (e.g. shell and command modules).

• Reuse tools that the community uses. Follow conventions (e.g. deployment architectures and development tools) that are recommended by the community.

IaC enables seamless deployments without the need for (a lot of) manual actions.

• Codify everything and avoid putting additional instructions in a separate document. The configuration files should be the single source of truth of all infrastructure specifications.

• Package applications for deployment. Packaging applications (e.g. as .wars or Docker images) reduces the amount of code needed in the later stages of configuration management.

• Do not violate idempotence of IaC programs. An IaC program should carry out the same actions every time it runs. Skipping tasks in Ansible plays is therefore not recommended.

Changes to infrastructure configuration should be done incrementally.

• Use a version control system. Any changes to the configuration should be managed, tracked, and reconciled. Version control systems provide an audit trail for changes, the ability to collaborate, and peer review IaC code. Ideally, the configuration should use semantic versioning.

• Favour versionable functionalities. Try to use as many versionable files and modules as possible. This makes it easier to reason about the effects of a change.

Certain practices can be used to prevent errors in IaC code.

• Use the correct quoting style to prevent accidental or incorrect interpolation of values.

• Avoid unexpected behaviours whenever possible. Different solutions exist for different IaC languages, but in general it helps to define variables as close as possible to their usage and minimise the impact of unexpected execution orders.

• Use proper values for variables, e.g. a valid 4-digital octal value should be used for file modes.

Mistakes are inevitable, but you can make them less likely.

• Write tests as you code. Standard testing practices like static code analysis, unit testing, functional testing, integration testing, and test automation should also be applied to infrastructure code.

• Do not ignore errors. All errors should be handled to ensure that the infrastructure is always left in the desired state. The state of resources should also be checked explicitly whenever possible.

• Use off-the-shelf testing libraries and tools, e.g. to create virtual test environments.

• Monitor your environment continuously so you can always keep an eye on the state of the managed infrastructure.

IaC programs should be appropriately documented.

• Code as documentation. Source code should be treated as documentation. Do not include additional instructions in a manual, as this may lead to inconsistencies and non-reproducible environments.

• Use document templates to produce consistent templates for infrastructure code.

Proper organisation of IaC repositories makes collaboration easier and improves understandability of the program.

• Modularise repositories. It’s generally recommended to use a single version-controlled repository per organisation that’s separate from the application source code repository.

• Use standard folder structures that are recommended by your IaC tool.

Configuration data need to be managed in the right way.

• Use configuration data source. As the number of managed components increases, it becomes more important to use a separate storage system for configuration data about those components (e.g. user names, server IPs).

• Modularise configuration data to improve their maintainability and usage, e.g. make sure that the same IaC program can be used to deploy test, development and production environments.

• Select data sources wisely. IaC languages provide different options to store configuration data, along with guidelines that help you make the right trade-offs.

• Use configuration templates. Externalise all template variables as input parameters to create self-contained, reusable templates.

Finally, make sure that the IaC configuration is secure:

• Isolate secrets from code. Passwords and keys should never be hard-coded, but only be injected into the deployment workflow when necessary.

• Protect your data at rest by encrypting them in a vault when they are stored.

• Use facts from trusted sources to make decisions about the infrastructure.

• Use standard secure coding practices like secure logging and the principle of least privilege.