Bad practices in continuous integration pipelines and processes
Continuous integration (CI) enables development teams to detect integration issues as early as possible. Some teams also use continuous delivery (CD), which allows for frequent code releases.
Adoption of CI can lead to a significant increase in productivity and customer satisfaction, so it’s probably not surprising that CI has become one of the most widely used software engineering practices. Nevertheless, adopting CI properly can be quite a challenge. Developers may struggle with automating the build process or debugging pipeline failures, or apply CI incorrectly.
This week’s paper empirically investigates what bad practices developers encounter when using CI in their daily development activities. Its authors conducted semi-structured interviews with 13 developers and manually analysed over 2,300 Slack Overflow posts. The result is a chonky catalogue of anti-patterns for continuous integration, which was evaluated using a survey.
The catalogue lists bad smells related to seven different categories: the repository, infrastructure choices, build process organisation, build maintainability, quality assurance, delivery process, and culture.
Repository
The first ten bad smells are related to the way the source code repository is organised and misuse of the version control system in the context of CI.
Survey respondents believe that only some of these bad smells are important in practice, particularly differences between local and CI workspaces, and not versioning pipeline resources.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Project decomposition in the repository does not follow modularisation principles | 39 | > | 30 |
| Test cases are not organised in folders based on their purposes | 22 | < | 48 |
| Local and remote workspace are not aligned | 57 | > | 30 |
| Number of branches do not fit the project needs/characteristics | 27 | < | 41 |
| A stable release branch is missing | 45 | = | 45 |
| Feature branches are used instead of feature toggles | 45 | > | 30 |
| Divergent branches | 48 | > | 38 |
| Generated artifacts are versioned, while they should not | 18 | < | 50 |
| Blobs are unnecessarily checked in at every build instead of being cached | 23 | < | 41 |
| Pipeline-related resources are not versioned | 64 | > | 18 |
Infrastructure choices
The second category of bad smells is related to a sub-optimal choice of hardware or software components for CI pipelines, e.g. poor allocation of resources or incorrect usage of CI tools.
Somewhat surprisingly, most smells in this category were received negatively by respondents. It’s likely that the increased availability of cheap computing and networking resources has essentially reduced some bad smells to non-issues.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Resources related to the same pipeline stage are distributed over several servers | 14 | < | 64 |
| The CI server hardware is used for different purposes other than running the CI framework | 24 | < | 48 |
| External tools are used with their default configurations | 28 | = | 28 |
| Different releases of tools/plugins versions are installed on the same server | 48 | > | 40 |
| Different plugins are used to perform the same task in the same build process | 28 | < | 48 |
| A task is implemented using an unsuitable tool/plugin | 32 | < | 40 |
| Use shell scripts for a task for which there is a suitable plugin available | 20 | < | 44 |
Build process organisation
The authors identified 29(!) bad smells that are related to poor configuration of a CI pipeline.
A fair number of the smells in this category were received positively by survey respondents, which is not surprising as many of the bad smells in this category directly impact the usability of a CI pipeline.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Inappropriate build environment clean-up strategy | 40 | > | 28 |
| Missing package management | 58 | > | 25 |
| Wide and incohesive build jobs are used | 40 | < | 44 |
| Monolithic builds are used in the pipeline | 56 | > | 28 |
| Independent build jobs are not executed in parallel | 36 | < | 48 |
| Only the last commit is built, aborting obsolete and queued builds | 24 | < | 44 |
| Build steps are not properly ordered | 48 | > | 40 |
| Pipeline steps/stages are skipped arbitrarily | 46 | = | 46 |
| Tasks are not properly distributed among different build stages | 35 | < | 48 |
| Incremental builds are used while never building the whole project from scratch | 36 | = | 36 |
| Poor build triggering strategy | 50 | > | 33 |
| Private builds are not used | 36 | > | 32 |
| Some pipeline tasks are started manually | 44 | = | 44 |
| Use of nightly builds | 48 | > | 36 |
| Inactive projects are being polled | 17 | < | 52 |
| A build is succeeded when a task is failed or an error is thrown | 64 | > | 28 |
| A build fails because of some flakiness in the execution, whereas it should not | 67 | > | 12 |
| Dependency management is not used | 44 | > | 40 |
| Including unneeded dependencies | 44 | > | 36 |
| Some tasks are executed without clearly reporting their results in the build output | 56 | > | 24 |
| The output of different build tasks are mixed in the build output | 48 | > | 17 |
| Failure notifications are only sent to teams/developers that explicitly subscribed | 40 | < | 44 |
| Missing notification mechanism | 56 | > | 28 |
| Build reports contain verbose, irrelevant information | 56 | > | 24 |
| Time-out is not properly configured | 52 | > | 36 |
| Unneeded tasks are scheduled in the build process | 44 | > | 28 |
| Build time for the commit stage overcomes the 10-minutes rule | 52 | > | 22 |
| Unnecessary re-build steps are performed | 40 | > | 20 |
| Authentication data is hard-coded (in clear) under VCS | 52 | > | 32 |
Build maintainability
Build configurations can be harder to maintain, because changes to configuration generally incur more relative churn than changes to source code.
Reception to the bad smells in this category was mixed, possibly because some bad smells do not apply to all types of projects.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Absolute/machine-dependent paths are used | 65 | > | 31 |
| Build scripts are highly dependent upon the IDE | 58 | > | 27 |
| Environment variables are not used at all | 31 | < | 35 |
| Build configurations are cloned in different environments | 29 | < | 50 |
| Build jobs are not parametrised | 36 | = | 36 |
| Lengthy build scripts | 28 | < | 40 |
| Missing smoke test, set of tests to verify the testability of the build | 44 | > | 24 |
| Missing/poor strict naming convention for build jobs | 46 | > | 15 |
Quality assurance
Testing and static analysis are important steps in most CI pipelines, but both are easy to mess up.
Most respondents agreed that these bad smells are harmful in practice and should be mitigated whenever possible.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Lack of testing in a production-like environment | 75 | > | 8 |
| Code coverage tools are run only while performing testing different from unit and integration | 35 | > | 30 |
| Coverage thresholds are fixed on what reached in previous builds | 30 | < | 35 |
| Coverage thresholds are too high | 25 | < | 46 |
| Missing tests on feature branches | 50 | > | 31 |
| All permutations of feature toggles are tested | 37 | > | 32 |
| Production resources are used for testing purposes | 52 | > | 36 |
| Testing is not fully automated leading to a non-reproducible build | 65 | > | 27 |
| Test suite contains flaky tests | 69 | > | 19 |
| Bad choice on the subset of test cases to run on the CI server | 40 | > | 28 |
| Failed tests are re-executed in the same build | 20 | < | 52 |
| Quality gates are defined without developers considering only what dictated by the customer | 42 | > | 29 |
| Use quality gates in order to monitor the activity of specific developers | 52 | > | 36 |
| Unnecessary static analysis checks are included in the build process | 42 | > | 28 |
Delivery process
This category of bad smells is about the artefacts that are generated after a successful pipeline build.
All smells in this category were assessed positively by respondents, i.e. they should be taken into account by teams that want to improve their delivery process.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Artefacts locally generated are deployed | 58 | > | 35 |
| Missing artefacts’ repository | 46 | > | 35 |
| Missing rollback strategy | 62 | > | 15 |
| Release tag strategy is missing | 50 | > | 27 |
| Missing check for deliverables | 62 | > | 25 |
Culture
Some bad smells are not related to the CI pipeline itself, but the people who use it.
Again, all bad smells in this category are believed to be at least somewhat important by survey respondents.
| Bad smell | Agree (%) | ? | Disagree (%) |
|---|---|---|---|
| Changes are pulled before fixing a previous build failure | 43 | > | 30 |
| Team meeting/discussion is performed just before pushing on the master branch | 36 | > | 32 |
| Developers and operators are kept as separate roles | 44 | > | 24 |
| Developers do not have a complete control of the environment | 46 | > | 19 |
| Build failures are not fixed immediately giving priority to other changes | 58 | > | 12 |
| Issue notifications are ignored | 64 | > | 16 |
Summary
- I can’t and am not going to summarise 79 different bad smells. Just look at the tables and do whatever you need to do, okay?! 😩