Chuniversiteit logomarkChuniversiteit.nl
The Toilet Paper

Bad practices in continuous integration pipelines and processes

This catalogue of 79 bad smells can help you make your continuous integration processes less shitty.

Someone crawls into a giant nose
Oh, nose!

Continuous integration (CI) enables development teams to detect integration issues as early as possible. Some teams also use continuous delivery (CD), which allows for frequent code releases.

Adoption of CI can lead to a significant increase in productivity and customer satisfaction, so it’s probably not surprising that CI has become one of the most widely used software engineering practices. Nevertheless, adopting CI properly can be quite a challenge. Developers may struggle with automating the build process or debugging pipeline failures, or apply CI incorrectly.

This week’s paper empirically investigates what bad practices developers encounter when using CI in their daily development activities. Its authors conducted semi-structured interviews with 13 developers and manually analysed over 2,300 Slack Overflow posts. The result is a chonky catalogue of anti-patterns for continuous integration, which was evaluated using a survey.

The catalogue lists bad smells related to seven different categories: the repository, infrastructure choices, build process organisation, build maintainability, quality assurance, delivery process, and culture.

Repository

Link

The first ten bad smells are related to the way the source code repository is organised and misuse of the version control system in the context of CI.

Survey respondents believe that only some of these bad smells are important in practice, particularly differences between local and CI workspaces, and not versioning pipeline resources.

Bad smell Agree (%) ? Disagree (%)
Project decomposition in the repository does not follow modularisation principles 39 > 30
Test cases are not organised in folders based on their purposes 22 < 48
Local and remote workspace are not aligned 57 > 30
Number of branches do not fit the project needs/characteristics 27 < 41
A stable release branch is missing 45 = 45
Feature branches are used instead of feature toggles 45 > 30
Divergent branches 48 > 38
Generated artifacts are versioned, while they should not 18 < 50
Blobs are unnecessarily checked in at every build instead of being cached 23 < 41
Pipeline-related resources are not versioned 64 > 18

Infrastructure choices

Link

The second category of bad smells is related to a sub-optimal choice of hardware or software components for CI pipelines, e.g. poor allocation of resources or incorrect usage of CI tools.

Somewhat surprisingly, most smells in this category were received negatively by respondents. It’s likely that the increased availability of cheap computing and networking resources has essentially reduced some bad smells to non-issues.

Bad smell Agree (%) ? Disagree (%)
Resources related to the same pipeline stage are distributed over several servers 14 < 64
The CI server hardware is used for different purposes other than running the CI framework 24 < 48
External tools are used with their default configurations 28 = 28
Different releases of tools/plugins versions are installed on the same server 48 > 40
Different plugins are used to perform the same task in the same build process 28 < 48
A task is implemented using an unsuitable tool/plugin 32 < 40
Use shell scripts for a task for which there is a suitable plugin available 20 < 44

Build process organisation

Link

The authors identified 29(!) bad smells that are related to poor configuration of a CI pipeline.

A fair number of the smells in this category were received positively by survey respondents, which is not surprising as many of the bad smells in this category directly impact the usability of a CI pipeline.

Bad smell Agree (%) ? Disagree (%)
Inappropriate build environment clean-up strategy 40 > 28
Missing package management 58 > 25
Wide and incohesive build jobs are used 40 < 44
Monolithic builds are used in the pipeline 56 > 28
Independent build jobs are not executed in parallel 36 < 48
Only the last commit is built, aborting obsolete and queued builds 24 < 44
Build steps are not properly ordered 48 > 40
Pipeline steps/stages are skipped arbitrarily 46 = 46
Tasks are not properly distributed among different build stages 35 < 48
Incremental builds are used while never building the whole project from scratch 36 = 36
Poor build triggering strategy 50 > 33
Private builds are not used 36 > 32
Some pipeline tasks are started manually 44 = 44
Use of nightly builds 48 > 36
Inactive projects are being polled 17 < 52
A build is succeeded when a task is failed or an error is thrown 64 > 28
A build fails because of some flakiness in the execution, whereas it should not 67 > 12
Dependency management is not used 44 > 40
Including unneeded dependencies 44 > 36
Some tasks are executed without clearly reporting their results in the build output 56 > 24
The output of different build tasks are mixed in the build output 48 > 17
Failure notifications are only sent to teams/developers that explicitly subscribed 40 < 44
Missing notification mechanism 56 > 28
Build reports contain verbose, irrelevant information 56 > 24
Time-out is not properly configured 52 > 36
Unneeded tasks are scheduled in the build process 44 > 28
Build time for the commit stage overcomes the 10-minutes rule 52 > 22
Unnecessary re-build steps are performed 40 > 20
Authentication data is hard-coded (in clear) under VCS 52 > 32

Build maintainability

Link

Build configurations can be harder to maintain, because changes to configuration generally incur more relative churn than changes to source code.

Reception to the bad smells in this category was mixed, possibly because some bad smells do not apply to all types of projects.

Bad smell Agree (%) ? Disagree (%)
Absolute/machine-dependent paths are used 65 > 31
Build scripts are highly dependent upon the IDE 58 > 27
Environment variables are not used at all 31 < 35
Build configurations are cloned in different environments 29 < 50
Build jobs are not parametrised 36 = 36
Lengthy build scripts 28 < 40
Missing smoke test, set of tests to verify the testability of the build 44 > 24
Missing/poor strict naming convention for build jobs 46 > 15

Quality assurance

Link

Testing and static analysis are important steps in most CI pipelines, but both are easy to mess up.

Most respondents agreed that these bad smells are harmful in practice and should be mitigated whenever possible.

Bad smell Agree (%) ? Disagree (%)
Lack of testing in a production-like environment 75 > 8
Code coverage tools are run only while performing testing different from unit and integration 35 > 30
Coverage thresholds are fixed on what reached in previous builds 30 < 35
Coverage thresholds are too high 25 < 46
Missing tests on feature branches 50 > 31
All permutations of feature toggles are tested 37 > 32
Production resources are used for testing purposes 52 > 36
Testing is not fully automated leading to a non-reproducible build 65 > 27
Test suite contains flaky tests 69 > 19
Bad choice on the subset of test cases to run on the CI server 40 > 28
Failed tests are re-executed in the same build 20 < 52
Quality gates are defined without developers considering only what dictated by the customer 42 > 29
Use quality gates in order to monitor the activity of specific developers 52 > 36
Unnecessary static analysis checks are included in the build process 42 > 28

Delivery process

Link

This category of bad smells is about the artefacts that are generated after a successful pipeline build.

All smells in this category were assessed positively by respondents, i.e. they should be taken into account by teams that want to improve their delivery process.

Bad smell Agree (%) ? Disagree (%)
Artefacts locally generated are deployed 58 > 35
Missing artefacts’ repository 46 > 35
Missing rollback strategy 62 > 15
Release tag strategy is missing 50 > 27
Missing check for deliverables 62 > 25

Culture

Link

Some bad smells are not related to the CI pipeline itself, but the people who use it.

Again, all bad smells in this category are believed to be at least somewhat important by survey respondents.

Bad smell Agree (%) ? Disagree (%)
Changes are pulled before fixing a previous build failure 43 > 30
Team meeting/discussion is performed just before pushing on the master branch 36 > 32
Developers and operators are kept as separate roles 44 > 24
Developers do not have a complete control of the environment 46 > 19
Build failures are not fixed immediately giving priority to other changes 58 > 12
Issue notifications are ignored 64 > 16

Summary

Link
  1. I can’t and am not going to summarise 79 different bad smells. Just look at the tables and do whatever you need to do, okay?! 😩