Studying the impact of adopting continuous integration on the delivery time of pull requests (2018)

A completely empty highway
Merging is a lot easier when you’re in green fields

Continuous integration is a practice that promises more frequent releases, which means that new functionality can be delivered more quickly to end-users. Bernardo, Da Costa, and Kulesza analysed over 160,000 pull requests from 87 GitHub projects and suggest that you may want take those claims with a grain of salt.

Why it matters

Adopting continuous integration (CI) means that each change to an application is automatically built and verified: the entire development workflow is optimised for merging new features as quickly as possible into a project’s main VCS branch. This makes it less likely that (hard to resolve) merge conflicts arise, and makes it possible to release new versions more often.

Studies have already shown that open source projects merge pull requests more quicklyBut only when submitted by core developers… and discover bugs in less time once CI is adopted. CI also appears to change the way commits are made: practices like “commit often” and “commit small” are often adopted together with CI.

None of this is interesting to end-users however, who just want new functionality as fast as possible. Do features really make their way to end-users in less time when CI is adopted?

How the study was conducted

The authors searched GitHub for projects that:

and ended up with 87 projects that for further statistical analysis.

What discoveries were made

Many projects deliver merged pull requests more quickly after adoption of continuous integration, but not as many as you might have expected.

Merging pull requests

Any new change to an existing codebase needs to wait two times before it can be delivered to end-users:

The delivery delay is shorter in only 51% of the projects after adopting CI, which is surprising given that projects often adopt CI in an attempt to shorten these delays.

That’s not the only surprise however:

These findings suggest that things take a bit longer when you use CI.

Delivery activity after CI adoption

It’s interesting to know why projects need more time to deliver changes to end-users after adopting CI.

The authors therefore look at the number of pull requests that are submitted, merged, and delivered per release; both before and after CI adoption.

It turns out that the number of submitted pull requests greatly increased for 71% of the projects after adopting CI: projects submitted a median of 15.3 pull requests per release before and 42.6 pull requests per release after CI adoption. It makes sense that it would take more time to review, merge, and deliver this many pull requests.

The number of delivered pull requests and pull request contributors per release also increase after CI adoption.

Strangely there’s no increase in the number of releases.

Factors that affect delivery time

Finally, the authors looked at some characteristics of pull requestsThe complete list can be found in the original article. before and after CI adoption to learn how they affect delivery time.

The authors discovered that the merge workload – which is the number of pull requests that are created and waiting to be merged by a core contributor – has the strongest impact on delivery time before adoption of CI practices.

After CI adoption the primary factor becomes the position of a pull request in the pull request “queue”, i.e. pull requests that are merged later within a release cycle have a shorter delivery time.

There’s one factor that’s influential both before and after CI adoption: if pull requests by a contributor were delivered more quickly, then the same is likely to happen to future pull requests by that same contributor.

The important bits

  1. Only half of open source projects are able to deliver new features more quickly when adopting continuous integration
  2. Adoption of continuous integration appears to be correlated with a higher number of contributions and contributors