I’m leaving you, Travis: A continuous integration breakup story

Continuous integration services allow developers to automatically build and test their code with ease. Each of those services has its own strengths and weaknesses. Widder, Hilton, Kästner, and Vasilescu studied GitHub projects that no longer use Travis CI to understand what makes developers abandon a continuous integration service.

Continuous integration (CI) services like Travis, Circle, and Scrutinizer automate compilation, building, testing, and deployment of software. This leads to improvements in team productivity and code quality.

It takes a lot of effort to properly set up a CI pipeline however. So why do some developers choose to leave their CI service, given that they have likely spent significant amounts of effort on configuration (side note: This is obviously a sunk cost fallacy, but… well, you know…)?

There are two types of factors that could entice developers to leave their current CI service for a competing offering or abandon CI altogether:

• things that developers dislike about their current service and therefore push them away from it;

• things that developers particularly like about some competing service and therefore pull them towards it.

Understanding these factors helps us build CI tools and services that are a better fit for developers.

The authors conduct their study using a (pre-existing) dataset of 38,214 public GitHub projects that have completely disabled Travis CI using a configuration setting.

Projects with the following characteristics were filtered out to reduce noise:

• Projects in which development activity ceased shortly after Travis abandonment;
• Projects that use niche programming languages.

The resulting sample contains 1,819 projects. Each of these projects is matched with three other projects in the dataset that have similar characteristics, but have not abandoned Travis CI.

Finally, a binomial logistic regression model is constructed that estimates the likelihood of Travis CI abandonment, given a set of predictors:

• Project age in days

• Number of distinct project contributors

• Number of distinct contributors to .travis.yml (side note: This text file is committed as part of the project repository and is used to configure Travis builds.)

• Number of changes made to .travis.yml

• Number of pull requests

• Number of commits

• Build duration, in seconds

• Number of spawned jobs per build

• The dominant programming language

Continuous integration services – and Travis CI in particular – aren’t necessarily suitable for all languages. Some languages are used within contexts for which testing does not make much sense (side note: e.g. Puppet for configuration management), while other languages are primarily intended for platforms that Travis does not support (side note: e.g. C# for Windows environments). This makes a project’s dominant language an important predictor for Travis abandonment.

A possibly more surprising result is that projects that take longer to build are less likely to abandon Travis. The authors speculate that a longer build duration is an indication of build complexity, which is a sign that such projects have successfully managed to adapt Travis to their needs (side note: This of course assumes that no one in their right mind would needlessly create complex build configurations).

Another interesting finding is that community factors (like the number of contributors) have limited or no effect on Travis abandonment. There are two exceptions:

• Projects with more pull requests are less likely to abandon Travis, possibly because Travis is invaluable for evaluating pull requests;

• Projects with more commits are more likely to abandon Travis, because they become too large to work well with Travis or need more tailored solutions.