Chuniversiteit.nl
The Toilet Paper

I’m leaving you, Travis: A continuous integration breakup story

What makes people abandon their continuous integration service?

Travis’ girlfriend likes Jenkins more
It’s a bit of a travesty

Continuous integration services allow developers to automatically build and test their code with ease. Each of those services has its own strengths and weaknesses. Widder, Hilton, Kästner, and Vasilescu studied GitHub projects that no longer use Travis CI to understand what makes developers abandon a continuous integration service.

Why it matters

Continuous integration (CI) services like Travis, Circle, and Scrutinizer automate compilation, building, testing, and deployment of software. This leads to improvements in team productivity and code quality.

It takes a lot of effort to properly set up a CI pipeline however. So why do some developers choose to leave their CI service, ?

There are two types of factors that could entice developers to leave their current CI service for a competing offering or abandon CI altogether:

  • things that developers dislike about their current service and therefore push them away from it;

  • things that developers particularly like about some competing service and therefore pull them towards it.

Understanding these factors helps us build CI tools and services that are a better fit for developers.

How the study was conducted

The authors conduct their study using a (pre-existing) dataset of 38,214 public GitHub projects that have completely disabled Travis CI using a configuration setting.

Projects with the following characteristics were filtered out to reduce noise:

  • Projects in which development activity ceased shortly after Travis abandonment;
  • Projects that use niche programming languages.

The resulting sample contains 1,819 projects. Each of these projects is matched with three other projects in the dataset that have similar characteristics, but have not abandoned Travis CI.

Finally, a binomial logistic regression model is constructed that estimates the likelihood of Travis CI abandonment, given a set of predictors:

  • Project age in days

  • Number of distinct project contributors

  • Number of distinct contributors to

  • Number of changes made to .travis.yml

  • Number of pull requests

  • Number of commits

  • Build duration, in seconds

  • Number of spawned jobs per build

  • The dominant programming language

What discoveries were made

Continuous integration services – and Travis CI in particular – aren’t necessarily suitable for all languages. Some languages are used within , while other languages are primarily intended for . This makes a project’s dominant language an important predictor for Travis abandonment.

A possibly more surprising result is that projects that take longer to build are less likely to abandon Travis. The authors speculate that a longer build duration is an indication of build complexity, .

Another interesting finding is that community factors (like the number of contributors) have limited or no effect on Travis abandonment. There are two exceptions:

  • Projects with more pull requests are less likely to abandon Travis, possibly because Travis is invaluable for evaluating pull requests;

  • Projects with more commits are more likely to abandon Travis, because they become too large to work well with Travis or need more tailored solutions.

Summary

  1. Projects with languages that are not clearly suitable for or supported by a CI service are more likely to abandon it

  2. Projects with complex, tailor-made build configurations are less likely to abandon a CI service

  3. CI services possibly offer more value for projects with many pull requests, but less for projects with more commits