I’m leaving you, Travis: A continuous integration breakup story (2018)

Travis’ girlfriend likes Jenkins more
It’s a bit of a travesty

Continuous integration services allow developers to automatically build and test their code with ease. Each of those services has its own strengths and weaknesses. Widder, Hilton, Kästner, and Vasilescu studied GitHub projects that no longer use Travis CI to understand what makes developers abandon a continuous integration service.

Why it matters

Continuous integration (CI) services like Travis, Circle, and Scrutinizer automate compilation, building, testing, and deployment of software. This leads to improvements in team productivity and code quality.

It takes a lot of effort to properly set up a CI pipeline however. So why do some developers choose to leave their CI service, given that they have likely spent significant amounts of effort on configurationThis is obviously a sunk cost fallacy, but… well, you know…?

There are two types of factors that could entice developers to leave their current CI service for a competing offering or abandon CI altogether:

Understanding these factors helps us build CI tools and services that are a better fit for developers.

How the study was conducted

The authors conduct their study using a (pre-existing) dataset of 38,214 public GitHub projects that have completely disabled Travis CI using a configuration setting.

Projects with the following characteristics were filtered out to reduce noise:

The resulting sample contains 1,819 projects. Each of these projects is matched with three other projects in the dataset that have similar characteristics, but have not abandoned Travis CI.

Finally, a binomial logistic regression model is constructed that estimates the likelihood of Travis CI abandonment, given a set of predictors:

What discoveries were made

Continuous integration services – and Travis CI in particular – aren’t necessarily suitable for all languages. Some languages are used within contexts for which testing does not make much sensee.g. Puppet for configuration management, while other languages are primarily intended for platforms that Travis does not supporte.g. C# for Windows environments. This makes a project’s dominant language an important predictor for Travis abandonment.

A possibly more surprising result is that projects that take longer to build are less likely to abandon Travis. The authors speculate that a longer build duration is an indication of build complexity, which is a sign that such projects have successfully managed to adapt Travis to their needsThis of course assumes that no one in their right mind would needlessly create complex build configurations.

Another interesting finding is that community factors (like the number of contributors) have limited or no effect on Travis abandonment. There are two exceptions:

The important bits

  1. Projects with languages that are not clearly suitable for or supported by a CI service are more likely to abandon it
  2. Projects with complex, tailor-made build configurations are less likely to abandon a CI service
  3. CI services possibly offer more value for projects with many pull requests, but less for projects with more commits