Calculating the bus factor

Most software is developed in teams. Work and knowledge are rarely distributed evenly in such teams. This can cause problems: work that is done by engineers with low expertise on a (part of the) system is often more prone to bugs. More importantly, sudden departure of key project members can make a project come to a grinding halt when its remaining members no longer have a thorough understanding of a significant part of the project.

The bus factor (side note: Also known as the truck factor, bus number, or lottery factor) is a metric that was originally defined as the minimum number of developers that would have to be hit by a bus before the project stalls. Small values correspond to higher existential risks for projects, while larger values imply a relatively even distribution of knowledge.

While the bus factor is easy to understand, it is hard to compute. Various attempts have been made (side note: The original article discusses these in more detail.) to translate the informal definition of the bus factor into a bus factor estimation algorithm.

The most accurate method of determining the factor likely involves asking stakeholders, but this only works for small projects. A more common approach relies on data from version control systems (VCS), but is also flawed: knowledge is shared and created not only through code, but also through other methods, like code reviews, meetings, and issue trackers. Thus a VCS-based bus factor algorithm would severely underestimate the importance of a senior engineer who rarely writes code, but may still have the most project knowledge of anyone in the team.

The authors have tried to design a new bus factor estimation algorithm that also uses code review and meetings data from JetBrains Space.

The paper provides a formal description the new bus factor algorithm. I haven’t implemented math typesetting yet for my blog, so I can’t share the algorithm here, but I’ll describe how it works using words (side note: Like some kind of plebe).

First, we need to compute a so-called degree of authorship (DOA), i.e. the contribution level of an engineer. This metric is based on several file-based variables:

• Whether the engineer was the first author;
• The number of commits that the engineer has made;
• The number of reviews conducted by the engineer;
• The number of minutes spent on meetings about commits, with an upper limit of 240 minutes.

This DOA metric assumes that knowledge from a contribution decays exponentially and halves every five months. The resulting score is a number that indicates to what extent an engineer is a major contributor to a file.

The bus factor can then be computed by iteratively removing top authors until the current engineers’ knowledge covers less than half of the files in a project. The number of removed engineers represents the bus factor.

The authors conducted an online survey among 269 respondents in order to better understand how knowledge distribution and decay work in a project, and how these things impact the bus factor of a project.

Only 19% of respondents have worked on a project for which the bus factor was explicitly communicated. This is not necessarily a problem though, especially in smaller projects where team members simply “know” what the bus factor is.

Most respondents consider the bus factor to be the most important metric of project health, out of seven types of collective development problems (which include vague responsibilities, lack of documentation and broken fixes, among a few other issues). Low bus factors are seen as the fourth most common problem type, which further highlights the importance of the bus factor as a project health metric.

Many respondents believe that knowledge decays with time, but there is little agreement about the speed at which this happens, nor what the granularity level of bus factor calculations should be, i.e. per project, module or team.

Respondents believe that bus factor data could be used to address project risks in several ways, for example by:

• trying to gain more knowledge about modules that are at risk;
• attracting more key engineers;
• refactoring and improving the code quality to make it more accessible;
• designing the system so that responsibilities are divided from the beginning;
• sharing knowledge by writing documentation and giving talks;
• rotating engineers between different parts of the project.

It’s important to note that usage of bus factor data may be somewhat controversial. Respondents raised concerns about the ethical consequences of the metric. Key engineers may feel irreplaceable, while those not listed as key engineers may feel unvalued. Engineers in general may feel discouraged from working on the project altogether due to the perceived risk. Technical concerns include paying too much attention to “risky” modules or too little, when overestimation of the bus factor leads to a false sense of security.

Finally, the authors conducted a survey among 14 JetBrains engineers from 13 projects of varying sizes to validate how well the results of the bus factor algorithm agree with the perception of engineers.

The new algorithm seems to make slightly better estimations than the baseline version that only takes VCS data into account. A significant variance was found in the reported bus factor for large projects (where such a metric would probably be most useful).