JIT feedback – what experienced developers like about static analysis (2018)

A driver’s navigation warns them about the speed limit while the driver is stuck in a traffic jam

Running static analysis tools in an integrated development environment is like having a peer next to you that continuously reviews and criticises your code. Tymchuk, Ghafari, and Nierstrasz studied whether and why developers think these tools are helpful.

Why it matters

Manual code review is error-prone and time-consuming.

Fortunately static analysis tools can be used to automatically detect some bugs and code smells, which could save developers a lot of time.

But studies show that programmers don’t fully benefit from static analysis: many do not go through the effort of setting up a static analysis tool, and those that do might not use it systematically.

The authors argue that for static analysis to be effectively adopted by developers, it needs to be shipped and enabled in their IDE by default, and provide just-in-time feedback (rather than post hoc, on a CI server).

How the study was conducted

The study focusses on static analysis for Pharo, a Smalltalk-based programming language and IDE.

Pharo already included a static analysis tool that could be used on demand: developers had to actively run the tool in order to get results.

The authors added a new just-in-time analysis tool that automatically shows critiques about the context that’s currently being viewed.

To evaluate the impact of the tool on developer productivity, they conducted semi-structured interviews with 14 Pharo developers.

What discoveries were made

Most interviewed developers were highly experienced with both Pharo and programming in general. All developers understood the concepts of code quality and code smells, while all except one developer had used static analysis tools before.

Tool usefulness

Overall, developers said they like the new static analysis tool.

The good

Unsurprisingly, the authors learned from the interviews that if you want developers to adopt static analysis tools, the tools should work out of the box and run automatically without any configuration whatsoever.

One thing that developers liked in particular is the scoped nature of the new static analysis tool:

The tool helps developers find and resolve all kinds of (potential) bugs before they make it into code reviews, or the CI or production environments.

Having said that, the tool is also useful during code reviews: since it automatically highlights all questionable pieces of code, reviewers can pay more attention to the implementation of actual functionality.

The bad

One of the most annoying things about static analysis tools, is the high number of false positives. The just-in-time static analysis tool also suffers from false positives, but since it only runs on a small part of the codebase at any given time, they’re more easily dealt with.

Then there are also critiques that are technically correct, but perceived as annoying nonetheless, either because developers can’t easily address them or because they disagree with the rationale behind the critique.

Senior developers often still valued annoying critiques because they motivate one to rethink their implementation. Less experienced developers on the other hand often claimed that their code was well-written and simply dismissed the critiques.

Sometimes the issue isn’t so much with the critiques, but with the way they’re explained: unclear critiques aren’t actionable and therefore don’t lead to any improvements.

Finally, about a third of the interviewed developers strongly dislike having to disable context-specific critiques using annotations.

Rule usefulness

Developers tend to dislike rules that:

Impact

Half of interviewed developers believe that the introduction of the just-in-time static analysis tool has changed their programming habits.

The main benefits of such tools include:

The important bits

  1. Static analysis tools should be included and enabled in IDEs by default and “just” work, without any setup
  2. A static analysis tool is most effective when it provides immediate feedback on code that is being worked on by the developer
  3. Just-in-time analysis should not point out issues that cannot be easily resolved or aren’t relevant to the developer (yet)
  4. A static analysis tool can provide many benefits to developers, provided that they accept and resolve its critiques