JIT feedback – what experienced developers like about static analysis

Running static analysis tools in an integrated development environment is like having a peer next to you that continuously reviews and criticises your code. Tymchuk, Ghafari, and Nierstrasz studied whether and why developers think these tools are helpful.

Manual code review is error-prone and time-consuming.

Fortunately static analysis tools can be used to automatically detect some bugs and code smells, which could save developers a lot of time.

But studies show that programmers don’t fully benefit from static analysis: many do not go through the effort of setting up a static analysis tool, and those that do might not use it systematically.

The authors argue that for static analysis to be effectively adopted by developers, it needs to be shipped and enabled in their IDE (integrated development environment) by default, and provide just-in-time feedback (rather than post hoc, on a CI server).

The study focusses on static analysis for Pharo, a Smalltalk-based programming language and IDE.

Pharo already included a static analysis tool that could be used on demand: developers had to actively run the tool in order to get results.

The authors added a new just-in-time analysis tool that automatically shows critiques about the context that’s currently being viewed.

To evaluate the impact of the tool on developer productivity, they conducted semi-structured interviews with 14 Pharo developers.

Most interviewed developers were highly experienced with both Pharo and programming in general. All developers understood the concepts of code quality and code smells, while all except one developer had used static analysis tools before.

Overall, developers said they like the new static analysis tool.

The good

Unsurprisingly, the authors learned from the interviews that if you want developers to adopt static analysis tools, the tools should work out of the box and run automatically without any configuration whatsoever.

One thing that developers liked in particular is the scoped nature of the new static analysis tool:

• Developers no longer need to wade through swathes of critiques about other parts of the system, which they don’t care about;

• All feedback is only about the thing the developer is currently working on, which makes it easier to understand;

• Feedback is near-instantaneous, which not only saves time, but also makes it easier to process.

The tool helps developers find and resolve all kinds of (potential) bugs before they make it into code reviews, or the CI or production environments.

Having said that, the tool is also useful during code reviews: since it automatically highlights all questionable pieces of code, reviewers can pay more attention to the implementation of actual functionality.

The bad

One of the most annoying things about static analysis tools, is the high number of false positives. The just-in-time static analysis tool also suffers from false positives, but since it only runs on a small part of the codebase at any given time, they’re more easily dealt with.

Then there are also critiques that are technically correct, but perceived as annoying nonetheless, either because developers can’t easily address them or because they disagree with the rationale behind the critique.

Senior developers often still valued annoying critiques because they motivate one to rethink their implementation. Less experienced developers on the other hand often claimed that their code was well-written and simply dismissed the critiques.

Sometimes the issue isn’t so much with the critiques, but with the way they’re explained: unclear critiques aren’t actionable and therefore don’t lead to any improvements.

Finally, about a third of the interviewed developers strongly dislike having to disable context-specific critiques using annotations.

Developers tend to dislike rules that:

• are based on metrics: these often require significant effort to resolve;

• are not just-in-time friendly: checking for unused variables, uncommented classes, debugging code, etc. are all valid critiques, but mainly serve as an annoyance during debugging sessions;

• check for idioms: these can be very helpful – especially for novice developers – but also annoying if you know why you choose not to implement something idiomatically (side note: A fairly common case in many C-family languages is the application of intentional fallthrough in a switch statement.).

Half of interviewed developers believe that the introduction of the just-in-time static analysis tool has changed their programming habits.

The main benefits of such tools include:

• Better adherence to best practices;

• Detection of simple issues;

• Teaching optimisation and idioms to novice programmers;

• Teaching style and ways to make code more portable to senior developers; and

• Notifying experienced developers of API changes that were recently introduced.