Beyond the code itself: How programmers really look at pull requests (2019)

GraphQL logo with eye, surrounded by rays of light
Cousin, let’s go pulling!

Approval (or rejection) of pull requests is mostly done based on its contents. But it’s not just about the code – especially when the changes are proposed by someone you don’t know. This week’s paper examines the signals that developers look at when reviewing pull requests from unfamiliar contributors.

Why it matters

Software development is not just about code – it’s also about people.

The change that’s proposed in a pull request might make sense and adhere to whatever contribution guidelines you have set up, but if it’s made by someone whom you’re not familiar with, you’ll probably do some research on the submitter first: What’s their level of expertise? What have they worked on? Could they have nefarious intentions?

Contemporary Git hosting services, such as GitHub, facilitate such research by providing user profile pages that clearly show users’ online identities (avatar, username, and name), interests, and contributions.

We’ve known for quite a while that reviewers make use of such information, thanks to studies on pull review acceptance. Sadly, those studies often suffer from issues with validity, as they are either based on interviews with developersHumans tend to forget or misremember things pretty quickly. or analysis of GitHub data.

How the study was conducted

The authors recruited 42 students from an advanced computer science course for a controlled experiment with eye-tracking glasses.

Each participant was asked to review a pull request for a small mock project and indicate the likelihood that they would accept it on a 5-point Likert scale.

The pull request came in two variants: a reasonable version that works just fine and an unreasonable, buggy version. This pull request was “submitted” by one of three personas with identical experience levels: Abby, Tim, and Pat (who has an ambiguous gender).

Profile pictures used for each of the three personas
Abby, Tim, and Pat. None of them are mentioned in the Results section. ¯\_(ツ)_/¯

Participants were interviewed afterwards about their reviewing behaviour and their attitude towards profiles in technical communities.

What discoveries were made

Eye-tracking data reveal that developers have the tendency to under-report their fixations in interviews: less than half of participants mentioned looking up information about the submitter, even though in practice every single participant has done so.

Overall, participants spent most of their time (about 60%) on reviewing the actual codeAs they should!, but also looked at technical (30%) and social signals (10%).

Further analysis of participants’ (self-reported) experience levels and fixation patterns yields some interesting results:

Adapting profiles

There are several things that developers take into consideration when creating and maintaining profiles:

The important bits

  1. Developers often underestimate how much they make use of social signals when reviewing pull requests
  2. The proposed code changes receive the most attention during a review, but developers also try to learn more about the submitter
  3. A good profile provides the right social cues and avoids cues that may reflect badly on its owner