Expressions of sentiments during code reviews: Male vs. female

This study looks at the emotions that men and women express in their review comments, how they differ, and possible consequences for gender diversity in software engineering.

Until the 1960s programming was largely done by women. Since then, the field has become increasingly male-dominated through professional associations, hiring campaigns that discouraged women, and personality tests that were biased against female applicants.

The few female software developers that remain in the field today report that they experience sexual harassment, get fewer opportunities, and receive more rejections of their pull requests (side note: Some therefore argue that it’s “better” for women to hide the fact that they are female.).

But it’s not just about getting pull requests to be approved: the language used in the reviews matters as well.

Some believe that review comments – like other forms of feedback – should be seen as a gift. It’s easy to see how this can be true if a comment contains encouraging or heartwarming words. It gets a bit harder when it includes criticism, snark, or is full of expletives. (How) do comments between males and females differ?

The code reviews of six popular open-source (C, C++ and/or Java) software projects were mined for review comments.

Due to practical reasons, the authors only focus on 3,570 so-called non-casual developers, i.e. people who have contributed at least five changes to a project.

The authors used the genderComputer tool to guess developers’ genders. Since most developers are male, the authors then manually validated the gender of all 937 (supposedly) non-male developers by looking up their profile pictures in the review tool or on social media.

The SentiSE sentiment analysis tool is then used along with SentiSense, a lexical sentiment database, to determine the sentiment of each developers’ review comments.

More than 85% of review comments do not express any sentiments. Of the comments that do, most have a negative tone. Comments that are written by women are less likely to express any positive or negative sentiments. However, women do express sentiments more often when they interact with other women.

Men seem to express their sentiments more consistently, although in three of the projects, men write considerably more negative comments for their female colleagues, while withholding positive comments. It is likely one of the many reasons why so many women leave the computing field after a few years.

When we look at the type of sentiment words that are used, we can see that men are more likely to use words that are related to strong emotions, like surprise, sadness, love, joy, hate, disgust, fear, and anger. In three of the projects, comments written by women seem to show more anger and digust, presumably due to negative attitudes from their male colleagues (side note: I can’t tell if this is conjecture or something that the researchers have actually seen and confirmed in some way).

Some comments contain emoticons. Positive emoticons are used more often than negative ones, and mostly by men. This is somewhat surprising, as many studies show that women generally use more emoticons than men. The authors suspect that this may be due to a lack of diversity: because women are often misinterpreted by men and perceived as emotional, they are taught that it’s best to show as little emotion as possible.

Review comments rarely contain expletives or swear words, which is not very surprising considering that these are all large open-source projects (side note: And none of them is the Linux kernel). Male developers appear to swear more often than female developers. Words like “crap”, “damn”, and “screw” are commonly used by both genders. Expletives that are highly offensive (especially towards females), like “bitch”, “bastard”, “fuck”, and “jerk” were only used by males.