Enhancing the pull request experience with meta-bots

Nowadays bots are often used in the pull request review process as an interface between users and other tools. Such bots can reduce workloads of maintainers and contributors by automating repetitive tasks, like repairing simple bugs, refactoring source code, fixing static analysis violations, suggesting code improvements, and predicting defects. In some project bots even perform more activities than humans!

Nevertheless, in practice messages created by bots are sometimes seen as disruptive and noisy, especially when they overburden developers who already suffer from information overload.

In some domains, meta-bots are used to integrate and moderate interactions of multiple bots. This study explores how meta-bots can be used to reduce information overload in pull requests.

The paper describes a design study that consists of two parts. First, the researchers organised a series of Design Fiction sessions with experts in bot design, in which they explored strategies to overcome the information overload that bots can cause. Based on the input gathered during these sessions, a set of emerging strategies were prototyped and evaluated.

The Design Fiction sessions yielded 22 design strategies, which can be organised into five categories: information management, newcomers’ assistance, notification management, spam and failures management, and platform support.

Information management is primarily about mitigating noise from too many bot messages:

• Summarisation of bot content can help mitigate information overload by concisely reporting an overview of the pull request status.

• Aggregating bot comments is the easiest way to merge outputs from multiple bots into a single comment on a pull request.

• Prioritisation of bot outputs based on tasks and based on issues helps users focus on what is really important.

• Categorisation of bot comments (e.g. testing, security, information) lets users decide for themselves what kind of information they want to see.

• To avoid polluting pull requests with an avalanche of comments from the meta-bot, the meta-bot should only keep the most recent information, i.e. create a single comment and continuously update it with information from other bots.

• Design session participants also envisioned a meta-bot that can interact with users through natural language, understands users’ requests and can answer their questions.

• To promote diversity and inclusion, a meta-bot would ideally provide support for internationalisation.

Information provided by bots can affect newcomers’ perceptions and success:

• By explaining rules, instructions and requirements a bot can guide newcomers and let them know what they need to do to get their pull request approved. This could be done in a welcoming message.

• Participants envision the meta-bot providing information interactively, by displaying information step by step so as not to overwhelm newcomers.

• Another useful strategy is related to notifying maintainers about new pull requests from newcomers, to encourage human activity from the maintainer.

Several design strategies can be used to allow the meta-bot to control different aspects of bot notifications:

• Meta-bots should notify developers through a pre-specified channel, i.e. wherever the developer wants to receive notifications.

• Ideally it is possible to schedule bot notifications so that the meta-bot avoids notifying developers when they don’t want interrruptions.

• Alternatively a meta-bot could notify developers in their idle times.

• Another option could be to not notify maintainers until a condition is satisfied, e.g. when all tests have passed.

• Moreover, a meta-bot should notify only interested developers. Not all developers may be interested in activities for a particular repository, issue or pull request.

Three design strategies provide control over unforeseen problems created by bot interactions:

• To prevent repetitive bot activities a meta-bot should detect bots that generate repetitive outcomes and be able to silence them.

• There should be a mechanism for spam messages notification, so that developers can be made aware of repetitive bot messages that might be considered spam.

• Finally, if there is a bug with a specific bot, the meta-bot should be able to contact bot maintainers to provide a bug report.

A few changes in the platform interface could improve the experience of working with bots. For instance, one could:

• Separate bot comments from human comments by relegating them to a space reserved for bot interactions.

• Create a bot configuration dashboard, where developers can customise their preferences for viewing bot interactions.

In the second phase of the study, the researchers developed a prototype (side note: Screenshots can be found in the paper.) that incorporates the above strategies, and used interviews and a Technology Acceptance Model questionnaire to collect feedback from 15 practitioners who have experience working with software bots.

Overall reception to the meta-bot prototype was positive. Most participants found the designed meta-bot interface useful and easy to use. Almost all participants agreed that if the designed interface were available in the future, they would use it.

The participants also suggested a number of things that could be improved:

• Completely separating bot comments is probably not a good idea, as it may result in loss of context and makes it harder to quote bot comments in the main pull request conversation. It may also make bot comments less discoverable.

• Not all developers appreciate interactive meta-bots. Some might prefer static versions.