Learning from mistakes: An empirical study of elicitation interviews performed by novices (2018)

A television crew interviews someone with two microphones, but the camera is still on the ground

Interviewing is one of the most versatile tools for requirements elicitation. Sadly, it’s also notoriously hard to master. Bano, Zowghi, Ferrari, Spoletini, and Donati studied interviews conducted by postgraduate students, and categorised the different types of interviewing mistakes that one should avoid.

Why it matters

Interviews can provide a wealth of information about a problem domain, and its stakeholders and their needs.

It’s important that interviews are well-conducted: if mistakes are made before or during the interview, they can lead to incorrect requirements and costly rework down the line.

Novice interviewers should therefore be aware not only of best practices, but also of common pitfalls.

How the study was conducted

The paper describes an experiment with 110 postgraduate students of a requirements engineering class at University of Technology Sydney.

Students worked together in teams of 3–4 members to develop a software requirements specification for a fictional project, based on a brief, one-page project description and an interview with a “customerPlayed by the course instructor”. Each group was asked to submit minutes of their interview.

What discoveries were made

Novices predictably make a lot of mistakes, which affects the quality of their elicited requirements.

Common mistakes

Seven major categories of mistakes were identified during observations and analysis of audio recordings that were made during the interviews.

Question formulation

Questions should be clear and unambiguous: a badly phrased question will elicit questions that are less or even not useful at all.

The authors identified three major types of mistakes in this category:

Other less frequent mistakes include questions in which the customer is asked for a solution and questions that are way too long.

Question omission

While every project is unique in its own way, there are some questions that should always be asked. Forgetting to ask these questions (or even purposely omitting them) may result in missing or wrong requirements.

Specific examples of such mandatory questions include:

Order of interview questions

Questions should be asked in a logical order, so that they make sense and are felt to be appropriate to the customer.

The authors recommend the following order:

  1. Building rapport with the customer;
  2. Understand the existing business process;
  3. Understand the problems currently faced by the customer;
  4. Summarise the findings to the customer to confirm that everything was understood correctly.

Communication skills

Ideally, an interview should be like a conversation: a natural exchange of information rather than a verbal “survey”, an interrogation, or a monologue by the interviewers. It’s important that interviewers speak the language of the customer, carefully listen to what is said, and are able to deviate from the “script” if necessary.

Analyst behaviour

Non-verbal aspects also matter: an overconfident interviewer may unjustly think they understand the problem domain, which causes them to overlook alternative or contradictory information that might have resulted in better requirements. Other possible issues include unprofessional attitude and nervousness.

Customer interaction

The interviewer must create an environment in which the customer feels at ease.

Typical mistakes include:

Teamwork and planning

If there are multiple interviewers, one should make sure that interviewers work as a team and do not interrupt each other. It might also be advisable to divide tasks among interviewers.

Impact

The minutes that were submitted by the student groups were used to assess how well they understood the answers given by the customer.

In general, groups that made mistakes during the interview also articulated their understanding poorly in the minutes. Conversely, most (but not all) groups that did fairly well in the interviews also submitted reasonably good quality minutes.

The important bits

  1. Build rapport with the customer before starting the actual interview. Summarise your interpretation of their answers afterwards
  2. Interviews must be prepared in advance, so no questions are asked that are irrelevant, unclear, or even intimidating
  3. A badly conducted interview will likely lead to poor understanding of stakeholders’ needs