Learning from mistakes: An empirical study of elicitation interviews performed by novices

Interviewing is one of the most versatile tools for requirements elicitation. Sadly, it’s also notoriously hard to master. Bano, Zowghi, Ferrari, Spoletini, and Donati studied interviews conducted by postgraduate students, and categorised the different types of interviewing mistakes that one should avoid.

Interviews can provide a wealth of information about a problem domain, and its stakeholders and their needs.

It’s important that interviews are well-conducted: if mistakes are made before or during the interview, they can lead to incorrect requirements and costly rework down the line.

Novice interviewers should therefore be aware not only of best practices, but also of common pitfalls.

The paper describes an experiment with 110 postgraduate students of a requirements engineering class at University of Technology Sydney.

Students worked together in teams of 3–4 members to develop a software requirements specification for a fictional project, based on a brief, one-page project description and an interview with a “customer (side note: Played by the course instructor)”. Each group was asked to submit minutes of their interview.

Novices predictably make a lot of mistakes, which affects the quality of their elicited requirements.

Seven major categories of mistakes were identified during observations and analysis of audio recordings that were made during the interviews.

Question formulation

Questions should be clear and unambiguous: a badly phrased question will elicit questions that are less or even not useful at all.

The authors identified three major types of mistakes in this category:

• Vague questions can be interpreted in multiple ways or are so broadly formulated that it’s not clear what the interviewer actually wants to know;

• Technical questions erroneously assume that the customer has sufficient understanding of technical concepts. Asking technical questions may intimidate the customer and lead to bad rapport;

• Questions about things that are not within scope of the project or appropriate for the customer’s role are irrelevant and incorrect. They waste time and might even contribute to erroneous or redundant requirements.

Other less frequent mistakes include questions in which the customer is asked for a solution and questions that are way too long.

Question omission

While every project is unique in its own way, there are some questions that should always be asked. Forgetting to ask these questions (or even purposely omitting them) may result in missing or wrong requirements.

Specific examples of such mandatory questions include:

• questions that aim to identify relevant stakeholders;

• follow-up and probing questions;

• enquiring about existing systems or business processes;

• asking the customer to prioritise features;

• asking questions about the problem domain.

Order of interview questions

Questions should be asked in a logical order, so that they make sense and are felt to be appropriate to the customer.

The authors recommend the following order:

1. Building rapport with the customer;

2. Understand the existing business process;

3. Understand the problems currently faced by the customer;

4. Summarise the findings to the customer to confirm that everything was understood correctly.

Communication skills

Ideally, an interview should be like a conversation: a natural exchange of information rather than a verbal “survey”, an interrogation, or a monologue by the interviewers. It’s important that interviewers speak the language of the customer, carefully listen to what is said, and are able to deviate from the “script” if necessary.

Analyst behaviour

Non-verbal aspects also matter: an overconfident interviewer may unjustly think they understand the problem domain, which causes them to overlook alternative or contradictory information that might have resulted in better requirements. Other possible issues include unprofessional attitude and nervousness.

Customer interaction

The interviewer must create an environment in which the customer feels at ease.

Typical mistakes include:

• Asking questions without properly introducing themselves;

• Asking ice-breaking questions in a formulaic or disinterested manner (side note: In many cases students even reiterated complete phrases that were used as examples in an online interviewing course!);

• Vehemently disagreeing with something the customer says.

Teamwork and planning

If there are multiple interviewers, one should make sure that interviewers work as a team and do not interrupt each other. It might also be advisable to divide tasks among interviewers.

The minutes that were submitted by the student groups were used to assess how well they understood the answers given by the customer.

In general, groups that made mistakes during the interview also articulated their understanding poorly in the minutes. Conversely, most (but not all) groups that did fairly well in the interviews also submitted reasonably good quality minutes.