Recruitment screening: avoiding leading questions

We are careful to avoid leading questions in research; are we doing the same when screening participants?

The way we ask questions in research is different from the way we do in everyday conversation. It has to be because the goal is not the exchange of ideas. It is gathering data.

Likewise, screener questions help us gather information to establish the suitability of participants. We may filter out people with irrelevant experience, exclusions that may skew data, and people who are trying to make a quick buck from research without any interest in providing genuine answers.

The way we ask screener questions therefore requires the same rigour as research to make sure we don’t inadvertently reveal the ’correct‘ response to potential respondents.

User research with non-representative participants is a waste of time and money. Here are some questioning methods from research which can be applied to the screening process to make sure you recruit the ‘right’ participants.

1. Leave the answer out of the question

Instead of “Are you a driver?” ask, “What do you do for work?”

Obscuring the purpose of the question makes things less straightforward so you may need to ask further questions to match against your recruitment criteria, but it should yield more reliable data than giving participants a 50/50 chance of getting the answer ’right’.

In this case for example, you might need to follow up with “What does that job involve?” or “What are you responsible for?” and decide if the answer corresponds with your criteria.

2. Don’t prompt the participant

“What apps do you use? Like Facebook, Instagram or Path, for example?”

Even for well-intentioned participants, it’s easy to latch on to what is given and cut off a train of thought. Ask “What apps do you use?” and wait.

Don’t read out multiple choices either.

3. Use triangulation questions to verify their claims

Triangulation is a way of using different sources or methods to check the credibility of research findings. Findings are likely to be credible if the same results are obtained from disparate sources.

In this context, we ask follow up questions to see if answers converge.

Sometimes, you get a feeling that an answer was shaky or you slipped up and might have given away the answer—a questionable “Err…Facebook?” perhaps. Trust your instincts and follow up with “What do you use Facebook for?” or “What did you last do on Facebook and when was this?” If they can’t decide (might have been a very long time ago) or worse, can’t name a correct Facebook function, you have your answer.

These techniques can quickly improve your pool of participants, but you may still find the odd miss-recruit that slipped through. Remember that as a researcher, you reserve the right to send a participant home if they have misrepresented themselves during screening.

It takes an attention to crafting questions and a bit of intuition to extract truthful answers. If you want to recruit better participants, don’t just check things off on your screener, ask questions just like you would in research.