High quality research needs to employ carefully phrased and expressed questions that give participants the opportunity to say what they really think, without bias or misinterpretation. Minimising bias in research is essential to get results that are as accurate, objective and actionable as possible. Biases can take many forms and can be introduced at any stage of the process from initial design, through recruitment, right down to reporting the findings.
Implementing innovative research techniques or carrying out ‘agile’ research does not mean that potential bias becomes less important. Quite the opposite in fact. The biggest bias risk of all is creating a research paradigm that leads participants to the answers you (and/or the client) want(s) to hear.
Take for example an actual user experience research exercise in which a co-design workshop was used to explore user views and attitudes towards driverless cars. Participants were invited to attend the three-hour workshop at the Coventry Transport Museum. Whilst this might be a prefect venue from the perspective of facilities, the subject of the research was related to that of the Museum. Inevitably, this could introduce a degree of participant conditioning and affect their attitudes and behaviour. A neutral value reduces distractions and locational biases.
A more significant bias was introduced when participants were asked to place a cut-out of themselves on an ungraded physical positive/negative scale under the question; “Do you think driverless cars are a good idea?”
Whilst the idea of asking participant to engage with a physical scale may be appropriate, the problem is that the question is skewed to the presumption that driverless cars are a positive development and pushes respondents to think about the question framed in those terms.
Not only was the question leading, it was open to wide interpretation also. What exactly does “a good idea” mean? A conceptual “idea” as opposed to real world reality – the “idea” is good, but it’ll never work? “Good” relative to what? And so on. Participants each have their own norms and values and careful questioning is essential to help to align these to provide meaningful results.
This question was asked twice, once when respondents joined the workshop and again at the end of the workshop, after they had had the ‘advantages’ of driverless cars explained to them, to see to what extent there was change to placement on the scale. So, a risk of acquiescence bias was introduced during the workshop.
Even the photograph depicting the question results showed only the “A very good idea” end of the scale. So, additional reporting bias added distortion to the cumulative effects already built-in. A classic example of confirmational bias.
If you layer this with the client’s identity being fairly obvious to participants, thereby introducing bias around attitudes towards the sponsoring organisation, the problems just keep multiplying.
You may be wondering why this matters? It matters because there are so many types of potential bias that can mislead in all sorts of directions. In just this one example the more research participants were conditioned, the more ways a question could be interpreted, the more difficult it could become to analyse and interpret the results objectively. When research findings become a matter of subjective interpretation of biased data, they become little more than a lamp-post used by someone who has had one-too-many on a night out; a form of support, rather than of illumination. Worse still, the findings can end up making no sense.
No research methodology can ever be 100% bias-free. But to achieve high-quality results means asking the right participants, the right questions, at the right time, in the right place, in the right way, whilst identifying potential and actual sources of bias at every stage and eliminating, reducing or accounting for these in every way possible. And that includes the researcher’s own biases.