Cognitive bias among forensic experts is a significant issue that can lead to unsupportable conclusions and miscarriages of justice when undetected. It may also result in the exclusion of evidence at trial that one of the parties was relying upon as a method of proof. Such was the case in a June 2023 Northern Ireland prosecution under the Terrorism Act. This case will serve as the backdrop for discussing cognitive bias in a broader context.
What is Cognitive Bias?
Bias may be defined as being unfairly prejudiced against a particular individual or group or demonstrating an unjustified concentration of interest on a particular target or subject. It is described in scientific literature as “[a]ny process at any stage of inference which tends to produce results or conclusions that differ systematically from the truth.” [1] One type of bias is motivational bias, which involves conscious preferential choices whereas cognitive bias is less overt, often operating at the subconscious level, but no less problematic. [2,3] Cognitive bias includes processes that may lead to inaccurate judgments, interpretations, or decision-making. [4,5] It can impact the observation and perception of data, how data is tested, and how the results are interpreted. [3] There are different types of cognitive bias and in this article, I will focus on contextual bias and role effects.
Contextual bias refers to the subconscious bias that can develop when experts are given irrelevant information as part of their instructions and during their forensic work. In the image comparison context, wherein an expert is tasked with comparing questioned images of a person, vehicle or object with known images, the expert may be given information that is important to the instructing party but is completely irrelevant to the expert’s task. Examples include extraneous proof that the questioned and known subjects are the same (confession, DNA, fingerprints, eyewitnesses, and other corroborative evidence), that the defendant is a known criminal, police theories of the case, unnecessary emotive details about the incident being examined, etc. None of this information is helpful to the forensic imaging expert, nor is it necessary to convey.
Cognitive bias arises when irrelevant contextual information causes an aspect of the analysis to be overweighted, underweighted, or disregarded. [3] An imaging expert who has been given improper contextual information by the instructing party (eg., the “answer”) may intuitively be more likely to assign a stronger level of support (overweighting) to the comparison than might otherwise be warranted by the unadorned data (images). Similarly, irrelevant contextual information may cause an expert to overlook or underweight the absence of data, not confirm results that are confirmable, or not consider alternative explanations. [3] These problems can arise when the expert is given unnecessary information about the enormity of the case or its human impact. The expert may subconsciously be more inclined to go down an irrelevant rabbit hole to get to a certain conclusion.
Role effects are particularly associated with an adversarial system that pits one party against another. Itiel Dror refers to this as “adversarial allegiance” and “myside bias”, wherein experts are biased towards finding a result that favours “their side”. [3] Though it is correctly said that an expert’s duty is to the court and not the instructing party, subconsciously there may be a desire to please the instructing party. The fact that many forensic services are performed by state agencies accentuates this problem. Equally, in a private instruction environment, the desire to offer something of value to a potential client cannot be overlooked.
A further complicating factor that enhances the combined effects of contextual bias and role effects is when the expert crosses the line from being an independent and objective expert to becoming embedded in the investigation being undertaken by the instructing party. It is common for experts in some areas of expertise to be consulted during an investigation for the purpose of obtaining preliminary information and guidance from them. However, doing so can create the real risk of role effect bias. Both contextual and role effect biases will be further discussed below.
R. v. Gleeson (Northern Ireland)
In R. v. Gleeson (Northern Ireland Crown Court, 2023), the defendant was charged with a series of counts arising from his alleged attendance at meetings that involved members of a proscribed organization who were engaged in conspiracies to possess firearms and ammunition and to prepare terrorist acts. Part of the evidence relied upon by the prosecution arose from several expert reports prepared by two experts (referred to herein by their firm name “JPF”) instructed by the prosecution who opined that there is “strong support” for the proposition that the defendant was one of two individuals speaking at the two meetings in question. These conclusions arose from forensic voice analysis wherein the JPF experts conducted a voice comparison between questioned recordings obtained by covert listening devices placed at the meetings by security services and known recordings of the defendant speaking on the telephone while in prison after his arrest for these offences. There was no issue with the authenticity of the recordings or their content. The sole issue was whether the questioned recordings were properly attributed to the defendant.
The Court received a series of expert reports from JPF as well as testimony from one of their experts. It also received reports from a professor of linguistics and a forensic speech consultant, both retained by the defence. The forensic speech consultant testified at trial and challenged the conclusions of the JPF experts on the basis that their work was vulnerable to confirmation bias and that their methodology was flawed. The defendant challenged the admissibility of the evidence of the JPF experts and sought to have it excluded. The defence raised four issues with the court. The first three issues dealt with the admissibility of the known recordings and whether the voice comparison was in compliance with a Court of Appeal judgment in a previous case. I will focus on the fourth issue – cognitive bias.
Cognitive bias was a significant component of the defence’s challenge to the voice comparison evidence presented by the prosecution. The defence relied upon the extensive guidance provided by the UK Forensic Science Regulator (FSR) and the judge quoted extensively from this guidance. Among this guidance is the following:
1.4.1 The most powerful means of safeguarding against the introduction of contextual bias is to ensure that the practitioner conducting the analysis has only the information about the case that is relevant to the analysis…
4.2.6 Contextual bias is where someone has other information aside from that being considered, which influences (consciously or subconsciously) the outcome of the consideration. Psychological research has demonstrated that perception is responsive to both the individual’s psychological and cognitive state along with the environment in which they are operating. For example, a scientist working within a police laboratory could be influenced by knowing that the detectives believe that they have a strong suspect, or that the suspect has already confessed to having committed the crime. Provision of information not required by the scientist to undertake the evaluation and that potentially influences this type of bias has been termed “psychological contamination” or “cognitive contamination”, as opposed to the more widely understood issue with forensic science “physical contamination”.
4.2.7 Role effects are where scientists identify themselves within adversarial judicial systems as part of either the prosecution or defence teams, and this may introduce subconscious bias that can influence decisions, especially where some ambiguity exists…
The defence noted that when the JPF experts were provided with the questioned recordings, they were also given transcripts in which the police attributed specified content to the defendant. The defence argued that the seeds of potential bias were sown at this early stage and cited UK case law that criticized the provision of attribution transcripts to voice experts and the impact that this approach can have on their independence. The JPF experts countered that their role was to test the police attribution theory by applying their forensic expertise.
Further complications arose from the allegation that much of the early analysis of the JPF experts took place in the presence of investigating officers. While this early work did not involve the defendant, it was done for the purpose of providing an “intelligence report”. It was anticipated that having provided this intelligence work, the JPF experts would be engaged for the purpose of doing substantive evidential work that would be used at trial. The defence argued that there was an implicit plan to elevate the intelligence work undertaken by the experts to an evidential standard. Other concerns were also raised regarding the independence of the JPF experts given their relationship with the police during the investigation.
During the hearing, one of the JPF experts testified that she was aware of the risks of confirmation bias but was adamant that it did not arise in this case. She stated that she approached her work independently and considered the risks of bias. The Court was satisfied that the expert was convinced of the validity of her methodology and the validity of her opinion. There was no question of intentional misconduct or deliberate bias. However, the Court found that the risks of cognitive bias in the case were real and obvious. Not only were there risks of contextual bias and role effects, but also confirmation bias, anchoring effects, and expectation bias. These risks arose primarily from the fact that the experts were given an annotated transcript in with the police had already attributed content to the defendant. These risks were exacerbated by the very close relationship the experts had with the police throughout the investigation. No steps were taken to mitigate these risks by distancing the experts from the investigation and to otherwise protect the independence of their findings. Accordingly, the evidence of the JPF experts was excluded.
Though it was not the role of the Court to give directions for future work, the Court did state that forensic voice comparison experts should not be given annotated transcripts. Further, once instructed, experts must ensure that they are appropriately distanced from the prosecuting authorities (police and prosecutors). Mitigating steps such as peer review and independent verification are also important.
Significance of R. v. Gleeson
Though the Gleeson ruling is a Northern Ireland trial ruling and is not binding on other trial courts, let alone other jurisdictions, it should stand as a clarion call to experts, investigators, and counsel alike. For the sake of argument, assuming that the opinion of the JPF experts was in fact correct, the circumstances in which it came to fruition were fatally flawed and the prosecution therefore lost the ability to rely upon evidence that was an important plank in their case. Thus, there was a loss to the prosecution resulting from the immediate consequences of inadmissibility, but there is a broader issue to consider. Had the tainted evidence not been excluded, it may have contributed to a miscarriage of justice if the court relied upon unsound evidence to support a conviction.
Combatting Cognitive Bias
Gleeson is not an outlier. Expert witness bias is a topic that frequently comes up in cross examination and final argument at trial because it is a live concern in many cases. The Supreme Court of Canada stated that expert witnesses have a special duty to the court to provide fair, objective, and non-partisan assistance. A proposed expert witness who is unwilling or unable to carry out this duty is not qualified to give expert opinion evidence and should not be permitted to do so (White Burgess v. Haliburton, 2015). Proposed expert evidence has been excluded in Canada on this basis.
Biases can be identified and mitigated, but they cannot be eliminated. It is essential that experts focus solely on relevant data and guard against being compromised in their work. The FSR provides extensive guidance on combatting cognitive bias and in relation to contextual bias cautions against instructing parties giving experts irrelevant and unnecessary information and expresses a preference for written briefings rather than direct verbal communication between the investigator and the expert. (11.4.1)
Experts must actively combat the risk of bias creeping into their work. Steps that can be taken by experts to mitigate the risks of confirmation bias and role effects include the following:
a) Experts have control over what information is given to them and the form in which it is provided. Experts must decline to receive any contextual information that is irrelevant to their stated task. This includes information as to what the police or the instructing party believe the facts reveal and their case theories. Cautious and regulated receipt of information is in the best interests of the expert, the instructing party, and the case.
b) Instructing parties must avoid contaminating experts with irrelevant bias-inducing information.
c) Instructions should be provided in writing so that there is a record of what was requested and what information was provided.
d) Experts must document all information that was provided to them by the instructing party and any other source.
e) Experts should not become embedded in the investigation and then also serve as the evidential expert at trial. This is a recipe for disaster as was seen in Gleeson and other cases. There may be a legitimate need for an expert to be part of the investigation and in such cases a second independent expert should be instructed for trial purposes and communication between the two experts must be restricted.
f) Independent peer review is a helpful method of objectively assessing methodology and conclusions.
Conclusion
Cognitive bias leads to partisan truths. The goal should be to find non-partisan, objectively verifiable truths. Given the duty that experts have to the court to provide fair, objective, and non-partisan assistance, experts must strive to be independent from the interests of the instructing party. Experts can provide a valuable service to their client without joining the “team”. Combatting and minimizing cognitive bias is in the best interests of truth seeking and the administration of justice.
References
- David Sackett, ‘Bias in Analytical Research’, Journal of Chronic Diseases 32 (1979): 51-63.
- Gilberto Montibeller and Detlof von Winterfeldt, ‘Cognitive and Motivational Biases in Decision and Risk Analysis’, Risk Analysis 35, no. 7 (2015): 1230-51.
- Itiel Dror, ‘Cognitive and Human Factors in Expert Decision Making: Six Fallacies and the Eight Sources of Bias’, Analytical Chemistry 92 (2020): 7998-8004.
- Glinda Cooper and Vanessa Meterko, ‘Cognitive Bias Research in Forensic Science: A Systematic Review’, Forensic Science International 297 (2019): 35-46.
- Carsten Stahn, ‘From Preliminary Examination to Investigation: Rethinking the Connection’, in Quality Control in Criminal Investigation, edited by Xabier Agirre, Morten Bergsmo, Simon De Smet, and Carsten Stahn (Brussels: Torkel Opsahl Academic EPublisher, 2020), 37-66.