What constitutes evidence varies with decision makers and decision processes. For example, in some contexts, evidence is viewed as the results of research obtained through data collection and analysis. In policy making contexts, however, evidence used in decision-making processes should adopt a holistic approach that identifies multiple streams of knowledge and efficiently synthesizes information to support best possible outcomes. In this post, we outline various perspectives about evidence in selected contexts. We examine the technical basis of evidence development in addition to noting how evidence is interpreted, used, and ultimately put into practice in decision-making processes.
Evidence Derived from Qualitative Studies
The approaches for developing and synthesizing qualitative evidence can be difficult to define, in part because confusion can arise in classifying data used in a qualitative evidence synthesis (QES), an umbrella concept that encompasses over 30 different methods. In a recent paper about “evidence that goes beyond ‘what works,’” Fleming et al. (2021, p. 1) highlight the most common themes found in templates for evidence reviews and they consider the nuances of synthesizing data based on different research needs rather than using the same methods. Qualitative evidence reviews often adopt the following common outline: question formulation, protocol development, a literature search about inclusion and exclusion criteria, assessment of methodological limitations, data extraction, methods of synthesis, reporting of results, and the role of key stakeholders and consumers. This structure is helpful as it provides a roadmap for different organizations in developing a QES to suit their research needs best. A key element in reviewing qualitative data is inclusion of stakeholders and consumers because of the importance of engaging multiple perspectives and ensuring the findings are grounded in experience. Key stakeholders can offer insights for framing issues and developing the context for the focus of a review. In turn, engagement of stakeholders creates a domino-effect for involving various perspectives in establishing the criteria about what to include in a QES and also helps to determine the methods selected in a review and to formalize the overall structure of the end product and evidence.
Evidence that is “Good Enough”
When research-based evidence is combined with other types of knowledge, it is important to clearly articulate what is meant by labelling evidence as “good enough.” In their book chapter, Nutley et al., (2019) highlight concerns with identifying and weighing accumulated findings obtained from alternate forms of evidence for use in specific policy and practice interventions. Typically, discussion about such syntheses begins by identifying review bodies and labelling evidence before further steps are taken. The standards about what counts as evidence can vary among government departments even when they are a part of the same network or organization, e.g., the Canadian Coast Guard is a unit within the national Department of Fisheries and Oceans. Different departments may adopt a variety of approaches for assessing evidence that is considered by policymakers. For example, the categories “promising,” “good practices,” “effective,” “proven,” “evidence-based,” etc. represent a hierarchy of evidence. Research quality is assessed and challenged in peer review prior to publication of results. Additionally, informal professional standards about research and publication continue to be applied even after peer review is complete. Nutley et al. note that the main criteria for assessing the quality of evidence are transparency, accuracy, purposivity, utility, propriety, and accessibility. On the basis of these criteria, a government department can evaluate the quality of evidence by comparing characteristics of particular evidence to the department’s measure of what constitutes low to high quality.
Evidence in Environmental Management
When evaluating evidence in environmental policy contexts it is important to consider perception, use, and barriers to various forms of evidence by practitioners. For example, Lemieux et al. (2021) state that it is important to account for protected areas in gathering information about biodiversity for development of policy and in management activities. They demonstrate that use of all forms of evidence is in decline and perception and difficulties with availability of information are becoming greater issues. Additionally, barriers to evidence use are increasing over time. Using survey data collected in 2013 and 2019, the study explores the overarching concept of evidence complacency, introduced by Sutherland & Wordley (2017), through the lens of organizations that influence decision-making about protected areas. Evidence complacency is defined as “a way of working in which, despite availability, evidence is not sought or used to make decisions, and the impact of actions is not tested” (Sutherland & Wordley, 2017, p. 1215). To remedy this situation, Lemieux et al. (2021), outlined four key recommendations aimed at transformational change in the conservation community. The first is building a stronger evidence base through greater emphasis on evidence-synthesis. The second is expansion of the perspectives obtained beyond the common forums used by practitioners, such as input from Indigenous peoples, NGOs, and the news media. The third is to bring Indigenous knowledge to the forefront within protected area conservation to foster knowledge co-production. Finally, Lemieux et al. recommend extending transparency and accountability through improved reporting measures and legislative mechanisms.
Evidence in the UK Parliament
Politicians learn about and gain an understanding of legislation in their work in parliamentary committees. Geddes (2021) describes how Members of Parliament (MPs) in the United Kingdom obtain evidence, learn about the policy processes, and make decisions. The experience of Canadian MPs is similar to British MPs due to Canada’s Westminster style of parliamentary government and similar political climates. Geddes explains that most policy learning occurs in committee meetings where the testimony of witnesses influences many of the decisions made by parliamentarians. The role of parliamentary staff in distilling information is important to the decision and evidence-based process. Geddes’s primary finding is that provision of evidence is often restricted to those who have access to MPs and the committee process, e.g., public service scientists and recognized academics. Evidence can sometimes be disregarded by political staff when writing briefings or key players may not have access to decision-makers. This scenario creates evidence disconnects in the decision-making processes.
Evidence in Fisheries Management
Understanding how information flows from scientists to policymakers and the relationships that accompany the flow of evidence is important especially because the pathways are often complex. Soomai (2017) describes the flow of evidence within Canada’s Science Advisory Secretariat (CSAS) and the Department of Fisheries and Oceans (DFO), and the international, intergovernmental Northwest Atlantic Fisheries Organization (NAFO). Soomai focused on how scientists and policymakers interact in a national department and an international regional fisheries management organization. Relationships among scientists and managers and effective managerial and science communication enable policy decision processes. Typically, science information flows within DFO’s CSAS to the wider department and from scientists to managers at NAFO; however, the pathways of evidence are often not straightforward and barriers in relaying information can result in misinterpretations of evidence or the transmission of some information being impeded. To combat these hurdles, Soomai (2017) encourages policymakers to engage with scientists and vice versa, to build understanding of the scientific and policymaking processes, and to work at overcoming administrative and communication silos.
Conclusion
It can be concluded from the readings considered in this post that the process of searching for, selecting, and synthesizing evidence encompasses a wide spectrum of issues, questions, or contexts. Practitioners operating in different policy spaces should remember that the definition and interpretation of evidence can affect how it is perceived and used by decision-makers. For example, how scientists use evidence to inform decision-makers can alter policy outcomes. Moving forward, it is paramount that scientists and policymakers utilize holistic approaches to obtain and synthesize evidence which accounts for varying perspectives. Scientists and decision makers play key roles in policymaking; however, the way(s) that evidence is used to inform policy decisions can have long-term affect on the success of policy outcomes.
References
Flemming, K., & Noyes, J. (2021). Qualitative evidence synthesis: Where are we at? International Journal of Qualitative Methods, 20, 160940692199327. https://doi.org/10.1177/1609406921993276
Geddes, M. (2020). The webs of belief around “evidence” in legislatures: The case of select committees in the UK House of Commons. Public Administration, 99(1), 40-54. https://doi.org/10.1111/padm.12687
Lemieux, C. J., Halpenny, E. A., Swerdfager, T., He, M., Gould, A. J., Carruthers Den Hoed, D., Bueddefeld, J., Hvenegaard, G. T., Joubert, B., & Rollins, R. (2021). Free fallin’? The decline in evidence-based decision-making by Canada’s protected areas managers. FACETS, 6(1), 640-664. https://doi.org/10.1139/facets-2020-0085
Nutley, S. M., Davies, H. T. O., & Hughes, J. (2019). Assessing and labelling evidence. In A. Boaz, H. T. O. Davies, A. Fraser, & S. M. Nutley (Eds.), What works now? Evidence-informed policy and practice (pp. 225-249). The Policy Press.
Soomai, S. S. (2017). The science-policy interface in fisheries management: Insights about the influence of organizational structure and culture on information pathways. Marine Policy, 81, 53-63. https://doi.org/10.1016/j.marpol.2017.03.016
Sutherland, W. J., & Wordley, C. F. (2017). Evidence complacency hampers conservation. Nature Ecology & Evolution, 1(9), 1215-1215. https://doi.org/10.1038/s41559-017-0244-1
Authors: Cailey Dyer, Camille Haddad, and Sydney Francolini
This blog post is part of a series of posts authored by students in the graduate course “The Role of Information in Public Policy and Decision Making” offered at Dalhousie University.
