Eyes on Evidence III: still almost nothing to see here

Evidence for Democracy, a not-for-profit organization that explores the factual basis of government decision-making in Canada, has just released its third report on the transparency of policy-making processes. This series, called Eyes on Evidence, applies a framework to assess the information that is used to develop, implement, and review a chosen policy, ranking each step according to an observer’s ability to identify and confirm the relevant information

Previous reports had conducted such an analysis on dozens of federal policies, but this new entry considered the role of evidence in provincial policy. More specifically, the organization’s researchers looked at 133 policies announced by the governments of Ontario, British Columbia, and Saskatchewan. Ranked on simple scale of 0 to 3, based on publicly available citations of evidence used at each stage of policy formation, the results show that the vast majority of these policies offered very little in the way of such references.

“Most of the time, the policies being read through don’t mention evidence at all,” Caitlin Fowler, Senior Research Associate with Evidence for Democracy, told Research Money.

She noted that policies would often be accompanied by some enthusiastic rationale, such as desire to support the economy or defend the interests of a particular group, but seldom were these claims backed up with supporting facts. In some cases, what appeared to be such facts were not necessarily relevant. The report cites an announcement by Ontario to invest $7.7 million in language and skills training to improve the employment prospects of newcomers to the province.

“There were statistics provided (with no supporting sources) regarding the high rate of job loss among immigrants,” states the report, “but no evidence pointing towards language barriers as the cause.”

Another challenge was the use of jargon or otherwise confusing language that raised questions about what kind of evidence might have been employed in a particular policy. The report points to BC government announcements as being especially problematic in this regard.

“In particular, for the Ministry for Health, we chose to discard joint statements which referred to a new policy change that had been announced in a separate policy announcement on the same day,” the report explains. “In contrast, policies issued by the Government of Saskatchewan were very easy to read, but rarely mentioned the underlying evidence, and lacked features common to policy announcements from Ontario and British Columbia (such as ‘Quick Facts’, a ‘Backgrounder’ or a list of associated links or resources).”

Fowler pointed out that as part of the task of assembling the report, the Evidence for Democracy team approached members of the public service to determine what kind of barriers could be getting in the way of a more transparent accounting of policy statements. These interviews revealed a pattern, in which less and less detailed information was offered at each level of decision-making, to make life more efficient for those in positions of greater authority.

“There’s a certain level of trust by the higher-ups that those lower down have done the job of making sure that this is supported by science or evidence,” she said. “You’re losing evidence at every turn, so by the time you get up to the top, the focus is on the impact, the outcome, not necessarily on the evidence that supported it. It doesn’t necessarily make it to the public, because by the time they get there, that evidence was several meetings or briefing notes ago.”

Evidence for Democracy will be hosting a webinar, at 12:30pmEDT on Monday 31 October, to offer an interactive discussion of this latest report. Register for this event here.

R$