Dr Gilles G. Patry

Quest for research outcomes requires new approaches

By Dr Gilles G. Patry

Since the late 1990s, Canada has seen tremendous growth in government investments in university research. Not only did the Government of Canada initiate programs such as the Canada Research Chairs, the Canada Excellence Research Chairs, the Vanier Canada Graduate Scholarships and Banting Postdoctoral Fellowships, but it also continued investing in its research-funding agencies and created entities like Genome Canada and the Canada Foundation for Innovation (CFI). Combined, these investments have nurtured a vibrant research ecosystem that is producing the knowledge, the talented people and the facilities that are making Canada more innovative and globally competitive.

But with increased government investment come increased accountability and a growing need for accurate ways to measure tangible results that are meaningful to Canadians.

Since the inception of the CFI, its board of directors and management team have always recognized the importance of evaluation and public reporting, not only for accountability but also to generate in-depth insights into the social and economic impacts of publicly funded research. As a result, the CFI has spent the past four years developing and applying a new tool in R & D evaluation in Canada — the outcome measurement study (OMS) — that was published recently in the UK-based journal, Research Evaluation¹.

Historically, evaluating public investments in R & D has tended to focus on the quantity and quality of short-term research outputs, such as number of publications and quantifiable development and commercialization outputs, including licensing contracts and patents. Although these remain important indicators, they have not provided a complete picture of the true impacts of public R&D investment.

Many organizations, including the Organisation for Economic Co-operation and Development and the National Science Foundation, have advocated for more holistic approaches to evaluation that focus more on medium- and long-term impacts, highlight the relationship between strategic goals and outcomes, and involve the scientific community as a partner in the assessment. This is exactly what the OMS aims to do.

The OMS looks at both quantitative and qualitative information. As an analytical tool, it employs five categories of outcomes to assess a multidisciplinary research theme at a given institution: strategic research planning; physical and human research capacity; training of highly qualified personnel; research productivity; and innovation and intrinsic benefits. This offers a richness of analysis that would not be possible with evaluations focused on a single outcome. Looking across an institution and a specific theme, rather than examining an individual project or program, adds another layer of information.

The research theme — a grouping of projects linked through a field of study, such as human genetics, aquatic sciences and nanotechnology — allows the CFI to assess the impacts of, and interactions between, related infrastructure projects and investments over time. The theme is chosen in an area in which there has been significant CFI investment that is linked to the institution's strategic research plan — a key eligibility requirement of CFI funding.

The OMS process involves an in-depth questionnaire submitted to institutions, with a follow-up validation by a visiting expert panel whose report is the key output of the exercise.

To date, this process has been repeated at 26 institutions from coast to coast, providing the CFI with valuable information on the outcomes of its investments.

The OMS also point to a "facility effect", where integrated suites of research infrastructure have led to a significant increase in multidisciplinary and cross-sector research cooperation. In addition, many institutions demonstrate strong outcomes through an "organization effect", where they have deliberately focused their strategic research plans and translated them into the design of their facilities and related research, training and knowledge-transfer activities.

And there's more. The OMS has demonstrated that CFI investments have played a role in allowing institutions to, among other things:

* think more strategically about their research and the efficient use of research infrastructure;

* increase their research capacity and improve the technical and operational capabilities of research infrastructure;

* increase the number of highly qualified personnel trained each year at the masters, doctoral and post-doctoral levels; and,

* conduct research that has social and economic benefits, including improvements in health care, regulatory measures and public policies, new standards, new and improved products and process and environmental benefits.

But the OMS has proved useful for institutions in other ways. Although the process involves many resources, time and effort for institutions, the OMS expert panel reports have offered advice that has helped institutions further evolve projects, plan for future CFI applications and develop new strategic research plans. The reports have also identified new opportunities for advancing research and research training and have discovered gaps in the institutions' outcome data. Institutions have used the expertise of the panels to create new linkages and opportunities, nationally and internationally, that they had not seen before.

The OMS process has also been of benefit to other partners in the funding equation. Since the CFI funds up to 40 percent of a project's costs, the other funding partners are included as observers or presenters at each OMS visit. Piggybacking on the process, these partners become privy to the benefits and impacts of their investments in ways they may not have anticipated.

In the end, the OMS not only gives the CFI and its funded institutions important insights into the return on investment but also unearths ideas on how to better capture research impacts. With feedback from the institutions and partners, the process has evolved and will continue to evolve.

There is no single gold standard for assessing research outcomes and impacts, but the OMS goes a long way to helping reveal the different modes of innovation and socio-economic impacts resulting from CFI-funded infrastructure.

Dr Gilles G. Patry is president and CEO of the Canada Foundation for Innovation.

¹ G. Tremblay, S. Zohar, J. Bravo, P. Potsepp, M. Barker; Research Evaluation, 19(5), December 2010.