Evaluation of the CIHR-Institute of Population and Public Health's Centres for Research Development Program (Final Report) - December 2011

Executive summary

Context

CIHR's Institute of Population and Public Health (IPPH) announced the Centres for Research Development (CRD) program in September 2002. With the aim of improving Canadian capacity in population and public health research, the thematic focus of the Centres' program was to understand and address the impacts of physical and social environments on health. After peer review, seven of 21 applications were approved for funding of up to six years (2004-2010) (non-renewable), receiving a maximum of $400,000 per year. The Centres were funded as an innovative research development and capacity building by IPPH.

In preparation for the 10-year International Review of CIHR, this summative evaluation aimed to provide an overall assessment of the gains from the Centres' program for population and public health (PPH) research capacity in Canada, and of the Centres' program model as a tool for research capacity development.

Evaluation strategy

This evaluation was guided by a framework developed in 2004 based on a consensus reached by the funded Centres. It was intended to provide a program-level focus to support IPPH's assessment of the overall initiative. While the Centres varied greatly in their contexts and approaches, this evaluation design did not treat them as in-depth individual case studies, but rather sought to identify program-level learnings across the seven funded Centres, using Centre-specific accomplishments as examples of these learnings. Thus, the report is not intended to provide a detailed documentation of all Centre results, but rather a summative evaluation of the overall Centres' program, including an assessment of the Centres' program model. Centre directors can be contacted for additional information about any of the Centres.

The evaluation used two main data sources:

• Document review: of background documents on the Centres' program and its context in PPH research over the life of the program; progress and mid-term self-evaluation reports and final output reports submitted by all Centres; a mid-term evaluation cross-learnings report; and a peer review committee report from a mid-term review of individual Centres.
• Key informant interviews: with 20 key stakeholders familiar with the Centres' program and/or individual Centres. These interviews aimed to assess the results of the overall program and funding approach. Interviewees represented a cross-section of PPH research stakeholders who were sufficiently familiar with the program so as to be able to assess its overall effectiveness.

Findings

Research capacity development

Overall, the Centres have made clear and significant contributions to PPH research capacity in theme areas of the Centres' program. This is measurable in terms of the attraction of new scholars and disciplines to the PPH field and the establishment of research platforms both within and connected to the Centres that had not existed before. Interdisciplinarity has been strengthened, although not in all directions: there has been reinforcement from the social sciences and to some extent the physical sciences, but capacity development in areas such as economics, political science and health services/administration has not been as strong. Centres have been successful in accessing funds from a very broad set of sources, helping to move capacity closer to increased success in CIHR's open competition. In total, about $124M in peer-reviewed research funds from top-tier agencies (excluding chairs and STIHRs) were obtained by Centres during the funding period. This is about 7.4 times the $16.8M program investment. There were significant synergies with the Strategic Training Initiatives for Health Research (STIHRs) and Applied Public Health Chairs that contributed to mutual strengthening, as well as with the development and reinforcement of other important initiatives in population health intervention research and training. Infrastructure and equipment grants awarded to Centres had significant structuring effects on the PPH research platforms that the CRD program intended to develop. These platforms have supported scholars and students, in many cases positioning them for successful careers in PPH research. The Centres, through a variety of mechanisms at the researcher, Centre and/or institutional level, have established sustainable or at least longer-term infrastructure for maintaining both research and training capacity including, but not limited to ongoing funding of the Centre itself. Overall, PPH research capacity has been significantly advanced since and because of the CRD program.

Collaboration

Networking and collaboration have also grown through the Centres' activities. Over the funding period, the nature of networks and partnerships fostered by the Centres have expanded both within and beyond their universities, to newly include research entities and communities regionally, across Canada and to some extent internationally. The disciplinary diversity of Centre-engaged individuals has also increased significantly. Many types of collaborative relationships have flourished as well with non-academic partners including various levels of government, non-governmental and community-based organizations and associations in health and other sectors. These were enabled by structures developed by the Centres to support collaboration, including co-governance, co-funding, knowledge translation (KT) events and activities, and applied research projects funded directly by research users.

Knowledge translation and exchange

Centres' engagement with research users was impressively varied, and users surveyed see the Centres' work as useful and relevant. There has been a great deal of activity in KT and experimentation of techniques and technologies that would be expected to have impacts, and Centres are able to provide compelling examples of policy and practice impacts. Longer-term evaluation would be required to capture the full impact of these Centres on population health.

Centres' program model

The relatively large and long-term funding commitment of these Centre grants represented a strategic choice in the early years of IPPH. The evaluation findings show that the Centre program's key features were operationalized by the funded Centres in ways that advanced IPPH's overall aim of research capacity development in PPH, as well as achieving the specific funding program objectives. Particularly important were the commitment of multi-year funding on a flexible, thematic basis, and the close involvement with stakeholders and research users. The Centres' program is highly regarded and has been used as a model for at least two other CIHR programs. It was described as visionary, gutsy and innovative. Moreover, there was strong consensus among all stakeholders that the choice to invest in Centres had been somewhat risky, but that it has paid off handsomely in developing PPH research capacity, and has been well worth any opportunity costs. Overall, the findings from this evaluation suggest that the Centres' program and model were highly successful as a mechanism for funding population and public health research capacity
development.

It is recommended that the success of the Centres' program be communicated clearly and openly to all stakeholders within and outside of CIHR, including IPPH's Institute Advisory Board, host institutions for Centres, Centre partners, and the Centres themselves.

Prepared by: Natalie Kishchuk, PhD
Évaluation de programmes et recherche sociale appliquée