Mid-term Evaluation of CIHR’s Centres for Population Health and Health Services Research Development in HIV/AIDS Program, June 2009 - May 2012

Phase 3 Evaluation Report (Final)
Executive summary

Context and purpose

CIHR's HIV/AIDS Research Initiative's 2008-2013 Strategic Plan, developed by the CIHR HIV/AIDS Research Advisory Committee (CHARAC) recognized a systemic lack of Canadian capacity in HIV/AIDS health services and policy research and population health research. This led to the launch of the Centres for Population Health and Health Services Research Development in HIV/AIDS Program. Following the competition held in 2009, two centres were funded for five years (2009-2014), both in the general stream of the competition. Although the funding program will not end until 2014, the HIV/AIDS Research Initiative and CHARAC will decide by January 2013 whether it should be continued, redesigned, or ended. The Institute of Infection and Immunity is thus conducting an external mid-term evaluation of the Centres Program, covering the first three years of operation. The evaluation will provide evidence to be used in that decision.

Evaluation methods

The evaluation was conducted in phases, with several lines of evidence from external data collection and synthesis combined with a self-evaluation conducted by the centres and an external peer review. This document presents results from all phases of the evaluation, with data from six lines of evidence:

  • surveys of program stakeholders and participants, in two groups: "researchers, research-involved collaborators and research users" (n = 109 individuals, 119 surveys (some people being involved in both centres)) and "students, trainees and other types of learners" (n = 27 individuals, 30 surveys); overall response rate 54%;
  • key informant interviews with program stakeholders (24 people in 21 interviews; response rate 86%);
  • case studies of four illustrative Centres Program initiatives, involving document review and 18 interviews (some with key informants in the preceding group);
  • secondary analyses of CIHR-held administrative data on applications and applicants to research grant competitions in population health and health services and policy research in HIV/AIDS between 2004 and 2012;
  • data extracted from the funded centres' progress reports, prepared for their mid-term Peer Review;
  • Peer Review Panel's reports about each of the funded Centres.

Key findings and conclusions

Overall, the evaluation evidence suggests the program is on track to meeting its objectives and is proving a valuable contribution to developing population health and health services and policy research capacity in HIV/AIDS in Canada.

  • The Centres Program is being successfully implemented. The Program has created and put in place processes and mechanisms to support knowledge development in health services and policy and population health research in HIV/AIDS, garnering collaboration from a wide range of stakeholders. The first sets of results from these processes, in the form of research protocols submitted for and having obtained funding from the Centres and CIHR, are now visible. Some significant successes have been scored and some significant disappointments have provided opportunities for improvement and learning. There is some evidence that centre members have become better positioned in terms of receiving CIHR research grant funding.
  • Capacity development is occurring through training and mentoring as well as strategies that are helping affected communities engage with research.
  • The program is operating through a fully integrated KTE model; it has also developed and implemented a number of innovative KTE mechanisms and tools.
  • Overall, the program appears to contribute to improved cohesion and coordination of the HIV/AIDS research agenda in these areas, while ensuring that capacity is present to tackle that agenda.

Although the Centres Program's funded entities are facing the normal challenges of focusing and executing ambitious research agendas, maintaining engagement of participants, and developing plans for sustainability, these are all in the process of being addressed. The evaluation data nonetheless underscore some areas of particular challenge, already known to the centres and CIHR, for the funded centres to face going forward into the final period of funding:

  • There is likely a need to strengthen the Centres Programs' capacity to address CIHR's HIV Research Initiative's research priority of health services and health policy;
  • A second key challenge is to ensure engagement of the research and practice communities across Canada, with a particular concern about the francophone and Québec communities.
  • While the Aboriginal focus of the Centres Program has been addressed by significant efforts from the funded centres and the Aboriginal HIV/AIDS and health research communities, more can be done, and resources remain stretched for those communities.
  • There may be a risk, now that both funded centres have passed their mid-term peer review and have so clearly demonstrated their early success, that they will focus on consolidating their strengths and in particular bringing existing research, training and collaboration initiatives to fruition so as to best position themselves for a future competitive application process.

Considerations for the program post-2014

In terms of the post-2014 direction of the Program, the following considerations emerge:

  • The Centres Program, fully consistent with CIHR functioning in general, rewarded excellence. The researchers and stakeholders involved in the two funded centers were already making significant contributions to this research area, and they have continued to do so. The program design appears to be very effective in supporting these developments.
  • The marginal gain of the Centers Program funding to health services and policy and population health research is hard to ascertain. Ideally, a renewed program would prioritize moving farther beyond what would have happened anyway.
  • A renewed program could apply learnings from the Center Program to date by encouraging the following features:
    • A greater emphasis on theory-driven intervention development and testing, in order to improve HIV/AIDS programs, services and practice, as well as to contribute to population health and health services and policy theory, through the vehicle of HIV/AIDS.
    • More innovative and perhaps experimental ways of engaging with policy, program and practice decision makers;
    • Structures and governance models that can be fully inclusive of the potential across the country;
    • Continued thoughtful development and experimentation of innovative integrated KTE strategies, including requirement for their formal evaluation;
    • Training programs that can enable training more learners, from a broader range of background disciplines and regions;
    • Adequate support to allow maximal engagement of the research capacity in the Aboriginal HIV/AIDS sector; and
    • Targeted capacity development to move more promising young researchers into positions as independent scholars, notably but not limited to, scholars in Aboriginal HIV/AIDS research.
  • In parallel, CIHR can fuel the potential for the program to achieve its objectives in two important ways:
    • Strengthening Aboriginal HIV/AIDS research capacity in general, by ensuring dedicated funding so that Aboriginal HIV/AIDS organizations have the time and resources to enable their fullest possible participation in the Centers Program and other research opportunities;
    • Clarifying with the Public Health Agency of Canada how the HIV/AIDS Research Program and PHAC's activities can best fit together, complementing and supporting each other.

Submitted to:

Canadian Institutes of Health Research
160 Elgin St, Room 97
Ottawa, Ont. K1A 0W9

Submitted by:

Natalie Kishchuk Evaluation and Research Inc.
4360 St Ambroise
Montréal QC
H4C 2C7

For more information, or to request a PDF file of the complete report, please contact: HIVAIDS-VIHSIDA@cihr-irsc.gc.ca.

Date modified: