Evaluation of the Pandemic Preparedness Strategic Research Initiative

Final Report 2014

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada
Canadian Institutes of Health Research

Acknowledgements:

This report was authored by Carmen Constantinescu. The evaluation was carried out by an Evaluation Working Group which, during the different phases of the evaluation, included: Jennifer Raven, Michael Goodyer, Martin Rubenstein, David Peckham, Kwadwo Bosompra, and Danielle Halloran.

Special thanks to the all the participants in this evaluation – survey respondents, interview and case study participants, CIHR Management and Institute of Infection and Immunity (III) Management. Additional thanks to R.A. Malatest & Associates Ltd. and Science–Metrix for assisting with data collection and analysis as part of the evaluation.

Table of Contents

Foreword

In 2006, the Government of Canada (GoC) allocated $422 million in funding to the health portfolio to support preparedness for avian and pandemic influenza. From this, the Canadian Institutes of Health Research (CIHR) received $21.5 million for pandemic influenza research, which it used to create the Pandemic Preparedness Strategic Research Initiative (PPSRI), under the leadership of the Institute of Infection and Immunity (III). Through the efforts of a Task Group formed of experts in the field of influenza and public health, four strategic research priorities for the initiative were identified: vaccines and immunization research; the virus; prevention and treatment; and ethics, legal, and social contract. Throughout these key priority areas were the themes of building research capacity, preparing for outbreak research, fostering collaborations and networks, and enhancing communication and knowledge translation.

CIHR III showed tremendous leadership in developing the PPSRI. By forming linkages with provincial, federal, and international stakeholders, III increased the total amount of available funds for research from $21.5 million to $43.3 million. As a result, 92 projects involving 345 researchers were funded in total. This represents a significant investment in, and commitment to, influenza and pandemic preparedness research in Canada.

This bibliometric evaluation indicates that the PPSRI was successful in its goal of creating knowledge. Furthermore, the outcomes produced by PPSRI-supported projects have been substantial, with 44% of survey respondents reporting new research methods, 30% reporting the development of new practices (including clinical practice, tools, and procedures), 18% of projects resulting in new or changed policies, 16% generating new vaccines or drugs, and 14% generating patents or licences. This represents a significant advancement in the field of influenza and pandemic preparedness, particularly considering that the long-term outcomes are not fully recognized.

PPSRI-investigators engaged a large number, and a wide range, of knowledge users and other stakeholders in their research projects. More than half (55%) involved healthcare practitioners, 30% involved federal or provincial government representatives, 20% engaged community or municipal organizations, and 15% included patients and healthcare consumers in their studies. This engagement of various groups demonstrates a commitment to knowledge translation and the timely and effective update of their research findings.

Although the initiative as a whole ended in 2011, many of the projects have made a lasting impact and are continuing. One key example is the PHAC-CIHR Influenza Research Network (PCIRN). This network was initially funded in 2009 to carry out research on, and inform public policy related to, seasonal influenza vaccines. Once the H1N1/09 pandemic was declared, PCIRN was swiftly tasked with evaluating the pandemic vaccines in numerous populations. In addition to renewed funding from CIHR and PHAC in 2012, PCIRN has formed several partnerships with industry groups for the testing and evaluation of influenza vaccines, clearly demonstrating the value of this network to the scientific community.

Preparing for infectious disease outbreaks remains a priority for both CIHR and III. In the 2009 Health Research Roadmap 5-year plan and in its current refresh, CIHR listed “preparing for and responding to existing and emerging threats to health” as a priority investment area. In its new 2013-18 strategic plan, III also highlighted a commitment to supporting research in this area. These pledges will ensure that Canada continues to build upon the successes achieved through the PPSRI, and assure advancement in this area.

Finally, I would like to thank Dr. Bhagirath Singh and Carol Richardson, who initiated work on the PPSRI, for their leadership. Since the transition of III to Quebec City, PPSRI has been under the competent leadership of Dr. Jennifer Raven. The success of the initiative is dependent on the positive collaborations with many stakeholders and I would like to highlight the positive interactions with Dr. John Spika; Director General of the Centre for Immunization and Respiratory Infectious Diseases at PHAC. This report is the result of excellent collaborative work between Jennifer Raven at III and Carmen Constantinescu and Martin Rubinstein from the CIHR Resources Planning and Management portfolio. Their efforts and dedication are much appreciated and this report is an important step to demonstrate the impact and capture the outcomes of the contributions of Canadian researchers in the field of pandemic preparedness.

Marc Ouellette PhD
Scientific Director,
Institute of Infection and Immunity, CIHR

Executive Summary

Pandemic Preparedness Strategic Research Initiative (PPSRI)

Influenza is an infectious disease that has caused millions of deaths in the past, and which continues to cause the death of several thousand people worldwide each year. In response to the pandemic threat posed by the emergence of a highly pathogenic strain of avian influenza (H5N1) capable of transmission from birds to humans, in May 2006 the Government of Canada committed a total of $1 billion over 5 years to improve Canada’s ability to respond to pandemics, involving the following organizations: Public Health Agency of Canada (PHAC), the Canadian Food Inspection Agency (CFIA), Health Canada (HC) and the Canadian Institutes of Health Research (CIHR).

As the Canadian funding agency for health research, CIHR received $21.5 million over 5 years (2006-2011) to support pandemic influenza research.1 The Institute of Infection and Immunity (CIHR-III) created the Pandemic Preparedness Strategic Research Initiative (PPSRI) to develop a coordinated and focused pandemic preparedness research program, and to build national research capacity in this area. By forming partnerships with other federal and provincial agencies, CIHR-III increased the total funds available to $43.3 million. Ninety-two research projects in total were funded to address the strategic priority areas of the initiative.2

Evaluation purpose and scope

The purpose of this evaluation was to assess the relevance and performance of the PPRSI to inform decision-making regarding future similar initiatives by CIHR and to address the Treasury Board Secretariat (TBS) requirements for the specific core evaluation issues as described outlined in the TBS evaluation policy suite.3

The evaluation covers the entire period of the PPSRI (2006-2011) and builds on, and adds to, the findings of a mid-term evaluation conducted in 2009. In particular, the evaluation explored the extent to which PPSRI has met its objectives and highlights lessons learned in relation to program design and implementation. PPSRI’s success was assessed in terms of: knowledge creation, capacity building, knowledge translation, collaboration and partnership as well as leveraging additional funds.4

Key Findings

Knowledge creation

PPSRI funded projects were successful in generating new knowledge in the field of pandemic and influenza research. As compared to the ten most productive countries in the field of pandemic research for the period from 2001-2010, Canada has increased its specialization, as well as its scientific impact, in this area, since 2006 when the PPSRI began. Also, the scientific impact per paper based on Average of Relative Citations (ARC) for CIHR researchers funded through the PPSRI is significantly greater (1.30) than that of the rest of Canadian health researchers publishing in this area (0.68).

Across the world, the number of pandemic research papers has increased over the years for the period 2001-2010, and PPSRI funding contributed to Canada maintaining its middle rank within the ten most productive countries in the area of pandemics research. Over the period from 2008 to 2010, the percentage of pandemic research papers supported through PPSRI funding relative to the total of Canadian pandemic research papers increased from 27% to 62% with a steep increase in the number of pandemic papers produced in Canada from 37 to 103.

PPSRI-funded researchers produced a variety of publication types and attracted significant attention from the media. On average each PPSRI funded project produced 3.6 refereed journal articles; 6.9 conferences, symposium presentations and posters; 3.0 newspaper articles; 3.4 radio reports and interviews; and 2.3 internet articles. As some of the funded PPSRI projects are ongoing or have just finished, it is expected that more publications will be produced in the coming years.

Capacity development

PPSRI contributed to building capacity in Canada in the area of pandemic research by increasing the number of research staff involved in this field, this way ensuring that a pool of researchers will be able to respond to future pandemic threats. Each funded project involved an average of 11.2 research staff and it is estimated that a total of 1043 research staff were involved through all the funded projects5.

This evaluation demonstrated increases across all measures of training and mentorship since the mid-term evaluation. Nearly all (90%) NPIs reported that their projects included mentoring of students in the field of pandemics research. The mean number of students/trainees per project who completed their qualifications during the tenure of their mentor’s grant was 2.5. The projected total of students completing their qualifications was 224, compared to 173 in the mid-term evaluation.

A substantial proportion of both NPIs (83%) and COPI/OPIs (58%) considered their PPSRI project to have made a positive contribution to their career. The initiative focused the work of 68% and reoriented the careers of 58% of NPIs on influenza and pandemic preparedness research.

Knowledge translation

PPSRI funded projects produced a variety of research and commercialization outcomes in each research area of interest. Over a third of projects researching vaccines and immunization produced results that could lead to a new vaccine or drug. A couple of examples are: early detection of influenza variants and their correlation with variation in vaccine effectiveness, studies which identified that current vaccines are not as effective in the elderly, leading to the development of better vaccines for this age cohort. Three-quarters of the projects researching virus biology and diagnostics resulted in new research methods, patents and licenses, or intellectual property claims.

At the time of this evaluation, an increasing number of researchers funded under the PPSRI (88%), reported that they had worked on projects which included stakeholders, compared to the mid-term evaluation (66%). The most frequently reported stakeholders involved were: health system/care practitioners, study stakeholders (who were formally listed in the grant application), federal/provincial representatives and health system/care professional organizations. Generally, stakeholders were most commonly involved in data collection and the implementation of projects.

Examples of PPSRI-funded research impacts include: significant contributions to the world’s pandemic response systems by better informing disaster and humanitarian response programs of organizations such as: the Bill and Melinda Gates Foundation (BMGF), Public Health Agency of Canada (PHAC), World Health Organization (WHO) and the Ontario Health Plan for an Influenza Pandemic; research findings being adopted by the Colleges of Physicians and Surgeons in Ontario and incorporated into Nova Scotia’s policies for health care workers; PPSRI research findings stemming from mathematical modeling in infectious disease and influenza being used nationally and internationally to predict the rate of spread of H1N1.

Collaborations, partnerships and leveraging of additional funds

CIHR-III was effective in generating early collaborations with other stakeholder organizations and establishing funded partnerships for this initiative, attracting 44% of PPSRI’s financial resources from external partners. The Institute was effective in communicating with researchers by providing meetings and workshops to promote the initiative and help them with the application process.

Almost all NPIs (85%) and COPI/OPIs (82%) reported that their PPSRI grant had helped them or their teams to collaborate with other researchers involved in influenza and pandemic preparedness research. The proportion of papers published by PPSRI-funded researchers in collaboration with other researchers is higher than Canadian and world averages for pandemic research at both national and international levels.

At the project level, NPIs (45%) and COPI/OPIs (24%) were successful in leveraging additional funds as a result of their PPSRI grants. Survey respondents reported leveraging almost $25 million in total; approximately $8 million of that from additional CIHR grants. Apart from CIHR, the most frequent source of additional funding was provincial government organizations ($3 million) but the largest amount of external funding came from industry ($6.5 million) and was due primarily to a single partnership focused on vaccine evaluation.

Program design

PPSRI had several strengths including: its innovative format to rapidly respond to the need of improving pandemic preparedness and response in Canada; and, the availability of flexible funding (e.g., catalyst grants, team grants, operating grants, development and planning grants) which allowed researchers to shift focus if required. A total of 31 competitions were launched (including some priority announcements (PAs) through CIHR Open Operating Grants Program OOGP), which resulted in 92 funded grants that addressed the identified research priorities. Researchers were positive about the PPSRI grant application and decision process and many commented that the competition process was effective and administered well by CIHR.

A few suggestions for improvement were made. Partners commented on the need to improve the dissemination of the information offered to decision makers by providing it in a format that is more appropriate for them, such as synthesized summaries of key findings by theme.

Most NPIs comments related to funding - the need for additional funding, funding renewal, or funding for longer time periods, given that the PPSRI grants durations varied on average between one to maximum of three years. These findings support the earlier suggestions made in the mid-term evaluation where key informants also noted the length of the grants as a challenge and suggested that this type of initiative to include a two-year planning and implementation phase followed by five years of funding.

Another suggestion coming from researchers was to promote the creation of a consortium of vaccine manufacturers that would be capable of rapidly collaborating with researchers to create new vaccines or other therapeutics during a pandemic.

Program relevance

PPSRI’s mission was clearly aligned both with CIHR’s mandate and with the Government of Canada’s priority to fund pandemic preparedness research and all interviewed key informants considered it to be the role of the federal government to support influenza and pandemic research.

PPSRI provided additional funding on top of CIHR’s existing programs, thus supporting projects that would not otherwise have been funded in the needed area of pandemic research.

Researcher capacity, networks and collaborations have increased as a result of the PPSRI. While it is not clear how many of these gains will be sustained when funding ends, there is nevertheless a larger pool of trained people who can be called on in the event of another pandemic. NPIs emphasized the need to continue funding in pandemic preparedness research as the threat of pandemics remains given the biology of viruses.

Conclusions

The PPSRI initiative was successful in generating new knowledge, increasing research capacity and in building networks of researchers with expertise in pandemic and influenza research. Research was supported that most likely would not have otherwise been supported through CIHR’s usual funding mechanisms due to a lack of appropriate funding tools. The occurrence of the H1N1 pandemic midway through the initiative, and the fact that the PPSRI aided in Canada’s rapid response to the pandemic, demonstrated the ability of such an initiative to successfully support a national response to a public health emergency.

Qualified, multidisciplinary research teams were supported through a variety of grant types (e.g., catalyst grants, team grants, operating grants, development and planning grants) providing more flexibility to researchers. Research teams and networks were able to rapidly receive funding and respond to the pandemic threat. The process supported a breadth of projects, methodologically ranging from curiosity-driven laboratory research to large networks and team projects.

PPSRI was effective in facilitating the development of collaborations and partnerships between researchers, health service providers, and policymakers. These collaborations enabled findings from PPSRI funded research projects to have a direct impact on policy development and service delivery during the H1N1 pandemic. CIHR-III established funding partnerships with other organizations including other government departments, industry and NGOs; however, improvement could be made by better involving partners in the dissemination of information decision makers.

The most frequently identified challenge was related to the potential lack of sustainability of the infrastructure and networks established during the PPSRI and the short duration of the grants. Nevertheless, PPSRI contributed to building capacity in Canada in the area of pandemic research by increasing the number of research staff in this field and by making positive contributions to their careers, this way ensuring that a pool of researchers is positioned to rapidly respond to a future pandemic threat. With a view towards sustainability, it would be useful to clearly articulate to all stakeholders, at the launch of the strategic initiative, the funding options that would be available upon the sunset of the initiative.

Lessons learned

Findings from the evaluation identified a number of lessons learned from the PPSRI that have the potential to inform future CIHR funding initiatives. Overall, the evaluation shows that a targeted funding mechanism such as the PPSRI is an effective way to create knowledge and increase research capacity in a specific area. In particular, the following key lessons learned were identified:

Recommendations

The five-year term of the initiative has ended, yet, the following recommendations are being made for the benefit of similar programs that may be introduced in the future and also to address the continued need for pandemic preparedness research identified by the federal government, researchers, partners and other stakeholders.

  1. PPSRI has helped position Canada within the top five countries globally in the field of pandemic research and would be useful to maintain or improve upon this status. Given the ongoing threat of a future pandemic and the importance of having the capacity to respond to this threat, regular environmental scanning and assessments with international benchmarks should be conducted to determine the relative global position of Canada in the field of pandemic research in the absence of the PPSRI. This would include examining the Canadian investment in the field of pandemic research, tracking the subsequent careers of PPSRI principal investigators and trainees, and monitoring the application and use of the research on partners and research users to better understand the initiative’s influence on the field of pandemic research.
  2. CIHR-III was effective at establishing funded partnerships for PPSRI, attracting almost half of the initiative’s financial resources from external partners; however, partners suggested that improvements could be made for the dissemination of information to decision makers by providing it in a format that is more appropriate for them, such as synthesized summaries of key findings by theme. In order to ensure the effective dissemination of information, future initiatives of this type should:
    1. engage the partners/knowledge users in designing of the new initiatives;
    2. include an integrated knowledge translation (iKT) plan to engage and involve partners and knowledge users in order to understand and address their needs related to the research to be undertaken and for dissemination and application of research results.

Management Response

Recommendation Response
(Agree or Disagree)
Management Action Plan Responsibility Timeline
1. Given the ongoing threat of a future pandemic and the importance of having the capacity to respond to this threat, regular environmental scanning and assessments with international benchmarks should be conducted to determine the relative global position of Canada in the field of pandemic research in the absence of the PPSRI. This would include examining the Canadian investment in the field of pandemic research, tracking the subsequent careers of PPSRI principal investigators and trainees, and monitoring the application and use of the research on partners and research users to better understand the initiative’s influence on the field of pandemic research. Agree Bibliometric analysis provides a reliable measure of a nation’s global position in terms of publications and productivity in a given subject area. The conduct of such analyses at intervals of 3 years will allow CIHR to track Canada’s productivity, relative to other countries, in pandemic research, and to determine the long-term effects of changes in funding availability on research output. CIHR Institute of Infection and Immunity 2016
2. In order to ensure the effective dissemination of information, future initiatives of this type should:
  1. engage the partners/knowledge users in designing of the new initiatives;
  2. include an integrated knowledge translation (iKT) plan to engage and involve partners and knowledge users in order to understand and address their needs related to the research to be undertaken and for dissemination and application of research results.
Agree During the development of future large-scale initiatives, CIHR will develop an integrated KT plan to effectively engage, involve, and influence initiative partners, knowledge users, and the general public. Initiative plans will be developed in consultation with the CIHR Research and Knowledge Translation Portfolio , and with the full participation and involvement of initiative partners and knowledge users to ensure that the plans will fully meet their needs. Initiative Scientific Lead & Associate VP, Research and Knowledge Translation Concurrent with MOU/CA development, prior to the launch of the first RFA of a new initiative.

1. Knowledge Creation

One of the main objectives of PPSRI was to support the development of pandemic research in Canada. This initiative was intended to support research in areas including: developing diagnostics, antiviral drugs and vaccines; understanding and preventing disease transmission; and identifying ethical ways to use limited resources during a pandemic. It was expected that the outcomes of the research would lead to increased knowledge, and ultimately contributing to the body of knowledge used to develop the national pandemic preparedness plan.6

In addition to traditional methods of data collection such as online surveys and interviews with funded researchers, a bibliometric analysis was employed to assess the knowledge created in relation to the funding offered through PPSRI. Bibliometrics seeks to measure, among other things, the volume of a researcher’s publications and the relative frequency with which they are cited as a proxy for an article’s impact. The assumption underlying the use of bibliometric indices as measures of knowledge creation is that academic papers published in widely circulated journals facilitate access to the latest scientific discoveries and advances, and are seen as one of the most tangible outcomes of academic research (Goudin, 2005; Larivière et al., 2006; Moed, 2005).

For the purpose of this study, bibliometric analysis measured the publications’ scientific impact as well as the volume of publications produced, by searching Thomson Reuters’ Web of Science (WoS) for all papers having the words “influenza” and “pandemic” in their titles as well as all the papers published in the journal Influenza and other Respiratory Viruses, for all researchers funded through PPSRI (N=345 including PIs=61, and COPIs/OPIs =284).7 The results were compared with similar data for the top ten most productive countries in the area of pandemic research.

Scientific Impact

An important indicator of scientific performance is the impact that the published papers have on the scientific community. The Average of Relative Citations (ARC) is an indicator of the number of times that a paper is cited relative to the world average. Specialization Index (SI) is an indicator of the relative research intensity of publication of a country in an area of interest relative to the intensity of the world in that specific area. Looking at these two indicators gives us a graphical representation of the relation between knowledge creation and scientific impact. The scatterplot diagrams presented in Figure 1-1 compares the ARC and SI of the top ten productive countries in pandemic research for the period before PPSRI was implemented (2001-2005) with the period after its implementation (2006-2010). The comparison of these two figures shows that Canada has increased its specialization as well as its impact since the PPSRI started. From 2001-2005 to 2006-2010, Canada’s SI in the area of pandemic increased from 1.18 to 1.31 and its ARC, from 0.81 to 1.16.

Figure 1-1: ARC and SI of the top ten productive countries in pandemic research, 2001-2010

Figure 1-1 long description

Analyzing the scientific performance within Canada, the scientific impact per paper, based on ARC for CIHR researchers funded through PPSRI, is greater8 than that of the rest of Canadian health researchers publishing in the pandemic area. PPSRI-funded researchers published papers with an ARC of 1.30, as compared to 0.68 for Canadian researchers who did not received PPSRI funding, and who published in the field of pandemic preparedness.

Volume of publications

Volume of publications, although not an indicator of quality of the papers, reflects research capacity by showing the volume of papers being produced by researchers in a given research area. It is used generally as a basic indicator for the outputs that result from investing in a given program.

For the period from 2001 to 2010, Canada was situated in the “middle” of the ten most productive countries in the field of pandemic research with an average of 5.9% of the world’s pandemic papers; the United States led with the highest average percentage of pandemic papers published (Table 1-1). Over this period, the number of pandemic papers increased for the top five most productive countries (including Canada), and PPSRI contributed to Canada keeping pace with the United Kingdom, Japan and China (Figure 1-2).

Table 1-1: Ranking of the top ten most productive countries, based on the average percentages of the world pandemic papers for the period 2001-2010

Group Average for 2001-2010
United States 43.5%
United Kingdom 10.0%
Japan 7.6%
China 7.5%
Canada 5.9%
Australia 5.6%
France 4.7%
Netherlands 4.3%
Germany 4.2%
Italy 3.8%
World 100.0%

Figure 1-2: Number of pandemic papers of top five most productive countries, for the period 2001-2010

Figure 1-2 long description

Over the ten year period from 2001 to 2010, the number of Canadian pandemic papers increased from a low of 13 in 2001 to a high of 103 in 2010, with a steep increase in the last three years when the PPSRI funding was available from 37 papers in 2008 to 103 papers in 2010 (Figure 1-3). The percentage with which researchers funded through the PPSRI contributed to the overall number of Canadian pandemic papers ranged from a low of 42% in 2005 to a high of 73% in 2008; however, there was a gradual increase in the proportion of papers authored by recipients of PPSRI funding with PPSRI-supported papers representing 27% of the Canadian publications in the area of pandemics in 2008, to 55% in 2009 and 62% in 2010 (Figure 1.3).

Figure1-3: Percentage of Canadian pandemic papers authored by CIHR-PPSRI-funded researchers, 2001-20109

Figure 1-3 long description

Types of research outputs produced

The online survey of PPSRI-funded researchers also reveals data about researchers’ publication outputs. Conferences and publications were the most frequently reported outputs for PPSRI-funded grants (96% and 78% of projects, respectively) (Table 1-2). Overall, at the time of the survey, a total of 180 refereed journal articles have been produced, with a mean of 3.6 per project, and a total projected number of 325 refereed journal articles for all funded projects.

When comparing the overall means, the statistically significant highest rates of refereed journal articles were reported for projects with higher-value grants (greater than $500,000) (median = 9, mean = 8.8)10.

Table 1-2: Number of Publications per Project

Type of output % of Projects that produced the output (n=51) MeanFootnote I Standard DeviationFootnote I Total # produced ProjectedFootnote II total to all projects (92)

Source: NPI online survey QA6 and C3 reported for 51 projects. See Appendix D-6 for breakdowns by project categories.

Footnotes

Footnote I

For some projects, some NPIs did not provide the precise number of outputs; these cases were excluded from the calculation.

I

Footnote II

Projections are extrapolated from the total number of outputs produced (Projection = (total / 51) x 92).

II

Conferences, symposium presentations & posters 96% 6.9 10.9 345 622
Refereed journal articles 78% 3.6 5.3 180 325
Master's thesis / Doctoral dissertation 43% 0.6 1.1 30 54
Newspaper articles 43% 3.0 15.4 127 229
Radio reports/interviews 41% 3.4 15.1 150 271
Multi-media content (incl. non-refereed articles) 35% 0.8 3.0 35 63
Reports/Technical reports 33% 1.1 3.0 56 101
Television reports/interviews 33% 1.8 8.1 77 139
Prizes/Professional awards 25% 0.1 0.3 6 11
Magazine articles 24% 0.3 1.6 12 22
Books/Book chapters 20% 0.3 0.8 12 22
Internet articles 18% 2.3 7.5 89 161

PPSRI used different types of grants (also called “funding mechanisms”) to support research in the pandemic area. These include: catalyst grants, operating grants, team grants, the Influenza Research Network (IRN) and other planning and development grants. Each of these grants had different characteristics in terms of the objectives, duration and the amounts awarded11. To account for the differences in dollars expended and grant duration between the different types of grants, the average number of journal articles published per grant was normalized by dividing the total number of articles per grant by expenditures, grant duration and the number of researchers involved at the time of application, for three types of grants: catalyst, operating and team grants; IRN and the other planning and development grants, although reported publications, were excluded from this analysis due to their specific characteristics (i.e. grant size and duration), which makes for an unsuitable comparison. (Table 1-3).

The results show that there is no statistically significant difference between these three types of grants for all three control variables.12 Although the researchers funded under the team grants produced more refereed journal articles, when we control for dollars expended, grant duration and number of researchers involved at the time of application, the differences disappear. The small number of team grants (n=5) should also be considered when interpreting these results. However, the same kind of results were reported in the evaluation of CIHR’s Open Operating Grants Program (OOGP); the publication productivity for most researchers was very similar across all CIHR pillars when grant duration was controlled for (Source: Evaluation of OOGP, 2012).

The availability of similar publication data for CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI)13 and OOGP allows for a comparison of average number of refereed journal articles with PPSRI. Overall, PPSRI funded grants generated an average of 1.6 journal articles per $100K, compared with 1.5 for RMNI and 2.6 for OOGP. When normalized by grant duration, PPSRI funded grants generated an average of 2.5 journal articles per year of grant, comparing with 2.8 for RMNI and 2.3 for OOGP. These comparisons should be treated with caution due to important differences between the programs in terms of: program duration (e.g., RMNI funding period was from 2004 to 2010 compared to the PPSRI funding from 2006 to 2009); diversity of funding mechanisms used (e.g., PPSRI had the greatest variety of grant types: catalyst, operating, team, planning and dissemination); and different areas of research supported (e.g., respectively pandemics, regenerative medicine and nanomedicine, and open research).

Table 1-3: Normalized Average Number of Journal Articles for PPSRI, RMNI, OOGP

PPSRI RMNI RMNI RMNI OOGP
Overall Catalysts GrantsFootnote VI Operating Grants Team Grants Overall Catalyst Grant Team Grant Grant
n=39Footnote III n=16 n=18 n=5 n=26 n=12 n=14 (N=510; N=531)

Footnotes

Footnote III

Does not includes Influenza Research Network (IRN), other development and planning grants & grants < $ 95,000.

III

Footnote IV

Based on N=510 grants with commitments >= $100,000.

IV

Footnote V

Based on N=531 grants with at least one year in duration.

V

Footnote VI

Seven catalyst grants had commitments between $95,000 - $100,000.

VI

Footnote VII

Based on N=10 grants with commitments >=$100,000.

VII

Dollar range of grant value Majority < $250K Majority < $250K All > $500,000 Majority < $250K $1-2 M Average grant $ 300K
Grant average duration 1 year 2 years 3 years 2.5 years 4.5 years 2.5 years
Average number of journal articles normalized by: Mean ± SD Mean ± SD Mean ± SD Mean ± SD Mean ± SD Mean ± SD Mean ± SD Mean ± SD
Journal articles by $100k (at time of survey) 1.6 ± 2.6 1.9 ± 3.8 1.5 ± 1.6 0.7 ± 0.7 1.5 ± 1.9 1.3 ± 0.8Footnote VII 1.4 ± 0.9 2.6 ± 2.8Footnote IV
Journal articles by grant duration (at time of survey) 2.5 ± 3.9 3.4 ± 5.7 1.5 ± 1.4 3.0 ± 3.2 2.8 ± 2.6 1.1 ± 0.6 4.4 ± 2.7 2.3 ± 7.1Footnote V
Journal articles by researchers involved (at time of application) 1.1 ± 1.5 0.9 ± 1.3 1.2 ± 1.4 1.6 ± 2.6 2.1 ± 1.5 1.4 ± 1.1 2.8 ± 1.6 -

In interviews with funded researchers, a common theme was that outputs for PPSRI projects were still being produced.14 Accordingly, the full impact of this program will not be known for some time. For example, one funded researcher reported that their project had so far resulted in 24 journal articles, with an additional 10 to 15 articles still being prepared. The evaluation of the CIHR OOGP program, which is the largest CIHR funding program, found that the number of publications peak three years after the competition year and can continue for up to nine years after the competition year. The same report shows that the average time to publish the first journal article from the start of the grant was 2.18 years (OOGP Evaluation Report, 2012). Assuming similar publication behavior between PPSRI and OOGP funded researchers, it is likely still too early to fully capture all the publication outputs generated by PPSRI funding, with the reported publications in the survey an underestimation of the total PPSRI publications.

Researchers and partners also reported attracting significant attention from international media and health organizations even though they are still in the publishing phase. The H1N1 pandemic focused media attention on the research in a way that may not have otherwise occurred. For example, one of the interviewed NPI’s was a media spokesperson for the pandemic. Another reported receiving “effective lay communication through TV, radio and print media” about a training module which aimed to prepare hospital workers to face the stresses of the pandemic.

2. Capacity Development

One of the overarching objectives identified for PPSRI was the need to build research capacity in Canada, so that in the event of a pandemic outbreak there would be sufficient research expertise necessary to address the issue. Some of the funding opportunities had as part of the eligibility requirements that researchers must include trainees on their grants.15

In assessing capacity development the evaluation looked at the number of research staff involved in the funded projects, number of trainees receiving formal qualifications on PPSRI projects and researchers’ perceptions about the influence that PPSRI had on their careers.

Number of Research Staff Working on PPSRI-Funded Projects

According to the CIHR Electronic Information System (EIS) data base, PPSRI involved a total of 345 investigators: 61 NPIs and 284 COPIs and OPIs were registered on the funding applications.

The online survey provided more information about the number and variety of the research staff working on PPSRI projects (NPIs reported on up to two projects).16. A total of 573 research staff worked on the 51 projects surveyed: an estimated 1,043 across all 92 PPSRI projects (Table 2-1)

The median number of research staff per project was 5.5, although teams varied from solo investigators to large networks (mean of 11.2). The larger research teams were generally associated with those projects funded under team or catalyst grants, the Influenza Research Network,17 and those with CIHR commitment values of greater than $250,000.

Clinical projects, which included the Influenza Research Network, generally had greater numbers of post health professional degree holders, and Master’s students, compared to the projects funded under the other CIHR pillars. Projects funded under health systems and services tended to involve more research assistants and technicians.18

Table 2-1: Number of Research Staff Working on PPSRI Projects

Type of Research Staff % of Projects with Research Staff (n=51) MeanFootnote VIII Standard DeviationFootnote VIII Total # of Investigators ProjectedFootnote IX total to all projects (N=92)

Source: NPI online survey QA8 and C5 reported for 51 projects. See Appendix D-7 for breakdowns by project categories.

Footnotes

Footnote VIII

For some projects, NPI did not provide the precise number of investigators; these cases were excluded from the calculation.

VIII

Footnote IX

Projections are extrapolated from the total # of investigators involved (Projection = (total / 51) x 92).

IX

Research assistant(s)/
technician(s)
98% 3.7 7.1 190 343
PhD students 65% 1.2 1.4 60 108
Undergraduate students 63% 1.2 1.7 62 112
Postdoctoral fellows (post-PhD) 49% 1.1 1.8 55 99
Master's students 45% 1.0 1.9 51 92
Post health professional degree (e.g. MD, BScN, DDS) 31% 2.6 9.9 133 240
Fellows not pursuing a Master's or PhD 22% 0.4 1.1 22 40
Other staff involved in the projects 0.4% 0.1 0.6 5 9
Overall 100% 11.2 17.0 578 1043

Data from the bibliometric study show an increase in the number of Canadian authors publishing in the area of pandemics for the period 2001-2010, with the steepest increase occurring after 2008, from 106 Canadian authors in 2008 to 407 in 2010 (Figure 2-1).19 The onset of the H1N1 pandemic should be also considered as a factor contributing to this increase.

Figure 2-1: Number of distinct Canadian authors publishing in the area of pandemics, 2001-2010

Figure 2-1 long description

Students Trained

Nearly all (91%) of NPIs reported that their projects included mentoring of- students in influenza and pandemic preparedness research. Most NPIs reported that the projects provided interdisciplinary training to students (80%, to some or a great extent) and increased the number of trainees in influenza and pandemic preparedness research (84%). Student mentorship was provided by partners as well as by NPIs and their teams. One PhD student interviewed for a case study compared his/her involvement in the program to an apprenticeship and indicated that the experience provided direction to his/her doctoral research. A funded researcher reported a post-doctoral fellow achieved their qualifications midway through their project and attained a faculty position. In their new position, the student-turned-faculty member continued to collaborate on the project and brought a student from their own lab onto the project.

The mid-term evaluation report concluded that, although there was no baseline for comparison, it appeared that capacity building was occurring through training and mentorship on PPSRI projects.20 This evaluation result demonstrates an increase across the same measures of training and mentorship (Table 2-2).

Table 2-2: NPI Agreement with Statements about Training/ Mentorship (Mid-term vs. Initiative End Evaluation)

Mid-term Evaluation
% of NPI (n=21)
Agree/Strongly Agree
Initiative End Evaluation
% of NPI (51 projects)
Agree/Strongly Agree

Source: NPI Online survey QA7 and QC4 - asked this question about up to two projects per NPI.

Note: Mid-term Evaluation E-survey of researchers - asked this question of all researchers in regards to all their projects.

My PPSRI project has included mentoring students/fellows in influenza and pandemic preparedness research 73% 90%
My PPSRI project has increased the number of trainees in influenza research and pandemic preparedness research 68% 84%
My PPSRI project has included interdisciplinary training for students/fellows 64% 80%

At the time of the survey, 124 students had completed their formal qualifications while working on the 51 projects reported by NPIs. The average number of students/trainees per project who achieved qualifications was 2.5 (Table 2-3), similar to the data reported for the mid-term evaluation. However, the projected total of students achieving qualifications was 224, compared to 173 at the mid-term evaluation.21

Table 2-3: The Number of Students Achieving Qualifications on PPSRI Projects

Type of Trainee % of Projects with Trainee (n=51) MeanFootnote X Standard DeviationFootnote X Total # Trained ProjectedFootnote XI total to all projects (N=92)

Footnotes

Footnote X

For some projects, NPI did not provide the precise number of investigators; these cases were excluded from the calculation.

X

Footnote XI

Projections are extrapolated from the total # of investigators involved (Projection= (total / 51) x 92).

XI

Undergraduate students 41% 0.9 1.7 44 79
Master's students 41% 0.4 0.6 20 36
PhD students 37% 0.5 0.8 22 40
Postdoctoral fellows 35% 0.4 0.7 20 36
Post health professional degree (e.g., MD, BScN, DDS) 22% 0.2 0.7 9 16
Fellows not pursuing a Master's or PhD 10% 0.2 0.6 9 16
Overall 75% 2.5 3.0 124 224

Relatively more postdoctoral fellows and Master’s students than doctoral students completed their formal qualifications while working on projects with a high CIHR commitment-value grant.22

Achievement of qualifications is not the only measure of student training for PPSRI-funded projects. One of the case study projects developed an internship program, where postdoctoral students worked for three to six months at internships in Vancouver, Toronto or Montreal. The internships provided students with an opportunity to receive training and mentorship from PPSRI NPIs. The internships provided students with access to “real-life” data from the H1N1 pandemic, excellent supervision, and networking opportunities.

PPSRI Contribution to Researchers’ Careers

A substantial proportion of both NPIs (83%) and COPI/OPIs (58%) considered their PPSRI project to have made a positive contribution to their career (Figure 2-2). PPSRI re-oriented the careers of 58% and focused the work of 68% of NPIs on influenza and pandemic preparedness research. This was the case for fewer COPI/OPIs, possibly because many were earlier in their careers or were specialists in associated fields such as public health. Researchers (NPIs) who felt that the PPSRI contributed to a re-orientation of their career increased to 58% from 30% in the mid-term evaluation.23

Figure 2-2: Contribution of PPSRI to Researchers’ Careers

Figure 2-2 long description

Source: NPI online survey QF3 and COPI/OPI online survey QC1; percentage of researchers who reported “yes”.

In interviews, researchers explained that the PPSRI grants provided them with opportunities to broaden the scope of their research, helped their career progression, and helped recruit students who had not previously worked on influenza and pandemic research.

One researcher reported that the program allowed them to undertake high-risk activities and study topics previously considered “controversial” in their field. Another researcher, involved in vaccine research, reported that the PPSRI grants gave them greater independence from industry which enhanced their credibility in the public health universe.

Also during the interviews, many researchers noted that while PPSRI funding provided them with the opportunity to take on and train students, they faced a challenge to retain those students after the end of PPSRI funding. One partner noted that graduate students had developed areas of expertise while working on PPSRI projects and the demand for these skills declined at the end of funding.

3. Knowledge Translation

At CIHR, knowledge translation (KT) is defined as a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system. This involves a complex system of interactions between researchers and knowledge users which may vary in intensity, complexity and level of engagement depending on the nature of the research and the findings as well as the needs of the particular knowledge user.24

KT was a central component of the PPSRI. Funded researchers were expected to collaborate with research users, such as public health practitioners, policy makers, industry. This was to ensure that the research findings and outcomes of funded projects were applied in concrete practices and procedures, new or updated policies, commercial products like vaccines and other therapeutics, patents or licenses, and new spin-off companies.25 KT is critically important because while discoveries and generating new knowledge have the potential to result in improvements to health and health systems, these benefits will not be realized unless knowledge is put into action (Graham & Tetroe, 2007).

In assessing PPSRI’s success in knowledge translation, this evaluation looked at the type of KT and research outcomes produced, the extent of stakeholders’ involvement, and the influence that PPSRI-funded research had on these stakeholders and research users.

KT and Research Outcomes Produced

The survey of PPSRI-funded researchers reveals a large variety of KT and research outcomes reported by these researchers for their projects (Table 3-1). The most commonly reported outcomes, as reported by NPIs, were new research methods (44%), new practices or theories (30%), and changed policies or programs (18%).

Data available from the recent evaluation of RMNI and OOGP programs (Source: RMNI evaluation, 2013) allow benchmarking; however these data should be interpreted carefully due to the average grant size and the funding period of these programs: RMNI invested $80M to support research in the fields of regenerative medicine and nanomedicine, and held funding competitions between 2004-2010, and the OOGP is CIHR’s biggest open funding program which has been running since 2000.

Table 3-1: Type of research outcomes reported by NPI

Category Percent of Projects
PPSRI
(n=49*)
RMNI
(n= 26)
OOGP
(n=596)

Source: For PPSRI- NPI online survey QA5 and QC2 for 50 projects. For OOGP- CIHR Research reporting System (RRS) data, March 2012. For RMNI- survey with funded researchers, May 2012. *Note: For one PPSRI project the NPI did not provide answers for this question. One project did but was categorized as outside the four CIHR themes and is not included in this table.

Research findings/knowledge creation 98% 100% 94%
New research method 44% 92% 57%
New practices (clinical, tool instruments, procedure/technique) 30% 38% 22%
New theory 30% 50% 63%
Adaption of research findings 28% 35% -
New or changed policy/program 18% - 5%
New vaccine/drug 16% 8% 5%
Replication of research findings 16% 23% 49%
Patents/licenses 14% 46% 12%
Intellectual property claim 12% 38% 12%
Direct cost savings (individual, organization, system or population level) 4% 12% 6%
Software/database 4% 15% 8%
Spin off company 0% 8% 4%

Different research areas and pillars produced different types of results. Over a third of projects researching vaccines and immunization produced results that could lead to a new vaccine or drug. Nearly half of these projects also resulted in new practices that could include tools, instruments, procedures, and techniques. Three-quarters of projects researching virus biology and diagnostics produced new research methods, 42% produced patents and licenses, and 33% produced intellectual property claims.26 Slightly over two-thirds of projects focused on prevention and treatment produced new theories. Projects focused on ethical, legal or social aspects of influenza research and pandemic preparedness were most likely to replicate and adapt research findings; influence the development of new, or change pre-existing, policies and programs; and result in new practices.27 Only biomedical projects reported creating new vaccines/drugs and new products/licenses, and made intellectual property claims. Projects falling within CIHR’s clinical pillar produced new research methods and new practices. Most health systems/services projects reported new or changed policy or programs. Half of social/cultural/environmental/population health projects (n=7) reported developing a new theory.28

The following examples of project achievement drawn from key informant interviews and case studies demonstrate the diversity of the projects funded by PPSRI and their achievements.

Knowledge Translation to Key Stakeholders

While much of the knowledge translation from PPSRI projects occurred through the traditional channels of peer reviewed publications, presentations or posters at conferences and symposia, and workshops and annual meetings, examples of other approaches to knowledge translation described during the key informant interviews included:

Partnerships were also an effective way to disseminate information. NPIs reported that partnerships with decision makers and health service providers, or serving on advisory boards had led to PPSRI findings, particularly “real-time findings,” having a direct impact on policy development and service delivery. The ability to form partnerships was underlined by the H1N1 pandemic through the need to rapidly move research findings and expert advice into decision-making venues. NPIs reported that 18% of their projects had resulted in changed policies or programs.

Translating results to the broader public seemed to be somewhat dependent on researchers pre-existing aptitudes and connections. Some PPSRI researchers also had key roles as spokespeople during the H1N1 pandemic and used the media as a channel for disseminating information and research findings. For example, an interviewed researcher reported that his and their colleagues’ membership on different immunization committees in Canada and the US facilitated communication with the public.

Many of the interviewed researchers emphasized that knowledge translation was ongoing. Some researchers interviewed reported having received grants that focused on knowledge translation which contributed to successful dissemination of results.

A few researchers interviewed and surveyed cited a lack of infrastructure as hindering effective knowledge translation and 5% of surveyed researchers suggested specific improvements in this area, such as making data sharing between research institutions mandatory or producing a regular newsletter to report on projects’ progress and findings.

Stakeholders involved in the PPSRI-funded projects

In the mid-term evaluation,29 66% of researchers reported that research users were directly involved in their projects and a further 21% indicated that research users would eventually become involved. In the present evaluation, 88% of all researchers reported they had worked on projects involving stakeholders and 45% had listed stakeholders in their grant applications (Table 3-2).

Table 3-2: Percentage of Individuals/Groups Involved in the PPSRI Projects

Category of Partner Investigator's Role
Percent of NPI (n=40) Percent of COPI/OPI (n=106)

Source: NPI online survey QB1 and QD1 and COPI/OPI online survey QB1; Select all that apply type of question.

Note: Data were combined for up to two NPI projects. COPI/OPI reported generally on all the projects they had been involved in.

Health system/care practitioners 55% 69%
Study Stakeholders (who are formally listed in the grant application) 45% 48%
Federal/provincial representatives 30% 37%
Health system/care professional organizations 28% 25%
Health system/care managers 25% 33%
Community/municipal organizations 20% 19%
Patients/consumers of health system/care 15% 33%
Industry 8% 14%
Media 8% 1%
Consumer groups/charitable organizations 5% 5%
Other 5% 9%
Not applicable: No stakeholders were involved 28% 6%
Don't know / No response 5% 2%

Those stakeholders who were involved in PPSRI projects as formal partners (i.e., listed in the grant application) appeared to be involved in all or most aspects of the projects. Stakeholders provided valuable input to projects through the provision of data, in-kind support, access to blood samples, and access to health services and systems. One research team reported being the first team to have access to data from Mexico City about the H1N1 pandemic. Generally, stakeholders were most commonly involved in data collection and project implementation (Figure 3-1), particularly so for health system and health care practitioners.

Figure 3-1: Nature of Stakeholder Involvement in PPSRI Projects as Reported by Researchers

Figure 3-1 long description

Source: NPI online survey QB2 and QD2 and COPI/OPI online survey QB2; Select all that apply type of question.

Note: Data were combined for up to two NPI projects. COPI/OPI reported generally on all the projects they had been involved in.

Influence of PPRSI-Funded Research on Stakeholders

Survey results show that health system and health care practitioners were the group who benefitted the most from PPSRI projects (Figure 3-2). One interviewed researcher’s project created a 10-hour course on pandemic preparedness for hospital workers with a focus on anticipating and coping with an outbreak to decrease interpersonal problems on the front lines and increase a sense of self-efficacy. Moreover, researchers in the field of epidemiology (modeling the spread of disease) attributed effective pandemic responses by practitioners and decision makers to the data their projects had generated.

Figure 3-2: Stakeholders Reported as Influenced to Some/Great Extent by PPSRI Projects

Figure 3-2 long description

Source: NPI online survey QB3 and QD3 and COPI online survey QB3.

PPSRI-Funded High Impact Research Projects

The following two examples illustrate the broad impact generated by PPSRI-funded research. The impacts were captured by conducting case studies of two PPSRI funded projects which included interviews with researchers, partners, research users and trainees involved in the research projects.

4. Collaboration & Partnership

Fostering collaborations and partnerships was a central goal of the PPSRI. Through Team Grants, and both the Seed and Collaborative Research Project Grants under the International Opportunity Program, CIHR–III strived to bring research communities and stakeholders together to create innovative approaches to address important research questions.

For example through the International Opportunity Program, collaborations between Canadian researchers and researchers all over the world were encouraged. The purpose of the seed grants was to assist Canadian researchers in exploring, developing, and establishing new international collaborations with foreign researchers. The Collaborative Research Grants were designed to enable Canadian researchers to participate in international research projects approved for funding by foreign entities in which the Canadian participants must secure their own funding.30

CIHR – III collaborative activities and partnerships

Over the life of PPSRI, CIHR–III collaborated with a wide variety of stakeholders to fund research in the identified strategic areas, and to organize a series of workshops and meetings to promote the initiative. Stakeholder organizations included: the Alberta Heritage Foundation for Medical Research (now Alberta Innovates – Health Solutions), the Canadian Food Inspection Agency (CFIA), Canada's Research-Based Pharmaceutical Companies (Rx&D) Health Research Foundation, the Public Health Agency of Canada (PHAC), CIHR Ethics branch, CIHR Institute of Aboriginal Peoples’ Health (CIHR-IAPH), CIHR Institute of Health Services and Policy Research (CIHR-IHSPR), CIHR Knowledge Synthesis and Exchange branch, and the National Natural Science Foundation of China.31

A series of workshops and meetings (Table 4-1) were organized by CIHR-III, involving various stakeholders and pandemic experts from different research settings across Canada. The goal of these meetings was to help researchers share their research findings, identify gaps in research knowledge, and also to foster networking and collaboration. The purpose of the application development workshops was to assist researchers in gaining an understanding of the application process and the key components of the application. The majority of researchers who participated in these workshops and who completed the workshops’ evaluation forms rated the overall effectiveness and usefulness of the workshops as good, very good or excellent.

Table 4-1: Collaborative activities organized by CIHR-III

Meetings and workshops organized by CIHR-III Date of the event Number of participants

Source: CIHR website; PPSRI publications

Canadian Pandemic Preparedness Meeting: Outcomes, Impacts and Lessons Learned November 2010 N=140
Canadian Pandemic Preparedness Meeting: H1N1 Outbreak Research Response July 2009 N=180
Canadian Pandemic Preparedness Meeting: From Discovery to Frontlines November 2008 N=150
Influenza Research Network Application Development Workshop February 2008 N=20
PPSRI Application Development Workshop March 2007 N= 50

Over the course of the initiative CIHR-III attracted 44% of the initiative financial resources from external partners. Table 4-2 presents the financial commitments for each of the PPSRI’s partners.

Table 4-2: PPSRI external partners’ financial commitments:

Partner Organizations for PPSRI Financial commitments
Public Health Agency of Canada $17,163,717
Canada's Research-based Pharmaceuticals (Rx&D) Health Research Foundation $994,624
CIHR/Rx&D Collaborative Research Program $250,000
Canadian Food Inspection Agency $225,000
Alberta Heritage Foundation for Medical Research (now: Alberta Innovates - Health Solutions) $200,000
National Natural Science Foundation of China $90,000
Total $18,923,341

The mid-term evaluation reported that nine organizations had expressed an interest in partnering with CIHR on this initiative, and concluded that CIHR had been effective in developing solid and productive partnerships with national agencies. Partner stakeholders interviewed for the mid-term evaluation viewed the partnerships very positively because they enabled them to support research in identified priority areas and to capitalize on the CIHR’s peer review system. The PPSRI was regarded as effective in the production, coordination and integration of national research programming, as well as avoiding duplication.32 Although representatives from four partner agencies were interviewed as part of this evaluation, it was difficult to obtain more detailed perspective on the partnerships as in most cases the interviewed representatives had not been involved in the early stages of the partnership development. However, partners were generally very positive about CIHR’s effectiveness in administering the competition process, managing the grants, coordinating pandemic research nationally, the value of the PPSRI in establishing a research infrastructure, and the expectation that the research had or would create new knowledge. Some of the suggestions mentioned by partners were to have up to date information about the achievements of PPSRI projects, and synthesized information for decision makers.

PPSRI-funded researchers collaborative activities

PPSRI-funded researchers were also encouraged to collaborate with other researchers or different stakeholders through their projects. The survey, the interviews and the bibliometric study provide evidence of the extent of funded researchers’ collaborations.

Survey results show that almost all NPIs (85%) and COPI/OPIs (82%) considered that the PPSRI grant had helped them or their teams to collaborate with other researchers involved in influenza and pandemic preparedness research. Many collaborations were formed during PPSRI funding and have continued following the end of the grant (NPIs 83% and COPI/OPIs 67%). Collaboration was also increased by the H1N1 pandemic, which brought together researchers, decision makers and front-line staff to respond to the threat.

Table 4-3: Collaboration on PPSRI Projects (Mid-Term versus Initiative End Evaluation)

Mid-term Evaluation Percent agree/ strongly agree Initiative End Evaluation Percent agree/ strongly agree
NPI (n=21) COPI/OPI (n=56) NPI (n=40) COPI/OPI (n=104)

Source Mid-Term Evaluation E-survey of Researchers Q12, Q13 and Q15; NPI Online survey QF3; COPI/OPI Online survey QC1.

Footnotes

Footnote XII

NPI n=20.

XII

Footnote XIII

Mid-term: "Because of the PPSRI, my research work involves researchers from areas or disciplines that were not involved with my work before."

XIII

Has helped me/my team collaborate with international researchersFootnote XII 35% 25% 68% 43%
Has helped me/my team collaborate with other researchers involved in influenza and pandemic preparedness research 71% 70% 85% 82%
Has helped me/my team collaborate with other researchers NOT involved in influenza and pandemic preparedness researchFootnote XIII 67% 59% 73% 47%
Has resulted in new collaborations that have continued since the funding ended n/a n/a 83% 67%

In interviews, NPIs reported that the CIHR workshops and meetings had provided valuable opportunities for networking and collaboration. For some researchers the workshops were an opportunity to exchange data and keep up-to-date on developments of the PPSRI projects at other institutions outside their immediate research area and across disciplines. One researcher reported that networking led to collaboration with another researcher half-way across the country. Both researchers were able to leverage their own local human resources and laboratory capacities, and ultimately produce joint conference presentations and publications. Though not all researchers interviewed reported attending workshops, all were aware of the opportunities.

In the mid-term evaluation, 67% of NPIs and 59% of COPI/OPIs considered their work to involve researchers from areas or disciplines that were outside of influenza and pandemic preparedness work before, suggesting an increase in new collaborations.33 The percentage of researchers crediting the PPSRI with increasing their international collaboration has increased from 35% of NPIs and 25% of COPI/OPIs in the mid-term evaluation to 68% of NPIs and 43% of COPI/OPIs in this evaluation (Table 4-3).

Successful collaboration with public health agencies was a common theme in interviews with researchers. National and provincial agencies acted as conduits for knowledge translation, sources of feedback, and linkages to the broader research community. One researcher reported the compilation of project data by provincial agencies was critical to their project. Another NPI reported that in light of the H1N1 pandemic, their research team altered the goals of their project to better align with public health priorities, and designed studies based on the feedback they received. This researcher also reported that public health agencies acted as a bridge to international agencies and national agencies in other countries, which facilitated collaboration and the development of common protocols.

National and international collaboration rates

The bibliometric analysis was also used to assess the extent of researchers’ collaborations both nationally and internationally by looking at the number of papers published by PPSRI-funded researchers in collaboration with at least one other researcher from a different institution or from a different country. The findings indicate that the proportions of papers published by PPSRI-funded researchers in collaboration with other researchers are higher than Canadian averages or the world averages for pandemic research at both national and international levels (Table 4-4). The bibliometric network analysis revealed that the following Canadian institutions are the most central nodes of collaboration in the area of pandemic research: University of Toronto, Hospital for Sick Children – Toronto, Mount Sinai Hospital-Toronto, and the University Health Network.

Table 4-4: National and international collaboration rates for PPSRI-funded researchers

National collaboration International collaboration
2008 2009 2010 2008 2009 2010
Source: Bibliometric Analysis, Observatoire des sciences et des technologies (Thomson Reuters - Web of Science). October 2010
World (pandemic papers) n/a n/a n/a 18.6% 20.2% 19.8%
Canada (pandemic papers) 51.4% 56.5% 65.0% 37.8% 33.3% 38.8%
PPSRI (funded pandemic papers) 70.0% 78.9% 85.9% 40.0% 23.7% 40.6%

5. Leveraging additional funds

Leveraging additional funds is another indicator used to measure the success of funded projects, the assumption being that the additional funds will contribute to the continuation or expansion of the research projects. Nearly half of the NPIs (45%) and close to one-quarter (24%) of the COPI/OPIs reported that their involvement in their PPSRI grant had helped them to secure additional funding (Table 5-1).34

Table 5-1: Sources and Value of Additional Funding Leveraged by PPSRI Researchers

Source of Additional Funding NPI (n=40) COPI/OPI (n=105)
Percent Receiving Additional Funds Percent Receiving Additional Funds Total Amount of Additional FundsFootnote XIV

Source: NPI online survey QE1, QE2 and QE3 and COPI/OPI online survey QB4, QB5 and QB6.

Footnotes

Footnote XIV

The total is the amount reported by researchers who took part in the survey. One NPI reported to have leveraged 12 grants from industry valued at over $5,347,000 in total.

XIV

Canadian Institutes of Health Research (CIHR) 30% 10% $7,903,086
Industry 3% 4% $6,447,000
Provincial government organization 23% 11% $2,959,277
Other Canadian federal organization 8% 2% $1,161,930
Canadian not-for-profit organization 5% 3% $1,420,000
National Institutes of Health (NIH) 3% 0% $1,480,000
Canada Foundation for Innovation (CFI) 3% 0% $1,001,000
Networks of Centres of Excellence (NCE) 3% 1% $1,018,000
Natural Sciences and Engineering Research Council of Canada (NSERC) 8% 1% $664,000
Hospital/Public Health 5% 3% $480,000
Canadian university or research centre 3% 1% $126,000
Other International organization 0% 3% $175,000
Private Donor 0% 2% $85,000
Uncategorized Source 0% 2% $10,000
Total $24,930,293

Sources of additional funding varied, with the most commonly reported source being provincial government organizations. However, the most money was leveraged from industry. Five researchers reported having leveraged additional funding from industry sources. One researcher leveraged 12 grants and/or awards from industry with a total value of over five million dollars. The other researchers reported leveraging between one and four industry grants and/or awards with total values ranging from $300,000 to $500,000.

As well as the sources and value of additional funding leveraged by PPSRI researchers summarized in Table 5-1, researchers reported leveraging resources in the form of in-kind support. For example, three researchers interviewed reported receiving in-kind support from public health providers and other partners, including one researcher who ultimately leveraged support from four multinational pharmaceutical manufacturers.

Fund-leveraging appeared to be constrained by several factors. In an interview, one researcher explained that the sources of additional funding were limited and commented on the ethical implications of obtaining funding from industry. Another interviewed researcher commented on the credibility that having non-industry sourced funding provided to his work. A common theme among interviewed researchers was a competitive market for dedicated public funding towards influenza and vaccine research. Examples included reports that:

6. Program Design

The starting point for the design of the PPSRI was the Influenza Research Priorities Workshop organized jointly by CIHR-III, PHAC and the Canadian Association for Immunization Research and Evaluation (CAIRE) in September 2005. Following the workshop, the CIHR-III, in consultation with its Institute Advisory Board (IAB) and key members of the research community, formed a Task Group to identify the research priorities to focus on, as well as the funding mechanisms to be used. The priorities identified are described in more detail in the Program Background (Section 8) of this report, but here should be underlined that these priorities were validated through consultation with the Canadian Rapid Research Response Team (C3RT) and other stakeholders in pandemic-related fields.35

Beginning in fall 2006, PPSRI offered several types of funding opportunities: catalyst grants, team grants, operating grants, development and planning grants. A total of 31 funding competitions were launched through PPSRI, generating 92 funded projects. Some of these funding opportunities were offered through CIHR’s ongoing slate of programs such as priority announcements, others were offered in partnership with other CIHR institutes and other organizations. Overall most of the funding opportunities included a relevance review process of a letter of intent. Partner organizations were included in the relevance review process to ensure that they could identify applications that aligned with their priorities. A description of the funded projects is offered in the Program Background, Appendix 1.

To assess the effectiveness of PPSRI’s design and delivery the evaluation explored initiative’s strengths and challenges as they were perceived by the funded researchers, as well as different aspects related to grant application and decision process.

Strengths of PPSRI

For NPIs, the main strength of PPSRI related to the infusion of funding targeted to influenza and pandemic research (Table 6-1). Approximately one-third of NPIs noted specific aspects of the grant mechanism such as the design and timelines as strengths. In the key informant interviews, one NPI noted the advantage of being independent from industry funding. PPSRI was also seen as broadening the scope of what CIHR normally funds and providing flexibility to set and change the research agenda. COPI/OPIs were less involved in the grant application and decision process, and tended to identify the ability to build networks and collaborations as a key strength of the initiative.

Table 6-1: NPI, COPI/OPI Perceptions of the Strengths of PPSRI

NPI Percent (n=40) COPI/OPI Percent (n=108)

Source: NPI online survey QF6 and COPI/OPI online survey QC3

Funding (availability, targeting and speed of the process) 50% 24%
Mechanism/Design (efficient mechanism, timely response to pandemic) 35% 24%
Research (breadth and scope, innovative, ability to modify the research approach) 28% 27%
Building Networking/Collaboration 20% 32%
Teams (building teams, multidisciplinary teams) 18% 16%
Results/Achievements (preparedness for a pandemic, capacity to respond, coordinated response) 15% 18%
Training/Mentoring 5% 7%
Other elements 18% 11%

Key program strengths, identified during interviews with NPIs, were increased research and response capacity. Researchers appreciated that the PPSRI initiative had increased the number of people working on influenza and pandemic research. Multiple researchers reported that the initiative was well-timed and capitalized on public interest in pandemic preparedness. They also credited CIHR for investing in projects which they reported would have been unlikely to attract such substantive funding through normal funding mechanisms.

Challenges and Suggestions for Improvement

The most frequently identified challenge was related to the potential lack of sustainability of the increased researcher capacity and the networks and collaborations that had been developed through the PPSRI. Most NPIs comments related to funding - the need for additional funding, funding renewal, or funding for longer time periods (Table 6-2). The PPSRI grant durations varied from one up to a maximum of three years: catalyst grants were one year, operating grants two years and team grants three years in length.

These findings are consistent with the earlier suggestions made in the mid-term evaluation where key informants also noted the length of the grants as a challenge and suggested that this type of initiative to include a two-year planning and implementation phase followed by five years of funding.36

Other suggestions included involving other national organizations in the competition process and setting up a consortium of manufacturers capable of responding in collaboration with the research community to produce new vaccines.

Table 6-2: NPI, COPI/OPI Suggestions for Improvements to PPSRI

NPI Percent (n=40) COPI/OPI Percent (n=108)

Source: NPI Online survey QF7 and COPI/OPI online survey QC4

Funding (additional funding, increased amounts and duration, sustainability, faster delivery) 43% 28%
Research (broaden scope, sustained focus, focused priorities) 15% 12%
Mechanism/Design (develop a permanent infrastructure, involve stakeholders in the process, improve accountability) 5% 18%
Networking/Collaboration (suggested changes) 5% 8%
Training/Mentoring (suggested changes) 5% 2%
Teams (allow more time to form teams) 0% 2%
Improve knowledge translation (mechanism for translation, transfer to industry) 3% 6%
Other changes 5% 7%

Aside from the dominant theme of funding-related issues, some interviewed researchers cited access to data and resources as an area in need of improvement. One researcher noted that barriers to data sharing may have been more related to the policies of research institutions than an aspect of the initiative. Another researcher reported that there are a limited number of Level 3 “High Containment” laboratories in Canada, and that H1N1 had been designated a Level 2.5 risk, thus a greater proportion of project funding could be directed toward increasing the number of facilities.

Also consistent with researchers’ comments, interviewed partners suggested the following improvements: the need to sustain the gains in capacity that had been made; involve more industry partners; and increase support for knowledge dissemination and translation.

CIHR partners suggested improving the information available to end-users by ensuring organizations received annual reports.37 One partner noted that while there were opportunities for disseminating information to researchers, information was needed in a form appropriate to decision makers, such as synthesized summaries of key findings by theme. Information would also better meet the needs of partners if partners were more involved in the competition design stages of future initiatives.

Grant Application and Decision Process

NPIs were positive about the grant application decision process (Figure 6-1). While still perceived as effective, an area identified as needing more clarity was the application guidelines. In the interviews, there were also some comments made about timelines and challenges in applying, for researchers who did not have previously established teams or networks.

Figure 6-1: NPI Perceptions about the Application Process

Figure 6-1 long description

Source: NPI online survey QF1

NPIs were also positive about the decision process (Figure 6-2) and few had any suggestions to make about improvements to the process. However, the speed of responses by CIHR staff to requests for information and the transparency of the peer review process were areas where potential improvements could be made. While the overall attitudes of NPIs towards the peer review process were positive, one interviewed NPI noted the challenges associated with finding Canadian peer reviewers for large network grants.

Figure 6-2: NPI Perceptions about the Decision Process

Source: NPI online survey QF2

Figure 6-2 long description

Research Outputs

PPSRI used a variety of funding mechanisms to support research in the pandemic area: catalyst grants, operating grants, team grants, Influenza Research Network (IRN) and other planning and development grants. In order to compare the productivity of these different types of grants, an analysis of research outputs per $100,000 invested was conducted using the survey results (of 51 PPSRI projects) for the following indicators: the number of refereed journal articles, number of research staff and trainees involved in the projects, and the number and amount of additional funding leveraged. This analysis provides insight into productivity of researchers holding different types of grants at a basic level for PPSRI and allows comparability with other similar studies; however, the comparisons between the funding mechanisms should be treated with caution given contextual factors and potential confounds related to differences between funding mechanisms and areas of research. For these reasons, the IRN and other planning and development grants (n=10) are excluded from this comparison due to their specific characteristics and the fact that at the time of the survey IRN project was still active.

As shown in Table 6-3, in terms of productivity per $100,000, overall PPSRI grants produced 1.1 refereed journal articles, involved 3.0 research staff/ trainees and leveraged 0.5 grants/awards. For every dollar invested in PPSRI projects, researchers were able to leverage $1.14 dollars in grants/awards. When comparing by funding mechanisms: catalyst grants produced 1.4 refereed journal articles, involved 7.2 research staff/trainees, leveraged 0.9 grants/awards and for every dollar invested it leveraged another $2.06 dollars in grants/awards. They are followed by operating grants which produced 1.3 refereed journal articles, involved 2.5 research staff/ trainees, leveraged 0.7 grants/awards and for every dollar invested it leveraged another $1.80 dollars in grants/awards. Appendix 3 presents further details for this analysis.

The PPSRI data can be compared with similar analyses conducted for the RMNI and OOGP–Research Reporting System (RRS) data as part of the evaluation of the RMNI conducted in 2012 (Table 6-3). Considering the funding mechanisms characteristics, the most suitable comparison would be between PPSRI and RMNI catalyst grants and PPSRI and OOGP operating grants. Table 6-3 illustrates that RMNI catalyst grants produced 1.2 refereed journal articles, involved 4.3 research staff/trainees, leveraged 1.2 grants/awards and for every dollar invested it leveraged another $5.22 dollars in grants/awards; whereas, OOGP projects funded published 2.8 articles and trained 2.6 research staff/trainees per $100,000. As stated previously, this comparison should be treated with caution due to differences between the programs (PPSRI, RMNI and OOGP), the funding mechanism used (catalyst grants, team grants, operating grants) and different areas of research supported (pandemics, regenerative medicine and nanomedicine, open research). As such, this type of analysis provides a benchmark for future CIHR evaluations and studies.

Table 6-3: Research outputs per $100,000 invested

Research Outputs per $100K: PPSRI RMNI OOGP
Overall PPSRIFootnote XV Catalyst Grants N=17 Operating Grants N=18 Team Grants N=5 Overall RMNIFootnote XVI Catalyst Grants N=12 Team Grants N=14 OOGP GrantsFootnote XVII N=440

Footnotes

Footnote XV

This analysis does not include Influenza Research Network and other development and planning grants.

XV

Footnote XVI

Source: The RMNI evaluation, the results are based on a survey with RMNI funded researchers held in 2011, using expenditures of the grants at the time of the survey.

XVI

Footnote XVII

Source: Research Reporting System (RRS) Pilot Survey - for grants awarded over the period 1991-2008; money based on grant commitments.

XVII

Number of research staff and trainees involved 3.0 7.2 2.5 1.1 2.1 4.3 1.9 2.6
Number of refereed journal articles published 1.1 1.4 1.3 0.7 1.4 1.2 1.4 2.8
Grants and Additional Funding Leveraged:
Number of grants and awards leveraged per program grant 0.5 0.9 0.7 0.1 0.4 1.2 0.3 -
Dollar amount of grants and awards leveraged per $1 of program grant funding $1.14 $2.06 $1.80 $0.17 $1.77 $5.22 $1.44 -

For PPSRI, further analyses have been completed to compare the productivity and leveraging of PPSRI projects by CIHR pillars and area of research funded. The analysis shows that PPSRI projects funded under health systems and services pillar were the highest in terms of both productivity and leveraging: for every $100K invested, these projects produced 3 refereed journal articles, involved 11.3 research staff/ trainees, leveraged 2.1 grants/awards for every PPSRI grant, and leveraged $3.15 dollars in grants/awards for every PPSRI grant dollar invested (Table 6-1, Appendix 3).

The analysis by area of research found that the highest productivity and leveraging was achieved by the PPSRI projects funded under prevention and treatment area of research: for every $100K invested, these projects produced 2.0 refereed journal articles, involved 7.4 research staff/ trainees, leveraged 1.1 grants/awards for every PPSRI grant and leveraged $2.52 dollars in grants/awards for every PPSRI grant dollar invested (Table 6-2, Appendix 4).

7. Program Relevance

Continued need for Pandemic Preparedness Research in Canada

The present evaluation assessed also the extent to which PPSRI addressed the need for the pandemic preparedness. PPSRI funding has resulted in knowledge creation by projects that would likely not otherwise have been funded. In the absence of PPSRI funding, over two-thirds (68%) of NPIs believed that their project would not have been undertaken at all and approximately one-quarter (28%) felt their project would have been completed on a smaller scale or differently. Very few NPIs (3%) reported that their project would likely have been completed.38 This pattern was similar across all PPSRI theme areas.

Some researchers reported difficulty in getting funding from other sources because their work was at the interface between disciplines, or that they had tried other sources but were unsuccessful. Despite initial difficulties, some NPIs had been able to leverage additional funding as a result of their PPSRI grants, both from CIHR and other sources.

PPSRI was seen as an innovative initiative. One NPI commented that it was good that it had been put in place before the H1N1 pandemic, as the infrastructure and networks that had been established were able to be rapidly mobilized when the pandemic broke out.

The area of research where NPIs and COPI/OPIs thought PPSRI had had the most impact was research into vaccines and immunization (Figure 7-1). This was also the research area in which the most money funded through this initiative had been invested. NPIs and COPI/OPIs had differing views on the extent to which PPSRI had contributed to research on the ethical, legal or social aspects of influenza and pandemic research. Of note is the relatively high proportion of NPIs and COPI/OPIs who did not know the extent of PPSRI’s contribution to research as a whole. This may be because the dissemination of knowledge from projects is ongoing and/or that researchers are not aware of PPSRI contributions outside of their own field.

Figure 7-1: PPSRI’s Contribution to Influenza Research and Pandemic Preparedness in Canada

Figure 7-1 long description

Source: Perceptions of Contribution - NPI Online survey QF4 and COPI/OPI online survey QC2
Percentage of Invested dollars - Based on CIHR data only for NPIs who completed the online survey

These findings are consistent with the results of the 2009 mid-term evaluation for the researchers’ ranking of the pandemic research priorities for the following five years at that time. This supports the idea that the research priorities identified by the PPSRI Task Group were relevant for the field of pandemic research.

The PPSRI increased researcher capacity, formed networks, and established collaborations as shown in sections 2 and 4. It is not clear how many of these would have been formed without the PPSRI or how many will be sustained in the long term. However, there is evidence that there is now a larger pool of trained people who can be called on in the event of another pandemic.

In interviews, one researcher noted that in the time since the H1N1 pandemic, there had been a loss of focus on influenza and pandemic research possibly as a result of “fatigue” with the topic. Others emphasized that, although some progress had been made, there was a need to continue funding for research in this area and that the PPSRI needs to be an ongoing initiative.

Alignment with government priorities and departmental strategic outcomes

CIHR’s mandate is to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system."39 Through the research it funds, CIHR helps to respond quickly and effectively to health crises such as outbreaks of infectious diseases, by rapidly mobilizing researchers to address different area of health research. PPSRI responds directly to the CIHR strategic directions described in the CIHR’s Strategic Plan (2009-2014): Health Research Roadmap by investing in world-class research excellence and addressing health and health system research priorities through preparing for and responding to existing and emerging threads to health.40

Alignment with role and responsibilities of the Federal Government

In May 2006, the Government of Canada committed a total of $1 billion over five years to support Canada’s ability to respond to pandemics involving other government departments as Public Health Agency of Canada (PHAC), Health Canada (HC), the Canadian Food Inspection Agency (CFIA) and CIHR.41 In response to this, CIHR–III one of the 13 virtual institutes of CIHR, that supports research and building capacity in the area of infectious disease and immunity, established PPSRI, with the mission to identify strategic research priorities and to support pandemic preparedness research to address any potential pandemic threat.

In 2010, the Standing Senate Committee on Social Affairs, Science and Technology undertook a review of Canada’s response to the 2009 H1N1 influenza pandemic underlining the importance of pandemic readiness and the need to maintain a focus on pandemic planning. One of the recommendations of this study was that the Government of Canada ensures that future federal pandemic preparedness and response budgets provide dedicated and sustained funding for research (Canada’s Senate, 2010).

All researchers, partners and stakeholders interviewed believed that the federal government had a role in supporting influenza and pandemic research. The reasons cited by interviewees included the following: a pandemic is a national issue; the credibility that federal government funding gives to a research group at the international level; the need for a coordinated agency-wide approach to a pandemic; the federal government’s ability to make data available nationally and internationally; and, the lack of another body with the funding resources or the ability to respond quickly.

References

CIHR - Pandemic Preparedness Strategic Research Initiative – Report on Activities & Outcomes.

Institute of Infection and Immunity (III) – About the Pandemic Preparedness Strategic Research Initiative (PPSRI), Retrieved on May 2012.

Campbell D., Picard-Aitken M., Côté G. et al. (2010). Bibliometrics as a Performance Measurement Tool for Research Evaluation: The case of Research Funded by the national cancer Institute of Canada. American Journal of Evaluation, 31; 66-83.

Goudin, B. (2005) The Impact of Research Grants on the Productivity and Quality of Scientific Research [ PDF (291 KB) - external link ]. Accessed on December 14, 2011.

Graham, I.D., and Tetroe J. (2007). How to translate health research knowledge into effective healthcare action. Healthcare Quarterly, 10 (3), 20-22.

Larivière, V., Archambault, E., Gingras, Y., & Vignola-Gagné, E. (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities. Journal of the American Society for Information Science and Technology, 57, 997-1004.

Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR (May 2009), Retrieved on December 2012.

Moed, H. F. (2005). Citation Analysis in Research Evaluation. Dordrecht, The Netherlands, Springer.

McDavid, J C. and Hawthorne, L.R.L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

PHAC (2011) - Evaluation of the grant and contribution components of the Avian and Pandemic Influenza Preparedness Initiative - just draft available.

Senate Canada (2010) – Standing Senate Committee on Social Affairs, Science and Technology - Canada’s Response to the 2009 H1N1 Influenza Pandemic [ PDF (465 KB) - external link ], Retrieved on December 2012.

Appendices

Appendix 1: Program background

Recognizing the potential threat of a pandemic influenza, in May 2006, CIHR-III established PPSRI to address the need to develop a coordinated and focused research effort and build research capacity in the field of pandemic influenza in Canada. The mandate of the PPSRI was to identify strategic research priorities and support pandemic preparedness research. PPSRI was guided by PPSRI Task Group formed by researchers who carried out pandemic research and research users who would use the knowledge generated through this initiative.

The following PPSRI research priorities were identified by the PPSRI Task Group:

CIHR received $21.5 million over five years (2006-2011) to support pandemic influenza research and CIHR-III managed to attract another $19 million from external partners (Table 8-1).

Table 8-1: PPSRI Financial Allocations

Funding Source Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Total
2006/2007 2007/2008 2008/2009 2009/2010 2010/2011 2011/2012 2012/2013

Source: III administrative data. Information in this table may differ from that in the mid-term evaluation as it represents actual funding received.

PPSRI Core funding 1,722,901 1,919,102 5,449,041 6,343,628 5,806,950 0 0 $21,241,622
CIHR Institutes 0 350,000 907,500 671,667 549,568 395,000 170,000 $3,043,735
External partners43 0 1,258,882 1,337,500 7,632,213 4,999,862 3,844,884 0 $18,923,341
Total 1,722,901 3,527,984 7,694,041 14,647,508 11,356,380 4,239,884 170,000 $43,358,698

Funding was awarded to a nominated principal investigator (NPI) who worked on the project alone or with co-principal investigators (COPIs) and other principal investigators (OPIs). Other researchers as well as highly qualified personnel (HQP) such as graduate students, postdoctoral fellows and research associates may also have been involved in supported projects as they developed.44

The following types of grants were launched through PPSRI45:

Catalyst grants

27 projects

12% of budget

Catalyst grants served as a first step towards more comprehensive funding opportunities. They supported original projects with the potential to generate high impact results, such as new research tools, inventions, or methodologies. These grants had an average duration of 1 year, and an average value of $190,701.

Operating grants

30 projects

16% of budget

Operating grants helped develop and maintain Canadian health research capacity by supporting projects or teams in all areas of health. They contributed to the creation, dissemination and use of health-related knowledge. These grants had an average duration of 2 years and an average value of $219,887.

Team grants

10 projects

31% of budget

Team grants supported collaborative, multi-disciplinary research teams. Their goal was to enhance understanding of interactions between different areas of health, address major health research questions, and foster the application of knowledge. These grants had an average duration of 3 years and an average value of $1,293,034.

Influenza research network

33% of budget

Three years project (2009-2011) launched in collaboration with the Public Health Agency of Canada (PHAC), to conduct influenza and pandemic preparedness research, foster knowledge exchange between researches and decision makers and train the next generation of researchers. It involved over 130 researchers from different institutions. Grant value of $13,974,459.

Other development and planning grants

24 projects

8% of budget

Other grant programs under which PPSRI projects were funded include the China-Canada Joint Health Research Initiative, two Applied Public Health Chairs, the International Opportunities Program – Development and Planning Grant and other planning and collaborative research grants. With the exception of the two Applied Public Health Chairs which have a 4 and 5 year duration, the average grant duration is 1.4 years, with an average grant value of $131,510.

Figure 8-1: PPSRI’s Contribution to Influenza Research and Pandemic Preparedness in Canada

Figure 8-1 long description

Appendix 2: Evaluation Methodology

To ensure that the study findings were robust, credible, and consistent with Treasury Board Secretariat policy and recognized best practices in evaluation (McDavid et al., 2006), this evaluation used multiple lines of evidence, involving both quantitative and qualitative data. The PPSRI evaluation framework which depicts in detail the indicators, methods, and sources of data collection employed for each evaluation question is presented at the end of this section.

Data Collection and Analysis Activities46

Document review and data analysis

CIHR-III documents related to the mid-term evaluation as well as the reports on PPSRI activities and outcomes (available on CIHR-III website) were consulted for this evaluation. CIHR Electronic Information System (EIS) data analysis was conducted when necessary.

On line/telephone survey

All PPSRI-funded PIs and COPIs/OPIs were invited to complete an on line survey. Telephone follow-ups were conducted to remind researchers to complete the survey or assist them in completing the survey over the phone. The short questionnaire captured information about: project outputs, capacity building, knowledge translation, partnerships and collaboration, and general attitudes towards the PPSRI. Responses were received from 40 NPIs and 108 COPI/OPIs providing an overall margin of error of +/-6.1% (Table 9-1).

Table 9-1: Responses to the Online Survey

Person Role Original Sample IneligibleFootnote XVIII Eligible Complete (R) Percent Gross RR (R/Original Sample) Percent MRIA RRFootnote XIX

Footnotes

Footnote XVII

Ineligible respondents included those who had left their position, had invalid email addresses (where a telephone number was not able to be found), or were unavailable during the survey period.

XVIII

Footnote XIX

Marketing Research Intelligence Association Response Rate = Complete/Eligible and is the industry-wide standard for market research and program evaluation.

XIX

NPI 66 4 62 40 61% 65%
COPI/OPI 285 24 261 108 38% 41%
Total 351 28 323 148 42% 46%
Key-informant Interviews

Eighteen in-depth interviews with key informants were conducted. The key informants were asked to answer somewhat different questions due to the nature of their roles, therefore unique interview guides were devised for each group:

Case Studies

Two case studies were conducted with PPSRI-funded projects in order to better understand the accomplishments that were achieved through the PPSRI-supported grants, and the factors that contributed to the projects’ successes. Case studies involved interviews with the NPIs, partners/research users and trainees involved in the projects. Eight interviews were conducted to complete the two case studies.

Bibliometric analysis

The bibliometric study provided data on the scientific productivity and impact of the funded PPSRI researches (N= 345, including 61 NPIs and 284 COPs), compared with other Canadian researchers and other researchers publishing in the top most productive countries in the pandemic field for the period 2001-2010. The search was done using the words “influenza” and “pandemic*” in their titles to search the Thomson Reuters’ Web of Science (WoS) and the US national Library of Medicine PubMed database, as well as all the papers published in the journal Influenza and other Respiratory Viruses. Indicators used for this analysis were: Average of relative citation (ARC), Average of relative impact factor (ARIF), Specialization Index (SI), collaboration rate and networking analysis.

Limitations

One challenge encountered in data collection was obtaining information from PPSRI partners through the key informant interviews. Due to personnel changes in partner organizations, many of the partners contacted had not been involved in the early stages of the project. Others had not received CIHR reports and so had limited awareness of the project’s achievements.

In order to assess the impact of PPSRI which had a relatively short life span, 2006-2011, and considering that this evaluation was conducted right after the initiative ended, some of the research outcomes and impacts (e.g. publications, citations) may not be fully covered, as they are ongoing.

PPSRI Evaluation Framework

Evaluation Question Indicators Methods and sources of data collection
Program Performance
Knowledge creation
1. To what extent has PPSRI had an impact on the development of research in the pandemic preparedness field in Canada and internationally?
  • Scientific outputs of PPSRI-funded researchers:
    • Total number of publications
    • Average of relative citations (ARC)
    • Average of relative impact factors (ARIF)
    • Research Intensity (specialization index)
    • Research productivity (paper per researcher)
    • Percentage of Canadian papers
    • International and inter-institutional collaboration
    • Comparison of scientific impact of PPSRI-funded researchers with national and international averages over time
Bibliometric analysis
  • PPSRI-funded researchers
  • Canadian researchers in relevant fields
  • International researchers in relevant fields
  • # of publications resulting from PPSRI funding
  • Influence and major contributions of PPSRI towards the development of the research fields of pandemic preparedness in Canada and internationally
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
  • Partners and stakeholders
  • Profiles of high impact research resulting from PPSRI-funded projects
Case studies
  • PPSRI-funded researchers
  • HQPs
  • Partners and stakeholders
Capacity building
2. To what extent has PPSRI built capacity to conduct research and train new researchers in the area of pandemic preparedness?
  • Number of investigators involved in conducting PPSRI research: pre-post assessment of how many researchers were working in the field of pandemic preparedness
  • Number of students and HQP trained and mentored in the area of pandemic preparedness
  • Extent to which PPSRI contributed to the reorientation of the research focus for PPSRI-funded researchers, students, and HQP.
  • Extent to which PPSRI-funded researchers, students, and HQP attribute their career advancement to their involvement in PPSRI
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
Case studies
  • PPSRI-funded researchers
  • HQPs
Knowledge translation
3. To what extent has PPSRI facilitated the translation of knowledge into practice?

Extent to which knowledge was translated to key stakeholders including the research community, media and parliamentarians for rapid and effective integration of research findings into pandemic preparedness strategies:

  • Number of and type of research users (i.e. pharmaceutical companies, Government, NGO, etc.) involved in PPSRI projects
  • Degree of involvement of the research users in the specific PPSRI projects (i.e. integrated KT)
  • Extent to which PPSRI-funded research produced KT and commercialization-related outcomes, e.g.:
    • New vaccines/drugs
    • New practices
    • Patents
    • New policies/procedures
  • Extent to which research users (i.e. pharmaceutical companies, Government, NGO, etc.) have been influenced by PPSRI-funded research
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
  • PPSRI management/Task Group
  • Partners and stakeholders
Case studies
  • PPSRI-funded researchers
  • HQPs
  • Partners and stakeholders
Collaboration
4. To what extent has the PPSRI facilitated collaboration and networking among researchers conducting pandemic preparedness research?
  • # of workshops and events organized by III to promote the PPSRI
  • # of stakeholders involved by III in these workshops and events
Document review
  • Program records, strategic plans, mid-term reports, etc.
  • Workshops attendees’ evaluation forms
  • The extent to which PPSRI-funded researchers collaborated successfully on pandemic preparedness research with other researchers and knowledge users within Canada and internationally
  • The extent to which PPSRI-funded researchers participated in committees/networks
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
Partnership
5. Has PPSRI generated sustainable partnerships?
  • Number of partners attracted by the PPSRI
  • Diversity of partners
  • Funds committed by partners
  • In–kind support from partners
  • Duration of partnership
  • Partners’ feedback about working together with III on PPSRI
  • Identified strengths and challenges for PPSRI partnership
Document review
  • Program records, strategic plans, mid-term reports, etc.
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
  • PPSRI management/Task Group
  • CIHR Institute SDs/Ads
  • Partners and stakeholders
Leveraging additional funds
6. To what extent has PPSRI enabled funded researchers to leverage additional or subsequent funding and in-kind support?
  • Extent to which researchers report that their PPSRI funding contributed to leveraging additional/subsequent funding in terms of:
    • # of grants/awards received
    • $ amount
    • Source (organization)
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
Program design and effectiveness
7. Was the PPSRI designed and delivered effectively?
  • Assessment of the cost-effectiveness of the competition and awards management process
  • Identified strengths and challenges of PPSRI
  • Potential improvements to the design and implementation of future initiatives
  • Unanticipated outcomes that resulted from PPSRI
Survey
  • PPSRI-funded researchers
Key informant interviews
  • PPSRI-funded researchers
  • PPSRI management/Task Group
  • Partners and stakeholders
Document review
  • Program records, strategic plans, mid-term reports, etc.
Case studies
  • PPSRI-funded researchers
  • PPSRI management/Task Group
Program Relevance
8. Has PPRSI addressed the need for pandemic preparedness research in Canada?
  • PPRSI’s appropriateness to address the need for pandemic preparedness research in Canada
Document review
  • PPSRI mid-term evaluation
  • PHAC evaluation
  • PPSRI report on outcomes - 2010
Survey
  • PPSRI-funded researchers
  • Canadian influenza researchers
Key informant interviews
  • PPSRI-funded researchers
  • Canadian influenza researchers
  • Partners and stakeholders

Appendix 3: Research outputs per $100,000 invested - Comparison of PPSRI, RMNI and OOGP by type of funding mechanism

PPSRI RMNI OOGP
Overall without IRN and other devel. & pl. grantsFootnote XX Catalyst Grants Operating Grants Team Grants Overall RMNIFootnote XXI Catalyst Grants Team Grants OOGP GrantsFootnote XXII

Footnotes

Footnote XX

This analysis does not include Influenza Reaseach Network and other development and planning grants.

XX

Footnote XXI

Source: The RMNI evaluation, the results are based on a survey with RMNI funded researchers held in 2011, using dollar amonts spent for the grants at the survey time.

XXI

Footnote XXII

Source: Research Reporting System (RRS) Pilot Survey- for grants awarded over the period 1991-2008; money based on grant commitments.

XXII

Footnote XXIII

17 cases excluded due to no response identified across all categories of research staff and trainees.

XXIII

Dollar range of grant value Majority < $250K Majority < $250K All > $500,000 Majority < $250K $1-2 M Average grant $300K
Grant average duration 1 year 2 years 3 years 2.5 years 4.5 years 2.5 years
Grants effective date & expiring date 2008-2011 All between 2008-2010 All between 2008-2010 All between 2008-2011 2004-2010 All between 2004-2010 All between 2004-2010 All between 1991-2008
Number of funded research grants sampled 40 17 18 5 26 12 14 440Footnote XXIII
Dollar amount invested over lifespan of projects $14,460,651 $3,838,934 $4,151,488 $6,470,229

$22,237,859

$1,967,549

$20,270,309

$118,090,449Footnote XX

Number of research staff and trainees involved 446 275 102 69 472 84 388 3,019
Number of refereed journal articles published 153 54 53 46 315 24 291 3,300
Number of grants and awards leveraged 69 35 27 7 89 23 66 -
Dollar amount of grants and awards leveraged $16,438,528 $7,902,972 $7,464,279 $1,071,277 $39,366,929 $10,270,288 $29,096,641 -
Per $100K:
Number of research staff and trainees involved 3.0 7.2 2.5 1.1 2.1 4.3 1.9 2.6
Number of refereed journal articles published 1.1 1.4 1.3 0.7 1.4 1.2 1.4 2.8
Number of grants and awards leveraged 0.5 0.9 0.7 0.1 0.4 1.2 0.3 -
Per 1$:
Dollar amount of grants and awards leveraged $1.14 $2.06 $1.80 $0.17 $1.17 $5.22 $1.44 -

Appendix 4

Tableau 6-1 : Analyse des produits de recherche de l'IRSCIP par tranche de 100 000 $ d'investissement, selon le thème des IRSC

PPSRI
Biomedical ClinicalFootnote XXIV Health systems/services Social/ Cultural/ Environmental/ Population Health

Footnotes

Footnote XXIV

Without Influenza Research Network - $ 13,974,459. One project had no pillar specified; $768,172.

XXIV

Number of funded research projects sampled 23 4 6 6
Dollar amount expended (at time of survey) $8,234,596 $1,113,864 $1,349,113 $2,994,906
Number of research staff and trainees involved 145 45 153 79
Number of refereed journal articles published 65 7 41 35
Number of grants and awards leveraged 17 10 29 13
Dollar amount of grants and awards leveraged $7,729,279 $2,594,000 $4,253,972 $1,861,277
Per $100K:
Number of research staff and trainees involved 1.8 4.0 11.3 2.6
Number of refereed journal articles published 0.8 0.6 3.0 1.2
Number of grants and awards leveraged 0.2 0.9 2.1 0.4
Per 1$:
Dollar amount of grants and awards leveraged $0.94 $2.33 $3.15 $0.62

Table 6-2: Research outputs per $100K invested analysis for PPSRI projects by area of research

PPSRI
Vaccines and immunizationFootnote XXV Virus biology diagnostics Prevention and treatment Ethical, legal or social aspects Other areas

Footnotes

Footnote XXV

Without the Influenza Research Network.

XXV

Number of funded research projects sampled 10 10 12 7 1
Dollar amount expended (at time of survey) $5,128,601 $3,585,803 $2,707,436 $2,270,639 $768,172
Number of research staff and trainees involved 89 68 201 64 24
Number of refereed journal articles published 28 44 54 22 5
Number of grants and awards leveraged 12 18 31 8 0
Dollar amount of grants and awards leveraged $3,841,277 $5,489,279 $6,818,972 $289,000 $0
Per $100K:
Number of research staff and trainees involved 1.7 1.9 7.4 2.8 3.1
Number of refereed journal articles published 0.5 1.2 2.0 1.0 0.7
Number of grants and awards leveraged 0.2 0.5 1.1 0.4 0.0
Per 1$:
Dollar amount of grants and awards leveraged $0.75 $1.53 $2.52 $0.13 $0.00

Footnotes

Footnote 1

The five-year term of the initiative has ended, with the exception of Influenza Research Network (IRN) which was renewed in 2012 for another three-year term.

1

Footnote 2

PPSRI funded projects are described in Appendix 1 – Program background.

2

Footnote 3

For further details on the TBS policy suite see: Evaluation Policy Suite (Effective as of April 1, 2009).

3

Footnote 4

There is no overarching objective statement for the whole PPSRI; rather objective statements were prepared for each type of grant (also called “funding mechanism”). A complete description of the evaluation methodology is provided in Appendix 2.

4

Footnote 5

See the methodology used for estimation on Table 1-2 of this report.

5

Footnote 6

Institute of Infection and Immunity (III) – About the Pandemic Preparedness Strategic Research Initiative (PPSRI).

6

Footnote 7

For a complete description of the bibliometric analysis methodology see the Technical Report - Methods for the compilation of bibliometric indicators on the pandemic influenza field.

7

Footnote 8

A Mann Whitney test was applied between two groups (Funded PPSRI, n = 112, arc = 1.30; Canada without PPSRI, n = 71, ARC = 0.68), with p < 0.05.

8

Footnote 9

Bibliometric analyses in this report are based on data for publications produced by CIHR funded researchers while supported by PPSRI grants. This data should be interpreted as these grants have a significant contribution to the research output (e.g. Campbell et al, 2010), and not an actual attribution.

9

Footnote 10

A non-parametric test Kruskal–Wallis was applied to test the statistical significance, with p < 0.05. No statistical significant difference by pillar, grant type or research area.

10

Footnote 11

For a complete description of each funding mechanism see Program Background section.

11

Footnote 12

A non-parametric test Kruskal –Wallis was applied to test the statistical significance, with p < 0.05.

12

Footnote 13

The RMNI is a CIHR strategic initiative of over $80M to support multi/transdisciplinary and high-impact research, funded between 2004-2010, which was evaluated in 2012.

13

Footnote 14

Influenza Research Network, one of the funded projects has 2009-2012 as the funding period.

14

Footnote 15

“Applications must include requests for trainees at the Master's, Doctoral and/or Fellowships level. Trainees funded through these grants must be eligible to receive CIHR training awards”. (Operating Grant: Pandemic Preparedness Research - Influenza Diagnosis, Transmission, Ethics Review and Antivirals, Team Grant: Pandemic Preparedness - Influenza Transmission and Prevention, Team Grant: Pandemic Preparedness - Influenza Biology, Vaccines, Ethics, Legal and Social Research).

15

Footnote 16

Some NPIs had more than one project funded through PPSRI, but for the purpose of this survey they were asked to report about two of their most important projects.

16

Footnote 17

According to the online survey, the Influenza Research Network included 73 investigators, or 13% of all investigators funded under the PPSRI. However the project’s NPI was unable to estimate the number of the undergraduates and fellows not pursuing a degree.

17

Footnote 18

Data not shown. A non-parametric test Kruskal–Wallis was applied to test the statistical significance, with p < 0.05.

18

Footnote 19

According to OST bibliometrics study, this is an estimation of the number of Canadian authors publishing in the area of pandemics using the following assumptions: when all the addresses in a given paper are those of Canadian institutions, we can assume that all the authors of this paper are Canadian, when the first address is a Canadian one, we assume that the first author is a Canadian, the address of the corresponding authors gives the specific address for at least one author on several papers. Also, several Canadian authors have been identified thanks to the publication files we already have reconstituted for many Canadian authors. Finally, for paper published in 2008 and after, the links provided by Thomson were used. It should be mentioned that these estimates were produced in order to give a rough idea of the evolution of the number of Canadian authors in the area of pandemic and that they were not fully validated. However, the methodology used here (described above) is reliable and we should emphasise that it cannot produce an overestimation. If there is an error, the only possible one is an underestimation.

19

Footnote 20

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009, p. 36.

20

Footnote 21

Idem.

21

Footnote 22

See Appendix 3 for number of students trained by type per project compared by grant characteristics. A non-parametric Kruskal–Wallis test was applied to test the statistical significance, with p < 0.05. No statistically significant difference by pillar, grant type or by area of research.

22

Footnote 23

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009.

23

Footnote 24

CIHR - More About Knowledge Translation at CIHR - Retrieved on May 2012.

24

Footnote 25

Pandemic Preparedness Strategic Research Initiative – Report on Activities & Outcomes - Retrieved on May 2012.

25

Footnote 26

Data not shown.

26

Footnote 27

Idem.

27

Footnote 28

Idem.

28

Footnote 29

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009, p. 40.

29

Footnote 30

Pandemic Preparedness Strategic Research Initiative - Report on Activities & Outcomes (June 2008 - March 2009).

30

Footnote 31

Pandemic Preparedness Strategic Research Initiative (PPSRI) Collaborations - Retrieved on May 2012 from: Pandemic Preparedness Strategic Research Initiative (PPSRI) Collaborations.

31

Footnote 32

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009. (p. 24).

32

Footnote 33

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009.

33

Footnote 34

Of the remainder, 30% of NPIs and 27% of COPI/OPIs stated that their PPSRI grant had not helped them secure additional funding. The remaining NPIs and COPI/OPIs either did not know or said the question was not applicable to them.

34

Footnote 35

Report on the Pandemic Preparedness Research Initiative, III, CIHR.

35

Footnote 36

Institute of Infection and Immunity, Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR, 2009, (p. 27).

36

Footnote 37

Although annual reports were intended delays were reported and at the time of the interviews reports were overdue.

37

Footnote 38

An additional 3% did not know.

38

Footnote 39

CIHR’s Mission – Retrieved on December 2012.

39

Footnote 40

CIHR CIHR’s Strategic Plan (2009-2014): Health Research Roadmap: Creating innovative research for better health and health care.

40

Footnote 41

PHAC implemented the Avian and Pandemic Influenza Preparedness Initiative to strengthen the federal government’s capacity to prepare for and respond to two major inter-related animal and public health threats – the possibility that a highly-pathogenic strain of avian influenza would spread to wild birds and domestic fowl in Canada and that a human-adapted strain of this (or another new) virus would arise triggering a human influenza pandemic (PHAC, 2011).

41

Footnote 42

About the Pandemic Preparedness Strategic Research Initiative (PPSRI) - Retrieved on May 2012.

42

Footnote 43

Other government departments and Not-for-Profit Organizations, see a complete list on Table 4-2 of this report.

43

Footnote 44

This refers to HQP and researchers that were involved in the projects but not listed in the funding application.

44

Footnote 45

Request for Applications (RFAs) for different types of grants, retrieved from CIHR website on April 2012:

45

Footnote 46

A more detailed description of the data collection methods employed by this evaluation is available in the technical reports prepared by R.A. Malatest and Associates Ltd. and Observatoire des sciences et des technologies (OST).

46

Date modified: