Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative
3.1 Effectiveness of priority setting
Effectiveness of PPSRI research priority setting processes
Overview of the steps in priority setting
Priority setting workshop. The process of priority setting for the PPSRI began with an Influenza Research Priorities Workshop, held by CIHR in collaboration with the Public Health Agency of Canada (PHAC) and the Canadian Association for Immunization Research and Evaluation (CAIRE) in September 2005. The workshop was attended by 66 individuals from Canada, the United States and Europe, representing researchers, public health systems and organizations at every level, and the private sector. It produced a series of 10 priority areas, with a rationale and an analysis of the infrastructure required for each. The themes areas were:
- Influenza virus characterization and ecology
- Influenza virus transmission
- Public health preventive measures
- Improving rapid diagnostic tests
- Clinical management of influenza patients
- Development and optimal use of antiviral drugs
- Surge capacity of the health care system
- More effective and acceptable influenza vaccines
- Immunization programs
- Preparation for a pandemic vaccine
The key subsequent step identified in the workshop was for CIHR and PHAC, in 2006, to "develop strategic approaches for funding mechanisms and opportunities and national/international liaison for identified research priorities".
The evaluation results for the workshop were quite positive, with for example, 34 out of 44 questionnaire respondents agreeing that the overall objectives of the workshop had been achieved and distinct next steps were identified. A need to further synthesize and prioritize within the list of ten priority areas was mentioned in the evaluation report, based on comments that the workshop-generated list was too exhaustive for the level of funding available.1
Task Group. The Task Group was constituted in June 2006 and met eight times between June 16 and August 31. It was initially composed of four members, including a member of III's IAB, two other researchers, and a representative from PHAC. The Task Group identified a need for additional expertise in social science, health services research and ethics, resulting in the addition of another researcher. III's Scientific Director was an ex-officio member, and five CIHR and PHAC staff members also participated in meetings.
Following on the Influenza Research Priority Workshop, the Task Group's main task was to develop a strategic set of research priorities for the PPSRI. This was accomplished over several meetings. An initial large list with eight categories2 was refined based on members' comments and on consultations with key agencies, including the Rx&D Heath Research Foundation; Canadian Food Inspection Agency: the International Development Research Centre – Asia Research Partnership on Pandemic Influenza.3 The Task Group then sent a draft of the Pandemic Preparedness Strategic Research Priorities for consultation to the Canadian Rapid Research Response Team (C3RT)4 and C3RT ad hoc members on pandemic preparedness, a total of 21 individuals representing 12 organizations. Sixteen responses to the consultation request were received, with a majority of respondents agreeing that each of the draft areas identified by the Task Group was a priority for Canadian research5. This resulted in the priority areas listed in section 1.2.1., some of which regrouped two or more of the categories identified in the workshop, and which included an explicit focus on capacity building.
Stakeholders' view of the priority setting process
Key informants interviewed for this evaluation were generally of the opinion that the priority setting process had been effective. The inclusivity of the process, in particular that of the 2005 priority setting workshop, was cited very positively as having allowed input from not only the core groups of researchers and stakeholders who had been historically associated with influenza research in Canada, but other researchers and stakeholders "outside the regular group". The only exception to inclusivity mentioned by key informants were a possible lack of adequate representation from health services and population health research, and a focus on human, as opposed to animal health or human-animal interaction. The timing of this workshop was also mentioned positively, as it coincided with reflection and mobilization being undertaken by partner organizations, notably PHAC and the Canadian Food Inspection Agency (CFIA). At the same time, a drawback to the inclusivity was also noted, in that the resulting priorities were very broad, "more like categories than priorities"; "all we really did was identify all the unanswered research questions for influenza". While ensuring that most of the potential research community could see themselves within these priority areas, some respondents argued that they were not sufficiently narrow to accomplish strategic targeting of investments. This was echoed in one peer review committee's comments.6
Findings from the interviews as well as from the document review7 suggest that the overall strategy adopted for the PPSRI for setting priorities tended toward risk aversion, seeking to support work that would produce high-impact results in the short term. The broadness of the priorities was also seen as a conservative strategy, in that by placing fewer limits on the fundable fields, the possibilities of achieving breakthroughs in knowledge could be maximized. This seems to have been responding to both the potential urgency of preparing for a pandemic as well as the time constraints on funding.
Appropriateness and effectiveness of the Task Group
Key informants who had been involved with or members of the Task Group noted that it had been formed based on recommendations from III's IAB, recognizing the need to involve all four CIHR pillars, including clinical and social science research. They agreed that its composition was appropriate.
One key informant noted that there had been quite a long interval between the priority setting workshop and the launch of funding opportunities, suggesting that this had somewhat disappointed the high expectations at the end of the workshop. It was noted by others, however, that the workshop had been held before funding was confirmed, and due to the time required to obtain Treasury Board approval, the process had moved as quickly as could have been expected.
Key informants familiar with the Task Group's processes were of the view that the group had functioned effectively, contributing an intense effort over the summer of 2006. Task Group members observed that although they were specialists in different areas, they took their role seriously and pulled together so as to produce a priority statement that reflected collective interest for pandemic research and not just their own. For some, an overarching driver was not repeating problems that had been experienced during the Canadian SARS outbreak and research response.8 Stakeholders external to the process praised its results and particularly the efforts of III staff in terms of developing connections with partner agencies that led to the production of a coordinated research agenda.
Conflict of interest in priority setting
Although not a majority view, a concern raised in the key informant interviews was of possible perceived or actual conflict of interest in the priority setting process, in that some people who had had significant input into the priorities would also be likely to receive funding under these priorities. For example, one Task Group member commented that it was unusual from his/her organization's perspective for researchers who would likely be receiving funding to be involved in directing priorities. This was also of concern to III: because of the small size of the Canadian research community in this area, researchers who would be eligible for funding also participated in the priority setting workshop, the Task Group and the peer review committees (in the latter two cases, following the usual CIHR conflict of interest guidelines)9. Thus, one of the elements assessed in the researcher survey was respondents' perceptions of the conflict of interest in the design of the initiative and the adjudication of the applications.
The survey results showed that 23% of respondents stated that they did not know whether there was conflict of interest in the way the PPSRI was designed, and 40% did not know whether there was conflict of interest in the way it was adjudicated. This is not surprising, as most researchers would not have been directly involved in either of these processes. Among those who were able to answer, just under 15% agreed that there had been conflict of interest in the program design or adjudication processes (Table 7). As might be expected, unfunded researchers' views were more negative than funded researchers' (data not shown because of small Ns). Although it is easy to dismiss unfunded researchers' views as reflective of their disappointment in the funding decisions, given the relative size of the unfunded group, any level of perceived conflict of interest is perhaps of concern. However, overall, close to 85% of the PPSRI research community does not perceive there to have been conflict of interest.
Relevance and adequacy of the priorities chosen
Key informants noted that the set of priorities retained for the PPSRI adequately and "respectfully" reflected the priority setting process, and that all the priorities included were important. Several respondents again commented on the broadness of the priority areas, noting that possible consequences may have been first, to spread funding too thinly, and second, to lessen its potential impact by failing to focus on specific and highly targeted research questions. At the same time, respondents noted some gaps in the priorities. It was also pointed out that the coverage of the priority areas in the actual distribution of grants was not even, so that some areas received more funding than others. According to key informants, this may have been a reflection of existing capacity: within the five-year timeframe, it was not possible to stimulate research in areas where applications where not received, and so proportionately more grants were awarded in areas where applications were received and passed peer review. As well, as suggested in key informant interviews, the peer review process in funding opportunities that involved multiple disciplines may have contributed to uneven allocations. Ethics was an area mentioned by some key informants as having received a perhaps inordinate amount of research attention, although others applauded its inclusion and stressed its importance.
In the researcher survey, 13 funded researchers (16%), 18 unsuccessful applicants (40%), and two non-applicants (total 33, 24% of the total) agreed that important research priorities had not been included in PPSRI. These priorities included research on (listed in order of frequency of mention):
- immune systems and immunization;
- specific methodologies, including modeling;
- international issues in pandemics or involving international collaborations;
- surveillance systems and applied epidemiology;
- animal-human transmission;
- epidemic management; and
- behavioural, social or ethical dimensions of pandemic.
These areas are similar to the gaps mentioned by key informants, which included: animal/human interaction and animal issues; surveillance research, clinical research on prevention and treatment; and legal issues.
Among the priority gaps identified by both researchers and by key informants are themes that had been part of the initial Task Group discussions10 but which were not directly emphasized in the final PPSRI priorities: research on epidemic management, animal-human transmission, and modeling. The latter two were also mentioned in key informant interviews as having been considered by the Task Group and excluded from the funded research.
Researchers were also asked which priorities are the most important for pandemic preparedness research over the next five years. They were asked to rate the importance of the existing PPSRI priorities, as well as suggest other priority areas and rate their additions to the list. The data shown in Table 8 confirm relatively strong support for the existing priorities, with "Prevention and treatment" receiving the strongest endorsement. One half (50%) of these respondents (of whom 39% declared themselves to be engaged in these research areas) endorsed "Ethical, legal or social aspects" of pandemic preparedness as being of high priority.
A number of other priority areas (some of which could be considered subareas of the existing ones) were also identified by 60 (38%) of respondents. These were regrouped into several categories. The most prevalent category was research related to aspects of influenza and pandemic management from a public health perspective; i.e., the areas identified in the priority setting workshop as "public health preventive measures" (defined as non-pharmaceutical, interventions to reduce exposure to influenza and avoid infection) and "surge capacity" (defined as capacity of the health care system to respond to unprecedented demand in a pandemic).11 This area may also include research on real-time surveillance and surveillance capacity, also mentioned by several survey respondents, and identified in key informant interviews as a gap. Research priorities relating to use of specific methodologies were also frequently mentioned, as were research on behavioural, social or ethical dimensions of pandemic. Basic immune system research was identified as a priority by several respondents ("Effects of treatment on immune responses to infection" was included as an element of the priority on development of effective antivirals in the priority setting workshop12, but not mentioned in the funding opportunity).13 Global/international issues and environmental issues were also seen as emerging priority areas. Appendix 5 provides the complete list of priorities for the next five years identified through the researcher survey.
National and international duplication of effort versus complementarity
Stakeholders who commented on this issue agreed that the PPSRI has generally succeeded avoiding duplication of effort in funding of pandemic research: there appeared to be no outright duplication and little direct overlap among the research priorities themselves, and good coordination of funding efforts with some national agencies due to partnerships established (discussed further below). Through the international collaborative research it funded, the PPSRI allowed researchers to work in complementary areas. However, the PPSRI did not contribute directly to international complementarity of research funding. Key informant interviewees reported that there had been significant challenges in trying to share research agendas across countries: "we tried on a couple of fronts (vaccine development and antiviral) to share information, leverage and coordinate internationally with some countries, but it was impossible to coordinate and keep track of things like trials and the hundreds of pieces of research going on." The PPSRI did benefit from contacts with American research agencies in the same area, becoming familiar with their research agendas and having this inform the Canadian initiative.
3.2 Effectiveness of partnership development
Success in building national and international partnerships
Through the consultation process with stakeholders led by the Task Group, nine organizations indicated interest in partnering in one or more of the research areas14. These were:
- American Red Cross
- Canadian Food Inspection Agency (CFIA)
- Association of Medical Microbiology and Infectious Disease Canada / Canadian Foundation for Infectious Diseases
- Canadian International Development Agency (CIDA)
- Emerging Infectious Disease Research Network
- First Nations and Inuit Health Branch, Health Canada FNIHB)
- Rx&D (an association of Canada's research-based pharmaceutical companies) Health Research Foundation (HRF)
- International Development Research Centre (IDRC)
- Public Health Agency of Canada (PHAC).
Some organizations were interested in partnering to carry out research while other organizations were interested in partnering to support research. Formal partnerships involving joint funding opportunities were developed with PHAC, CFIA, HRF and IDRC. These organizations were most ready to develop partnerships, as they had either worked with CIHR on the priority setting workshop in 2005 or had already developed their own pandemic research agendas or priorities. Within CIHR, several institutes and portfolios also partnered in the PPSRI, participating in joint funding opportunities: CIHR Institute of Aboriginal Peoples' Health (IAPH); CIHR Institute of Health Services and Policy Research (IHSPR); CIHR Institute of Population and Public Health (IPPH), CIHR Knowledge Translation Branch; the Ethics Office of CIHR, and CIHR Partnerships and International Relations Branch. A research funding partnership was formalized with the Mathematics of Information Technology and Complex Systems (MITACS: a Network of Centres of Excellence (NCE) for mathematical sciences) for the 2006 Operating Grant funding opportunity, but no joint funding occurred with this organization.
Impacts of partnerships
Coordination and integration of national research programming
The PPSRI's partnerships with PHAC, CFIA and HRF resulted in joint funding of research activities. Stakeholders interviewed from these external partner organizations viewed these partnerships as very successful, noting several advantages gained from their perspectives. These included ensuring the partner's research funds would support research with clear, carefully developed priorities; and bringing partners together in a coordinated initiative that avoided duplication of effort. Capitalizing on CIHR's peer review system was also seen as an important benefit of the partnerships, as it ensured that the research would undergo rigorous review, without the partners having to develop the necessary infrastructure and expertise. These findings suggest that the PPSRI was effective in producing coordination and integration of national research programming, as well as avoiding duplication.
Although IDRC partnered on a funding opportunity for joint research on pandemic preparedness by Canadians and researchers from developing countries, the uptake on the funding opportunity was such that no awards were made. Although it praised the PPSRI's intentions and efforts to find common ground, IDRC felt that it had not been involved early enough in the funding opportunity development process to ensure that it could be oriented towards its research agenda and the specific needs of capacity development in developing countries. From its point of view, the resulting applications were too biomedical in orientation to garner IDRC support, and did not fulfill IDRC's aim of partnering foreign researchers in lower-capacity settings with Canadian researchers who had experience in developing countries as equal partners. According to key informants, this would have required consultation with developing countries' research and policy communities, as well as a capacity/building proposal development process to forge relationships among researchers who had not worked together before. The time constraints of the PPSRI, along with a complicated process to develop a Memorandum of Understanding between CIHR and IDRC, rendered these steps impossible. From III's point of view, the partnership presented challenges in terms of the difference in mandates of the two organizations: while they shared an interest in pandemic preparedness research, CIHR's mandate focuses on research excellence regardless of setting, while IDRC's focuses on strengthening research capacity in developing countries. Finding and operationalizing the two agency's common ground may have required a longer period than was available within the time pressures to launch PPSRI funding opportunities.
The partnership with PHAC was regarded by key informants as having been very successful, and a significant advance over the research experiences encountered with SARS. PHAC representatives reported having benefitted from the close collaboration with CIHR on the PPSRI, through organizational learning about research processes. PHAC's involvement with the Task Group, the priority setting processes and the structuring of funding opportunities were key components for ensuring coordination and integration of their joint research programming. At the same time, some key informants outside of PHAC expressed concern about the extent to which PHAC's research agenda had been adequately addressed through the PPSRI, because of the strong investigator-driven orientation of influenza research in Canada, and because of the small size of the influenza research community. Sharing this concern, PHAC reserved some of its research funds for directed research under contract. The Influenza Research Network funding opportunity was also seen by some key informants as providing a necessary focus on research efforts to ensure that those efforts will contribute directly to "on-the-ground" pandemic preparedness, i.e., PHAC's preparedness mandate.
There was consensus among key informants from external agencies – including IDRC and PHAC – that the PPSRI had developed an effective partnership structure. Its success was attributed largely to III's approach to developing the partnerships: stakeholders praised its success in bringing organizations together who had not partnered on research before– an "incredible accomplishment'. Partners were especially appreciative of III's approach in listening and responding to partners' needs, and its willingness to adjust its processes and projects to accommodate partners' interests: "their willingness to include other objectives that were important for us as partners – we clearly felt that they wanted to collaborate with us... a true partnership rather than just using us as a bank". Cited here, as examples of III's effectiveness in partnership, were the relevance reviews for several PPSRI funding opportunities that involved partners, and processes whereby partners could ask for clarifications from applicants. III representatives acknowledged that adjustments to their processes had been made so as to ensure that partners could and would remain involved. The Canadian Pandemic Preparedness Meeting: From Discovery to Frontlines meeting held in November 2008, a component of the PPSRI's Knowledge Translation Strategy (see section 3.5) was also cited as an exemplar of partnership process to develop coordination and integration of research programming, although mainly at the national level.
Indeed, above and beyond the experience with IDRC, international partnership structures were seen by key informants as not having developed as far as would have been desired. Although some discussions with international agencies were described positively, key informants reported that there had been no real response from the WHO despite it having been approached, and that significant barriers existed to using the existing channel of CIHR's China-Canada initiative. It was noted that because the contexts of each country's research are so different, tending to focus on their own internal issues, developing collaboration takes time, and likely more time than available to a five-year initiative such as the PPSRI.
As can be seen from Figure 3, partners' direct contributions resulted in doubling the resources available for pandemic research. Partners' allocations account for 51% of all PPSRI funds over the period 2006/07 to 2012/13. External partners will contribute 45% of the total funds, while internal partners (other Institutes and units of CIHR) will contribute 6%. These contributions may be underestimated, as they do not include in-kind contributions and only includes those funds administered through CIHR. Thus, the PPSRI has resulted in significant resource leveraging for pandemic preparedness research.
Figure 3: PPSRI and Partner Contributions to Pandemic Research through CIHR
Data provided by III, as of December 2008.
As noted above, funding partners were keen to contribute research funds to the PPSRI, to achieve leveraging from their perspective especially in benefiting from CIHR and III's infrastructure and to ensure a coordinated research effort.
The PPSRI's reporting strategy aimed to keep national and international partners transparently apprised of developments in the initiative. However, few key informants were aware of the reports or other elements of the strategy, and so could not comment on its effectiveness.
As of March 30, 2009, the III website has a link to the following PPSRI publications:
- Proceedings of the Influenza Research Priorities Workshop (September 2005): [ PDF (557 KB) | Help ]
- PPSRI Application Development Workshop Report (March 2007): [ PDF (425 KB) | Help ]
- Pandemic Preparedness Strategic Research Initiative: Report on Activities and Outcomes (June 2006 - June 2008): [ PDF (943 KB) | Help ]
- Report on the Influenza Research Network Application Development Workshop February 2008): [ PDF (1.2 MB) | Help ]
3.3 Appropriateness of program design
Success of the suite of funding opportunities
Key informants were asked to what extent the set of funding opportunities offered through the PPSRI was contributing to achieving the initiative's objectives. There was consensus that the program components, through the funding opportunities, were addressing the priority areas identified in the priority setting process. It was noted, however, that the time required to develop and launch the opportunities had led to a compression of the time frame for the grants from five years to a maximum of three. According to key informants, this type of initiative would benefit from a two-year planning and implementation phase followed by five years of funding. It was also again noted that the level of funding available was not commensurate with the broadness of the priority areas.
In the researcher survey, PPSRI applicants who had heard of the PPSRI (86% of those completing the survey) were asked to indicate whether a set of factors had influenced their decision to apply or not apply to the program, as a measure of incentive and disincentive features of the suite of funding opportunities (Table 9). For those who did submit applications to one or more of the funding opportunities, their qualifications (94%) and the degree of fit of their research with the PPSRI themes (93%) were the most frequently selected incentive features, indicating that the program reached those with specific expertise in the targeted areas. The expected involvement of students (43%) and the relative prestige of the program (38%) were the least influential factors. Over and above the ubiquitous reason of already having enough research funding, among the 10 non-applicants who answered these questions, the view that the program seemed to be targeting specific groups of researchers was selected as a disincentive factor by eight. This is surprising, given that these non-applicants have a previous successful history of pandemic-related research applications with CIHR. However, six also indicated that a lack of fit with their research areas was a factor in their decision not to apply. Perhaps because of this, six also indicated that their chances of success would not have been good enough to be awarded funding.[ Table 9: Incentive and disincentive features for applications: Factors in applying or not applying ]
Effectiveness of strategic competition
A question of interest to III in the context of the PPSRI but also more generally, was whether funding opportunities in an area of strategic importance and in need of rapid advances in knowledge should focus on funding recognized front-runners in the area, or allowing all researchers to compete for funds.
All researchers, including those who had not heard of the PPSRI, were asked to provide their view on this question: their responses are shown in Table 10. Across all categories of respondents, there is stronger support for ensuring that all researchers who can make a contribution have the opportunity to compete for funds, than for ensuring that recognized leaders can continue to receive funding. As the funded applicants' category is the most likely to include Canada's recognized leaders in pandemic preparedness research, this is an important difference to note. However, almost 80% of funded researchers also agree that continued funding to recognized leaders was important – compared to 53% of unsuccessful applicants. A third strategy, of increasing research capacity overall by ensuring that researchers new to the field can access funding, elicited endorsement from 65% of unsuccessful applicants, and 48% of funded applicants.[ Table 10: Researchers' agreement with types of funding strategies ]
Effectiveness of the communication strategy
When asked about the PPSRI's communication strategy, those key informants in a position to comment noted that as is typical, those researchers who are most active and involved in the area were most aware of the initiative. III staff indicated that communications about the initiative had been made through lists compiled by III of about 3,500 infection and immunity researchers and stakeholders as well as a list of researchers who had been involved in pandemic and influenza related events such as the 2005 workshop. They noted that there was a possibility that they had reached mainly people already working in the area. It was also pointed out that as CIHR has so many funding opportunities, it is hard for the average researcher to keep track of them and know where they are coming from. Review of website information on the PPSRI tends to confirm this observation: there seems to be no one location or search that will identify all the PPSRI program components; those that are part of existing CIHR initiatives are especially hard to locate.
According to some key informants, the time pressure to launch the programs also affected communications. The need to produce, translate and disseminate program materials by dates that would allow adequate time for researchers to prepare proposals obviated possibilities for conducting more outreach activities or developing more extensive mailing lists. Lack of reach to the public health community was invoked as a possible explanation for the lack of response to the Transmission, Public Health Measures and Compliance Operating Grant funding opportunity (see next section).
Data from the researcher survey tend to corroborate the need for more effective communication. As indicated above, twenty-two (14%) of the respondents, including six (7%) of the funded researchers, indicated that they had never heard of the PPSRI or did not know if they had. Unsuccessful applicants were less likely than non-applicants to have heard of the program (76% versus 83%), raising a question about whether their lack of success was tied to a lack of awareness of the parameters and evaluation criteria of the funding opportunity they applied to. Among those who had heard of the program, about one-half of the respondents (54%) funded through it agreed that the communications about the program had been adequate (10, or 13%, did not know). These results may signal not only a need for more communication about the program, but also a need to ensure that researchers' attention is drawn to the funding opportunities' specific features.
3.4 Achievement of desired funding opportunity outputs
The 29 funding opportunities offered under the auspices of the PPSRI prompted a total of 149 applications, of which 71 were funded (Table 11). These were submitted by a total of 102 different NPIs, of which 44 were involved in one or more successful applications as either an NPI or an applicant. The number of applications per NPI ranged from one to eight, and the number of successful applications from one to three.
The overall success rate was 51% for reviewed applications and 92% for fundable applications (those that were rated 3.5 or higher by the peer review committees15). This means that over all the funding opportunities, the majority of eligible and scientifically meritorious applications were funded. This rate is higher than that generally seen in CIHR funding opportunities. Given that the same quality standards were applied in the peer review process as in CIHR peer review overall (although see below for some contrasting views), this may suggest that the amount of funds available vis-à-vis the number of fundable applications was relatively high, i.e., that the funding opportunities generated a relatively low number of applications. Although this may be partly due to a lack of interest in or awareness of the PPSRI, it may also support the identified need to build pandemic research capacity.
As Table 11 indicates, some of the program components did not generate any applications. These include the first round of Knowledge Synthesis Grants; all three rounds of Meetings, Planning and Dissemination Grant: End of Grant Knowledge Translation Supplement; and the first two rounds of Meetings, Planning and Dissemination Grants. Taken together, these results could suggest a lack of readiness for knowledge translation, perhaps because the research had not yet advanced enough to be able to identify what knowledge could be transferred. Alternatively, it could suggest a low level of interest among the pandemic research community in carrying out knowledge translation projects, a lack of familiarity or comfort with strategic funding in this area, or perhaps a lack of fit of these funding tools with the types of activities knowledge translation that are relevant to pandemic preparedness and influenza.
As well, after review, no applications were funded under the Operating Grant for Transmission, Public Health Measures and Compliance. Data from both the researcher survey and key informant interviews on the PPSRI priorities confirm that this is still an area with a significant research gap.
When asked about their views on the success of program uptake, about two-thirds of key informants were not aware enough to comment. Among those who were, some, including one external partner organization, felt that the funding opportunities had generated an acceptable number of applicants, from the right populations. Others felt that more applications would have been desirable to increase competitiveness, although the lack of capacity and the need to build it through capacity development initiatives such as workshops or training was also emphasized. Some areas where uptake had been weaker were noted: these included public health (notably the non-response to the public health funding opportunity, but more generally as well); risk perception and communication; and ethics review in pandemic situations.
It was also noted by key informants that uptake was not optimal in that not all applications sufficiently reflected the targeted nature of the funding opportunities. It was suggested that perhaps because of a widespread position in the research community that investigator-driven research will address knowledge gaps, researchers sometimes do not respond appropriately to targeted funding opportunities, including submitting applications that are not within the parameters of the funding opportunities, or failing to meet requirements such as inclusion of end-users and a clear knowledge translation component in each project. This is consistent with our interpretation of the researcher survey finding that unsuccessful applicants were less likely than successful applicants and non-applicants to have heard of the PPSRI – even though they had applied to it. (It is important to note that each funding opportunity announcement is accompanied by a detailed background document, with a program description, objectives, eligibility, application guidelines, and criteria used to evaluate applications). This is also supported by a finding that a majority of unsuccessful applicants were unaware of the evaluation criteria, discussed below.
Adjudication process. Some key informants also commented on the adjudication process (perhaps more generally than the PPSRI), noting that it had included more than just the "usual suspects", i.e., had been more inclusive. At the same time, the difficulty of ensuring adequate review without conflict of interest for grants with large numbers of investigators (such as the team grants) was noted. A consequence of this, according to key informants, is that applications are reviewed by only a subset of the full peer review committee; i.e., their review is not as thorough because some members must recuse themselves. The use of only committee members' reviews, without external reviewers, was also criticized by key informants as having resulted in less than adequate reviews.
The researcher survey also included questions on the PPSRI adjudication process. Although about two-thirds of applicants were not knowledgeable enough to answer the questions, among those who could offer an opinion, the results suggest some dissatisfaction overall with the adjudication processes and communication of results:
- a minority of both funded applicants (40%) and unsuccessful applicants (17%) agreed that the PPSRI's peer review committees had adequate expertise. This is a troubling finding if it reflects a veridical assessment of committee credibility, although it may represent a generalized tendency for researchers to believe that they are the most expert in their domain.
- less than half of funded applicants (44%), who would be expected to be most satisfied with their outcomes, agreed or strongly agreed that the funding decisions were fair.
- only about half (49%) of those funded agreed that they were aware of the criteria used to evaluate their applications. This proportion is even lower, 32%, among those not funded. Although respondents may not have been able to recall if they had been aware of the criteria at the time of the review or may have forgotten them by this point, it is also important to note that the evaluation criteria are published with every funding opportunity.
- About two-thirds of funded applicants (64%) and one-third of unsuccessful applicants (32%) agreed that they had been provided with sufficient information following the adjudication process.
However, it is not possible to say whether similar results would be obtained in researchers' assessments of CIHR peer review processes more generally.
Coverage of the intended research areas
Figure 4 shows investments by PPSRI priority area, as determined by III staff in reviews of funded applications. As of September 2008, the Vaccines and Immunization priority had received about 44% of PPSRI funds, followed by Virus Biology, at 21%. Prevention and Treatment, and Ethics, Legal and Social Contract dimensions have each received about 17% and 18% of the investments, respectively.16
Figure 4: Investment by PPSRI Priority Area, committed funds as of September 2008 ($ millions)
Key informants interviewed were generally of the opinion that the intended research areas were reasonably well covered by the PPSRI's funded activities. Some gaps were nonetheless noted, including clinical work, modeling, humanities and social sciences. It was also noted by a key informant that researchers involved in modelling appear to be discouraged, as few applications in this area were funded. As noted previously, modeling was one of the priority gaps identified by some researchers.
Research capacity development
According to key informants who were able to comment on this issue, the PPSRI is providing many new opportunities for capacity development in pandemic preparedness research. This includes the involvement of graduate students and post-doctoral fellows: key informants noted that the PPSRI initiative presents many opportunities for trainees to become involved for their dissertation research or as research assistants. Respondents who had attended the Canadian Pandemic Preparedness Meeting: From Discovery to Frontlines noted the presence of trainees, citing this as a very positive example of the PPSRI's impacts on developing pandemic research capacity. According to the registration file, 15 of 172 registrants (9%) named a supervisor, i.e., were trainees. As well, a perceived influx of new researchers over the last several years was remarked upon, into what had been seen previously as "a bit of an old boys' network".
It was suggested in key informant interviews that the PPSRI could develop pandemic specific training initiatives along the lines of the Strategic Training Initiative in Health Research (STIHR)17. However, the Task Group had considered this option and decided to recommend that inclusion of trainees be a requirement on core PPSRI operating and team grant funding opportunities because of the challenges that would be faced in putting together a training program application due to the limited number of principal investigators whose research area and expertise focused on pandemic and influenza.18
The importance of the proposal development workshops as a capacity development tool was made clear in several ways by key informants' responses. It was noted that the workshops had been useful for those preparing proposals, but also as a means to engage researchers who may be involved in other areas but could be "sold" on pandemic-related research. The absence of such a process for the international component of the PPSRI was identified as a key drawback that had probably contributed to the overall lack of depth in the applications involving international collaborations.
Researchers responding to the survey were asked to indicate the extent to which their funded project was contributing to capacity development for pandemic preparedness research through engaging researchers from areas or disciplines that had not been involved before, and by shaping new research foci. These data are summarized in Table 12, for the entire sample and by area of research (the categories "vaccines and immunization" and "prevention and treatment" were combined because of the large overlap in their membership). About 60% of funded researchers agreed that the PPSRI was contributing to involvement of researchers from new areas or disciplines. And, about 30% agreed that the PPSRI had contributed to a reorientation of their research focus. Although it is hard to set a target on this latter issue, it suggests that the PPSRI has, while ensuring funding to the core population of researchers already engaged in research relevant to the PPSRI priorities, also succeeded in helping some researchers to focus their research on pandemic-related areas, and in reframing or reorienting others' work to be in line with its strategic capacity development aims.
As no other data exist as yet on the integration of trainees into PPSRI research projects, questions about student involvement were included in the researcher survey. To avoid double-counting, only NPIs were asked to respond. A total of 53 trainees were involved in the 21 grants reported on by their NPI; on average, this works out to about 2.5 trainees per grant. Table 13 also shows a projection of the number of trainees that might be estimated to be participating in the entire PPSRI. If the respondents are representative of all PPSRI NPIs, we can estimate that the PPSRI has involved about 173 (more prudently, perhaps between 150 and 200) trainees. However, 14% of the grants involved no trainees at any level.[ Table 13: Number of trainees involved in PPSRI grants ]
According to six of the responding NPIs (33%), their trainees have been involved in international research collaborations.
The data in Table 14 show that the majority of funded PPSRI researchers agree that their project has involved interdisciplinary training (64%) or mentoring (73%) of students/fellows in influenza and pandemic preparedness research. As well, about two-thirds (68%) of respondents agreed that their project has increased the number of trainees in influenza and pandemic preparedness research. Together with the data in Table 13, these findings suggest that capacity development through training in pandemic research through the PPSRI is occurring. However, as there is no baseline or explicit target other than embedding of training throughout the funding opportunities, it is hard to assess whether this can be seen as in line with expectations.
3.5 Success of PPSRI networking and knowledge translation activities
Facilitation of communication and networking
In the key informant interviews, respondents praised the PPSRI for its success in facilitating communication and networking through the organization of the priority setting workshop and the Canadian Pandemic Preparedness Meeting: From Discovery to Frontlines, describing this as "fantastic", "excellent", "effective" and a "great example of value for money". III's approach in this was described as novel and innovative. It was noted that whereas the small research community in Canada is highly disparate and diverse, these events had succeeded in bringing people together who would not have done so otherwise. This resulted in increased awareness among researchers of each others' work, as well as creating national momentum in pandemic preparedness research: "there is no question that it has created momentum into now a major national effort ... . There is excitement in the community and the PPSRI has encouraged them to continue on in the area – many people who would not have worked together." The meetings provided an opportunity for researchers to learn about other branches of research with which they were less familiar, which was described as useful.
Some key informants noted that a mix of formal (such as the From Discovery to Frontlines meeting) and informal opportunities for networking is desirable. This would seem to support the offer of funds for meetings, and underscore the question about why uptake for these has been limited.
Finally, although the momentum gathering through the PPSRI speaks to the health of the initiative, some key informants expressed concern about whether that level of vigour can be maintained if a pandemic does not materialize and after targeted funding ends.
Impacts on enhanced collaboration and increased capacity
Over and above increased awareness of pandemic research activity, key informants noted several other impacts of the PPSRI's networking activities on enhanced collaboration and increased capacity. The participation of trainees in these activities was cited as a very important tool for increasing research capacity. For research partners, benefits were also noted in terms of making connections with other relevant agencies: "While the information presented at the conference is interesting and important, half the take-home is the connections we make at the conferences – PPSRI is doing this very well... . By attending this conference, it will change the way we do our work at (organization) – our position in the Canadian landscape and the impact that we can make. I now know who to contact to make this happen." Increasing the potential for collaborations between governments and universities, and between human and animal researchers, was described as an important impact of the PPSRI. While in some cases new collaborations have emerged, key informants also cautioned that it may take some time for these to result in increased research capacity. It was pointed out that having researchers who have not worked together before be ready to do so is an element of pandemic preparedness. Because the workshops and conferences increase their familiarity with each other, even if this does not lead directly to immediate new collaborations, it improves their positioning and readiness for future collaborations, even in emergency situations.
Researchers responding to the survey were asked to indicate the extent to which their funded project was contributing to capacity development for pandemic preparedness research through networking and collaboration with other researchers, including international collaborations. While the majority of respondents in all research domains agreed or strongly agreed that the PPSRI had facilitated networking and collaboration, this agreement was less pronounced for those who had indicated their research area involved ethical, legal or social aspects of pandemic preparedness. About one-quarter of respondents in all research areas (27% overall) agreed that the PPSRI had helped involve international research collaborators in their work.
Among the 21 respondents involved in international collaboration in their PPSRI research, 13 (62%) indicated their international collaborators were working jointly with them on the same problems; 11 (52%) indicated that they were working on different aspects of the same problem; and 6 (13%) indicated that their collaborators were working on problems pertinent to their own setting. The locations of the international collaborators were most frequently in the US (12, or 57%) and Europe (8, or 38%); with some collaborations also reported in China, other parts of Asia, Latin America and Africa It was noted in the key informant interviews that projects supported by the PPSRI for international work tended to involve collaborations among rich countries; the survey data support this observation.
Groundwork for effective knowledge translation (KT)
The PPSRI KT strategy consists of five components: integrated KT for team grants; research synthesis grants; KT end-of-grant supplements; meeting, planning and dissemination grants; and an annual meeting of researchers and knowledge users19. As shown above in Table 11, uptake of these funding opportunities specifically for KT (research synthesis grants; KT end-of-grant supplements; meeting, planning and dissemination grants) has been slow; as more funded grants move towards completion, it would be expected that uptake will increase.
The first meeting of researchers and knowledge users, the Canadian Pandemic Preparedness Meeting: From Discovery to Frontlines, was held November 6-8, 2008 in Winnipeg. According to the meeting registration data, among the 157 non-student registrants, 45 (29%) were potential research users: public health or heath care practitioners, industry representatives, program administrators or decision-makers. While some key informants noted with satisfaction the presence of end-users at the From Discovery to Frontlines workshop, others felt that more managers, policy-makers and decision-makers could have been present.
Key informants were asked for their views on the likelihood that there will be sufficient knowledge translation of the results from the funded projects. Many stated that it was too early to tell, and noted that while important, this area was challenging for many researchers. In particular, challenges were noted with integrating end-users into research teams, because they are fully committed in other organizations (and in some cases, notably provincial governments, do not seem to value participating in research), as well as the exigencies of the CIHR application and CV requirements. From III's perspective, the strategy of requiring projects to include end-users from the beginning of project design will hopefully pay off, but they too find it is too early to predict this outcome. However, the lack of uptake of the funding opportunities for planning and dissemination activities is worrisome to III as it may indicate a lack of interest or readiness to engage actively in knowledge translation, or a lack of clarity among biomedical researchers about what KT is.
Indeed, it was noted in key informant interviews that the priorities retained for the PPSRI reflect CIHR's strong tradition of curiosity-driven research, and that CIHR and III have only recently begun signaling the research community that they wish to support very specific research areas. According to this view, response from the research community was also mainly driven by investigator interests. This was seen as limiting the capacity of the PPSRI to contribute to pandemic preparedness: "I think the problem was that there is some difficulty in achieving true preparedness. In a sense CIHR is going about it in the wrong way... . – investigators apply for opportunities, but not necessarily apply to do things that were most relevant". This recognizes the challenge of "trying to move the community of scientists to the most relevant types of research to assist with pandemic preparedness". As previously noted, one of the main partner organizations noted that because they could not know which research questions would end up being addressed once the funds were awarded, they had kept aside a portion of the funds for directed research under contract, "so that we were sure some of our priorities would be covered".
Researchers responding to the on-line survey were asked to provide some indicators of knowledge translation activities in their work, including the involvement and roles of research users and the presence of KT plans. These questions had relatively high levels of 'don't know responses', about 15% of respondents overall. For those researchers who could respond, these data are shown in Table 16. More than three-quarters of respondents stated that research users either were (47, or 66%) or would eventually be (14, or 21%) involved in their PPSRI project. This proportion was similar for all three research areas. Almost three-quarters (73%) acknowledged having a knowledge translation plan in place. This proportion was highest, 82%, among respondents who indicated that their work was related to ethical, legal or social aspects of pandemic preparedness, whereas it was lower among researchers engaged in vaccine (68%) and virus (58%) research.[ Table 16: Knowledge translation in PPSRI projects (No. (%) agree or strongly agree) ]
Governments are the most frequently named type of research users currently involved in PPSRI projects (70%), followed at a distance by non-governmental organizations (24%) and other types of public-sector entities (health care professionals, laboratories, regulatory bodies, health agencies/institutions). Pharmaceutical companies are involved as research users in 19% of projects where users are currently involved. As would be expected, the vast majority (87%) of these research users' roles include application or use of the results. In addition, 55% of responding researchers said that users are involved in carrying out some parts of the research, 49% in developing the research questions, and 40% in interpretation of the results. These latter findings suggest that integrated knowledge translation, where research users and researchers dialogue on the research questions, methods and findings throughout the entire research process, is not widespread among PPSRI researchers.
[ next section ]
- Proceedings of the Influenza Research Priorities Workshop, August 31 - September 1, 2005.
- Influenza virus characterization and ecology; transmission and public health preventive measures; behavioural theme; diagnostics; antiviral; surge capacity; vaccines; immunization programs. Task group meeting minutes, July 6 2006.
- Task Group meeting minutes, July 6, July 13, July 20. 2006.
- This group was formed after the Canadian Severe Acute Respiratory Syndrome (SARS) Research Initiative, to allow for more rapid research response to health threats. It is composed of representatives from CIHR, the Institute of Health Services and Policy Research (IHSPR), the Institute of Population and Public Health (IPPH); Institute of Infection and Immunity (III) and senior representatives of Rx&D Health Research Foundation; PHAC, the Council of Chief Medical Officers of Health (CCMOH), the Canadian Food Inspection Agency (CFIA), National Institute of Allergy and Infectious Disease (NIAID) of the National Institutes of Health (NIH); Michael Smith Foundation for Health Research, and Health Canada. Ad hoc members include senior representatives of: the Canadian International Development Agency (CIDA); the International Development Research Centre (IDRC); the Public Health Agency of Canada (PHAC); NIAID; Mathematics of Information Technology and Complex Systems (MITACS); Canadian Association for Immunization Research and Evaluation (CAIRE); CIHR’s Knowledge Translation and Ethics Branches, and the Institute of Aboriginal People’s Health.
- ">Report on the Pandemic Preparedness Research Initiative, Institute of Infection and Immunity, CIHR
- Pandemic Peer Review Committee Meeting, December 16, 2008: End of Meeting Discussion Summary.
- Task Group meeting minutes, June 16 2006.
- The Evaluation of the Canadian SARS Research Consortium describes lessons learned
- Three of the five Task Group members applied for funding under the PPSRI and two were successful NPIs. Of the 79 distinct members of the PPSRI’s Peer Review Committees, nine were successful applicants and five were unsuccessful applicants.
- Task Group minutes, July 6 2006.
- Proceedings of the Influenza Research Priorities Workshop - Integration and Summation Session
- Proceedings of the Influenza Research Priorities Workshop - Integration and Summation Session
- Pandemic Preparedness Strategic Research Initiative - Report on Activities & Outcomes
- This rating, on a 5-point scale, is considered to be a "very good" level of scientific merit. Only proposals rated 3.5 or higher ("very good", "excellent", or "outstanding") are eligible for CIHR funding (see: The CIHR Peer Review Process: Policies and Responsibilities of Grants Committee Members: [ PDF (341 KB) | Help ] ). However, depending on the number of applications reviewed in a given competition and the amount of funds available, some applications that are rated 3.5 and higher may not be funded, as grants are awarded to the highest scoring applicants. The number of unfunded but meritorious (3.5+) grants in a competition is viewed as an indicator of the degree of competitiveness or excellence of the field, i.e, the strength of its research capacity.
- Overview of the Canadian Institutes of Health Research Funding Program
- Strategic Training Initiative in Health Research (STIHR). Some of the existing STIHR projects appear to be in areas related to the PPSRI themes.
- Task group minutes, August 31 2006. Also reported in interviews.
- Overview of the Canadian Institutes of Health Research Funding Program; Meetings, Planning and Dissemination Grant: End of Grant Knowledge Translation Supplement (2008);
- Date modified: