Evaluation of CIHR's Knowledge Translation Funding Program

Acknowledgements

This report was authored by Robert McLean, Lead Evaluator and Joanne Tucker, Junior Evaluator.

The evaluation was carried out by an Evaluation Working Group which included: Robert McLean (Chair), Ian Graham, Martha MacLeod, Jacqueline Tetroe, Joanne Tucker, Christopher Manuel, Adrian Mota, Yumna Choudhry, Stephanie Coen, Ryan McCarthy, Andrea Smith, and Kwadwo Bosompra.

Table of Contents


Executive Summary

Background

The following report presents key findings and recommendations resulting from an evaluation of the Canadian Institutes of Health Research (CIHR) Knowledge Translation (KT) Funding Program. Commercialization programming, although an important aspect of CIHR's KT strategy, is not a part of this study as it is being examined in a separate ongoing CIHR evaluation. For this evaluation, the KT Funding Program comprises six standalone CIHR funding opportunities.

When the evaluation was launched in early 2011, its objectives were twofold. Firstly, the evaluation was designed to assess the performance and impact of the KT Funding Program and identify areas for improvement. This was the first undertaking of such a study of KT at CIHR. Secondly, it was designed to fulfill CIHR's responsibility to the Treasury Board of Canada under the 2009 Policy on Evaluation. In the interim, CIHR initiated a reform of its open grants competition and its peer review processes. As a part of this reform, the decision was made by CIHR to integrate the KT Funding Program into new open schemes of funding. This evaluation – although not designed to inform the integration of the KT funding opportunities into the open grants competition – provides information that can be used to support evidence-based decision-making during this reform process.

This evaluation employed multiple lines of inquiry including: literature, document, and electronic information system (EIS) reviews; an international environmental scan; quantitative surveys; in-depth qualitative interviews; and case studies.

Key Findings

Achievement of outcomes

KT design and delivery

The evaluation identified key elements responsible for success within the KT funding opportunities. Along with these key elements, related challenges were uncovered. The identification of these factors is useful for understanding what can facilitate and what can impede KT. These elements are potentially relevant across alternative funding designs that include the objective of enabling KT.

The role of CIHR in KT

Recommendations

CIHR's KT Funding Program has performed well in terms of meeting expected outcomes. Moreover, it has helped to position CIHR for success in an area that is of increasing significance to health research funders across the world.

The following recommendations have been developed to support continued KT success for CIHR in the newly proposed Foundation and Project Schemes of research funding, as well as across CIHR strategic initiatives.

Appendix H of this report provides a detailed crosswalk of key elements for sustained KT success with detailed sub-recommendations and considerations for any mainstreaming process.

Recommendation 1

Recommendation 2

Recommendation 3

Management Response

Recommendation Response
(Agree or Disagree)
Management Action Plan Responsibility Timeline
1. CIHR should invest the required resources to sustain its role in enabling KT. Given the decision by CIHR to integrate the KT funding opportunities into the open research suite, both of the proposed Project and Foundation Schemes of research should retain the key elements identified in the evaluation as responsible for the success of the current standalone iKT and end of grant KT funding opportunities. (See Appendix H for sub-recommendations and detailed cross-walk with each identified key element.) Agree Agreed and in progress. One of the objectives of the open reforms is to capture excellence across different communities and different types of health-related research and/or KT activities, including integrated and end-of-grant KT approaches to research. Data on how excellence is assessed by different communities has been gathered and is being built into the structured review process of the new open funding schemes. The open reforms are also aiming to improve accessibility, from a technical and content perspective, of future funding opportunities to all areas and modes of health research. The peer review reforms, including the establishment of the College of Reviewers, aims to increase the number of Knowledge Users engaged in peer review so as to ensure the capture of key elements of merit. Furthermore, to mitigate risk of a gap during integration of the KT programs into the new open suite, most core KT programs will remain in place until early 2016 when the new open schemes are expected to be fully rolled out and accessible to the community. VP Research and KT Initial redesign of peer review and application processes will be complete by spring 2013 followed by testing and implementation by fall 2014. KT programs are included as part of the pilot testing.
2. To ensure the continued success of KT and address funding balance across all fields of research within the broad suite of open programs and strategic initiatives, CIHR should develop performance measures which accurately monitor and assess the integration of the KT Funding Program into the open research schemes. Data collected in this evaluation should be used as a baseline for the future study of CIHR success in KT under the newly proposed Project and Foundation Schemes of research. This will allow CIHR to measure whether or not the objective of mainstreaming – that there will be an increase in iKT research – has occurred. Agree Agreed. The development of performance metrics related to KT and a system of collection and analysis plan is currently being developed as part of the open reforms mentioned above. VP Research and KT Metrics will be established by summer 2013 as part of the open reforms implementation.

The analysis plan will be considered once the reforms have been implemented.
3. For university-based researchers, performing iKT research and conducting KT of research findings is not well aligned with the performance measures used by universities to judge their success. CIHR should initiate dialogue with the academy and researcher community in order to draw attention to and work toward mitigating this tension. Agree Agreed and in progress. Engagement with the research community and universities has already begun during the consultation period for the open reforms. CIHR will consider how to work with the universities to develop appropriate and meaningful performance indicators for non-traditional research outputs and activities. VP Research and KT Include as part of open reforms implementation for fiscal year 2013/14.

Program and Evaluation Description

KT at CIHR, and the CIHR KT Funding Program

At the Canadian Institutes of Health Research (CIHR), knowledge translation (KT) is defined as:

a dynamic and iterative process that includes synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products, and strengthen the health care system. (CIHR, 2012)

In more simple terms, KT at CIHR is about turning health research into improved health. KT has been an important aspect of CIHR's vision and work since the organization's inception in 2000, and is written into the Parliamentary Act which created CIHR (GOC, 2000).

The KT program at CIHR has evolved significantly since the year 2000. Today, this program encapsulates a diverse set of activities including funding opportunities, partner engagement activities, outreach, and research. Although it lists many ongoing and dynamic activities, Figure 1 provides a static snapshot of the CIHR KT program.

Figure 1 – The CIHR KT program

Program goal
CIHR advancing the science and practice of KT Supporting KT at CIHR Outreach to decision-maker community to solicit feedback on their needs (as mandated by the CIHR Act)

Note: This evaluation concentrates on six funding opportunities within the KT Funding Program: Synthesis, K2A, PHSI, DE, KTS, and KT Science.
Acronyms: PHSI- Partnerships for Health System Improvement, K2A- Knowledge to Action, DE- Dissemination Event, KTS- Knowledge Translation Supplement, OOGP- Open Operating Grants Program, DSEN- Drug Safety and Effectiveness Network, SPOR- Strategy on Patient-Oriented Research, EIHR- Evidence Informed Healthcare Renewal, RRS- Research Reporting System, CCV- Common Curriculum Vitae, EKS- Expedited Knowledge Synthesis, PHAC- Public Health Agency of Canada, NAPHRO- National Alliance of Provincial Health Research Organizations

Activities
  1. KT Funding Program (Synthesis, K2A, PHSI, KT Science, DE, KTS, etc.)
  2. Capacity Development Modules
  3. Open Access
  4. PubMed
  5. OOGP
  6. Directed Grants
  7. Evidence on Tap
  8. KT Guide
  9. Casebooks
  10. Presentations, workshops and webinars
  11. KT Evaluation
  12. Commercialziation Evaluation
  1. Support to DSEN, Institute Cluster, SPOR, Open Reforms
  2. Actively participating in EIHR
  3. Corporate Project (RRS, CCV)
  4. Impact stories (all FOs)
  5. Caseload Project
  6. Evaluation of individual initiatives (i.e. EKS, Science Policy Fellowship)
  7. Representing CIHR on external working groups (i.e. Rx for change)
  8. Lunch and Learn
  1. Relationships with provincial ministries
  2. Relationships with Health Canada and PHAC
  3. Relationships with NAPHRO partners
  4. Relationships with charities
  5. Relationships with research community

Through a more strategic lens, CIHR supports four interrelated but different types of KT. These are: synthesis, end of grant KT, integrated KT research, and KT science. The unique funding opportunities which make up the KT Funding Program are designed to encourage these four types of KT.

Synthesis is the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. It is a family of methodologies developed to determine what is known in a given area or field and what the knowledge gaps are. The underlying principle is the support of timely and accurate scientific knowledge being available to those who work in real-world settings requiring such evidence. Knowledge synthesis studies may be useful to policy-makers, industry, clinical and medical practitioners, amongst others. In some cases, knowledge synthesis research can be conducted with the participation of knowledge users (KUs) throughout the research process. CIHR refers to this approach as integrated KT. The Knowledge Synthesis funding opportunity (FO) is designed to support synthesis research, which is performed in an integrated KT (iKT) approach.

End of grant KT describes the process where the researcher develops and implements a plan for making knowledge users aware of the knowledge that was gained during a project. End of grant KT includes the typical dissemination and communication activities undertaken by most researchers, such as KT to their peers through conference presentations and publications in peer-reviewed journals. End of grant KT can also involve more intensive dissemination activities that tailor the message and medium to a specific audience, such as summary briefings to stakeholders; interactive educational sessions with patients, practitioners, and/or policy makers; media engagement; or the use of knowledge brokers to name a few. The commercialization of scientific discoveries is another form of end of grant KT, but as a specific strategy it is not being explored in this evaluation study. Commercialization is the focus of a separate CIHR evaluation which is currently underway. The Dissemination Events (DE) and Knowledge Translation Supplement (KTS) provide funding for end of grant KT.

Integrated KT research, involves engaging knowledge users through the entire research process. By doing iKT research, researchers and knowledge-users work together to shape the research process by collaborating to determine the research questions, deciding on the methodology, being involved in data collection and tools development, interpreting the findings, and helping disseminate the research results. This approach is designed to produce research findings that are more likely to be relevant to and used by end users. This approach is similar to those used in collaborative research, participatory, action-oriented research, co-production of knowledge, and Mode 2 knowledge production. The Knowledge Synthesis, Partnerships for Health Systems Improvement (PHSI), and Knowledge to Action (K2A) all support iKT.

KT science or research (also known as implementation science) is the study of the process of KT and the use of knowledge. KT science explores the factors which facilitate and hinder the sharing of knowledge between creators and users. While it often addresses issues such as the efficacy and effectiveness of certain KT strategies, it may also involve the development of new KT theory or practice. The KTR (knowledge translation research) panel of the Open Operating Grant Program (OOGP)1 is the panel that reviews KT science grant proposals received by CIHR.

Evaluation purpose

When the evaluation was launched, its objectives were twofold. Firstly, the evaluation was designed to assess the performance and impact of the KT Funding Program and identify areas for improvement. Secondly, it was performed to fulfill CIHR's responsibility to the Treasury Board of Canada under the 2009 Policy on Evaluation which requires that all federal spending be evaluated to investigate the value for money of the expenditure.

In the interim, CIHR initiated a reform of its open grants competition and its peer review processes. As a part of this reform, the decision was made to integrate the KT Funding Program into new open schemes of funding and across CIHR strategic initiatives. This evaluation – although not designed to inform the integration of the KT funding opportunities into the open grants competition – provides information that can be used to support evidence-based decision-making in this reform process.

Evaluation approach

To evaluate CIHR's KT program in a resource-constrained environment, the direct object of evaluation was limited to a sample of six KT funding opportunities (Synthesis, PHSI, K2A, DE, KTS, and KT science). For the purposes of this evaluation these are called the KT Funding Program. These were highlighted as particularly important by CIHR senior management, are financially material (approx. 64% of KT program financial commitment), and provide a full representation of the four types of KT (synthesis, end of grant KT, iKT research, and KT science). A full description of each funding opportunity – including financial outlays, application rates, and success rates – is available in Appendix C of this report.

Table 1 - Relevance and materiality coverage of the KT Funding Program
Funding opportunity KT type 2010–11 Financial commitment (CAD) % of 2010–11 KT program financial commitment
Notes:
  1. KT science grants fall under the OOGP financial classification.
  2. DE and KTS are grouped together in financial classification.
Synthesis Synthesis; iKT 1.76m 10.6%
K2A iKT 1.33m 8%
PHSI iKT 6.29m 37.7%
KT science KT science n/a n/a
DE and KTS End of grant KT 1.34m 8%
Total coverage 10.72m 64.3%

This evaluation employed an integrated, participatory approach. Evaluators, KT staff, CIHR senior management, CIHR Institute representation, and external researcher representation worked together to design and conduct this study (see Appendix G).

An early step in the design of this evaluation was the construction of a logic model (Figure 2) that was used to articulate the KT Funding Program Theory2 and also helped inform the development of the evaluation questions.

Figure 2 – Logic model for the KT Funding Program


Figure 2 long description

Through the evaluation process – particularly the many formative discussions and interviews with other funding agencies, researchers, CIHR senior management, and KT staff – it became clear that underlying the KT Funding Program Theory were three critical and ordered steps: the involvement/engagement of KUs within research, the influence/influencing of KUs through the research process, and finally, KUs taking real world action that is informed by research created knowledge. This simplified version of the KT Funding Program Theory is illustrated below.

Involve-Influence-Act

Evaluation questions

Eight evaluation questions3 were developed to guide this study:

  1. To what extent are KT funding opportunities achieving their expected outcomes?
  2. How effective is the KT Funding Program mix in achieving CIHR's expected outcomes? (iKT , End-of-grant KT, KT science, Synthesis)
  3. To what extent have KT funding opportunities reached a broad and diverse range of knowledge-users?
  4. What factors facilitate or inhibit the achievement of KT funding opportunity outcomes?
  5. To what extent are KT funding opportunities being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness?
  6. What role is there for CIHR in enabling/promoting iKT research, synthesis, end-of-grant KT, and KT science?
  7. What would be the effect on CIHR-funded researchers and knowledge-users if the KT Funding Program no longer existed? What would be the effect on the improvement of health, more effective health services and products, and the strengthening of the healthcare system?
  8. What are the unanticipated outcomes, positive or negative, resulting from the KT Funding Program?

Key Findings

This part of the report presents the key findings of the evaluation study and is organized into three sub-sections. Sub-section 1 presents findings related to the success of the KT Funding Program in achieving its expected outcomes. Sub-section 2 presents findings related to program design and delivery. Sub-section 3 examines what role exists for CIHR in enabling/promoting KT. Given the broad nature of evaluation questions 7 and 8, these are covered in each section where appropriate, rather than as a separate chapter of analysis.

1 - Achievement of expected outcomes

Evaluation Questions

  1. To what extent are KT funding opportunities achieving their expected outcomes?
  2. How effective is the KT Funding Program mix in achieving CIHR's expected outcomes?
  3. To what extent have KT funding opportunities reached a broad and diverse range of knowledge-users?

This sub-section reports on key findings from collected data. Specifically, it presents findings related to the achievement of expected outputs and outcomes for the KT Funding Program. The presentation of a KT Funding Program Theory enriches the assessment by identifying additional areas of investigation that are more relevant to the purpose and objectives of KT. These additional indicators are examined using currently available data. Key findings related to how KT funding opportunities work to achieve outcomes are discussed. The findings include the importance of meaningful partnerships, the value of supported end of grant KT, and the sustained benefits of partnering.

1.1 – Traditional measures of research funding success

CIHR has traditionally measured KT success in terms of the production of KT outputs (a predefined type of research output) and a defined set of research outcomes. Data on these outputs and outcomes are collected through an end of grant survey known as the Research Reporting System (RRS), which until very recently was only administered to OOGP4 funded researchers. To better understand the success of KT funding programs against these traditional measures, several of the questions posed in the RRS survey were replicated in this study's surveys of KT funded researchers.

1.1.1 - Research outputs

Traditional research outputs span three categories: academic outputs (e.g., peer reviewed papers, book chapters); engagement of highly qualified personnel (HQP) (e.g., students, post-docs), and KT outputs (e.g., conference presentations, webinars). The academic output and HQP categories, although not specific KT priorities, are seen as traditionally important measures of CIHR success. Data from the RRS survey is included to provide a point of reference for interpreting these data5.

Table 2 presents the individual iKT funding opportunities, the KT science funding opportunity, iKT as a group, the OOGP, and the OOGP pillars II-IV. It should be noted – here and throughout this report where OOGP and KT data are presented together – that the strategic design of the KT funding opportunities and OOGP are different, the two have different objectives, and the nature of the research supported may be different. Therefore, any comparative analysis between the two should be interpreted with these caveats in mind.

Table 2: Outputs of individual KT funding opportunities and the OOGP
CIHR $ investment per grant HQP Academic outputs KT outputs
Mean Total investment Mean #/100,000$ Mean #/100,000$ Mean #/100,000$
Notes:
  1. All KT figures include grants identified as completed by researchers;
  2. Two respondents were removed from Synthesis because data was unavailable;
  3. Rounding has been performed;
  4. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011; OOGP figures based on RRS Pilot data collected in 2009 on grants administered between 2000 and 2008; Investment data from EIS 2011 and matched to appropriate KT survey and RRS respondents to ensure accuracy of figures.

Synthesis $87,010 $4,089,482 5.3 6.09 3.6 4.14 13.4 15.40
K2A $151,115 $2,568,956 5.6 3.71 4.2 2.78 18.7 12.37
PHSI $118,407 $2,012,927 8.2 6.93 4.9 4.14 21.1 17.82
KT science $169,622 $2,713,948 4.6 2.71 3.1 1.83 8.5 5.01
iKT $107,054 $8,671,365 6.0 5.60 4.0 3.76 16.9 15.79
OOGP $258,284 $153,937,211 9.1 3.52 9.8 3.77 15.8 6.12
OOGP II-IV $221,986 $30,412,133 12.2 5.50 7.6 3.44 20.8 9.37

In terms of CIHR's traditional measures of success, a typical iKT grant resulted in a mean of six trainees, four academic outputs and approximately 17 KT outputs. On average, an OOGP funded research project produced approximately nine trainees, 10 academic outputs, and 16 KT outputs (Table 2).

Using the common denominator of dollar amount invested (we employed a 100K base), results indicate that iKT funded projects produce greater numbers of KT outputs than do the OOGP funded projects. The unexpected findings, however, are that iKT funded projects also do better at engaging HQP and in two out of three cases (PHSI and Synthesis) at producing academic outputs than do OOGP funded projects. OOGP funded projects do produce greater numbers of academic outputs than do K2A funded projects. KT science grants lag behind both the OOGP and iKT in each of HQP, academic, and KT outputs using this measure.

The results of this analysis could be interpreted as indicating that iKT funding opportunities represent a low-risk, high-return investment opportunity to CIHR. This same hypothesis was raised by several knowledge users (non-researchers) interviewed as a part of this study. Such an assertion from this group gives additional context and weight to this finding.

1.1.2 – Research outcomes

Apart from outputs directly resulting from the KT Funding Program, a wider range of research outcomes could also be linked to KT grants.

In Table 3, data are ranked by order of frequency of iKT researchers reporting "yes" to having produced the research outcome. In this format it provides an indication of the type of research outcomes being produced most commonly from grants obtained from the iKT funding opportunities. The most common of these were research findings being cited by others, the introduction of a new tool, technique, instrument or procedure and a new research method.

Table 3: Percentage of KT grants producing research outcomes
iKT KT science
Produced May in the Future Produced May in the Future
Notes:
  1. Data presented excludes respondents who reported having not completed their grant at the time of survey;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Findings cited by others 42.4 42.4 50.0 43.75
Tool, technique, instrument or procedure 44.7 22.4 37.5 43.75
Research method 32.8 35.3 25.0 43.75
Professional practice 28.2 44.7 41.7 50.0
Replication of research findings 24.7 38.8 18.75 56.25
Information or guidance for patients or public 24.7 31.8 12.5 43.8
Policies or programs 20.0 60.0 6.3 62.5
Theory 18.8 46.9 18.75 50.0
Software/database 11.8 16.5 12.5 25.0
Direct cost savings 9.4 37.6 0.0 62.5
Patient or public behaviors 8.2 29.4 6.3 37.5
Intellectual property claim 2.4 4.7 6.3 0.0
Vaccines/drugs 2.3 3.5 0.0 0.0
Patent 1.2 3.5 0.0 0.0
Product license 0 2.4 0.0 0.0
Spin-off company 0 1.2 0.0 0.0

Overall, the data relating to traditional CIHR measures of research outputs and outcomes identified in this evaluation confirm that the KT Funding Program is creating and disseminating knowledge.

1.2 - An appropriate KT evaluation approach; KU involvement, influence, and action

There is an extensive amount of academic literature which describes how the involvement of KUs in the research process will increase the uptake of research results and improve the relevance of research findings (Israel et al. 1998; Lomas 2000; Cargo & Mercer 2008; Macaulay et al. 2011). With a grounding in this theory, CIHR hypothesizes that KU engagement in research can lead to applications, such as informed decision-making, or the development of new tools or techniques, and that ultimately, this is will lead to health improvements at the individual and systems levels.

However, there is significant lack of evaluative work done on KT, especially KT funding interventions (Cordero et al. 2008; Tetroe et al. 2008). To address this challenge, a key step in assessing KT program performance in relation to expected outcomes was the development of a logic model (see Figure 2 on p.4). This logic model provides a visual representation of the KT Funding Program Theory, and accordingly, it provides a guide for identifying metrics of success.

A range of both quantitative and qualitative methods of investigation were employed in order to investigate and learn about the occurrence and intricacies of this KT Funding Program Theory. Results of this analysis are presented in the following sub-sections.

To begin this analysis, Table 4 provides quantitative survey data which provides a basic illustration of the correlation between involving and influencing KUs. The quantitative indicator used as a proxy for action is not included in Table 4 as the format of the survey question used to measure action did not allow for comparison across KU groups (as is done in Table 4). Data related to action is discussed in sub-section 1.2.3. Data related to involvement and influence is discussed in greater depth in sub-sections 1.2.1 and 1.2.2. Qualitative data is presented in each section, as appropriate, in order to corroborate and delve deeper into quantitative survey findings.

Table 4: Involvement of KUs in the research process and KU influence; % of iKT researchers reporting by KU group
Involved Influenced
Notes:
  1. "Involved" is a representation of researchers who did not identify the corresponding KU group as "not involved", it does not connote involvement in each stage of the research process or any discrete stage. Data related to each stage of the research process are presented in Table 5;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Other researchers/academics 74.4 58.5
Health system/care practitioners 84.1 53.7
Patients/consumers of health care 67.1 31.7
Health care managers 83.5 55.5
Health care professional organizations 70.1 36
Federal/provincial representatives 72 38.4
Community/municipal organizations 67.7 25
Consumer groups/charitable organizations 65.9 18.3
Industry 57.9 8.5
Media 65.2 15.9
1.2.1 Involving and engaging KUs with research

To examine KU involvement in the research process, recipients of iKT grants (Synthesis, PHSI, K2A) were asked to report on involvement of various types of KUs at various stages of their research projects6. Results are presented in Table 5 along with reference data from the OOGP.

Table 5: Involvement and engagement of KUs in the research process: OOGP and iKT (%)
Full engagement in research process Stages in the research process Not involved
Development of research idea/question Development of protocol Data collection phase/Project implementation Interpretation of findings End of grant KT
Notes:
  1. "Full Engagement in the research process" is a sum of respondents who checked each of the 5 "Stages in the research process" independently;
  2. The figure contained in brackets in the OOGP rows is the percentage based on only pillars II- IV RRS pilot data.
  3. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011; OOGP figures based on RRS Pilot data collected in 2009 on grants administered between 2000 and 2008.

iKT other researchers/ academics 51.2 70.1 72 64 69.5 59.8 25.6
OOGP other researchers/ academics 35.4
(40.9)
49
(51.1)
48.7
(51.8)
48.7
(50.4)
52.9
(54.7)
41.4
(47.4)
7.2
(5.8)
iKT health system/ care practitioners 9.8 52.4 48.2 47.6 31.1 50.6 15.9
OOGP health system/ care practitioners 3.9
(6.6)
12.2
(21.9)
8.7
(19)
15.4
(35)
12.9
(25.5)
19
(39.4)
63.1
(27)
iKT patients/ consumers of health care 6.1 15.9 12.2 17.7 22 24.4 32.9
OOGP patients/ consumers of health care 0.3
(1.5)
2.9
(6.6)
1.8
(5.8)
10.1
(29.2)
1.3
(5.8)
9.2
(24.8)
78.9
(46.7)
iKT health care managers 18.9 51.2 48.8 45.1 57.3 53 16.5
OOGP health care managers 0.5
(1.5)
2
(7.3)
1.5
(4.4)
4.4
(16.8)
2.2
(8)
5.5
(17.5)
88
(62)
iKT health care professional organizations 8.5 21.3 20.1 21.3 29.3 34.1 29.9
OOGP health care professional organizations 0.3
(1.5)
1.5
(5.1)
0.8
(3.6)
2.7
(8.8)
1.7
(5.8)
8.7
(25.5)
86.6
(62.8)
iKT federal/provincial representatives 6.7 26.2 24.4 21.3 29.3 38.4 28
OOGP federal/ provincial representatives 0.2
(0.7)
0.3
(1.5)
0.2
(0.7)
0.5
(2.2)
0.7
(2.2)
5
(14.6)
93
(81)
iKT community/ municipal organizations 8.5 15.2 14.6 15.9 19.5 22 32.3
OOGP community/ municipal organizations 0.2
(0.7)
0.7
(2.2)
0.2
(0.7)
1.3
(5.8)
0.7
(2.2)
6.5
(17.5)
92.1
(80.3)
iKT consumer groups/ charitable organizations 4.9 13.4 9.8 9.8 14 20.1 34.1
OOGP consumer groups/ charitable organizations 0.3
(1.5)
1.3
(2.2)
0.5
(2.2)
1.3
(5.1)
0.8
(2.2)
8.1
(18.2)
88.9
(78.8)
iKT industry 0.6 1.8 0.6 3 3 3.7 42.1
OOGP industry 0 1
(0.7)
1.5
(1.5)
1.7
(1.5)
1.5
(1.5)
7.6
(2.9)
87.9
(94.2)
iKT media 0 0.6 0.6 2.4 1.8 13.4 34.8
OOGP media 0 0.3
(0)
0.0
(0)
0.7
(1.5)
0.3
(1.5)
19.1
(37.2)
77.3
(56.9)

Overall, results indicate that iKT funded projects – grants which have a requirement for KU involvement in order to be funded – are more likely to have involved KUs in, and throughout, the research process than are researchers funded through the OOGP. IKT funding opportunities also tend to involve a greater variety of KU groups in research.

1.2.2 Influencing KUs with research

IKT researchers were also asked to assess the degree of influence that their research had on KUs. Table 6 presents survey responses related to influence.

Table 6: Research influence on knowledge users: OOGP and iKT (%)
Influenced Too soon to tell / Don't know Not at all / Not applicable
Notes:
  1. The figure contained in brackets in the OOGP rows is the percentage based on only pillars II- IV RRS pilot data.
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses, 2011, OOGP figures based on RRS Pilot data collected in 2009 on grants administered between 2000 and 2008.

iKT other researchers/academics 58.5 36 5.5
OOGP other researchers/academics 89.4
(87.7)
7.7
(10.5)
2.9
(1.5)
iKT health system/care practitioners 53.7 39.6 6.7
OOGP health system/care practitioners 31.4
(59.9)
26.2
(24.1)
42.4
(16)
iKT patients/consumers of health care 31.7 43.3 25
OOGP patients/consumers of health Care 24.3
(51.1)
26.3
(27.7)
49.3
(21.2)
iKT listed study stakeholders 84.2 13.4 2.4
OOGP listed study stakeholders 37.8
(44.4)
17.1
(20.4)
45.1
(35)
iKT health care managers 55.5 36.6 7.9
OOGP health care managers 9.7
(28.4)
29
(35.8)
61.2
(35.8)
iKT health care professional organizations 36 45.1 18.9
OOGP health care professional organizations 11.6
(28.4)
29.7
(37.2)
58.7
(34.3)
iKT federal/provincial representatives 38.4 47 14.6
OOGP federal/provincial representatives 7
(15.3)
32.7
(42.3)
60.2
(42.4)
iKT community/municipal organizations 25 46.3 28.7
OOGP community/municipal organizations 6.9
(17.6)
30.2
(38.7)
62.9
(43.8)
iKT consumer groups/charitable organizations 18.3 47 34.8
OOGP consumer groups/charitable organizations 12.1
(16.7)
28.2
(34.3)
59.7
(48.9)
iKT industry 8.5 43.3 48.2
OOGP industry 24.7
(10.2)
28.9
(27.7)
46.5
(62)
iKT media 15.9 48.8 35.4
OOGP media 19.1
(25.5)
27.2
(27.7)
53.7
(46.7)

Table 6 demonstrates that iKT researchers report influencing a diversity of KU groups with their research. In contrast to OOGP researchers and OOGP researchers from strictly pillars II-IV, iKT researchers were more likely to report influencing: listed study stakeholders, health system/care practitioners, patients and consumers of health care, health care managers, health care professional organizations, federal/provincial representatives, community/municipal organizations, as well as consumer groups and charitable organizations. OOGP researchers were more likely to report influencing other researchers/academics, industry, and media. 7

Additional noteworthy findings arise from these data. For instance, the KU categories where the OOGP researchers create greater influence than the iKT researchers are researchers/academics, industry, and media. Researchers/academics are well understood to be users of research and for this reason it is not surprising to see this category score well on the OOGP. However, the higher OOGP rates for the industry and media KU categories may be indicative of best practices for influencing industry and media KUs. The lesson seems to be: to best influence industry and media, use end of grant KT.

That is to say, a comparison of data in Table 6 with that in Table 5 illustrates that iKT researchers engage industry and media at higher rates, albeit marginally, throughout the research process with the exception of the end of grant KT stage. At the end of grant KT stage, OOGP researchers engage these KU categories more often. Although this finding may be a function of industry and media being more interested in research results funded through the OOGP, it may also demonstrate that researchers believe they can create influence in industry and media with end of grant KT activities. Those researchers who wish to influence industry and media, and likewise CIHR, should it wish to increase influence with these two groups, ought to consider and explore this finding further.

To probe deeper into these findings, we conducted in-depth interviews with both researchers and KUs. These interviews explored the issue of how influence happens in the research process, and how it happens best. This analysis is presented in the following subsection.

1.2.2.1 Meaningful partnerships as a catalyst for KT

In-depth, qualitative interviews, with both researchers and KUs, were used to further investigate and better understand not only if but also how iKT research might, and might not, lead to influence and eventual impact (n=29). Both funded researchers and KUs continually highlighted the concept of a "meaningful partnership" as a catalyst for creating influence and eventually action. This opinion was consistent among both researchers and KUs, and across each KT funding opportunity, which is particularly interesting as some grants do not require partnering (i.e., end of grant KT and KT science). The key finding of these interviews, however, was the contextual indications of how a "meaningful partnership" can crystalize. These are described in the following image.

The meaning of "meaningful partnerships"

Researchers and KUs describe building meaningful collaboration as key to research project success and impact, however, the term "meaningful" is quite nuanced and contextually-bound.

  • Meaningful partnerships are characterized by: mutual learning, mutual respect, mutually agreed upon roles and responsibilities, mutual recognition of efforts, and mutual exchange of information.
  • Mutual does not necessarily denote that partners give and receive equally, but that all parties play a role in negotiating roles and expectations.
  • Researchers and KUs have different understandings of the roles and responsibilities required of each team member in order to make collaboration meaningful.
  • Meaningful partnerships are negotiated based on many factors including, but not limited to: resources, external commitments, technical skills, and epistemology.

Another significant finding from these interviews, reported by both funded researchers and KUs, was that building these relationships is the most difficult aspect of doing research. More specifically, it was cited as the most costly and time-consuming activity related to doing a research project. This finding is elaborated on in greater depth in section 2 of the key findings section of this report.

1.2.3 Action informed by research

If knowledge users are involved in and influenced by research, the KT Funding Program Theory suggests that KUs would be compelled into action to apply the research findings. A proxy measure of action used in this evaluation was researchers' perceptions of their contribution to fulfilling the CIHR mandate and the KT program's ultimate outcome of: improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system. A major avenue by which researchers can contribute to the achievement of CIHR's mandate is through inducing stakeholders to apply their research findings, and thus, this metric was used to assess the extent to which relevant stakeholders have taken action. It is recognized that a measure which directly assessed action taken by the KU would enhance this analysis, however, no such quantitative measures are collected by CIHR. To mitigate this data limitation, this evaluation undertook case studies and in-depth interviews to triangulate and validate the quantitative data presented below. Indications of research induced action from case studies and in-depth interviews with KUs are provided in the following sections of the report.

Figure 3 below illustrates the perceived contribution made by each KT funding opportunity to each part of CIHR's mandate.

Figure 3: Reported contribution to CIHR mandate: KT funding opportunities

Involve-Influence-Act
Figure 3 long description

Notes:
  1. Percentage calculated by grouping "some extent" and "great extent" responses together;
  2. Final percentages do not take into account the response category of "May in the future" and may therefore downwardly bias results;
  3. Responses from researchers reporting grants as incomplete are excluded.
  4. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Figure 3 illustrates that the varied KT funding opportunities contribute quite similarly to the different parts of the CIHR mandate. Generally speaking, KT grants do best at creating the transformative outcomes of strengthening health care systems, creating more effective health services and/or products, and improving health for Canadians. They perform less well at creating new health knowledge and translating knowledge from research into real world applications. The finding that KT funding opportunity researchers report having translated knowledge into applications at lower rates than they report having produced the aforementioned transformative outcomes has no obvious explanation, but one interpretation is that these researchers identify their work as contributing to specific areas of health impact identified in CIHR's mandate, and doing so indicates that the KT has occurred.

When interpreting Figure 3 above and Figure 4 below, it is important to note that respondents in this evaluation were given an additional option of "may in the future" when identifying the contribution of their project to CIHR's mandate. It is quite possible that giving respondents the option to indicate that their research "may" contribute to CIHR's mandate "in the future" resulted in respondents understating present results. Table 7 is provided to mitigate this data interpretation limitation by presenting data for the "may in the future" category. Table 7 provides a detailed overview of how iKT funded projects (Synthesis, PHSI, K2A) and OOGP funded projects rank across the spectrum of CIHR's mandate.

Table 7: Reported contribution to CIHR mandate: iKT and OOGP
iKT OOGP
Great extent Some extent May in future Great extent Some extent May in future
Notes:
  1. Figures presented in brackets is OOGP pillars II-IV;
  2. For OOGP figures the RRS response categories "a little extent" and "some extent" have been merged to form "some extent" and the RRS categories "considerable extent" and "great extent" have been merged to form "great extent";
  3. For iKT figures responses from researchers who report grants as incomplete are excluded to improve comparability of the data;
  4. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011, OOGP figures based on RRS Pilot data

Creating new health knowledge 42.4 4.7 50.6 66.6
(60.6)
29.8
(36.5)
n/a
Translating the knowledge from research into real world applications 37.6 10.6 51.8 24.7
(40.1)
48.5
(46)
n/a
Improving health for Canadians 34.1 40.0 18.8 10.7
(16)
44.3
(57)
n/a
Creating more effective health services and/or products 31.8 32.9 23.5 11.1
(18.2)
32.2
(46.7)
n/a
Strengthening the Canadian health care system 35.3 38.8 20.0 9.2
(14.6)
29.8
(46.7)
n/a

Figure 4: Reported contribution to CIHR mandate: Synthesis, iKT, end of grant KT, KT science and OOGP

Involve-Influence-Act
Figure 4 long description

Notes:

  1. For KT figures, percentage is calculated by grouping "some extent" and "great extent" together.
  2. Final percentages do not take into account the response category of "May in the future" and may therefore downwardly bias KT results vs. OOGP results which did not have this response option;
  3. For OOGP figures, the RRS response categories "a little extent", "some extent", "considerable extent" and "great extent" have been merged to calculate percentage;
  4. iKT figure combines PHSI and K2A responses, End of grant figure combines KTS and DE;
  5. For iKT and synthesis figures responses from researchers who report grants as incomplete are excluded.
  6. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011, OOGP figures based on RRS Pilot data

The key finding of this analysis is the complementary role that projects funded through the KT Funding Program play in relation to projects funded through the OOGP. Figure 4 illustrates that, according to researchers, achievement of the CIHR mandate is realized through both OOGP and KT funding, and both sets of funding excel where the other is less effective. Together, these two funding streams produce a balanced approach toward the fulfillment of CIHR's mandate.

In summary, the findings presented in Figures 3 and 4 and Table 7 support the hypothesized KT Funding Program Theory – the notion that by being involved in research KUs are influenced to apply findings in action8. Further exploration, through in-depth and methodical statistical testing, of the relationships between the quantitative data points used as indicators of each step in this KT Funding Program Theory (i.e., Involve, Influence, Act) would be valuable to enhancing the understanding of the strengths, weaknesses, and nuances of this process.

1.3 The strategic value of the mix of KT funding opportunities within the KT Funding Program

In-depth interviews with researchers and KUs elicited data which support the strategic value of the mix of KT funding opportunities within the KT Funding Program, and also provide indications of how the breadth of the KT funding opportunity mix contributes to CIHR achieving the full breadth of its mandate. Two over-arching issues related to mix were identified:

The following two sub-sections will elaborate on findings related to two key aspects of the KT Funding Program mix: end of grant KT and iKT.

1.3.1 End of grant KT (DE and KTS)

As is described in section 1 of this report, end of grant KT is supported by CIHR through two funding opportunities, Dissemination Events and the KT Supplement. The parameters of these funding opportunities are provided in detail in Appendix C. Data presented earlier in this report provides KTS and DE results in terms of achievement of outcomes and impact. Here, we further the analysis to look for any significant divergence between the two, and look closer at how these funding opportunities achieved results.

To begin, end of grant KT (both KTS and DE) funding was highlighted by researchers as a particularly valuable and beneficial opportunity. Survey data shows that:

Qualitative evidence gathered from KTS and DE researchers provides an explanation of why these grants were so highly valued in the research community.

The value of end of grant KT funding

End of grant KT funding opportunities are highly esteemed for four key reasons:

  • End of grant KT funding opportunities satisfy a funding gap by enabling researchers to undertake their planned dissemination activities when initial grant funding is insufficient.
  • End of grant KT funding opportunities enable researchers to undertake emergent dissemination activities. It is difficult to identify and describe KT strategies during the grant writing process due to a lack of in-depth understanding of the research context and the findings.
  • The expedient grant delivery process is well matched to the needs of researchers operating in a dynamic environment that calls for accelerated adaptation.
  • Together, KTS and DE, fund a varied range of dissemination techniques, which are critical for meeting the needs of varied audiences.

To investigate divergence between DE and KTS an assessment was undertaken of which stakeholders each program was intending to engage and which stakeholders were being influenced. Figure 5 presents results of this analysis.

Figure 5: Intended audience vs. influenced audience; KTS and DE


Figure 5 long description

Notes:

  1. The "too soon to tell" response option was omitted for both DE and KTS data.
  2. The "intended audience" field for both DE and KTS was populated with all survey response data; the "influenced audience" field was populated with only those respondents indicating their respective DE or KTS grant was complete;
  3. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Figure 5 illustrates some interesting discoveries. For one, for every KU group except health practitioners, and in the case of DE consumer/municipal organizations and industry, the funding opportunities are reported to create influence at greater rates than was initially intended. This finding highlights the ability and potential of end of grant KT activities to influence KUs. Researchers experience also shows that certain KU groups such as health practitioners are more difficult to influence through end of grant KT methods, and others such as industry and consumer/municipal organizations are more reactive to research findings through KTS than DE.

Data also show that aside from industry and other researchers/academics, DE grants are intended to reach more KU audiences than KTS grants. The industry and researcher/academic KU groups are both traditional Pillar 1 audiences, and this data indicates the KT value the KTS tool may hold for this group of researchers. As an additional note in regard to KTS, for the first four KU categories (health systems practitioners, patients, managers and professional organizations), not one respondent replied "not at all".

Generally speaking, Figure 5 illustrates that DE and KTS show some individual strengths and weaknesses in terms of reaching and influencing KU audiences. However, both funding opportunities are reported to create greater than intended influence in the majority of cases.

1.3.2 iKT partnerships and leveraging

Here we present a supplementary look at how the strategic design of the three iKT funding opportunities attribute to each performing differently in terms of leveraging external investments – a CIHR objective.

To begin, Figure 6 shows how K2A and Synthesis funding opportunities leverage partner money and in-kind contributions in grant proposals. PHSI grants attract the largest amount of contributions because of their unique partnership criteria, which require that a minimum of either 20% or 30%, depending on the province or territory, of the grant budget must come from external partner sources (i.e., non-CIHR funds). A review of EIS records on funded PHSI grants confirmed this, so PHSI data is not included in Figure 6. However, both Synthesis and K2A grants have also leveraged partner money and in-kind contributions in grant proposals, although this is not a formal requirement of the funding opportunity.

Figure 6: Synthesis and K2A researchers who reported partner financial and/or in kind contributions


Figure 6 long description

Source: KT Evaluation Survey responses 2011

While the above figure suggests that iKT grants effectively leverage partner contributions on grant proposals, we performed further analysis to examine the average amount of money leveraged per grant. Figure 7 shows that PHSI grants attract the largest amount of partner contributions.

Figure 7: Estimated $ value of partner contributions as a percentage of CIHR $ contributions
Mean on-grant partner investment per grant CIHR investment per grant Partner contribution as a % of CIHR contribution

Notes:

  1. Calculations based on respondents that reported partner contributions on the grant proposal and does not include contributions made that were not written into the grant proposal;
  2. Figures include estimates of both cash and in-kind support; 3) Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Synthesis $25,139 $87,010 29%
K2A $70,436 $151,115 47%
PHSI $151,386 $118,407 128%

The findings presented in Figure 7 may be attributable to a variety of factors. Firstly, a PHSI partnership criterion, as discussed above, requires leveraging money from partners. It is also important to consider the partners that are engaged through the PHSI funding opportunity. PHSI partners are primarily provincial governments and/or provincial health research funders. According to reported partner contributions across iKT grants, these organizations appear to have the capacity and political will to provide larger in-kind and monetary contributions.

While K2A and Synthesis grants share the common goal of moving knowledge into action by linking researchers and KUs, these funding opportunities attract partners that contribute smaller amounts, such as Canadian universities and research centers as well as other Canadian organizations. While these grants do not leverage as much total contribution as PHSI, they do provide opportunities for smaller organizations to participate in partnered health research. In-depth interviews undertaken with KUs and researchers from K2A and Synthesis grants revealed that forming linkages with diverse partners is a source of fresh ideas and perspectives, and this is asserted to fuel innovation. In short, the true value of these contributions should not be discounted because they are not financially equivalent.

Finally, PHSI attracts more partners on average per grant than other iKT funding opportunities.

Figure 8: Partners listed on iKT grants
Total Mean Median

Note:

  1. Calculations based on respondents that reported partners on the grant proposal and does not include partners that were not written into the grant proposal;2) Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Synthesis 199 6.0 3
K2A 53 2.4 2
PHSI 356 8.0 3

1.4 The sustained benefits of partnered research

In-depth interviews and case studies elicited a further discovery about KT funding opportunity outcomes and impact, which can be summarized as the sustained benefits of partnered research. It was reported by both researchers and KUs that additional, post-funding activities occurred between project partners that would not have occurred without the initial grant funding. For instance, it was reported that many of the partnerships created through KT funding evolved over time into sustained long-term collaborations, including instances where a partnership either spawned continued research on a topic or research on entirely new topics with the same collaboration. In addition, networks created through CIHR funded partnerships were cited to have induced new relationships and collaborations. In essence, partnerships formed as a result of CIHR KT funding lasted and continued to produce transformative health impacts well beyond the sphere of the funded project.

Similarly, about half of the KU and researcher interviewees reported direct outcomes related to capacity building and ownership resulting from the researcher-KU partnership. Anecdotes from funded partnerships are provided to illustrate these concepts.

On the KU side, capacity building was described as KUs using research, or even doing their own research in the future. For example, in one project the KUs returned to their local organizations and provided training to staff on how to use research to improve their roster of services. In another case, KUs returned to the research team to ask them for additional research to be used to develop new health policy and programs. Given the success of this process, the researchers reported that KUs from other regions, across Canada, have contacted them to apply the research findings to their local context. Ownership was described by KUs as a situation where, following involvement in the research process, they found greater value in the knowledge created, and were more likely to utilize this knowledge as a result. As an example, a KU reported that their in-depth understanding of research results allowed them to advocate, with an evidence-base, for changes in the way pharmacists receive up-to-date information via handheld devices.

On the researcher side, capacity building was described as learning how to increase research relevance for partners and society at large. For example, in one project researchers described how the presence of KUs in the design of information management systems improved the efficacy of their product because they were constantly exposed to end-user needs and perspectives.

1.5 Summary findings

Review of the achievement of CIHR KT funding opportunities illustrates how the KT Funding Program has produced expected KT outputs, academic outputs, and has engaged HQP.

The KT Funding Program Theory, as reflected in this evaluation's logic model, is a guide for advancing efforts to evaluate KT, and particularly partnered research funding, more comprehensively. It should also support and inform the design of more appropriate KT-related performance indicators for ongoing data collection at CIHR.

The KT Funding Program is largely successful in achieving expected outcomes, including contributing to the achievement of the CIHR mandate. Substantial quantitative and qualitative evidence supports this finding. Quantitative researcher survey data indicates that the involvement of partners in research not only happens more often in projects funded through iKT funding opportunities (where partnerships are a requirement) than in projects funded through the OOGP, but that iKT-funded researchers are also more likely to report influence with KU partners, and they report that their research leads to real-world action. Qualitative data presented in this section supports these findings and suggests reasons for how this happens. The existence of a "meaningful partnership" is identified as a specific catalyst of impact. What "meaningful partnerships" look like in reality is explored further in the next sub-section of this report, where five examples of impact are presented in order to further explore and elucidate this finding.

Respondents identified end of grant KT funding opportunities as an important means of timely and appropriate dissemination of findings to the most relevant KUs. Nearly every researcher surveyed (99%) reported that the DE and KTS funding opportunities fill a gap in the overall CIHR suite of funding. Furthermore, end of grant KT is highlighted as an important device for OOGP researchers performing KT and several distinct KU groups: industry, media, and other researchers/academics.

An exploration of differences in design factors between iKT funding opportunities related to partnering and financial leveraging requirements provided interesting findings. Specifically, the requirement on the PHSI funding opportunity for partner financial contributions on grant applications (20–30% of funding must come from partners) was found to lead to more funds leveraged per grant, to funds being leveraged on more grants, and to more partners per grant. However, who partners are and how they interact with researchers, was highlighted as an important consideration in every method of this evaluation. The fact that the PHSI funding opportunity attracts more partners and money does not guarantee that the impact of the funding opportunity is optimal to that of K2A and Synthesis. In fact, the above investigation of researcher cited impact showed the PHSI, K2A and Synthesis funding opportunities to achieve quite similar results. Then too, the fresh and unique perspectives brought to a research project by the partners who are capable of participating (i.e., without the 20–30% requirement) on K2A and Synthesis grants have been suggested by researchers to be a driver of innovation.

Finally, the post-funding activities undertaken by CIHR-funded partnerships were identified as a lasting and sustained benefit of each KT funding opportunity. "Capacity building" by both KUs and researchers as well as the "ownership" of research findings by KUs were cited as the underlying causes of this ultimately positive outcome.

2 - Program design and delivery

Evaluation Questions

  1. What factors facilitate or inhibit the achievement of KT funding opportunity outcomes?
  2. To what extent are KT funding opportunities being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness?

This sub-section of the report presents identified facilitators and hindrances of CIHR KT success. It provides context as to why the results and impacts identified in the previous key findings sub-section are occurring, and in what ways they are being obstructed. Evidence from each line of inquiry has been examined, and triangulated key findings are presented.

2.1 Facilitators of success

In order to examine factors that induced and encouraged program success, a pathway case study analysis was undertaken. In this method, cases are chosen which exhibit a common starting point and a known end outcome – in our situation, an exemplary outcome (Gerring 2007). Such case study analysis provides detail about causal factors of change or the active ingredients of success within the funding opportunities.

To pick cases for analysis, a subset of the Evaluation Working Group undertook a comprehensive selection methodology. This process included review of nominated cases from KT program managers based on anecdotal reports of exceptional achievement of outcomes, a review of survey and interview responses related to outcomes achievement, and a review of end of grant reports submitted to CIHR by funded project teams. Following this, a narrowed set of projects was independently reviewed and scored by members of the Evaluation Working Group. Independent scoring was then aggregated to select final projects for case study analysis.

Summary narratives and key findings representing each KT funding opportunity are presented below9.


NPI: Dr. Esther
Suter


PDM: Grace
Mickelson

Synthesis: Can lnterprofessional Collaboration Provide Health Human Resources Solutions? A Knowledge Synthesis

Research team: Dr. Esther Suter, Ms. Grace Mickelson (KU), Dr. Siegrid Deutschlander, Ms. Zahra Nurani (KU), Ms. Jana Lait, Dr. Elizabeth Louise Harrison, Dr. Sandra Jarvis-Selinger, Dr. Lesley Ann Bainbridge, Ms. Sheila Achilles (KU), Dr. Christine Anne Ateah, Dr. Kendall Ho, Dr. Ruby Emily Grymonpre
CIHR Investment: $100,000

Overview

Between the shortage of health care providers and the lack of communication and collaboration amongst them, there is a clear need to change how health services are delivered and the way in which providers interact with one another. lnterprofessional Collaboration (IPC) has been identified as a way to address this need. With the support of a Synthesis grant the research team set out to answer the following questions: (1) In what ways do IP interventions at the post-licensure level influence IPC and workplace quality, provider satisfaction, reuutrnent, retention and turnover? (2) In what ways do IP interventions at the pre-licensure level influence graduate employment choice? (3) Is there any evidence of cost benefits of IP interventions?

Pathways to outcomes

Involving KUs throughout the project lifecycle is essential to performing an impactful synthesis

Having KU involvement at project outset was critical. The project team stressed that the findngs world not have gained the same traction in the KU community without the involvernent of the right knowledge users at the right stages of the research process. The formal iKT structure requred in the grant application (for example, requiring a specific Principle Decision Maker) solidified the importance of the process for the team.

Forming a KT sub-group within the project team created focus and drive for meaningful research

Tasking a smaller intemal group with continuous attention to the KT strategy was a fruitful practice. This team ensured that at each stage of the project KT was occurring or being planned for.

Team members matter

The individual expertise and experience of team members was important to project success. However, the team experience emphasized that perhaps the most important characteristic of a team member - in order to move research into action - is the connections they hold with external knowledge user and knowledge user networks. The result was the broadening of knowledge uptake.

Impacts

Project team members emphasized that the Synthesis project made KUs better proponents of evidence-informed change within their organizations and networks. Accordingly, the completed Synthesis study has made a significant and momentous contribution to the field of IPC and Health Human Resources (HHR) in Canada. Provincial organizations in Manitoba, Saskatchewan, Alberta, and B.C. have used the synthesis to improve HHR priorities, with the consequence of real improvement in the quality of health products, services, and outcomes for these provinces as a result of more integrated and coordinated HHR.


NPI: Dr. Judith
Bartlett


PDM: Dr. Catherine
Louise Cook

Knowledge to Action: Need to know too

Research team: Dr. Judith Bartlett, Dr. Catherine Louise Cook (KU), Ms Sheila Carter, Dr. Patricia Martens (KU), Ms. Rose Neufeld (KU), Dr. Brenda Elias, Dr. Julianne Sanguins. Research staff: Mr. Nathan Hoeppner
CIHR Investment: $200,000

Overview

Despite representing a significant part of the population of Manitoba, very little research in Manitoba had considered Metis health needs separately from other Aboriginal groups. A data linkage study called the 'Metis Health Status and Health Service Utilization' was conducted to address this gap in knowledge. Using a holistic approach, the team set out to meet the following objectives: 1) Facilitate Manitoba Metis Federation (MMF) and Manitoba Health (MH) Regional Health Authorities (RHA) to develop and sustain a knowledge network; 2) Implement and evaluate a novel KT model between the University of Manitoba, MMF and MH; and 3) Utilize this KT model to accelerate the translation of the Metis Atlas study outcomes in adapting MH programs and services to better meet the needs of Manitoba Metis.

Pathways to outcomes

Involving stakeholders with the right skills from across the Manitoba health care landscape

The project brought stakeholders from all levels of health policy and programming to the table. During the application preparation the team carefully identified individuals with the best skills and organizations best positioned to facilitate putting knowledge into action. This resulted in a team with a wealth of expertise that spanned the Manitoba health care landscape.

Using a knowledge translation framework

K2A projects require the utilization of a KT approach that is supported by empirical evidence. Through the use of a framework, participants negotiated their roles across the span of the project. The use of a KT model demonstrated that all parties were working towards the same end-goal: to better the health of Manitoba Metis.

Involving high-level KUs

Having KUs with sufficient organizational power on the research team ensured that knowledge flowed from research to its applied use in an efficient way. Additionally, having sufficiently empowered knowledge users involved meant the pathway of information flow was clear and concise, and very little was lost or misconstrued along the way.

Impacts

As the project expanded across the province, all seven other MMF Regions and 10 of the 11 RHAs have engaged with the implementation team to be involved in the KT model to inform Metis-specific health policy and programming in their region. Accordingly, the province funded the MMF as a key member of the researchers' implementation team to keep this process underway. Furthermore, requests for more research from Knowledge Networks spearheaded additional studies on chronic diseases, which will be used to inform future policy and programming. In essence, for the first time, MMF Regions and associated RHAs are engaged in collaborations aimed at improving Metis-specific health outcomes in their regions. They have negotiated a common-ground relationship that resulted in better understanding the health needs of Metis in the region, and have set up a sustainable knowledge-to-action framework that enables ongoing policy and program change.


NPI: Dr. Shoo Lee


PDM: Dr. Eizabeth
Whynot

PHSI: National health partnership for reducing infections in NICU

Research team: Dr. Shoo Lee, Dr. Eizabeth Whynot (KU), Ms Elaine Orrbine (KU), Dr. K.J Barrington, Dr, G Ross Baker, Dr. Joanne Langley, Dr. Arne Ohlsson, Dr. Nicola Shaw
CIHR Investment: $100,000

Overview

Hospital acquired infection is one of the leading causes of mortality and morbidity in neonatal intensive care units (NICU). Despite efforts by individual NICUs to reduce infection rates using Continuous Quality Improvement (CQI), there has been little to no significant change in the rate of hospital acquired infection in NICUs. An evidence-based quality inprovement method called the Evidence-based Practice for Improving Quality (EPIQ) was developed and evaluated and was shown to reduce hospital acquired infection. Following the successful implementation of EPIQ, the research team, with the support of the PHSI grant, started research with the following objectives: to test the generalizabiity of the EPIQ model to all Canadian NlCUs; to determine whether quality improvement gains using the EPIQ model are sustainable; and to establish and evaluate a national system that will last beyond this study.

Pathways to outcomes

Building organizational change using institutional staff

In traditional cases, organizational change is advocated by senior management, enforced by administration, and facilitated by external consultants (these are called top-down approaches). The PHSI team believed that sustainable change could be built from the bottom-up. This organic process ensured that when the official faciitation process ended, the agents of change remained within the organization.

Involve everyone who has a vested interest

In order to produce sustained change, the team involved everyone that had a stake in the intervention - from hospital administrators to the housekeeping staff. From the outset, all staff particpated in focus groups led by these clinical teams that helped them identify the barriers to change in their institution. After this informiation was gathered, the team identified effective change processes that met their unique institutional needs. This ensured that all staff bought into the process, and understood the impetus for change.

Obtain financial commitments from partners

The institutions provided support in the form of in-kind donations of staff hours from participating NICUs. This ensured that staff would have the time, resources, and support to carry out the work required throughout the duration of the project. It also created incentives for meaningful engagement from the provinces.

Impacts

This study had significant inpacts on health policy and decision making and has the potential to fundamentally change the current paradigm for quality improvement efforts in NICUs across Canada. The original eleven NICUs continued to have reductions in infection rates. It was clear that the EPIQ process was sustainable as the behavior and culture of the NlCUs changed to continue to meet the challenges of caring for premature infants. The team continues to build upon the knowledge gained from the PHSI grant. They are currently embarking on EPIQ III, a study that has increased the scope of previous studies.


NPI: Dr. Katherine
Boydell

KT science: Producing and communicating knowledge differently: Articulating the pathway to mental health care through artistic expression

Research team: Dr. Katherine Boydell, Ms. Brenda Gladstone, Ms. Angela Kertes, Ms. Elaine Stasilius, Ms. Bramilee Dhayanandhan
CIHR Investment: $74,953

Overview

First episode psychosis is an important period for intervention to prevent future impairments and disabilities. Many youth do not seek help during this critical period, and current social stigma that is associated with issues of mental illness are thought to be responsible for the lag in time it takes to seek help. Efforts to educate and inform populations about first episode psychosis and mental health have been largely unsuccessful. Arts-informed methods provide a new, accessible medium where participants are challenged to engage with mental illness. This CIHR-funded project aimed to accomplish two objectives: to better understand the science of knowledge generation and translation using art-based methods, and to further the understanding of first episode psychosis education and awareness from the perspective of a small group of young people who had experienced first episode psychosis and accessed care.

Pathways to outcomes

Using a participatory (iKT) and iterative process

The research design was iterative, meaning that it was built with the participation of all involved parties and flexible to the needs of the participants. Success was attributed to the integrated knowledge creation that took place. The design decisions for the mural were made by the participants, giving them complete control over the final product- a process that provided the research team with rich data. The process reflected an understanding of the emotional process that the participants went through to produce a product that effectively communicated their stories.

Team experience and skills

The research team believes that their combination of expertise contributed significantly to the success of the project. The team worked together for many years and had extensive experience in first episode psychosis in young people as well as strong methodological skills.

Leveraging project-specific resources

The funding mechanism for KT science grants is designed to be flexible to accommodate new and innovative projects. The KTR Panel does not require an official partner on the grant; however, the grant assisted them in effectively leveraging additional resources which contributed to the success of the project.

Impacts

This project acted as a stepping-stone toward a more evidence-based field of practice in arts-based methodologies. Today, the team continues to engage in research that explores the value of arts-informed methods, including collaborating with an intemational consortium that focuses on how arts-based knowledge translation works. Dr. Boydell presented this research at national and intemational conferences and was asked to publish on the topic in Nature magazine. Several participants, as well as the research team, continue to be involved in the evolution of this project as it travels to rural schools in Ontario and PEI.


NPI: Dr. Sherry
Lynn Grace

KT Supplement: Promoting Inpatient Cardiac Rehabilitation Referral

Research team: Dr. Sherry Lynn Grace, Dr. Caroline Chessex, Dr. Jack Ven Tu, Dr. Blair Joseph O'Neill, Dr. Robert Donald Reid
CIHR Investment: $100,000

Overview

Cardiovascular diseases are the leading causes of mortality and morbidity in Canada. Cardiac rehabilitation (CR) is an outpatient chronic disease management program designed to enhance and maintain cardiovascular health. It has been found that participation in CR programs reduces mortality and morbidity rates, however, these programs are poorly attended. The two leading reasons why patients do not access CR are referral failure and lack of provider endorsement. This KTS project aimed to increase referral and enrollment of indicated cardiac inpatients, based a on evidence-informed referral and enrollment targets established by the research team and endorsed by the Canadian Cardiovascular Society and the Canadian Association of Cardiac Rehabilitation. The project consisted of a multipronged KT approach that included educational components, audit and feedback systems, and informatics.

Pathways to outcomes

Tailoring both the message and the mechanism for the audience

KT pathways are not always linear and often researchers and KUs explore and implement varied approaches in order to achieve the best KT outcomes. To address this complexity, the team implemented three purposefully different KT models in order to bolster referrals and enrollment in cardiac rehabilitation.

End-of-grant implementation involves relationships with KUs

Although not a requirement for KTS funding, the research team actively engaged KUs throughout the project. In fact. the KTS funding was cited by the research team to enable the forging of important partnerships. By working with hospital administrators and team leads, the research team was granted access to frontline staff. The buy-in from top-level employees at the participating sites ensured that the KT models were accepted and implemented effectively. Clinicians and frontline staff also played integral roles in the successful implementation of the models.

Integrating feedback from KUs

Throughout the course of the project, the research team interacted with a myriad of institutional partners, ranging from hospital administrators to performance measurement units. This work informed the research in many ways, particularly by lending a nuanced perspective of the challenges and barriers to implementing KT models.

Impacts

This grant led to the effective implementation of KT models that enhanced CR enrollment and referral. Promotional and educational materials were created to inform patients of the services available at CR and these materials are available in the participating cardiac units, and translated into 4 languages. The team also developed a referral toolkit, which is available online to CR clinics across Canada.

Additionally, this project supported the development of the internationally endorsed Charter on Cardiac Rehabilitation. The newly formed Intemational Collaboration in Cardiovascular Prevention and Rehabilitation will present the drafted charter at the World Congress of Cardiology 2012 in Dubai.

(For additional details on each of the research projects highlighted in the above case narratives, links to further reading and resources are provided in Appendix E of this report.)

In summary, these case studies – which were purposefully selected based on their exemplary results – have been used to identify active ingredients of success within the KT Funding Program. Key elements can be summarized as: engaging KUs in and throughout the research process; assuring commitment and buy-in from partners (but not necessarily financial); working with the right expertise within both the researcher and KU contexts, and; tailoring and timing the dissemination of results to the audience(s).

In addition to these key elements, another ingredient of success identified in this evaluation is the inclusion of KUs in the review of research applications.

2.1.2 Merit Review as a facilitator of iKT research project success

Merit review is grant review process used at CIHR exclusively for iKT grant proposals. It is designed to increase the probability of the selection of projects that will excel both in scientific rigor (scientific merit) and real world impact (impact). Merit review requires that both researchers and KUs be grant reviewers.

Through interviews with both researchers and KUs, the strategic design of merit review (requiring both scientific merit and impact to be considered by researcher and KU committee members) was asserted to be a facilitator of eventual project success. Survey results show strong approval for merit review as over 82.3% of researchers were either satisfied or very satisfied with the merit review process.

In addition, variations of the Merit Review process are the standard at several high-profile private and public research funding agencies, including the Bill and Melinda Gates Foundation and a Government of Canada supported offshoot of the Gates Foundation: Grand Challenges Canada. Both organizations use non-researcher expertise in the review of grants in order to ensure the scalability (i.e. real world value) of funded research (Grand Challenges Canada 2010; Gates Foundation 2010).

For example, in their Global Health Strategy the Gates Foundation describes the objectives of their review process as follows:

"Our goal is to ensure that we are considering the widest range of funding opportunities and hearing diverse perspectives on the relative merit of those opportunities. The review process for all large grants involves input from a broad cross-section of outside experts, other funders, and other stakeholders. The vast majority of our grants, even many of the smallest, are shared with experts in an external review." (Gates Foundation 2010)

Further to this alignment with several innovative health research funders, during our interviews with other national research funding organizations anecdotal interest was shown in learning more about CIHR's Merit Review process. For instance, questions arose around how Merit Review could be implemented at the country level out of the stated desire of mimicking the process at their own national funding organization.

Additionally, the review of grants by potential users of research is very much in line with the post-Jenkins Report10 environment in which CIHR operates, including the direction for innovation laid out in the GOC Budget 2012. In this light, merit review presents significant value to CIHR as it moves forward in the current federal policy context.

Given the indications of the value of Merit Review uncovered in this evaluation, further analysis of the results and effectiveness of merit review would be both innovative and of high value to CIHR. Such analysis was not within the scope of this evaluation.

2.2 Hindrances of success

This evaluation also aimed to reveal any underlying challenges or weaknesses with/within the KT Funding Program. To perform this inquiry, a sequential mixed method approach was employed. Qualitative interviews probed researchers and KUs on four broad categories (internal, external, CIHR strategic, and CIHR program delivery) and through these interviews specific challenges under each of these categories were uncovered. Based on the range of issues unearthed in exploratory interviews, a series of survey questions were used to investigate the generalizability of these issues across the wider population. The most prevalent are discussed below, by category.

2.2.1 Internal to the funded project team
The additional effort required to involve KUs in research

The most frequently cited internal concern, by both researchers and KUs, was the amount of effort required to perform iKT research. Doing research in a partnership was considered to be more time consuming, expensive, and uncertain than doing the same project in a traditional research format. Moreover, creating and fostering a meaningful partnership – the type that is an identified driver of KT project success – was cited to require added effort and considerable tact. The additional burdens of time and finances are related, quite directly, to partnering activities such as more meetings and travel. Additional uncertainty was tied to matching the research to the KUs' timelines and the risk of a change in the KU environment that would affect the research process or results (e.g., staff turnover or changes in organizational mandates).

Related to each of the issues cited above is the finding that iKT research was considered to require diplomacy skills not typically associated with traditional investigator-driven research. For instance, negotiating roles and responsibilities, encouraging consistent and effective team member engagement, and ensuring the right researchers engaged the right KUs at the right instances (and vice versa) were all cited to require significant 'non-academic' skills. It is essential to note, however, that these skills are asserted by respondents to be vital for both researchers and KUs, as partnering is considered a dyadic process. Figure 9 below illustrates survey responses to a question developed around this topic. The results indicate this is a generalized challenge across the iKT funding opportunities.

Figure 9: Internal challenges


Figure 9 long description

Note:

  1. Values under 10 percent are not written into the figure, but are represented at scale;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

2.2.2 External to the funded project team
Timing research to meet KU needs

Several challenges were cited as interferences from the external environment, which was defined for our purposes as that which is external to the funded project team. The first of these was timing research to meet the needs of the KU.

Both researchers and KUs identified matching the research production cycle with the knowledge use cycle as a challenge of doing knowledge translation. This finding was consistent across both iKT interviewees and interviewees undertaking end of grant KT not using an iKT approach. In simpler terms, delivering research in a timely manner to KUs – such as industry leaders, policy-makers and/or health care practitioners – is challenging.

Two solutions are, however, implicit to the current range KT funding opportunities. On the one hand, in-depth case study analysis shows that a key to partnering and knowledge translation success is the engagement of KUs in an iKT research approach. Specifically, having KUs aware and involved through the research process increases the meaning, utility, and timeliness of research findings. It should be noted that this finding is not unique to the CIHR environment and is well supported by empirical evidence and theory on collaborative research methods (Israel et al. 1998; Macaulay et al. 2011). Both KUs and researchers interviewed as a part of our evaluation argued that this type of research would not happen in the absence of a strategically targeted funding opportunity supporting partnered research.

On the other hand, the use of end of grant KT funding opportunities (KTS and DE) was considered an important tool for rapid research dissemination. The relatively open and flexible application process paired with the prompt turnaround of funding decisions helps researchers to share results with relevant KUs on demand, and in varied and appropriate ways. Another key benefit cited was the ability to engage KU groups who might not have been identified as relevant at the outset of a research project.

Doing KT in the university setting

In both interviews and case studies a tension was reported between working in a university setting and pursuing KT activities. Two specific causes of this tension were identified. The first was spending the necessary effort on partnering in an iKT research approach. The second was related to the creation of KT outputs and outcomes. This includes the development of end products, such as clinical care pathways, practical decision-aids, policy documents, websites, and commercialized products and services.

Universities generally value the creation of knowledge through traditional paths and the larger financial grants associated with such research. Indeed, researchers identified that academic tenure and career progression is often tied, explicitly and implicitly, to traditional knowledge creation products – including peer-reviewed publications, books, and book chapters – as well as to the attraction and retention of larger grants and awards. Thus, a direct conflict exists for the university-based researcher who pursues and performs KT and who seeks career advancement and security at his/her university.

Figure 10: External challenges


Figure 10 long description

Notes:

  1. Values under 10 percent are not written into the figure, but are represented at scale;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Upon further investigation, this was not a unique research finding. Estabrooks et al. (2008) reach similar conclusions in an empirical study of university researchers. Of further interest is the research of Jensen et al. (2008) who found that scientists who engage with the public are just as likely to perform to excellent scientific standards. Further investigation of this issue, perhaps in collaboration with universities, may be of value to CIHR in order to improve incentives toward the pursuit of KT objectives.

2.2.3 CIHR strategic challenges
Length of time and amount of funds for iKT research

Closely related to the major internal challenge identified through this evaluation are the identified strategic challenges of iKT funding opportunity time-length and financial limitations11.

Figure 11: CIHR strategic challenges


Figure 11 long description

Notes:

  1. Values under 10 percent are not written into the figure, but are represented at scale;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

Funded researchers express a desire for strategic changes to be made to iKT funding opportunities that would increase funding and length of grants. To consider this desire further, and to examine an additional indicator beyond researcher perspective, an analysis of grant holder behavior is presented below.

To begin, Figure 12 illustrates the difference between funding opportunity length and actual length of time spent on the same project by grant recipients.

Figure 12: Comparison of expected iKT research grant length and average actual duration of iKT grants
Maximum length of grant on RFA (months) Average actual duration (months)

Notes:

  1. Figures represent months;
  2. Actual duration is the period between the effective date of a first version of a grant and the authorization to use funds date of the last version number.

Source: EIS data 2012

Synthesis 12 28.6
K2A 24 39
PHSI 36 47.8

These data indicate that researchers do take more time than they are allocated to complete their funded research projects, however, the reasons for this are less obvious and could be numerous. The qualitative and quantitative data gathered in this evaluation suggest that the considerable effort required to perform iKT research could be one significant underlying reason.

The desire to increase the financial contribution of these grants is also logically linked to the challenges of doing iKT research, and like the desire the increase time lengths this concept was reiterated through in-depth interviews with funded researchers. Examples cited of the explicit financial costs of partnered research surfaced in activities such as the co-design of the research process, consensus-seeking with partners, and the tailoring of research outputs to KU audiences. All of these activities can require significant travel, telephone conferencing, or simply the paid time of both researchers and KUs. Accordingly, a need for additional funds to finance these activities is perceivable. Figure 13 below indicates maximum and actual contributions made by CIHR to iKT research projects.

Figure 13: Comparison of maximum CIHR financial contribution and actual CIHR financial contribution
Maximum contribution listed on RFA Average actual CIHR contribution

Notes:

  1. Actual contribution figure does not include partner contributions;
  2. Actual contribution figure calculated per FRN.

Source: EIS data, 2012

Synthesis $100,000 $136,789.31
K2A $200,000 $167,389.75
PHSI $400,000 $201,692.98

Figure 14 indicates that on average only the Synthesis funding opportunity disburses more funding than is listed on the grant RFA. This data is contrary to what is noted in Figure 11, where researchers suggest that the amount of funding available in the grants is a hindrance to success. In addition, survey data tells us that the vast majority of researchers (88% across iKT funding opportunities) have received the funding that they request in their original grant proposal.

Of the 12% that did not receive the amount they requested, Figure 14 indicates that a minority of these researchers were able to finish their grants in the anticipated timeframe, however, the majority did complete their project within the intended scope. This finding is verified in Figure 13 above. Accordingly, these data indicate that the issues of funding contributions and time-length of the iKT funding opportunities may well be intertwined.

Figure 14: Impact of a reduction in requested funding: iKT funding opportunities


Figure 14 long description

Note:

  1. Multiple responses were allowed;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

However, in terms of funding contributions from CIHR, this is not adequate evidence to fully judge whether the current funding ceiling, per funding opportunity, is sufficient or insufficient to perform research activities, as numerous confounding factors could be skewing this analysis. Furthermore, very little additional evidence is available for triangulation; mainly for two reasons. First, traditional research funding offered at CIHR is an inappropriate comparison, given it is not designed to encourage collaborative approaches to research. Second, data from the international scan of KT funding activities uncovered very little comparable programming, and none that had been evaluated, making it impossible to establish an evidence-based indication of adequate funding requirements for partnered research.

Finally, it should be reiterated that iKT funding opportunities did quite well at achieving expected outcomes (see the key findings presented in sub-section 1). This begs the question of whether funding increases would affect results proportionately, disproportionately (upward or downward), or if they would have any effect at all. Data collected in this evaluation do not answer this question.

2.2.4 CIHR program delivery challenges
CV and application issues

As discussed earlier in this report, researchers and KUs are satisfied with the processes of KT funding program delivery. Specifically, the merit review process for iKT grants and the timely administration of end of grant KT funding applications have been highlighted as contributing to, and facilitating, research project success. However, while program delivery has not been identified as a systemic barrier to program success, one opportunity for improvement arose through in-depth interviews and tested similarly in a survey of funded researchers. The issue is grant application for KUs and for researcher-KU partnerships.

Firstly, it was identified that for KUs – non-academic applicants who are encouraged and expected to apply to iKT funding opportunities – the application process is perceived by respondents to be tedious and to include many unrealistic criteria. For one, these partnering KUs face an overly strenuous and awkward task when trying to match their CV to the CIHR format requirements. During interviews, both KUs and researchers argued that CIHR could improve the application process by allowing non-academic CVs to be accepted from non-academic applicants. Additionally, a few interviewees noted that the CV module utilized by CIHR is actually different from other funding agencies, such as NSERC, and a common tri-agency approach would improve the efficiency of application.

Secondly, both researcher and KU interviewees noted that the application template is especially tiresome in requiring partnership details that are often simply unrealistic or unavailable. A common example cited was the requirement for listing the number of hours per week each KU was committing to the project. These requirements, which essentially predefine how the partnership will be structured, are inconsistent with findings of this evaluation which indicate that flexibility and idiosyncratic design are elements underpinning a meaningful, and therefore, successful partnership. Figure 15 below illustrates survey responses on this issue.

Figure 15: External challenges


Figure 15 long description

Note:

  1. Values under 10 percent are not written into the figure, but are represented at scale;
  2. Sample sizes are detailed in Appendix B.

Source: KT Evaluation Survey responses 2011

These findings, although certainly helpful for CIHR program delivery, are not unique to KT funding opportunities as KU and partnership criteria are the same for OOGP applications. In fact, researchers identified that the Merit Review process, which includes KUs in the review, helps facilitate the fair assessment of partnerships and partners. It should also be considered that, the respondents providing their views received funding from 2005 onward. Over this period CIHR has undergone multiple changes to the CV modules and funding opportunity application requirements. In this sense, goal posts are moving.

2.3 Summary Findings

In-depth case study analysis of highly successful KT projects offer insight into understanding key elements of success across the funding opportunities that support synthesis, iKT, end of grant KT, and KT science. The common themes to emerge include: engaging KUs in the research process, assuring commitment and buy-in from partners (not necessarily financial), tailoring dissemination to the audience, and working with the right subject area experts (within both the researcher and KU contexts). The factors highlighted in these cases correspond well with findings from other methods of the evaluation. As a triangulated set, these themes provide a guide to understanding which factors should be encouraged and supported in order to stimulate successful KT. Individually, the case study narratives play a valuable role as they provide real-world evidence to support CIHR KT Funding Program Theory, and in doing so, demonstrate the performance and impact of CIHR funded research.

This section has also presented Merit Review as a key element of iKT funding opportunity success, and, notably, the use of similar methods of application review by other funding agencies as a means of enabling the translation of research into action.

Hindrances to success are also identified in this section of the report, and included: the substantial effort required to do iKT research , timing research with KU needs, doing KT in a university setting, the length of time and the amount of funds provided for iKT research, and CV and application issues related to KU partners and partnerships.

The inhibiting factors identified are useful for understanding what may be slowing or diminishing the success of the KT Funding Program and the individual funding opportunities comprising it.

3 - The role of CIHR in KT

Evaluation Questions

  1. What role is there for CIHR in enabling/promoting KT (iKT, synthesis, end of grant KT, and KT science)?

The over-arching purpose of this sub-section is to investigate the relevance of CIHR's position in promoting/enabling KT. To do this, this sub-section provides an overview of what CIHR aims to achieve through KT. Secondly, it presents a comparison of these aims to current Government of Canada (GOC) policy related to research and innovation. Thirdly, it presents key findings from an international environmental scan of KT funding to investigate the role CIHR plays within the Canadian and global contexts. Finally, it highlights data obtained through a scan of CIHR's EIS database and from evaluation survey results, both of which help to illustrate the use of KT funding opportunities by CIHR's client-base.

3.1 – What is CIHR pursuing through KT?

In health practice – including health products, services, and systems – a generally accepted desire exists for product and service provision to be informed by valid and reliable evidence. There is, however, an abundance of evidence indicating that health practice often lags behind knowledge and best practices established through health research (AHRQ 2001; Mitton et al. 2007). In this context, there is an increasing need for health research to be better linked to health innovation and its widespread application.

To address this issue, efforts have been made to promote the use of research-developed and tested evidence in practice. This concept of moving research into practice has become known by many names including KT (Straus et al. 2009; Tetroe et al. 2008). Since the creation of CIHR in 2000, the organization has been strategically involved in the effort to promote the use of evidence in practice through KT.

In other words, the KT program at CIHR has been offered with strategic intent. To support this, a precise definition of KT was developed and a set of activities, including standalone funding opportunities, were created and supported in order to increase the uptake of knowledge. Figure 1 in Section 1 of this report outlines the current KT program. Strategically, the program has included support for iKT, specific attention to innovative end of grant activities with knowledge users (KUs), and an acknowledgement of the needs of knowledge users for comprehensive syntheses of research evidence.

Although much evidence exists to support the need for KT, very little research exists that measures the performance and impact of KT interventions, especially when those interventions are funding mechanisms (Lavis et al. 2006; Tetroe et al. 2008; Cordero et al. 2008). The international environmental scan undertaken in this evaluation added further support to this finding. Of the 26 funding agencies who participated in the scan, not one had completed an evaluation study of their KT activities that would be comparable to those offered at CIHR. The need for such evaluative work is further demonstrated by Dalziel et al. (2012) in a recent systematic review of research impact studies. The authors find that this gap spans beyond the field of health research to impact studies across numerous research fields (Dalziel et al. 2012).

3.2 KT at CIHR and within the Federal Government context

The importance of KT to CIHR is clear. The concept is embedded in the CIHR mandate and written into the Parliamentary Act which created CIHR as it now exists:

The objective of CIHR is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system. (GOC Bill C-13 2000)

In addition, the desire for the application of research in society is an objective of significant importance to the Government of Canada (GOC Budget 2012; GOC 2012; Industry Canada 2007). As a dynamic and iterative process that leads to the ethically-sound application of knowledge, investments in KT enable researchers and KUs to move their discoveries and innovations into solutions to health challenges which in turn improve social welfare and economic competitiveness. Findings presented in this evaluation report validate that this is happening.

As such, these investments directly contribute to the vision outlined in the Federal Government's Budget 2012. Indeed, the GOC Budget 2012 indicates clearly that the Government supports investments in basic and applied research, and the translation of public research knowledge to applied uses. Chapter 3.1 of GOC Budget 2012 highlights "Knowledge Translation" as a specific area of importance to Canadians. Furthermore, as a research process, each KT funding opportunity directly aligns with priorities stated in the Federal Science and Technology Strategy, and specifically contribute to the Entrepreneurial Advantage and Knowledge Advantage priority areas outlines in the same Federal strategy (Industry Canada, 2007).

In sum, there does exist a close alignment between CIHR KT and current GOC plans and priorities for research and innovation.

3.3 The international KT context

In order to better understand the relevance and position of CIHR in promoting and enabling KT, an international environmental scan of research funding agencies was undertaken as a part of this evaluation. A full list of agencies examined in this scan is provided in Appendix A of this report. Figure 16 provides a contextual lens for positioning CIHR in KT, both nationally and internationally.

Figure 16: Overview of key statistics related to KT at 26 funding agencies
% Budget allocated to KT Annual budget for KT (CAD millions) Annual budget (CAD millions) Dedicated KT staff (FTE) KT prioritization

Notes:

  1. Full agency names provided in Appendix 1;
  2. Financial figures are in CAD, converted December 18, 2011;
  3. "KT prioritization" was self-reported by participating agencies and ranked on a 5 point scale (1= very unimportant; 2 = unimportant; 3 = neither unimportant nor important; 4 = important; 5 = very important);
  4. Empty cells denote information not provided by the agency.

Source: Interviews with senior officials performed in 2011. We cannot be certain all KT activities are captured as we did not engage every division of each agency. Figures have been approved by each agency.

Australia
NHFA 61.6 44 71 0 "High"
CCA 17.6 information not provided by the agency information not provided by the agency 1 information not provided by the agency
NHMRC information not provided by the agency information not provided by the agency information not provided by the agency 80 Very Important
Canada
SSHRC 7-9 19.4 350.9 2 information not provided by the agency
FRSQ 5 5 100 0 Between Neither Important or Unimportant and Important
AHFMR 2-6 0.34 + embedded 91.9 3 Very Important
CIHR 1-3 30 + embedded 1,000 15 Important
MSFHR 1.4 0.45 33 2 Very Important
CHSRF information not provided by the agency Embedded 15.2 Embedded Very Important
NCIC information not provided by the agency Embedded 41 Embedded Very Important
NSHRF information not provided by the agency N/A 4.9 1 Very Important
SHRF information not provided by the agency Embedded 6 0 Important
Netherlands
ZonMW information not provided by the agency information not provided by the agency information not provided by the agency 20 information not provided by the agency
Scandinavia
FSS Embedded Embedded 44 information not provided by the agency information not provided by the agency
RCN information not provided by the agency Embedded 1,261 information not provided by the agency information not provided by the agency
United Kingdom
HF 0.95 0.40 42 Embedded Neither Important or Unimportant
CSO 0.58 0.62 106 1 information not provided by the agency
NIHR HS&DR 13.3 2.4 18 2.5 Between Neither Important or Unimportant and Important
WT not fixed not fixed 968 45 Very Important
AS information not provided by the agency Embedded 124 Embedded Important
NHS HTA information not provided by the agency information not provided by the agency 14 information not provided by the agency Important
UK MRC information not provided by the agency Embedded 1,215 152 Very Important
United States
RWJF 85 340 400 35 Very Important
AHRQ information not provided by the agency 31 information not provided by the agency 300 Very Important
NIH-NCI information not provided by the agency information not provided by the agency information not provided by the agency 7 Important
VA information not provided by the agency Embedded 18 Embedded Very Important

The environmental scan indicated that KT is known by many different names around the world. Representatives of numerous funding agencies spoke of 'alternative' descriptive terms used to define what is 'commonly' known as KT including: translational research, implementation science, knowledge transfer, and knowledge exchange. The definition of KT pioneered by CIHR has been adapted by both funding and non-funding agencies in Canada and abroad. For instance, it is the official definition employed at the World Health Organization (WHO 2006).

Another indicator of the significance of KT as a strategy and/or objective is presented in Figure 17 below. Examined in time-series12, results of this scan provide telling indications of the growing importance of KT to research funders around the world.

Figure 17: Change in KT inclusion in mandate over time (2008 vs. 2011)


Figure 17 long description

Notes:

  1. "n" represents the number of funding agencies;
  2. "%" represents the percentage that include KT in their agency mandate.

Source: KT environmental scan, 2011

Figure 17 shows that the inclusion of KT in the mandate of research funding agencies has either remained the same or increased in occurrence across the six regions/countries examined in this scan. All of the funding agencies contacted in 2011 offer some form of targeted KT funding. These findings indicate that CIHR is well aligned with a current global trend of promoting/enabling KT in health research funding. Considering the time-series nature of this data, and that CIHR has included KT in its mandate since 2000, it becomes apparent that CIHR has been ahead of the curve in promoting and enabling KT as a research funding agency strategy and objective.

The following statement by Australia's Chief Scientist helps to illustrate the importance and prominence of funding KT as a part of research funding around the world.

3.4 Who is using the KT Funding Program?

Another evaluative indicator of CIHR's role in promoting/enabling KT is an examination of program use. In other words, the CIHR KT Funding Program may only be as relevant as it is useful to CIHR's client-base. Although KUs are a required part of iKT funding opportunities, the lack of data on KUs in CIHR's EIS database required this analysis to focus only on researchers. Figure 18 provides an illustration of the demographics of KT researchers per KT funding opportunity. A reference group of CIHR open operating grant program (OOGP) researchers is included.

Figure 18: Researcher profiles (%)

Pillar
I II III IV
Synthesis 2 26 39 33
K2A 0 18 54 29
PHSI 0 9 70 21
KT science 0 46 23 31
KTS 5 26 26 44
DE 21 16 32 36
Total KT 9 21 38 32
OOGP 70 14 6 10
Years of Experience
0-5 6-11 12–17 18+
Synthesis 26 25 13 27
K2A 11 25 36 29
PHSI 17 38 10 34
KT science 19 38 31 12
KTS 12 34 22 34
DE 23 29 20 27
Total KT 21 33 19 27
OOGP 2 32 18 47
Gender (sex in OOGP case)
Female Male

Note:

  1. OOGP "sex" data derived from EIS 2005–2012;
  2. OOGP "sex" data excludes 13 cases of "unknown sex";
  3. KT "gender" data derived from a survey question which used the wording "gender" rather than "sex". Accordingly caution should be taken in drawing comparisons between KT and OOGP figures.

Source: KT Evaluation survey responses 2011; OOGP based on RRS pilot data, except sex data which is based on EIS 2005–2012.

Synthesis 57 43
K2A 68 32
PHSI 57 43
KT science (OOGP KT review panel) 52 48
KTS 69 31
DE 56 44
Total KT 58 42
OOGP 30 70

In general, demographic data illustrates that KT grants are awarded to a different variety of researchers than is the case with the OOGP. For one, KT grants are held mainly by pillar II, III and IV researchers – whereas the OOGP is dominated by pillar 1 researchers. Secondly, KT grants are held by researchers of all experience levels, including new investigators (0-5 years experience) – the OOGP funds predominately more experienced researchers. Finally, across each KT funding opportunity, female grant holders outnumber male grant holders – OOGP data shows the reverse.

These demographic data highlight some further detail about who KT funding opportunity recipients are. Firstly, the majority of iKT researchers do report pillar III or IV as their primary research domain, however, 21% of KT researchers report pillar II as their primary research domain, and 9% report pillar I. The vast majority of pillar I KT funded researchers use end of grant KT funding opportunities DE and KTS13. Secondly, Figure 18 shows that although researchers awarded KT grants are often new investigators, many others are highly experienced; 46% report over 12 years of experience as an independent researcher and 27% report over 18 years of experience.

3.5 Summary findings

In addition, a review of researchers who have received KT grants shows that KT funding opportunities reach a broad range of researchers, and are particularly used by the three groups that are less represented in OOGP funded projects: female researchers, pillars III and IV researchers, and new investigators.

Ample evidence exists suggesting a lag or often complete failure in moving research evidence into practice (AHRQ 2001; Mitton et al. 2007). Since its inception through act of Parliament in 2000, CIHR has made KT its strategic response to this issue. A review of current GOC policy and strategic direction indicates a strong desire for research to be applicable to society, and the CIHR KT Funding Program is well suited to fulfill this aspiration.

An international scan of health research organizations illustrates the pursuit of KT by health research funding organizations is emerging around the world as an important strategic objective. It has also brought to light the leadership role CIHR has played in this process.

Methodology

To ensure that evaluation findings were robust and credible, and that valid conclusions were drawn about the performance of the programs, the evaluation used multiple methods and draws on both quantitative and qualitative evidence. A list of evaluation questions, indicators, and sources is available in Appendix B. For a comprehensive outline of the evaluation protocol, see: McLean et al. 2012.

Method Purpose Focus and approach
International environmental scan
  • To update and expand on Tetroe et al. 2008 in order to profile and compare, in the international context, KT at research funding agencies and its evaluation14
  • 26 major research funding agencies (Canada, USA, UK, Netherlands, Norway, Sweden, Australia)
  • Website and publications scan, followed by semi-structured interviews with each agency
Document, literature, and EIS data reviews
  • To provide information that describes and contextualizes KT programming, and its environment
  • To provide quantitative demographic data about program users and inform the sampling process for other data collection methods
  • CIHR publications, GOC publications
  • Academic and grey literature on KT and KT funding
  • CIHR administrative data; including Electronic Information System (EIS) data and grant files
Key informant interviews
  • To gather in depth perceptions and experiences of KT funding programs
  • KT funded researchers and knowledge-users (n = 29)
  • CIHR senior officials (n=8)
  • Semi-structured telephone and in-person interviews
Surveys
  • To gather more generalizable quantitative data to address evaluation questions
  • KT funded researchers (n = 379)
  • Online survey questionnaire, versioned by KT funding opportunity
Case studies
  • To provide rich contextual data related to KT projects.
  • To analyze, in-depth, mechanisms that facilitated project success.
  • Highly successful KT funded projects (Synthesis, PHSI, K2A, KTS, KT science) (n=5)
  • Site-visits (where possible), semi-structured interviews, and document review

References

Agency for Health Research and Quality. (2001). Translating research into practice (TRIP)-II. Retrieved February 11, 2012.

Canadian Institutes of Health Research. (2010). Knowledge Translation (KT) activities at CIHR. Retrieved October 18th, 2012.

Cargo, M., & Mercer, S. L. (2008). The Value and Challenges of Participatory Research: Strengthening Its Practice. Annual Review of Public Health, 29, 325–50

Cordero, C., Delino, R., Jeyaseelan, L., Lansang, M.A., Lozano, J.M., Kumar, S., Moreno, S., Pietersen, M., Quirino, J., Thamlikitkul, V., Welch, V.A., Tetroe, J., ter Kuile, A., Graham, I.D., Grimshaw, J, Neufeld, V., Wells, G., & Tugwell, P. (2008). Funding agencies in low- and middle-income countries: support for knowledge translation. Bulletin of the World Health Organization, 86(7), 524-534.

Chubb, I., (2012), Can Australia afford to fund translational research? Keynote Address to the BioMelbourne Network, April 3rd 2012, Melbourne, Australia.

Dalziel, M., Roswell, J., Tahmina, T. N., & Zhao, X. (2012). Impact of Government Investments in Research & Innovation: Review of Academic Investigations. Optimum Online; The Journal of Public Sector Management, 42(2), 19–32.

Estabrooks, C.A., Norton, P., Birdsell, J.M., Newton, M.S., Adewale, A.J., & Thornley, R. (2008). Knowledge translation and research careers: Mode I and mode II activity among health researchers. Research Policy, 37(6-7), 1066–1078.

Gates Foundation. (2010). Global Health Strategy Overview [ PDF (252 KB) - external link ]. Retrieved May 24, 2012.

Gerring, J. (2007). Is there a (viable) crucial case method? Comparative Political Studies, 40(3), 231-253.

Government of Canada. (2000). Government of Canada bill C-13. Retrieved February 13, 2012.

Government of Canada. (2012). Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report [ PDF (4.6 MB) - external link ]. Retrieved April 15, 2012.

Government of Canada. (2012). Budget Plan: Budget 2012. Retrieved April 15, 2012.

Grand Challenges Canada. (2010). Integrated Innovation. Toronto, ON: McLaughlin-Rotman Centre for Global Health.

Industry Canada. (2007). Mobilizing Science and Technology to Canada's Advantage. Ottawa, ON: Publishing and Depository Services.

Israel, B.A., Schulz, A., Parker, E., & Becker, A.B. (1998). Review of community-based research: assessing partnership approaches to improve public health. Annual Review Public Health, 19,173-202.

Knowledge Translation Branch, CIHR (2010). Knowledge Translation 10 year Report (draft). Ottawa, ON.

Jensen, P., Roquier, J. P., Kreimer, P., & Croissant, Y. (2008). Scientists who engage with society perform better academically. Science and Public Policy, 35(7), 527-541.

Lavis, J. N., Lomas, J., Hamid, M., & Sewankambo, N. K. (2006). Assessing country-level efforts to link research to action. Bulletin of the World Health Organization, 84(8), 620-628.

Lomas, J. (2000). Using 'linkage and exchange' to move research into policy at a Canadian foundation. Health Affairs, 19(3), 236-240.

Macaulay, A., Sirett, E., Bush, P. (2011). Community-Based Participatory Research. In Oxford Bibliographies 2011. Oxford: Oxford University Press

McLean, R.K.D., Graham, I.D., Bosompra, K., Choudhry, Y., Coen, S.E., MacLeod, M., Manuel, C., McCarthy, R., Mota, A., Peckham, D., Tetroe, J., & Tucker, J. (2012). Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science, 57(7).

Mitton, C., Adair, C. E., McKenzie, E., Patten, S. B., & Perry, B. W. (2007). Knowledge transfer and exchange: Review and synthesis of the literature. The Milbank Quarterly, 85(4), 729-768.

Straus, S. E., Tetroe, J., & Graham, I.D. (2009). Defining knowledge translation. Canadian Medical Association, 181(3-4), 165-168.

Tetroe, J.M., Graham, I. D., Foy, R., Robinson, N., Eccles, M. P., Wensing, M., Durieux, P., Legare, F., Nielson, C.P., Adily, A., Ward, J.E., Porter, C., Shea, B., & Grimshaw, J.M. (2008). Health research funding agencies' support and promotion of knowledge translation: An international study. The Milbank Quarterly, 86(1), 125-155.

World Health Organization (WHO). (2006) Bridging the "Know–Do" Gap; Meeting on Knowledge Translation in Global Health. Geneva, Switzerland.

Appendices

Appendix A – Organizational Scan Participants

Organizations which participated in 2011 environmental scan (n=26)

Australia
  • Cancer Council Australia (CCA)
  • National Heart Foundation of Australia (NHFA)
  • National Health and Medical Research Council (NHMRC)
Canada
  • Alberta Heritage Foundation for Medical Research (AHFMR)
  • Canadian Health Services Research Foundation (CHSRF)
  • Canadian Institutes of Health Research (CIHR)
  • Fonds de recherche en santé du Quebec (FRSQ)
  • Michael Smith Foundation for Health Research (MSFHR)
  • National Cancer Institute of Canada (NCIC)
  • Nova Scotia Health Research Foundation (NSHRF)
  • Saskatchewan Health Research Foundation (SHRF)
  • Social Sciences and Humanities Research Council (SSHRC)
Netherlands
  • Netherlands Organization for Health Research and Development (ZonMW)
Scandinavia
  • Danish Agency for Science, Technology and Innovation – Danish Council for Independent Research – Medical Sciences (FSS) – {Formerly: Danish Medical Research Council (DMRC)}
  • Norwegian Medical Research Council (RCN)
United Kingdom
  • Alzheimer's Society (AS)
  • Chief Scientist Office (CSO)
  • Health Foundation (HF)
  • National Health Service Health Technology Assessment (NHS HTA)
  • National Institute for Health Research, Health Services and Delivery Research (NIHR HS&DR) - {Formerly: National Health Service Delivery and Organization (NHS SDO)}
  • UK Medical Research Council (UK MRC)
  • Wellcome Trust (WT)
United States
  • Agency for Healthcare Research and Quality (AHRQ)
  • National Institutes of Health –National Cancer Institute (NIH-NCI)
  • Robert Wood Johnson Foundation (RWJF)
  • U.S. Department of Veteran Affairs (VA)

Appendix B – Survey Statistics

Funding opportunity Population of unique grant recipients
All KT grant recipients 833
All end of grant KT 401
All iKT 344
K2A 65
Synthesis 145
PHSI 134
KTS 146
DE 255
KT science panel on the OOGP 88
Survey response rates and CI, per KT type and funding opportunity
Funding opportunity # of respondents Confidence interval (95% CL, at 50%)
Entire sample
All KT 379 3.72
All end of grant KT 189 5.19
All iKT 164 5.54
K2A 28 n/a
Synthesis 89 6.48
PHSI 47 11.56
KTS 43 12.60
DE 146 5.31
KT science panel on the OOGP 26 n/a
Sample who indicated their grant was completed at the time of the survey
All KT 258 5.07
All end of grant KT 157 6.11
All iKT 85 9.24
K2A 19 n/a
Synthesis 49 11.43
PHSI 17 n/a
KTS 18 n/a
DE 139 5.62
KT science panel on the OOGP 16 n/a

Appendix C – Funding Opportunity Descriptions

Synthesis

Background

Knowledge synthesis grants provide funding to researchers to produce scoping reviews or synthesis that meet the needs of decision makers or knowledge users in all areas of health using an iKT approach. They require that knowledge users should identify synthesis questions in collaboration with researchers so that the answers to these questions can inform policy, programs and practice. They are also expected to increase the capacity of researchers to identify new, relevant avenues for exploration that have not yet been investigated that respond to decision makers/ knowledge users' needs (CIHR 2010). Finally, synthesis grants are intended to promote the process of mutual learning between researchers and knowledge-users.

Launched first in 2004, CIHR invites all forms of knowledge synthesis. Qualitative, quantitative and mixed methods approaches are accepted, as well as syntheses of knowledge gained through observation, testing, or reviewing of texts. Scoping reviews are also accepted; these are projects that explore the literature available on a topic, identifying the key concepts, theories, sources of evidence, and gaps in the research (CIHR 2010). Synthesis funding opportunities fall under the iKT focus area. Applications go through a merit review process.

Resources

Competitions are launched twice a year by the Knowledge Translation Branch in partnership with various institutes and initiatives, along with external partners. The maximum amount awarded for a synthesis is $100,000 for one year. The maximum amount awarded for scoping reviews is $50,000 for one year.

Success rates across Synthesis since 2005 (%)
2005 2006 2007 2008 2009 2010

Notes:

  1. Averages have been applied when multiple competitions were held in the same year;
  2. Renewals are removed.

Source: EIS data 2011

Synthesis 50.00 26.67 53.66 47.34 47.06 33.02
Synthesis grants awarded per year
2005 2006 2007 2008 2009 2010

Source: EIS data 2011

Synthesis 8 8 54 11 38 42

Partnerships for Health System Improvement (PHSI)

Background

The first CIHR PHSI competition was held in 2005. The PHSI funding program supports teams of researchers and decision makers/knowledge users interested in conducting applied and policy-relevant health systems and services research that respond to the needs of health care decision makers. Partnerships can be project specific (partners that the researchers identify themselves with and with whom they negotiate) and/or competition specific (CIHR negotiated competition partnerships). This funding opportunity requires financial or in kind partner contributions (CIHR 2008).

Calls for applications for the PHSI program are launched annually. PHSI falls under the iKT classification and uses the merit review process to evaluate the applications (CIHR 2008).

Resources

The maximum amount awarded by CIHR for a single grant is $400,000 for up to three years (partnership contributions are in addition to the CIHR amount). A minimum of either 20% or 30%, depending on the province or territory, of the grant budget must come from external partner sources (i.e., non-CIHR funds). There is no limit to partner contributions, and in-kind contributions are recognized, especially where they reflect meaningful collaboration that will increase the likely success of the project. It should be noted that funding and contributions may be received from stakeholders who are not members of the grant team.

Success rates across PHSI since 2005 (%)
2005 2006 2007 2008 2009 2010

Notes:

  1. Averages have been applied when multiple competitions were held in the same year;
  2. Renewals are removed.

Source: EIS data 2011

PHSI 53.85 49.76 66.67 34.55 33.90 36.76
PHSI grants funded per year since 2005
2005 2006 2007 2008 2009 2010

Source: EIS data 2011

PHSI 21 32 16 35 20 25

Knowledge to Action (K2A)

Background

K2A is designed to move knowledge into action by linking researchers and KUs and to increase the understanding of knowledge application through the process. By bringing both parties together, it is expected that research results will translate to actions that strengthen Canada's health care system and/or improve the health of Canadians.

K2A also aims to support the development, implementation and evaluation of cutting-edge KT research and approaches. As such, K2A establishes and strengthens common ground between the interests and expertise of the research community and the needs knowledge users. Applicants can request funding to support partnerships, knowledge and tools for implementation projects.

This program was first launched in 2005, and is currently launched annually. K2A falls under the iKT classification, and accordingly, all applications go through a merit review process.

Resources

The maximum CIHR contribution is $100,000 per year for up to two years. Applicants may increase funding for their proposal and further demonstrate the level of engagement of their partner(s) through cash or in-kind commitments, but a financial commitment from the partner is not a criterion for funding (CIHR 2007). Applicants are encouraged to apply for a renewal of their grant if they plan to scale up their implementation project, which might not be achievable within a two year time frame.

Success rates across K2A since 2006 (%)
2006 2007 2008 2009 2010

Notes:

  1. Averages have been applied when multiple competitions were held in the same year;
  2. Renewals are removed.

Source: EIS data 2011

K2A 66.67 22.22 36.36 29.41 36.00
K2A grants funded per year since 2006
2006 2007 2008 2009 2010

Source: EIS data 2011

K2A 10 12 18 5 18

Dissemination Events (DE) & the KT Supplement (KTS)

Background

Meetings, Planning and Dissemination (MPD) grants are intended to provide support for meetings, planning and/or dissemination activities consistent with the mandate of CIHR and relevant CIHR Institutes, Initiatives or Branches.

From 2004–2007, the KSE (now KT) Branch ran a version of this program, called Workshop/Symposia/Conference Support in the branch, prior to the launch of the MPD pilot by the Research Portfolio in 2007. The successful pilot was transformed into the full current program in 2008. MPD offers three tools for funding: Dissemination events (DE), Planning Grants and the KT Supplement (KTS). This evaluation only covers DE and KTS15.

DE supports the organization of events focused on the communication of health research evidence. DE has three annual competitions (February, June and October) and undergoes a peer review process. Eligible activities for DE include:

KTS supports KT activities that follow a peer-reviewed grant/award where further dissemination is appropriate. KTS has three annual competitions (February, June and October) and uses a peer review process. Eligible activities for the KTS include:

Resources

KTS and DE are non-renewable one year grants. However, multiple grants can be awarded to the same candidate in the same calendar year. KTS are funded up to $100,000, and DE are funded up to $25,000 (CIHR 2010).

Success rates across KTS and DE since 2005 (%)
2005 2006 2007 2008 2009 2010

Source: EIS data 2011

KTS & DE 47.62 n/a 60.16 87.29 63.51 80.83
DE and KTS grants funded per year since 2005
2005 2006 2007 2008 2009 2010

Source: EIS data 2011

KTS & DE 3 13 67 99 100 146

KT science panel on the OOGP

Background

KT science is funded through the OOGP being assigned to the KTR panel for review. Funded projects must conduct research directed toward developing theory, evidence and innovation to define the determinants, implementation and uptake of health research evidence into practice. This includes grants which aim to improve KT to consumers, health practitioners and policy makers, to examine the role of organizations as KT vehicles, to determine how to improve knowledge uptake potential during the research process, to develop/evaluate KT tools and/or methods, to contribute to KT theory and to improve knowledge uptake (Knowledge Translation Branch 2010).

Resources

These competitions provide grant funding for up to five years and have no funding limit or specific requirements for team size or composition. In total, 85 KT science projects have been funded by the KTR panel since 2000 for a total expenditure of over $27,000,000.

Success rates for KT science since 2005 (%)
2005 2006 2007 2008 2009 2010

Notes:

  1. Averages have been applied when multiple competitions were held in the same year;
  2. Renewals are removed.

Source: EIS data 2011

KT science 26.38 20.33 22.23 19.93 17.02 16.67
KT science grants funded per year since 2005
2005 2006 2007 2008 2009 2010

Source: EIS data 2011

KT Science 11 8 15 25 13 16

Appendix D – Evaluation Matrix

Evaluation Questions Indicators Methods Sources

1. What role is there for CIHR in enabling/promoting iKT, end-of-grant KT, and KT science?

  • Is the CIHR role consistent with the needs of Canadians, the improvement of health products and services, and the strengthening of the Canadian healthcare system?
  • Theory and empirical evidence related to the role of a funding organization in the KT process
  • Theory and empirical evidence related to the advantages and limitations of iKT, end-of-grant, and KT science strategies
  • Degree of alignment of CIHR KT strategy with theory and empirical evidence of KT success strategies
  • Organizational scan of comparable organizations nationally and internationally
  • Document review
  • Various forms of KT academic and professional publications
  • Indications of incentive induced behaviour of researchers and knowledge-users
  • Indications of unique or innovative KT strategies employed
  • Case studies
  • Exceptional funded projects
  • Application pressure (total applications per funding program)
  • Ratio of researchers funded versus applied
  • Ratio of researchers funded versus fundable but not funded
  • Degree of alignment with CIHR mandate and strategic vision
  • Degree of alignment with the government of Canada's plans and priorities? (i.e. S&T Strategy)
  • Document & EIS data review
  • EIS application records
  • CIHR guiding documents
  • Government of Canada documentation

2. To what extent are KT funding opportunities achieving their expected outcomes?

  • To what extent are immediate outcomes being achieved?
  • To what extent are intermediate outcomes being achieved?
  • To what extent are long-term outcomes being achieved?
  • Indications of immediate, intermediate, and long-term outcomes (using LM)
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional funded projects
  • The number of grants awarded by each program
  • # of partnerships created (iKT)
  • Comparison of application pressure across funding opportunities
  • Indications of intermediate and long term outcomes
  • Document and EIS data review
  • EIS application records
  • End of grant reports
  • Degree of alignment of CIHR KT strategy with theory and empirical evidence of KT success strategies
  • Document review
  • Various forms of KT academic and professional publications
3. What factors facilitate or inhibit the achievement of KT funding opportunity outcomes?
  • Indication of influence on program theory from:
  • Internal program processes
  • External environmental factors
  • Strategic level factors
  • Program delivery level factors
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional funded KT projects
  • Document & EIS data review
  • EIS application records
  • Final reports
4. How effective is CIHR's KT Funding Program mix in achieving expected outcomes? (Synthesis, iKT, end of grant KT, KT science; push, pull, linkage and exchange)
  • Perceptions of suitability of program mix for promoting/enabling effective KT
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Profiles of pathways to program outcomes
  • Case studies
  • Exceptional funded KT projects
  • Degree of alignment of CIHR KT strategy with theory and empirical evidence of KT success strategies
  • Literature review
  • Various forms of KT academic and professional publications
5. To what extent have KT funding opportunities reached a broad and diverse range of knowledge users?
  • Extent to which KT funding opportunities attract varied types of knowledge users
  • Number and type of knowledge users included per iKT grant
  • Perceptions of meaningful partnerships having been established
  • Indication of varied knowledge user engagement through MPD and KT science funding programs
  • Document & EIS data review
  • EIS application records
  • Final reports
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional funded KT projects
6. To what extent are KT funding opportunities being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness?
  • Indications of efficiency and effectiveness in the conversion of program activities into program outputs
    • Identified success and challenges of the merit review process
  • Document & EIS data review
  • EIS application records
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional funded KT projects
7. What would be the effect on CIHR-funded researchers and knowledge-users if the KT Funding Program no longer existed? What would be the effect on the improvement of health, more effective health services and products, and the strengthening of the healthcare system?
  • Perceived impact of absence of future KT funding on funded researchers, knowledge-users, and KT outcomes
  • Perceived future directions for funded researchers, knowledge-users, and KT outcomes in the absence of KT funding
  • Use of alternative funding sources by KT funded teams (leveraging)
  • Use of alternative funding sources by KT researchers and knowledge users not funded by CIHR (Knowledge User partners)
  • Organizational scan of similar organizations nationally and internationally
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional funded KT projects
  • EIS
  • EIS application records
  • Final reports
  • Document review
  • Various forms of KT academic and professional publications
8. What are the unanticipated outcomes, positive or negative, resulting from the KT Funding Program?
  • Identified unintended outcomes of KT Funding Program
  • Document and EIS data review
  • EIS application records
  • Final reports
  • Surveys
  • Funded researchers/knowledge users
  • Unfunded researchers/knowledge users
  • Key informant interviews
  • Funded researchers/knowledge users
  • Program staff
  • Case studies
  • Exceptional KT funded projects
  • Literature review
  • Various forms of KT academic and professional publications

Appendix E – Case Study Readings

Synthesis: Can Interprofessional Collaboration Provide Health Human Resources Solutions? A Knowledge Synthesis
Research leads: Dr. Esther Suter, Ms. Grace Mickelson (KU)

Knowledge to Action: Need to know too
Research leads: Dr. Judith Bartlett, Dr. Catherine Louise Cook (KU)

PHSI: National health partnership for reducing infections in NICU
Research leads: Dr. Shoo Lee, Dr. Elizabeth Whynot (KU)

KT Research Panel in the OOGP: Producing and communicating knowledge differently: Articulating the pathway to mental health care through artistic expression
Research NPI: Dr. Katherine Boydell

KT Supplement FO: Promoting Inpatient Cardiac Rehabilitation Referral
Research NPI: Dr. Sherry Lynn Grace

Appendix F – Glossary

Acronym Definition
CIHR Canadian Institutes of Health Research
CCV Common Curriculum vitae
CR Cardiovascular Rehabilitation
CQI Continuous Quality Improvement
DE Dissemination Event
DSEN Drug Safety and Effectiveness Network
EIHR Evidence Informed Healthcare Renewal
EIS Electronic Information System
EKS Expedited Knowledge Synthesis
EPIQ Evidence-based Practice for Improving Quality
FO Funding opportunity
FRN Funding Reference Number
GOC Government of Canada
HHR Health Human Resources
IP Interprofessional
IPC Interprofessional Collaboration
iKT Integrated knowledge translation
KT Knowledge translation
KT science Knowledge Translation science
KTPO Knowledge Translation and Public Outreach Portfolio
KTR Panel Knowledge Translation Research Panel
KTS Knowledge Translation Supplement
KU Knowledge user
K2A Knowledge to Action
MH Manitoba Health
MMF Manitoba Metis Federation
NAPHRO National Alliance of Provincial Health Research Organizations
NICU Neonatal Intensive Care Unit
NSERC Natural Sciences and Engineering Research Council
OOGP Open Operating Grant Program
PAA Program Activity Architecture
PDM Principal Decision Maker
PHSI Partnerships for Health System Improvement
RHA Regional Health Authority
RFA Request for application
RPP Regional Partnerships Program
RRS Research Reporting System
Rx Prescription
SHOPP Small Health Organization Partnership Program
SPOR Strategy on Patient-Oriented Research
WCICH Western Canadian Interprofessional Health Collaborative
WHO World Health Organization

Appendix G – Evaluation Working Group

Knowledge Translation Expertise

Ian Graham
Former CIHR VP of Knowledge Translation
Senior Scientist, Ottawa Hospital Research Institute

Martha MacLeod
Professor and Chair, School of Nursing, University of Northern British Columbia
Chair, CIHR Knowledge to Action Committee

Jacqueline Tetroe
CIHR Senior Advisor, Knowledge Translation

Adrian Mota
CIHR Manager, Process Stream

Yumna Choudhry
CIHR Senior Specialist

Stephanie Coen
Former KT Manager, CIHR Institute of Gender and Health

Ryan McCarthy
Former CIHR Director, Knowledge Translation

Andrea Smith
CIHR Manager, Major Initiatives

Evaluation Unit Project Team

Robert McLean
CIHR Lead Evaluator

Joanne Tucker
CIHR Junior Evaluator

Christopher Manuel
CIHR Junior Evaluator

Kwadwo (Nana) Bosompra
CIHR Evaluator

Appendix H – Evaluation Recommendations Crosswalk

Crosswalk of Evaluation Recommendation One with identified key elements of continued KT funding success

Recommendation 1

CIHR should invest the required resources to sustain its role in enabling KT. Given the decision by CIHR to integrate the KT funding opportunities into the open research suite, both of the proposed Project and Foundation Schemes of research should retain the key elements identified in the evaluation as responsible for the success of the current the standalone iKT and end of grant KT funding opportunities.

The following chart provides a crosswalk between what this evaluation has identified as key elements of KT success and important considerations for mainstreaming these into the open research suite. Both synergies and challenges are highlighted, and where appropriate recommendations for CIHR action are provided.

Key elements of KT success Considerations during mainstreaming
  1. Engaging KUs in and throughout the research process.
  2. Assuring commitment and buy-in from partners (not necessarily financial).
  3. Working with the right expertise (within both the researcher and KU contexts).

Potential Synergies:

  • The current iKT funding opportunities were identified by both researchers and KUs as limited in the amounts of funding and time-lengths provided. This created challenges to engaging partners in and throughout their research in meaningful ways. Funding iKT research projects through the Foundation and Project Schemes will lengthen the term of funded projects and increase the funding ceiling for iKT research well above that currently allowed through the standalone funding opportunities.
  • There is an opportunity with a shift to the Foundation and Project Schemes of research to develop application processes that better encourage and support meaningful partnerships. The evaluation suggests that a flexible application process which allows KU and researcher applicants to describe the process of partnership development and maintenance most appropriate to their research and KT goals could improve the iKT research funding mechanism. For example, the points were consistently raised by researchers and KUs that the KU curriculum vitae module should not be approached in the same way as the academic curriculum vitae module, and that describing a partnership at the application stage should not be a straightjacketing process. Addressing these issues during the reform process has the potential to save time for applicants and reviewers, and may improve the quality of review by making applications more accurate and comprehensive.
    • Recommendation 1.1: In the process of reform CIHR should amend and develop the application criteria for partnered research (with specific attention to KU curriculum vitae and partnership description).
  • The evaluation highlights that the standalone KT funding opportunities have been used by researchers from a diverse range of backgrounds. These include two groups of researchers CIHR has acknowledged interest in improving accessibility for via current reform processes: early career-stage researchers and pillar III and IV researchers. Through the integration of KT funding opportunities into the proposed Project and Foundation Schemes there is the potential to bring these groups of researchers into CIHR's larger open funding envelope. Diversity of both researcher and KU frame of reference and expertise is considered a driving force of innovation, and this evaluation has demonstrated indications of this occurrence. Furthermore, CIHR has aimed to enhance stakeholder diversity as a part of the current reforms.
    • Recommendation 1.2: CIHR should ensure that the integration of the KT funding opportunities is done in a manner that is inclusive to all traditional users of the KT funding opportunities.
  • The evaluation identified that the requirement for partner financial contributions within the PHSI funding opportunity can align researcher and KU incentives for success, and is a source of financial leveraging for CIHR.
    • Recommendation 1.3: Mainstreaming the PHSI funding opportunity should occur with due consideration of this unique feature and how it can be applied and encouraged in the larger Project and Foundation Schemes. The caveat to this consideration is the understanding that meaningful partner commitment is the vital objective, whether or not this includes leveraged financial contributions.

Potential Challenges:

  • This evaluation has demonstrated that requiring partners through a standalone funding opportunity is a successful strategy for achieving research impact and KT. It is uncertain whether the research community, or the KU community, will be as likely to engage in partnered research without this formal requirement. Given that partnered research is considered to require more time and be more costly, there is little incentive for researchers to choose this approach when competing against researchers who do not do partnered research.
    • Recommendation 1.4: To sustain the impacts achieved through the standalone funding design, steps should be taken to ensure researchers remain aware of their ability to, and the advantages of, applying for funding in a partnered format.
    • Recommendation 1.5: CIHR should explore how researchers who have not included a partner(s) in an application, and whose research would potentially benefit from doing so, will be encouraged to include partners.
  • There is no evidence to suggest an iKT research approach should be used on all CIHR-funded research projects. On the contrary, evidence of this evaluation indicates partner inclusion is an idiosyncratic process and should be conducted in different ways, if at all, from one research project to another. In a system where applications using both partnered and non-partnered approaches are made to the same funding competition it is essential that a level playing field is created between iKT research applications and "investigator-driven" research applications. However, removing the standalone application – the mechanism that has traditionally allowed researchers themselves to select a funding opportunity for partnered research – means that the onus moves from the applicant to CIHR to ensure iKT research applications and "investigator-driven" research applications are equally valued and are evaluated on distinct scales (e.g., a non-partnered application which does not require partners should not score lower in review for not engaging partners throughout the research process.)
  1. Tailoring and timing the dissemination of results to the audience(s).

Potential Synergies:

  • Evidence from this evaluation has shown that end of grant KT funding opportunities KTS and DE are deemed highly useful and effective by researchers from all four pillars of health research and have been used by these researchers to achieve research impact when grant funding ends. The benefits of these funding opportunities may continue to be realized by offering end of grant KT funding designs similar to DE and KTS to both Project and Foundation Scheme researchers (i.e., through a distinct process at the conclusion of the research project or the time proven appropriate by a funded researcher/research team).
  • End of grant KT funding opportunities are not only able to fill a gap when research funding dries up, they need not be granted for research that has not produced findings worthy of a significant investment in a dissemination strategy. The cost-effectiveness of such a strategy may continue in both of the proposed Project and Foundation Schemes of research through built-in, but discrete, end of grant KT funding.
  • There is little data in this evaluation demonstrating the value of having offered the two funding opportunities (KTS and DE) distinctly, and the reform process has the opportunity to improve this.
    • Recommendation 1.6: Providing a sliding funding scale for the full and diverse range of end of grant KT supported through the two funding opportunities as part of the Project and Foundation Scheme could achieve the same results with less internal cost for CIHR.

Potential Challenges:

  • Specific elements identified in this evaluation that are responsible for end of grant KT success include: the expedited application, review, and funds disbursement; multiple application intakes; the ability to support diverse dissemination techniques; and the ability to engage expected and unexpected KU groups with research results.
    • Recommendation 1.7: CIHR should ensure these elements are retained in the end of grant KT funding offered through both of the Project and Foundation Schemes.
  • KTS and DE funding opportunities were designed to allow CIHR to capitalize from health research investments from across and Canada and internationally. That is to say, they were not limited to applications that followed from CIHR funded research. Linking end of grant KT funding directly to the Project and Foundation Schemes will mean funding for the dissemination of research results is only available to CIHR-funded researchers.
    • Recommendation 1.8: If CIHR continues to place value on supporting end of grant KT activities that follow research funded via alternative investment sources a mechanism for doing this should be considered.
  1. Engaging both researchers and KUs in the review of funding applications for KT research.

Potential Synergies:

  • This evaluation indicates that partnered research projects, which are evaluated through the Merit Review process, have achieved transformative results. Broadening the scope of Merit Review (i.e., the inclusion of KUs in application adjudication) to the proposed Project and Foundation Schemes may allow the benefits of the process to occur across a wider field of applications, and resultantly, may spread the outcomes of KU involvement in review to a greater number of projects.
    • Recommendation 1.9: The principles of Merit Review should be sustained in the new open schemes of research. To enhance Merit Review in the new open schemes, KUs should be further engaged to assist in determining what information is considered essential in order to perform review of grant applications.

Potential Challenges:

  • Mainstreaming standalone KT funding opportunities into the proposed Project and Foundation Schemes will require CIHR develop mechanisms that internally determine which research proposals will be put to Merit Review.
    • Recommendation 1.10: CIHR should endeavor to create a system of application classification that ensures KT focused applications are evaluated by appropriately experienced reviewers including KUs.

Footnotes

Footnote 1

The CIHR open operating grant program (OOGP) provides funding for investigator driven research and amounts to approximately 44% of CIHR's annual grants and awards expenditures (2005–2010).

1

Footnote 2

The term 'program theory' is used to describe the underlying and causal sequence of events that are hypothesized to move a program along a pathway from internal activities to long-term environmental outcomes. A logic model is used to form a simplified linear representation of a program theory.

2

Footnote 3

In earlier iterations of these questions, instead of the term 'funding opportunity' (e.g., PHSI funding opportunity) the term 'funding program' (e.g., PHSI funding program) was used. Although the terminology has changed to reflect the current CIHR lexicon, the purpose of the question has not.

3

Footnote 4

The CIHR Open Operating Grant Program (OOGP) has traditionally provided operating funds to support research proposals in all areas of health research. The OOGP has no specific requirements in relation to team size or team composition; nonetheless, the role of each applicant must be clearly identified. No specific requirements or restrictions on the specific research activities to be undertaken. No constant maximum or minimum with respect to the term of funding or funding amount is imposed.

4

Footnote 5

Throughout this report KT related data and OOGP data are displayed in reference to one another, often this is furthered to OOGP pillars II-IV. The purpose of this analysis is to provide a context and reference for interpretation of (especially quantitative) data points related to KT. The purpose of furthering the reference to OOGP pillars II-IV is to provide a more accurate match between the types of researchers using KT and OOGP funding.

5

Footnote 6

A limitation of this analysis is that the data presented are self-assessed by researchers. The mixed-methods approach employed in this evaluation is strong mitigation strategy against any bias. In fact, data collected through alternative methods of inquiry employed in this evaluation support the findings presented. This inquiry included independently engaging KUs on their experiences in partnered grants.

6

Footnote 7

Although the "influence" question provided the same options for KU categories as that of "involvement", the two questions were asked independently of each other. This is important to note when comparing results and any relationship between results of the two questions.

7

Footnote 8

Further statistical testing of the relationships between involvement, influence, and action would strengthen this investigation and should be pursued in order to further understanding about the strength of any statistical relationships between the quantitative data points.

8

Footnote 9

The resource constraints of the evaluation did not allow for a case study to be performed on the DE funding opportunity, however, the breadth of other data sources allows for substantial analysis of DE in other areas of this report.

9

Footnote 10

The Jenkins report highlighted the importance of establishing a clear federal voice for innovation that focuses research activities that support commercially useful innovation (GOC 2012).

10

Footnote 11

In a program evaluation it is not uncommon that program users, be they researchers or not, identify increased funding as an opportunity for program improvement. This being said, the number who reported this fact in the survey is substantial, and the issue is logically linked to the challenge identified as an internal hindrance to success: the additional effort required to do iKT research.

11

Footnote 12

The environmental scan undertaken as a part of this evaluation is an update on a previous scan: Tetroe et al. (2008) Health Research Funding Agencies' Support and Promotion of Knowledge Translation: An International Study. All 2008 data in Figure 18 is from this research.

12

Footnote 13

This data corresponds to the findings presented on pages 11–14 of this report which suggest OOGP researchers are performing end of grant KT activities in order to engage and to influence KUs.

13

Footnote 14

Our sample includes only Northern organizations. This is a certain limitation to this method. Consideration was made of also duplicating the sample employed in similarly focused Southern study (Cordero et al. 2008), but resource limitations did not allow for this undertaking.

14

Footnote 15

Planning grants were removed from the purview of this evaluation as the objectives and theory of the FO would require a specifically tailored evaluation approach. In short, planning grants support pre-funding project activities whereas all other FOs apart of this evaluation support post-funding project activities.

15

Date modified: