Summary Report of the Interim Evaluative Study of the CIHR/RX&D Program

Other format

Evaluation and Analysis Unit
October 2005

Table of contents

  1. Introduction
  2. Limitations of This Study
  3. Nature of Rx&D Program
  4. Program Expenditures
  5. Logic Model
  6. Study Methodology
  7. Findings
    1. Strengths of the Program
    2. Weaknesses of the Program
  8. Recommendations
  9. Management Response and Action Plan
  10. Participating and Non Participating Companies

Appendix A - Evaluation Issues
Appendix B - Program Operation

1. Introduction

This report presents a summary1 of a small evaluative study of the CIHR/Rx&D program conducted December 2003 and ending in April 2004. This study is not considered a comprehensive formative or summative evaluation.

This report is based on an evaluation planning study carried out in 2002,2 for the CIHR/Rx&D program which identified both early and longer term evaluation issues. The longer term issues were proposed to be dealt with at a time when a reasonably large group of projects had been completed, but the planning study also suggested that a small interim evaluative study dealing with the design and delivery of the program could be conducted.

This study had two main objectives:

  1. To provide senior management with performance information on certain key aspects of the program pertaining to the design of the program; and
  2. To validate the performance indicators and evaluation issues and questions which will serve as the basis for a comprehensive evaluation to be conducted in the future.

2. Limitations Of This Study

This study was limited by the small sample size of only 28 completed interviews. Despite numerous attempts to contact representatives of the companies, many were unreachable, or declined to participate in the study. Awardees were easier to contact. However, having a much larger number of awardees than participating and non-participating companies could be unrepresentative and resources available to conduct this study were modest. Another limitation was the use of only one data collection method: that is, individual interviews of awardees, participants and non-participants.

Despite these two limitations, the study team believes that the study has yielded useful results. These results could therefore be of assistance to guide the program manager's intervention at this stage of the program, and to possibly prompt a further investigation. It is recommended that a full evaluation, consisting of a larger sample size, be conducted once a reasonably larger number of projects have been completed.

3. Nature Of Rx&D Program

The CIHR/Rx&D Program is a partnership between the Canadian Institutes of Health Research (CIHR) and Canada's Research-Based Pharmaceutical Companies (Rx&D) to support university-affiliated (academic) health research that is of interest to industry and the academic research community3. The program provides matching funding for research training and salary support (studentships, fellowships, investigator salary support, research chairs) and research grants (operating grants, RCTs, and workshops) that are supported by industry.4

The mission of the program is:

To enhance quality of life for the world's peoples by bringing together academia, industry, and government. We facilitate collaborative partnerships that create and transfer knowledge, driving forward the goals of scientific research consistent with Canada's strategic health research agenda, the pharmaceutical sector, and health care on an international scale.

The objectives of the program are:5

  1. Stimulate new academic health research activity in Canada.
  2. Contribute to the training of new researchers in Canada and the attraction and retention of experienced researchers.
  3. Strengthen the relationships between academic researchers and Canada's pharmaceutical companies.
  4. Contribute to economic development and job creation and to improved health and health care in Canada.

It is a key purpose of the program to stimulate research that would not otherwise have occurred. Research training occurs both as a result of specific CIHR/Rx&D training awards and as a result of the involvement of trainees in CIHR/Rx&D projects (e.g., as research assistants).

4. Program Expenditures

As shown in the table below, the Rx&D program provided increasingly larger amounts in each of the first five years. The percentage invested in this program relative to the total CIHR grants and awards expenditures therefore also increased. Appendix "B"; contains a brief description of the program operation.

Rx&D Expenditures FY 1999/2000 - FY 2003/2004

1999-2000 2000-2001 2001-2002 2002-2003 2003-2004
CIHR Expenditures - $(000) (excl. CRC & NCE) $275,209 $339,374 $448,530 $586,826 $575,583
Rx&D Grants & Awards 6 122 213 264 270
Rx&D Grants & Awards expenditures (actual) $13,228 $2,337,897 $4,910,538 $6,569,734 $6,303,979
% of CIHR's total expenditures 0.005% 0.689% 1.09% 1.119% 1.095%

5. Logic Model

Below is a logic model for the CIHR/Rx&D program developed in consultation with program managers?6. Note that the program objectives are among the program outcomes illustrated in the logic model. Intermediate outcomes which reflect stated program objectives are highlighted.

6. Study Methodology

The study adhered to the interim evaluation design proposed in the CIHR/Rx&D Program Evaluation and Performance Measurement Framework Project Plan, dated April, 2002.

Following the recommendations of the evaluation design report, the findings were based on three sets of interviews with key informants:

Interviews with participating university researchers (awardees) - university researchers who had acted as Principal Investigators for this program. The questions covered both program design/delivery issues and program outcome issues from their perspective.

Interviews with participating companies (participants) - representatives (e.g. Director of Research, Vice-President Scientific Affairs, Head of Clinical Research) of pharmaceutical research and development companies that have participated in the CIHR/Rx&D program - the questions covered both program design/delivery and program outcome issues.

Interviews with non-participating companies (non-participants) - representatives (e.g. Director of Research, Vice-President Scientific Affairs) of pharmaceutical research and development companies that have never participated in the CIHR/Rx&D program, primarily in relation to program design and delivery issues.

A list of participating and non-participating companies that were contacted is contained in Section 10.

Initial Target Contacts provided by CIHR Completed Interviews Percentage target completed
Awardees 10 24 11 110%
Participants 20 19 11 60%
Non-participants 15 15 6 40%

7. Findings

Overall, the findings of this study provide positive indications that the key parts of the program that were looked at in this study are on track and overall progress is being made towards the achievement of its objectives. Both researchers and participating companies view the structure and operations of the program as being largely positive, and the program appears to be having a number of positive early impacts.

The most significant area for improvement that was identified concerns program communication and marketing activities which are viewed as needing strengthening. The key findings of the interim evaluative study are identified below.

i. Strengths of the Program

This section of the report outlines the areas of the CIHR/Rx&D program that are running smoothly and have received positive reactions from awardees, participants, and/or non-participants interviewed.

  • The program is enabling awardees to carry out more research and to hire more people than they would have in the absence of the program. It is offering them funding both from CIHR and industry participants, as well as a validation of their research through the peer review process.
    (Source: - awardee interviews)
  • The program is providing industry participants with more opportunities for in-depth research in areas of strategic interest. Clearly, this area is key to industry and if the program can assist them strategically, they will be more inclined to participate. The program also motivates industry due to the reputation and credibility of CIHR, as well as the financial benefit.
    (Source: - participant and awardee interviews)
  • All of the respondents who have had contact with CIHR staff noted that they (being predominantly the CIHR/Rx&D Program Director) are helpful. Some of the respondents, however, mentioned that the Program Director is not always easy to reach.
    (Source: - participant and awardee interviews)
  • The application and review process is seen by awardees as comparable to similar programs in terms of timeliness, and the criteria used to assess applications is rated as satisfactory. There are no overall complaints with the process and it seems to be acceptable to awardees. For the most part, industry participants agree that the process is relatively timely and appropriate, with the exception of RCTs, since timeliness is more important than money in these cases.
    (Source: - awardee interviews)
  • Applications that were turned down are seen as justified by industry participants in the majority of cases. It was noted, however, that when an application was revised and re-submitted, it had to endure the same lengthy review process as the first time through.
    (Source: - participant interviews)
  • All industry participants and non-participants interviewed agree and are satisfied with CIHR's approach to intellectual property issues, with the exception of one individual who did not know enough about the program to comment. (Source: - participant and non-participant interviews)
  • Almost half of the awardees mentioned a positive effect in the way in which they viewed research being carried out in collaboration with the pharmaceutical industry. Some researcher opinions changed from one of skepticism of the pharmaceutical industry and their conflicting objectives to one of security and support.
    (Source: - awardee interviews)
  • Those industry participants who were involved in the predecessor program positively recognized improvements made to the program.
    (Source: - participant interviews)
  • Current funding ratios for training, salary support, and operating grants, are seen as fair by both industry participants and industry non-participants.
    (Source: - participant and non-participant interviews)
  • Both industry participants and industry non-participants see the eligibility of in-kind contributions as very beneficial. A couple of participants mentioned the need for clarification of what defines 'in-kind' contributions and how they are valued.
    (Source: - participant and non-participant interviews)

ii. Weaknesses of the Program

This section of the report outlines the areas of the CIHR/Rx&D program that may be inhibiting participation or impeding the process. These were based on opinions of those interviewed.

  • Most industry participants and non-participants do not feel that the program's communications and marketing activities are as effective as they should be. The non-participants stated that they do not know the benefits of the program and that CIHR has not effectively marketed the program, in terms of who can apply, what is the process, what are the benefits, etc. Even awardees demonstrated an overall lack of familiarity with the program (e.g. awareness of program changes, objectives of the program).
    (Source: - participant, non-participant and awardee interviews)
  • The application and review process is considered to be slow and burdensome to some of the industry non-participants and select industry participants, particularly those who engage in RCTs, where time is of the essence.
    (Source: - participant and non-participant interviews)
  • Industry non-participants generally viewed the peer review process as adding little or no value, because the expertise of the reviewers was not always apparent. Furthermore, industry participants mentioned that they were likely to execute their own internal review, regardless of the review conducted by CIHR.
    (Source: - participant and non-participant interviews)

8. Recommendations

  1. CIHR should improve communications and marketing activities in the following areas:
    1. General awareness and information: General program information should be made more broadly available. In addition to the benefits of the program, the program itself (what it is about, who can participate, what is the process for applying, etc.) should be better communicated and marketed to relevant communities.
    2. Benefits: Program benefits should be clearly articulated to awardees, participants, and non-participants. Benefits of program participation should be presented at forums, on the CIHR website, in marketing brochures and pamphlets, etc.
  2. Program communications and materials including the website should be improved in the following areas:
    1. General assistance: Assistance provided other than through the CIHR/Rx&D Program Director would permit all program participants to access important operational information in a more timely manner. This could be accomplished by providing an alternate contact person to field questions and provide operational assistance for the program.
    2. Program Specifications: While industry views the eligibility of in-kind contributions positively there remains some confusion on what constitutes an in-kind contribution and how the contribution is to be measured. Accordingly, a detailed clarification of in-kind contributions and how they are defined should be conveyed to all participants and potential participants alike. This could transform non-participants into participants.
  3. Program management should further analyze and study funding for RCTs to determine if and how they could be made more attractive to potential industry and academic participants. This issue should be revisited in a more comprehensive evaluation.
  4. The program could enhance efforts to increase contact and encourage interaction between awardees and industry during the course of the partnerships, thus building overall longer term collaborative capacity.

9. Management Response And Action Plan

Recommendation 1 - The program should improve communications and marketing activities in the following areas:

1.1 General awareness and information: General program information should be made more broadly available. In addition to the benefits of the program, the program itself (what it is about, who can participate, what is the process for applying, etc.) should be better communicated and marketed to relevant communities.

Agree

  • The impact of the program and its response to CIHR's Blueprint in general is not sufficiently represented on the CIHR website. This is being corrected in the new web architecture being prepared.
  • Although based on scientific excellence and successful peer review we should perhaps reconsider publicising the amount of money the program brings to CIHR. We are hesitant to do this since it may detract focus from enhancing excellent research opportunities but it is a fact that it is currently barely mentioned and is rarely referenced in the information we provide.
  • The program description and funding tools themselves were not obvious on the website. With the new web architecture, this has been improved substantially and is now integrated into an ongoing review. The industry partnered programs are now listed alongside the open competition programs on the master lists of funding opportunities. The titles were chosen specifically to enable this to happen.
  • Program descriptions now have direct links to the 'How to Apply' sections and 'Application Packages'. In November of 2005, Innovation and Industry Programs will have its own home page on the web. This resource will be a valuable tool for communication.

1.2 Benefits: Program benefits should be clearly articulated to awardees, participants, and non-participants. Benefits of program participation should be presented at forums, on the CIHR website, in marketing brochures and pamphlets, etc.

Agree

  • Marketing brochures have been created and new ones planned.
  • Scientific Liaison and Director have held focus groups to inform member companies of the benefits and opportunities of CIHR and Rx&D collaboration. These meeting were held in Montreal and Toronto. Plans to continue this with more targeted focus groups is anticipated in new Agreement.
  • The Director of the program typically promotes the benefits of the program during site visits/meetings with relevant communities, but admittedly is limited to how much can be done by one individual. Increased presence and communication with academic and member communities as well CIHR Institutes by Director and staff is being developed for new Agreement with Rx&D
  • Significant enhancement anticipated by better engaging Assistant Directors of CIHR's Institutes.

Recommendation 2 - Program communication and materials, including the website, should be improved in the following area:

2.1 General assistance: Assistance provided other than through the CIHR/Rx&D Program Director would permit all program participants to access important operational information in a more timely manner. This could be accomplished by providing an alternate contact person to field questions and provide operational assistance for the program.

Agree

  • As mentioned above, in the past the program descriptions were not easy to find on the web. The new architecture should help relevant communities locate the information they need.
  • All of the program descriptions were revised in the Spring of 2005. Program Delivery Officers and Coordinators were added as contacts for their respective programs. The Deputy Director remained the contact on the general Rx&D program description.
  • The master lists contain all of the specific programs; however the Rx&D program description itself is currently under "Other Funding Programs". It would benefit visibility if it was pulled out and moved to "Current opportunities organized by research area". This is where the links to all of the Institute home pages are located, as well as Ethics and International Cooperation". Once the Innovation and Industry home page goes live, we should be able to make this a reality.

2.2 Program specifications: While industry views the eligibility of in-kind contributions positively there remains some confusion on what constitutes an in-kind contribution and how the contribution is to be measured. Accordingly, a detailed clarification of in-kind contributions and how they are defined should be conveyed to all participants and potential participants alike. This could transform non-participants into participants.

Disagree

  • There is a section of the website that details eligibility of in-kind contributions ({http://www.cihr-irsc.gc.ca/e/22633.html#4-A4.6}[http://www.cihr-irsc.gc.ca/e/22633.html]).
  • This section is being reviewed and revised with consideration to a direct link from the program description and not only (as it is the currently) located in the Grants and Awards Guide.
  • Industry partners may be polled so that specific concerns and questions are captured and definitions clarified.

Recommendation 3: Program management should further analyse and study funding for RCTs to determine if and how they could be made more attractive to potential industry and academic participants. This issue was beyond the scope of this study and should be revisited in a more in-depth evaluation.

Agree

  • To sustain the integrity of the peer review, any change to RCT peer review of industry partner programs must be held to the same standards as other applications for funding in RCT. It is our opinion that an evaluation to ascertain consistency and effectiveness of process is a healthy and warranted exercise.
  • Furthermore we would need to ascertain whether or not changes (if any) would increase participation and application rate and benefit the program and communities.

Recommendation 4 - The program could enhance efforts to increase contact and encourage interaction between awardees and industry during the course of the partnership, thus building overall longer term collaborative capacity.

Agree

  • Networking and enhancing communication is a critical element of success not exclusive to this Program.
  • Enhancing bridging of researchers with Rx&D members will undoubtedly help build longer lasting relationships. This is an exercise that is never complete and can always be improved.
  • Institutes need to be increasingly engaged.

10. Participating And Non Participating Companies

The following 19 participating companies were contacted for the purpose of this review, of which 11 were interviewed.

  • Abbott Laboratories Canada
  • ALTANA Pharma Inc.
  • AstraZeneca Canada Inc.
  • AstraZeneca USA
  • Aventis Pharma
  • Bayer Inc.
  • Boehringer Ingelheim (Canada) Ltd.
  • Bristol-Myers Squibb Canada Inc.
  • Eli Lilly Canada Inc.
  • GlaxoSmithKline
  • Hoffman LaRoche Limited
  • Merck Frosst Canada Inc.
  • Neurochem Inc.
  • Novartis Pharmaceuticals Canada Inc.
  • Pfizer Canada Inc.
  • Pharmacia
  • Schering Canada Inc.
  • SOLVAY PHARMA Inc.
  • Wyeth

The following 15 non-participating companies were contacted for the purpose of this review, of which six were interviewed.

  • Actelion Pharmaceutiques Canada
  • Aeterna Laboratories Inc
  • AMGEN Canada Inc.
  • Berlex Canada Inc.
  • LEO Pharma Inc.
  • Lorus Therapeutics Inc.
  • Lundbeck Canada Inc.
  • Organon Canada Ltd.
  • Paladin Labs Inc.
  • Procter & Gamble Pharmaceuticals Canada
  • Purdue Pharma
  • Servier Canada Inc.
  • Shire BioChem Inc.
  • SYNX Pharma

Appendix A - Evaluation Issues

It identified a series of evaluation issues to be addressed during the life of the entire program, some in an interim evaluation and others in a comprehensive evaluation. The table below shows issues addressed in this interim evaluation study and those to be addressed in a future comprehensive evaluation.

Evaluation Issue To be addressed
What is the impact of the program on the amount and nature of pharmaceutical industry involvement in academic research in Canada? Comprehensive Evaluation
What is the quality (scientific merit) of the research that is supported by the CIHR/Rx&D program, and how does this compare with CIHR's other programs? Comprehensive Evaluation
What is the impact of the program on the formation and strengthening of relationships between pharmaceutical companies and academic researchers? Comprehensive Evaluation
To what extent have the research results generated, in research supported by this program, been useful to the industry partners, and how have they been used? Comprehensive Evaluation
To what extent have the industry partners made subsequent use of the relationships with researchers that are developed and strengthened through the program? Comprehensive Evaluation
To what extent have the students/recipients who have received training under this program been employed in the industry or in related applied employment? Comprehensive Evaluation
What is the impact of the industry participation in this program on the research programs of participating researchers? Comprehensive Evaluation
How can this program be designed and delivered so that, in addition to addressing its current objectives, it is also an effective vehicle for supporting the strategic research initiatives of the CIHR institutes? Interim Evaluation
Have the program design and delivery issues that were identified at the time the predecessor program was reviewed been satisfactorily dealt with in the CIHR/Rx&D program? Interim Evaluation
Is there a better (more efficient and/or more effective) alternative program model for achieving these same program objectives? Comprehensive Evaluation

Appendix B - Program Operation

In most cases there is an established relationship between an academic researcher or team of researchers and a company. The researcher(s) and the supporting pharmaceutical company work together to develop a research proposal - for a research project, fellowship, chair, clinical trial, studentship, etc. The first step in an investigator-initiated proposal is a letter of intent, which is submitted to CIHR approximately two months before the full proposal by the principal investigator. The letter includes background on the applicant, a project overview, and details of the company sponsor's level of support. The full application is then prepared using CIHR's regular application forms and the forms from the Industrial Partnership Module and submitted to CIHR for review. There are five application deadline dates each year. Applications are reviewed by a review panel made up of members of CIHR's University-Industry Peer Review Committee in accordance with the following peer review procedures.

  • The application is sent to at least three experts in the research area involved, who provide a written assessment.
  • A volume containing the essential information for all applications is sent to each panel member. Each application is assigned to two panel members for in-depth review.
  • The two reviewers report on their evaluation in a meeting of the panel.
  • The application is then rated by the full panel.

There are some variations in this process for very large projects and training awards. Proposals for very large projects may require site visits and review by a select panel of senior scientists. Applications for training awards do not require assessment by external experts. The factors used in the assessment of proposals are:

  • Importance and/or originality of hypotheses.
  • Importance and originality of expected results.
  • Appropriateness of methodology.
  • Adequacy of literature review.
  • Applicant's experience, training, and research productivity.

Applications which receive a rating of 3.0 or more (on a scale of 1 to 4.9) are approved for funding. In a company-initiated proposal, the company approaches CIHR and indicates that they would like to support a research program or a number of trainees or independent researchers in an area of interest. A Request for Application is developed jointly by CIHR and the sponsoring company. Both the company and CIHR publicize the funding opportunity through web site postings and direct mail-outs to solicit applications. At the time of this study, CIHR Institutes were just getting underway, and there was little formal program activity specifically related to the Institutes. It was planned that there would be more Institute-initiated projects supported through the CIHR/Rx&D program. As of 2003, there had been one strategic initiative with the Institute of Circulatory and Respiratory Health. For all research grants with duration longer than three years, progress reports are required after 2 ½ years. Progress reports for RCT projects can be requested on an annual basis.

  1. The full report is available upon request.
  2. CIHR/Rx&D Program: Evaluation and Performance Measurement Framework Project, BearingPoint, April, 2002, complete list of evaluation issues shown in Appendix A.
  3. A predecessor program to the CIHR/Rx&D program initiated in 1993 as a result of an agreement between the Medical Research Council of Canada and the Pharmaceutical Manufacturers Association of Canada, called the MRC-PMAC Health Program MRC expended approximately $40 million dollars on the MRC-PMAC program up through the end of 1999, with matching industry expenditures of $200 million.
  4. There are also a few projects carried out under this program for which CIHR provides no matching funding but simply a "seal of approval" as a result of the project having successfully passed through the CIHR peer review process.
  5. These have been derived from a review of program documentation and interviews of program personnel.
  6. Activities are the tasks that are carried out by CIHR program personnel. Outputs are the goods and services which are produced as a result of the activities – for example, research grants that are awarded to researchers. Outcomes are things which are done or experienced by others as a result of the outputs. Outcomes can be subdivided into immediate outcomes (e.g., research is conducted), intermediate outcomes (e.g., research findings are generated), and long-term outcomes (e.g., a new drug is developed and commercialised based on the research findings).