Mid-Term Evaluation of the Institute of Genetics (IG)

Executive Summary

December 2005

1. Executive Summary
    1.1 Overview of CIHR
    1.2 Overview of the Institute of Genetics (IG)
         1.2.1 Mandate
         1.2.2 Research Priority Themes
    1.3 Evaluation Objectives and Issues
         1.3.1 Methodology
2. Evaluation Results
    2.1 Relevance
    2.2 Effectiveness
    2.3 Delivery
3. Recommendations
4. Management Response

1. Executive Summary

1.1 Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

CIHR was created under The CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.

CIHR 's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR 's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).

1.2 Overview of the Institute of Genetics (IG)

1.2.1 Mandate

The mandate of the IG is to support research on the human and other genomes, as well as on all aspects of genetics (including the interaction of genes with physical and social environments), basic biochemistry and cell biology related to health and disease, including the translation of knowledge into health policy and practice, and the societal implications of genetic discoveries.

1.2.2 Research Priority Themes

The IG has identified six strategic research priority themes. These were selected as areas of fundamental/increasing importance, or having potential for significant impact to address a critical weakness or build on an area of Canadian strength:

Integrating the Physical and Applied Sciences into Health Research;
Proteomics and Bioinformatics;
Population Genetics, Genetic Epidemiology, and Complex Diseases;
From Genes to Genomic Medicine (including Clinical Genetics Research);
Health Services for Genetic Diseases; and
Genetics and Ethical, Legal and Social Issues.

1.3 Evaluation Objectives and Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:

to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.

The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:

Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR ?
Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

1.3.1 Methodology

The evaluation consisted of four main lines of evidence:

a review of documents and administrative data relating to the IG ;
40 key informant interviews with IG staff and IAB members, researchers and students, stakeholders and partners, including a focus group discussion with IAB members (held at an IAB meeting);
case studies of two IG initiatives; and
a telephone survey of 177 funded and 42 non-funded researchers affiliated with the IG.¹

2. Evaluation Results

2.1 Relevance

In general, the IG 's mandate receives broad approval from its researcher and stakeholder communities and most believe the strategic research priorities have been appropriately identified and have continued relevance. While the broad mandate and the numerous research domains that are encompassed by the Institute pose challenges in terms of managing expectations across a wide array of research and stakeholder communities, a perceived strength of the Institute is the breadth of disciplines that have been assembled into a genetics community.

One caution that was raised in the key informant interviews was the extent of awareness of IG and engagement of basic scientists who, given their historical connection to the Medical Research Council, more strongly identify with CIHR central and the open competitions. In the researcher survey, IG-affiliated researchers have a lower level of familiarity with the IG than CIHR researchers overall (referring to the Institute with which they are aligned) and also indicate less support for strategic research in general and the CIHR virtual model.

There is support among key informants that the IG is an appropriate and important mechanism to address capacity, research excellence and knowledge translation in the genetics and basic sciences fields (although, given the funding limitations, the IG is not the only mechanism). Key informants stressed the role of complementary organizations and granting avenues in meeting these objectives. Surveyed researchers are less uniform in their opinions: while seeing a need for the Institute in fostering research excellence and capacity development, there is a segment of the IG 's constituency that does not see a need for funding of strategic research. Few suggested alternatives to the IG mechanism were identified, however.

2.2 Effectiveness

The evaluation results indicate that the IG has contributed to the objectives and mission of CIHR in many ways and has made good progress in fulfilling its mandate and strategic priorities. Knowledge creation is a strength of the Institute. The Institute has issued a number of Requests for Applications (RFAs) in strategic areas, as well as developing a suite of continuous programs that focus on innovation and capacity. These efforts are consistent with the IG 's strategic goal of supporting the work of individual investigators. The Institute has also been a key contributor to CIHR joint strategic initiatives, being a strong force behind the Canadian Lifelong Health Initiative and the Regenerative Medicine and Nanomedicine Initiative.

The Institute sees knowledge translation (KT) as a key priority - its strategic goals and priorities clearly reflect this. The case study of the health services for genetic diseases research priority illustrates the Institute's KT activity. In addition, the Institute contributes to KT through RFAs, public education, and development of white papers. The Institute's researcher and stakeholder communities, however, are largely unfamiliar with the Institute's contribution in this area (stemming in part from varying definitions/understanding of KT), and IG-affiliated researchers are also less apt to have a KT component in their own research projects.

The IG has dedicated significant resources to capacity building using a variety of means such as the New Principal Investigator (PI) Meetings, awards, and Strategic Training Initiative in Health Research (STIHR) ² grants among others. Until now, IG activities have been focussed on strengthening weaker communities. The more recent focus, with "From Genes to Genomic Medicine", is to build on strengths. Key informants generally agree that the Institute has made a positive contribution to capacity development, and most researchers in the survey feel that IG has contributed, to at least some extent, to the development of people and the research environment.

The transformative vision of CIHR has been operationalized by IG through its focus on integrating disciplines, focusing attention on underdeveloped research communities and knowledge translation. Still, the bulk of open and strategic funding is directed to the IG 's primarily Theme 1 community, with much less funding devoted to the other Themes, though it should be added that some programs to which IG has contributed, such as Interdisciplinary Capacity Enhancement (ICE) or New Emerging Teams (NETs), require that the research teams be composed of various research themes. IG-affiliated researchers generally report a lower percentage of their research projects as being interdisciplinary.

Examples of IG 's contribution to the CIHR ethics mandate are numerous. The Institute has been a leader in this area through its Genetics and Ethical, Legal and Social Issues Priority and Planning committee, and devoted significant resources to funding initiatives related to ethics.

In terms of its own strategic priorities, the Institute has funded initiatives in all its strategic priority areas, adopting different strategies depending on the needs of the particular area or research community. To the extent that they are aware, the Institute's researcher and stakeholder communities perceive the IG to be "performing well" relative to its strategic priorities (within its funding capacity) and the majority of surveyed researchers believe that the overall mandate of the Institute has been achieved to at least some extent. CIHR investments in research related to the IG 's mandate have increased significantly over the past five years. Progress on the Institute's strategic priority related to population genetics/genetic epidemiology and complex diseases has suffered with the deferral of the birth cohort study, however. This is a complex area, with few researchers in Canada and internationally, and so has presented challenges for the Institute to advance this priority area. For other priorities, there are several noteworthy achievements, though progress is often slow in nurturing small, nascent disciplines.

In the area of partnerships, the Institute has been very active in inter-Institute activities (co-funding RFAs with other Institutes, as well as joint strategic initiatives, co-sponsoring New PI meetings). There has also been significant activity in international collaborations and the IG has developed some linkages with other organizations in its domain (Networks of Centres of Excellence, Genome Canada), though this is an area where key informants suggested a more expansive effort. While the IG has built a solid foundation for collaboration with Voluntary Health Organizations (VHOs), finding funding partners has been a challenge (the IG 's mandate does not directly align with the mission of any of the sizeable VHOs who associate more readily with a disease or organ-based Institute). Still, the Institute has partnered with organizations such as the Heart and Stroke Foundation and Foundation for Fighting Blindness and is working toward developing a project-based collaboration involving a consortium of VHOs.

According to key informants, the IG has demonstrated leadership in a number of areas: fostering of emerging areas; stimulation of cross-disciplinary work and providing new opportunities for different types of research through strategic funding through RFAs and the Institute's regular programs. The leadership of the SD was also praised. Most surveyed researchers believe the Institute is influencing the research agenda within its mandate at least to some extent. A minority of key informants questioned the ability of the Institute to influence the research agenda in light of its limited budget, however.

2.3 Delivery

There is broad approval for the planning and strategic mechanisms employed by the Institute - there is confidence among key informants and surveyed researchers that the IG has and continues to identify emerging priorities in the field. The IAB operates as an effective advisory body for the IG and the Priority and Planning (P&P) committee structure is widely praised as a means for consulting the broader research community and operationalizing the Institute's strategic priorities. The work of the committees provides an effective foundation for the Institute's decision-making.

Consultation is seen to be strength of the Institute. In addition to sponsoring workshops and meetings within its community, the Institute maintains a strong presence at scientific and professional meetings and conferences and also maintains an effective informal network.

Communications with the Institute's various audiences is undertaken primarily by e-mail, the Web site and public media, as well as informal networks and contacts. Communications was identified as an area for future further development. Some suggestions were operational in nature (primarily, improving the Web site). Some key informants (particularly stakeholders) desire more regular communications from the Institute. Others believe the Institute should have greater visibility (e.g., among elected officials, government, the public) and, in fact, media analyses show only modest coverage of the IG in the context of CIHR coverage overall. Finally, an issue with respect to communications is the latitude and capacity of the Institute to communicate its activities and successes. This likely implies the need for a clearer definition of roles vis-à-vis the communications efforts of CIHR central and specialized communications support from outside the Institute, given its current resource limitations. The need for a clearer definition and more guidance from CIHR central was also mentioned in the context of KT.

Funding of the IG was a prevalent theme throughout the evaluation study. Most key informants feel that the level of funding to the Institute is inadequate in light of its broad mandate. Some argue that the equal funding of 13 Institutes is not sustainable beyond the initial start-up and would support a variable funding model based, for example, on an Institute's mandate, size of its research community, or success of its programs. The majority view among key informants is that the 70:30 balance between investigator-initiated and strategic research funding is appropriate (though many argued for increasing the amounts in both pools). However, there is a substantial pocket of interviewees who favour more emphasis on investigator-initiated funding. Significant concern was raised by key informants about perceived growing application pressure in CIHR open competitions and the sustainability of researchers in the longer-term (e.g., beyond the developmental grants and seed funding offered by the Institutes), as well as Canada's international competitiveness in this area.

3. Recommendations

Following are recommendations that emerge from this mid-term evaluation of IG. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.

Recommendation 1:

The Institute has been found to be doing well and is encouraged to continue the following:

sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities, in concert with complementary organizations;
maintain the current structure and operation of the IAB;
maintain planning mechanisms, as they are effective;
maintain efforts in knowledge creation, within the Institute's reflective process that is based on lessons learned from prior efforts (e.g., taking into consideration readiness of the research community to respond, community saturation points, adequate RFA response lead time);
sustain its productive efforts in the ethics area; and
persist in its proactive efforts to be the voice of the basic sciences pipeline and to court leveraging opportunities vis-à-vis consortia of VHOs/other granting organizations.

The following areas are ones in which it is recommended the Institute take action to improve:

Recommendation 2:

Consultation and Communication - There is a continuing need for outreach to the basic medical scientist community to solidify their awareness, engagement and buy-in.

Recommendation 3:

Consultation and Communication - While internal resources are limited for this activity, the Institute should consider a more expansive approach to communications overall - with the support of CIHR central and stakeholder organizations - to clarify the ability and means for the Institute to raise its profile within its researcher/stakeholder constituency and the public, and to trumpet successes.

Recommendation 4:

Collaboration and Partnerships - There is a perceived need among the IG 's researchers/stakeholders for greater explanation of the role of the IG vis-à-vis other similar organizations (primarily Genome Canada) and partnership development with a broader community of stakeholders (e.g., private sector such as pharmaceutical/biotechnology).

Recommendation 5:

Knowledge Translation - The IG should continue to review its efforts in support of KT. The efforts expended by the IG in the area of KT do not always appear to have a large profile among IAB members, researchers & stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IG are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.

Recommendation 6:

Contribution to Transformative Vision - 1. The Institute should continue its efforts in contributing to the transformative vision of CIHR through its emphasis on inter-disciplinary work and focusing attention on nascent disciplines. 2. The Institute should continue to review what is being done, including all aspects to the transformative vision, in, particular, funding across the other three Themes.

Recommendation 7:

Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.

4. Management Response

Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:

Recommendation	Response	Action Plan
1. Consultation and Communication - There is a continuing need for outreach to the basic medical scientist community to solidify their awareness, engagement and buy-in.	Agree: YES	IG will continue to consult with the community, including the strong input though the Priority and Planning (P&P) committee structure. IG will continue to distribute funding announcements to the research community. IG will engage a consultant to develop an IG- specific communications proposal.
2. Consultation and Communication - While internal resources are limited for this activity, the Institute should consider a more expansive approach to communications overall - with the support of CIHR central and stakeholder organizations - to clarify the ability and means for the Institute to raise its profile within its researcher/stakeholder constituency and the public, and to trumpet successes.	Agree: YES	IG will engage a consultant to develop an IG- specific communications proposal. Implementation of a communication strategy will require assistance from CIHR. Continue to publicize results from IG funded programs. Continue to promote outcomes of IG funded projects. Continue to promote results from Maud Menten Prize for the New PIs. Continue to support national meetings, on average 10-20 per year. Continue to organize Annual NEW PI meeting, in partnership with another CIHR Institute.
3. Collaboration and Partnerships - There is a perceived need among the IG 's researchers/stakeholders for greater explanation of the role of the IG vis-à-vis other similar organizations (primarily Genome Canada) and partnership development with a broader community of stakeholders (e.g., private sector such as pharmaceutical/biotechnology).	Agree: ____ Disagree: YES	IG will continue to communicate our Institute activities and its roles and responsibilities to the entire stakeholder community. Continue to work with GC to organize consultative meetings with researchers and other stakeholders, in the areas of mutual interest. Continue to works with existing partners and explore new partnership opportunities, specific to Programs in progress and Regular IG Funding Programs
4. Knowledge Translation - The IG should continue to review its efforts in support of KT. The efforts expended by the IG in the area of KT do not always appear to have a large profile among IAB members, researchers & stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IG are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT..	Agree : YES	Our major effort will be to continue what we are doing, i.e almost all our activities are KT. In addition, we will: Continue partnership with CIHR KT, other Institutes Continue to explore KT strategies in partnership with VHO sector Explore partnering on the Case Book program, developed by IHSPR and IPPH Develop targeted KT strategy for IG , to be presented at future IAB meeting
5. Contribution to Transformative Vision - The Institute should continue its efforts in contributing to the transformative vision of CIHR through its emphasis on inter-disciplinary work and focusing attention on nascent disciplines. The Institute should continue to review what is being done, including all aspects to the transformative vision, in, particular, funding across the other three Themes.	Agree: YES	Continue to contribute to the transformative vision of CIHR through its emphasis on inter-disciplinary work trough investment into new programs and consultation and activities in all IG research priority themes. Continue to support development of nascent disciplines. Key examples include: Health Services for Genetic Diseases, Genetics and Ethical, Legal and Social Issues, Population Genetics and Complex Traits. Continue to build on strengths of the basic science community and clinical research community in next big RFA.
6. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.	Agree: YES	Continue to monitor performance of IG programs on a regular basis, report to IAB and adjust IG actions accordingly. IG , ICR are working with CIHR Evaluation Unit to develop the "End of the Funding Report", to be used for monitoring the outcomes of Grants and Awards. IG has developed the brief questionnaire, approved by CIHR Evaluation Unit, to survey the outcomes of a subset of IG -funded Grants and Awards whose grant has already terminated and which will not be captured by the "End of the Funding Report", due to later implementation.

Institute of Genetics Strategic Initiatives Budget (Nov. 25, 2005)

¹ Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
² STIHR (the Strategic Training Initiative in Health Research) consists of innovative and interdisciplinary training programs that support the development of new researchers. Each program can receive up to $310,500 a year for up to six years and at least 71% of these funds must be used for trainee stipends or travel between training locations.

Breadcrumb trail