Mid-Term Evaluation of the Institute of Neurosciences, Mental Health and Addiction (INMHA)

Executive Summary

December 2005

1. Executive Summary
    1.1 Overview of CIHR
    1.2 Overview of the Institute of Neurosciences, Mental Health and Addiction (INMHA)
        1.2.1 Mandate
        1.2.2 Strategic Priorities
    1.3 Evaluation Objectives and Issues
        1.3.1 Methodology
2. Evaluation Results
    2.1 Relevance
    2.2 Effectiveness
    2.3 Delivery
3. Recommendations
4. Management Response

1. Executive Summary

1.1 Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

CIHR was created under The CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.

CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).

1.2 Overview of the Institute of Neurosciences, Mental Health and Addiction (INMHA)

According to INMHA documentation, the INMHA is the largest of the 13 CIHR Institutes in terms of the number of scientists working in the research domains covered by its mandate,¹ and the number of potential partner organizations such as associated voluntary and professional organizations.² The number of research communities that are included under the mandate of the INMHA would represent six or seven³ individual Institutes of the National Institutes of Health (NIH) in the United States. Within this context, the INMHA developed a set of strategic priorities and strategic goals that supported this large mandate and mission.

1.2.1 Mandate

INMHA's mandate is to support research to: enhance mental health, neurological health, vision, hearing, and cognitive functioning and to reduce the burden of related disorders through prevention strategies, screening, diagnosis, treatment, support systems, and palliation. Associated research advances our understanding of human thought, emotion, behaviour, sensation (sight, hearing, touch, taste, smell), perception, learning and memory.⁴

In addition to the mandate indicated above, the INMHA has described its mission in the Strategic Plan developed in 2001 for the Institute. The mission of the Institute is to: foster excellence in innovative, ethically responsible research in Canada that aims to increase our knowledge of the functioning and disorders of the brain and the mind, the spinal cord, the sensory and motor systems, as well as mental health, mental illness and all forms of addiction. The INMHA seeks to translate this new knowledge into a better quality of life for all Canadians through improved health outcomes, health promotion and health care services.⁵

1.2.2 Strategic Priorities

As presented in the INMHA's Strategic Plan 2001-2005, the INMHA identified five strategic priority areas. While ensuring the priorities of the INMHA were in keeping with the four overall strategic directions for CIHR, the Institute developed the priorities to assist in the advancement of innovative cross-Institute and cross-theme approaches to research in neurosciences, mental health and addiction. As well, they were chosen to assist in establishing partnerships with stakeholders that maximize investments in health research. The stated strategic priorities of the INMHA are:

To foster and develop a capacity for innovation in research in neurosciences, mental health and addiction that will strengthen Canada's health research milieu in these fields and enhance its competitive position on the international scene. The four main areas of focus for this priority are:
- Training;
- Strategic initiatives;
- Research in emerging areas or where unique opportunities are arising;
- Research in bioethics.
To pursue and sustain collaborative partnerships with governmental, non governmental and volunteer health organizations as well as pharmaceutical and biotechnology industries that will enable the INMHA to share, develop, obtain or leverage resources required to accomplish its mandate.
To promote linkage and exchange between the research community, the municipal, provincial and national levels of decision-makers as well as users of research results, including non governmental and volunteer health organizations, through structured efforts aimed at knowledge translation.
To develop the INMHA's presence on the international stage through joint research, training and funding initiatives with scientific and research funding agencies in other countries.
To establish an organizational and an operational structure that will enable the INMHA to accomplish its goals.

The initial four priorities were designed to assist in the development of the INMHA's research agenda, while the last priority was to assist in development of the INHMA structure, support and governance of activities.

1.3 Evaluation Objectives and Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:

to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.

The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:

Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

1.3.1 Methodology

The evaluation consisted of four main lines of evidence:

a review of documents and administrative data relating to the INMHA;
41 key informant interviews with INMHA staff and IAB members, researchers and students, stakeholders and partners, as well as a focus group discussion with IAB members (held at an IAB meeting);
case studies of three INMHA initiatives; and
a telephone survey of 191 funded and 85 non-funded researchers affiliated with the INMHA.⁶

2. Evaluation Results

2.1 Relevance

In general, the INMHA mandate was viewed as broad, but appropriate. A number of challenges and opportunities were identified with having such a broad mandate. The main challenge identified was the limitation in funding while addressing and supporting the needs of a very large researcher and stakeholder community. The main opportunity identified was the potential for transdisciplinary research when so many different research and stakeholder communities are brought together under one Institute.

The evaluation found that researchers and stakeholders generally supported the strategic priorities identified by the INMHA. The main area that the evaluation identified as needing additional consideration in the future was the integration of the "psycho-social" perspective when developing Requests for Applications (RFAs).

Overall, the INMHA is viewed as an appropriate mechanism for developing and supporting health research in the areas of neurosciences, mental health and addiction. This perspective was often provided with the caveat that the CIHR virtual Institute model is still young, and will need more time before its efficacy, appropriateness and alternatives can be accurately assessed.

2.2 Effectiveness

The evaluation found that INMHA had made demonstrable progress in many areas, and had been able to establish promising foundations for progress in another area. The areas that were highlighted in which INMHA had made considerable progress were with respect to capacity building, developing an international presence, and development of partnerships and collaborations. The area identified in which INMHA has contributed strong foundations promising future progress was in creating new knowledge. As well, neuroethics was highlighted by staff and IAB members as an area that the Institute was taking an international lead in developing.

The INMHA appeared to have more mixed results with respect to the area of knowledge translation. The Institute provided examples of its efforts in knowledge translation, however, there appears to be some need for further clarification and demonstration as to what KT is, and how KT can be incorporated with research in the domains covered by INMHA.

2.3 Delivery

Overall, stakeholders and researchers viewed favourably the approaches used by the INMHA to plan and select strategic priorities. While researchers reported that the INMHA had been relatively successful in identifying and responding to emerging priorities, a few IAB members felt that with the limited funding available, it is challenging to appropriately address these emerging priorities.

The evaluation found that the IAB functioned well and appropriately. The structured working groups on the IAB were viewed positively, and frequently cited as one of the main contributing factors to the success the IAB had experienced.

Consultations and communications appear to be a strength of the INMHA. There were numerous examples cited with respect to the varied approaches that the INMHA uses for its communications. By employing both formal and informal mechanisms, the INMHA is able to communicate with a wide variety of audiences including the general public, students, researchers and stakeholders.

3. Recommendations

Following are recommendations that emerge from this mid-term evaluation of INMHA. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.

Recommendation 1:

The Institute has been found to be doing well and is encouraged to continue the following:

sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities;
maintain the current structure and operation of the IAB;
maintain current planning mechanisms, as they are effective;
maintain its varied approach to communication and consultation with a wide variety of audiences;
sustain its effective work undertaken to date in the area of collaboration and partnerships;
sustain its contribution to the transformative vision of CIHR through its IAB representation, its RFA requirements, and its support to strategic initiatives favouring the integration of researchers from non-biomedical and clinical research areas; and
maintain its leading work in development of neuroethics.

The following areas are ones in which it is recommended the Institute take action to improve:

Recommendation 2:

Ethics - While INMHA has conducted extensive activities in the area of ethics, awareness of those activities and results among INMHA's various stakeholder groups could be improved. Additional attention should be paid to communication of the INMHA's role in the implementation of the CIHR ethics mandate across all stakeholder groups.

Recommendation 3:

Consultation and Communication - Improved efforts should be undertaken to increase the visibility of the Institute among student researchers, and potentially the visibility of student researchers among university administrators. The INMHA may also have an increased role to play in not only increasing its own visibility among student researchers, but also in increasing the visibility of "star" student researchers in the INMHA research domains among university administrators. The Brain Star Award program appears to be a very good start in this direction.

Recommendation 4:

Knowledge Translation - The INMHA should review its efforts in support of KT. The efforts expended by the INMHA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area could be reviewed to determine to what extent the efforts are likely to result in the impacts that the INMHA are hoping to achieve.

Recommendation 5:

Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.

4. Management Response

Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:

Recommendation	Response	Action Plan
1. Ethics - While INMHA has conducted extensive activities in the area of ethics, awareness of those activities and results among INMHA's various stakeholder groups could be improved. Additional attention should be paid to communication of the INMHA's role in the implementation of the CIHR ethics mandate across all stakeholder groups.	Disagree See comments	Nov. 2005 International Panel with 19 representatives of 8 countries (Canada, Germany Italy, Japan, Sweden, Switzerland, U.K., U.S.A.) Washington, D.C. co-chaired by S.D. and Judy Illes, IAB member Dec. 2005: Chair in Neuroethics RFA launched. The Chair will allow a Canadian university or health research institute to support a leading scientist in the field of neuroethics in Canada. Planned specifically to increase awareness among stakeholders Dec 2005: SD co-author of "International perspectives on engaging the public in neuroethics" published in Nature Reviews Neuroscience May 2006 Symposium at ACFAS meeting at McGill Nov. 2006 Proposed Symposium at the Society for Neuroscience Meeting in Atlanta, USA (pending) Regular update in INMHA's newsletter, the Brain Brief, circulated widely (including other institutes) One new section on INMHA Web page dealing with neuroethics Discussion with and monitoring of the Neuroethics New Emerging Team from Halifax and future teams Preparation of a one-pager on Neuroethics Ongoing activity of IAB Focus Group
2. Consultation and Communication - Improved efforts should be undertaken to increase the visibility of the Institute among student researchers, and potentially the visibility of student researchers among university administrators. The INMHA may also have an increased role to play in not only increasing its own visibility among student researchers, but also in increasing the visibility of "star" student researchers in the INMHA research domains among university administrators. The Brain Star Award program appears to be a very good start in this direction.	Disagree Given the small number (3) of student surveyed the recommendation may not be justified See also comments	Report on INMHA- STIHR meeting with session ran by trainees and held in Montreal in Nov. 2005
		Work more closely with the University Delegates to publicize the Brain Star Program
		Production of a STIHR-INMHA brochure and distribution at national and international meetings as recommended during the STIHR meeting
		Regular update in INMHA's newsletter, the Brain Brief, circulated widely (including other institutes)
		STIHR-INMHA meeting focused on trainees planned every two years
3. Knowledge Translation - The INMHA should review its efforts in support of KT. The efforts expended by the INMHA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area could be reviewed to determine to what extent the efforts are likely to result in the impacts that the INMHA are hoping to achieve.	Agree	Oct. 2005 Workshop at INMHA meeting with NGOs on Knowledge Translation Creation of a Knowledge Translation Focus Group of the IAB Upcoming Partnership on a Pain KT Network with IMHA KT included in the Advancing Knowledge to reduce Nicotine Addiction and Tobacco Abuse" RFA Focus on KT in the next Strategic Plan (2006-2010) Canadian Mental Health Services Knowledge Translation Network whose goal would be the dissemination and uptake of knowledge to inform policy-making and improve the mental health system.
4. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.	Agree	Continue to publish a quarterly newsletter widely distributed to our stakeholders and CIHR Continue to publish a printed version of our annual report for our stakeholder Continue to publish annually a brochure featuring the Brain Star Awardees Continue to organize an annual meeting for researchers, stakeholders and trainees Update annually the performance report and monitoring results according to common indicators and strategic plan Continuous follow-up of teams funded by INMHA and its partners with targeted objectives (trainees, publications, KT) Target to increase student awareness of INMHA by 25% in the next 3 years. Aim to increase national and international partnership and visibility Increase by 25% the number of Graduate Students in INMHA areas Decrease stigma associated to neurological disorders, mental illness and addiction Partnership with Senator Kirby and Mental Health Commission Funding of two more teams on Suicide Prevention targeted to Aboriginal Peoples. Funding of one more team on Neuroethics Funding of three teams on Early Life Events and First Episodes of Brain Disorders Development of a RFA on comorbidity Regenerative Medicine and Nanomedicine Mental Health in the Workplace

¹ INMHA (2004) Annual Report 2003-2004: Institute of Neurosciences, Mental Health and Addiction. CIHR-IRSC
² INMHA (2004) The Brain Brief; 1(4). CIHR-IRSC
³ Quirion, R. (2002) A Canadian Experiment: INMHA. How to link up the brain via a virtual institute. Trends in Neuroscience] 25: 268-270.
⁴ INMHA Web site (3/10/2005)
⁵ INMHA (2001) Strategic Plan 2001-2005: Institute of Neurosciences, Mental Health and Addiction. CIHR-IRSC. p.10
⁶ Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.

Breadcrumb trail