Mid-Term Evaluation of the Institute of Gender and Health (IGH)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Gender and Health (IGH)
1.2.1 Mandate
1.2.2 Objectives
1.2.3 Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Gender and Health (IGH)
The mandate of IGH is to support research to address how sex (biological factors) and gender (socio-cultural experiences) interact with other factors that influence health to create conditions and problems that are unique, more prevalent, more serious or different with respect to risk factors or effective interventions for women and girls, men and boys.
IGH has the following five objectives:
Generate evidence with respect to health status and health behaviour of health services' use by females and males across the life span; influence of gender and sex on health status, health behaviours, and health services use, and their interactions with other determinants of health.
- IGH offers an unprecedented opportunity to generate evidence regarding the influence of gender and sex on health, their interactions with other health determinants, and the implications for interventions. Elucidating the concepts of gender, sex and health and their interactions through research, bridges major gaps in the study of women's and girls' health and men's and boys' health. IGH creates opportunities to generate excellent research that 1) bridges gaps in the knowledge base such as: a) health status, health behaviours and health services use of females and males across the lifespan; b) influence of gender and sex on health and their interactions with other determinants; c) relevant interventions, and 2) addresses major national and international health related issues.
Enhance the capacity to conduct gender and health research in Canada.
- IGH understands that new and established investigators representing the full spectrum of health research need to be attracted to the study of gender, sex and health. Furthermore, a gender and sex perspective will involve emerging and established researchers within both academic and community settings. The capacity of researchers and research groups to conduct gender-sensitive research across disciplines and research settings must be expanded.
Provide evidence to inform design of programs, policies and practices that enhance health of women, men, girls, and boys in Canada.
- Translation of new knowledge about gender, sex and health should be linked to syntheses of existing knowledge. Partnership with stakeholders in policy, program, and community arenas is important for knowledge transfer of gender and health research. IGH research priorities demand a transdisciplinary and transformative approach engaging diverse health and health-related disciplines, as well as policy, program, practice, and public domains. The goal is to provide evidence to inform the design of programs, policies and practices that enhance the health of women, men, girls, and boys in Canada.
Exert international impact.
- The influence of sex and gender on health is receiving worldwide recognition. Integrative and innovative research on gender, sex and health will have a national and international impact. Ongoing efforts on the international stage positions IGH as a world leader in advancing sex and gender in health research. The goal is to exert international impact by generating and communicating new knowledge re: gender, sex and health and building international collaboration.
Advance the gender and health perspective through strategic research and research capacity building.
- Strategic funding partnerships encourage organizations across research, policy, program, and community domains to "think gender and sex". Partnerships also provide unique opportunities to access diverse segments of the Canadian population. IGH encourages gender and sex sensitive analysis through all strategic research initiatives, providing partners with the opportunity to promote gender equity in health research. The goal is to advance the gender and health perspective nationally and internationally and facilitate the development of supportive environments for the conduct of gender, sex and health research.
IGH has, in consultation with its stakeholders, set the following five research priority themes:
- Access and equity for vulnerable groups/populations;
- Health equity
- Access to health services
- Violence, gender and health
- Health across the lifespan;
- Healthy child development
- Healthy aging
- Work, leisure and health
- Mental health in a changing society
- Reproductive and sexual health
- Promoting health in the context of chronic and infectious conditions;
- Chronic diseases and infections
- Disabilities
- Promoting positive health behaviours and preventing addictions; and
- Promoting positive health behaviours
- Preventing addictions
- Environment and health.
These research priorities, from a gender perspective, guided the launch of all strategic research initiatives and priority announcements, and funding of workshops of IGH.
1.3 Evaluation Objectives and Issues
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- A review of documents and administrative data relating to the IGH;
- 41 key informant interviews with IGH staff and IAB members, researchers and students, stakeholders and partners;
- Case studies of four IGH initiatives; and
- A telephone survey of 31 funded and 16 non-funded researchers affiliated with the IGH. 1
2. Evaluation Results
2.1 Relevance
The majority of interviewees feel that the Institute's mandate is appropriate, with some noting the breadth and comprehensiveness of a mandate that includes both genders and embraces all four CIHR research themes. This breadth is also seen to be reflected in IGH's research priorities, although there is some concern that the priorities are too broad.
Interviewees generally feel that since IGH's inception, there have been changes in gender-based health research, particularly a growing focus on gender in health research, for which the Institute was considered to be largely responsible. IGH researchers surveyed also feel that there is a need for CIHR and that IGH specifically is needed.
Interview results indicate that most see the Institute as the most appropriate mechanism to make a difference in developing Canadian capacity, research excellence and knowledge translation (KT), but some suggestions were made regarding how well it functions, such as more money for research instead of KT, and more support from other Institutes on gender and health issues. Most interviewees feel that the CIHR virtual model is good and sustainable, noting that an advantage is increased flexibility, but at the same time identifying communication issues as a considerable disadvantage. Therefore, for the model to be truly effective communication requires extra effort.
2.2 Effectiveness
Funded researchers surveyed and interviewees generally agree that IGH has contributed to the creation of knowledge at least somewhat, despite some concern that it may be too early to assess impacts. This outcome is achieved primarily through Requests for Applications (RFAs)2 and priority announcements through CIHR open competitions.
IGH has undertaken a number of activities in order to carry out KT, such as a Knowledge Translation Working Group, a KT component in all RFAs and research on KT, and generally interviewees consider KT to be a priority for IGH and that the Institute has contributed to it. Nevertheless, many think that the concept is poorly defined and that IGH would benefit from some CIHR corporate guidance. Most funded researchers surveyed believe that IGH activities support knowledge translation at least somewhat.
IGH has made considerable progress in capacity building. Related efforts on the part of IGH may be reflected in an increase of female PIs in CIHR open competitions from 2000-01 to 2004-05, as well as an increase in IGH-funded researchers who went on to apply to open competitions and were successful. The opinions of interviewees align with these data, as do the opinions of funded researchers surveyed.
Interviewees generally agree that IGH has also contributed to the transformative vision of the CIHR, given that it focuses on gender, sex and health, a departure from the old Medical Research Council (MRC) model. This focus tends to bring together researchers from a variety of different disciplines.
The Institute's contribution to the CIHR ethics mandate appears far less clear. Although most funded researchers believe that the CIHR provides leadership on ethics and supports the research necessary to address ethical issues, few interviewees were able to comment on how IGH has contributed to ethics. It would appear, however, that IGH is contributing an increasing amount of funding to ethics as time goes on.
Interviewees and survey respondents agree that IGH has made progress in terms of achieving its mandate and strategic priorities, but it may be difficult to determine this progress because IGH has had to establish a research community where one did not exist previously, making for extra foundational effort. Now that this has happened, the progress to come may be far clearer to see. Most IGH-funded researchers also think that the Institute has performed at least somewhat well with respect to its mandate and priorities.
Based on the evidence from this mid-term evaluation it appears that good collaboration is taking place among CIHR institutes, although this may be dependent in some part on whether funding is involved. Partnering is progressing (a number of organizations have been cited in IGH documentation), albeit slowly, according to interviewees. This may be due, in part, to the lack of natural partners for IGH at inception; the Institute has had to create all networks and affiliations "from scratch". Nevertheless, IGH data indicate that the Institute has partnered successfully with all of the other CIHR Institutes on more than one initiative, and most funded researchers think that IGH has been facilitating collaboration.
One method used by Institutes to influence research direction is the creation of funding opportunities in the areas of strategic and emerging research importance relevant to the Institute's research domains. The extent to which these opportunities generate responses from the research community is an indicator of the level of activity and interest in these research areas. Based on these data the Institute has generally received a good response to the RFAs issued. Consultations also provided input into strategic research priorities and helped focus attention on the need for ethical and sensitive research, and inclusive approaches that reflect sexual, cultural and economic diversity. Interviewees agree that IGH has positively influenced the research agenda in the gender and sex-related health research community, noting that IGH has focused on an area of research that had received insufficient attention previously, and has ensured that all research that it funds incorporates gender-based analysis. In doing this, the Institute is seen as providing much needed leadership on gender and health. Survey results reflect this as well, as nearly all funded researchers believe that IGH has been successful in this regard.
2.3 Delivery
Interviewees are generally satisfied with planning processes, although some respondents across all groups are not very familiar with them. Thorough consultations with stakeholders (especially at the outset), and the continued revisiting of the strategic plan every year were considered to be key means by which effective operational planning takes place, and were praised as being collaborative and consultative. Consultations involved activities such as brainstorming sessions, information and feedback sessions, meetings with individuals and groups, conference presentations, written submissions, and discussions with various Institute Scientific Directors and other potential partners. Interviewees feel that planning mechanisms have been successful in identifying areas of strategic and emerging research importance. Most funded researchers agree.
Most staff and IAB members think the IAB operates effectively and contributes to the achievements of the Institute, and they view it as a very dedicated and hard working team that is inclusive of different viewpoints and works to build consensus among members.
IGH has carried out some consultation exercises, including brainstorming sessions and a survey of stakeholders. A Communications Master Plan was also developed in 2002 and IGH undertakes numerous communications activities including newsletters, media interviews, and its Web site, among others. Interviewees are satisfied in general with Institute communication, although there was considerable recognition of the need to better communicate with audiences such as the general public. This opinion is supported by recent media analysis that shows IGH media exposure was very limited in 2004.
While IGH staff and IAB members who were able to comment were satisfied with CIHR central's contribution to the effectiveness of the Institute, most respondents were not familiar enough with the role of CIHR corporate, and there was some frustration with respect to responsiveness on the part of central CIHR.
There is considerable agreement among interviewees that funding is not sufficient, due to a number of compelling factors such as the need for IGH to be more influential and the discouraging rate of good projects being turned down for funding. Opinions were mixed among respondents about the current balance between funding for investigator-initiated research and strategic research. There was little consensus on the issue at all.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IGH. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities. The Institute should continue its efforts at building a cohesive research community in the area of gender and health research;
- maintain planning mechanisms, as they are effective;
- maintain its approach to consultation;
- sustain its current efforts in collaboration and partnering as they have proven effective to date;
- sustain efforts in knowledge creation;
- continue to promote gender based methodologies within CIHR;
- proceed with its current knowledge translation activities;
- sustain its contributions in the area of ethics; and
- maintain current efforts with respect to contributing to the transformative vision of CIHR.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Communications - The Institute should work - with the support of CIHR corporate - to identify mechanisms to raise its profile with the public and to better communicate its successes. The Institute should also endeavour to identify and communicate with researchers whose research may fit within the mandate of the Institute but who do not currently identify with IGH.
Recommendation 3:
Knowledge Translation - The IGH should review its efforts in support of KT. While the Institute has generally done an effective job in creating awareness of KT priorities, the efforts expended by the IGH in the area of KT do not always have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IGH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 4:
Contribution to Transformative Vision - While encouraged to sustain its efforts in contributing to the transformative vision of CIHR through its inter-disciplinary work, the Institute should endeavour to better communicate the benefits of the transformative vision to researchers.
Recommendation 5:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
| Recommendation | Response | Action Plan |
|
1. Overall - The Institute has been found to be doing well and is encouraged to continue the following:
|
Agree | The IGH- IAB, the SD and IGH staff are delighted by the positive feedback on the performance of IGH on virtually every dimension reflected in this recommendation. The overarching recommendations on Knowledge Translation and Transformative Vision (#3 and #4) seem to be to that we need to profile the successful strategies of the institute to a greater extent and to a wider audience. These are certainly timely and encouraging recommendations. IGH will sustain and build on this reported success in research excellence, research capacity building, strategic planning, consultation, partnership, knowledge creation and translation, ethics, transformative vision, and gender-sex based methodologies. * See also the Directions (long-term strategic plan for 2006-2010) in the Reflections and Directions report |
| 2. Consultation and Communication - The Institute should work - with the support of CIHR corporate - to identify mechanisms to raise its profile with the public and to better communicate its successes. The Institute should also endeavour to identify and communicate with researchers whose research may fit within the mandate of the Institute but who do not currently identify with IGH. | Agree |
IGH continues to collaborate with CIHR Corporate Communications Office and discuss new options for profiling IGH with the public:
* See also the Directions (long-term strategic plan for 2006-2010) in the Reflections and Directions report |
| 3. Knowledge Translation - The IGH should review its efforts in support of KT and work to improve knowledge translation at the level of its research community. The efforts expended by the IGH in the area of KT do not always appear to have a large profile among IGH IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IGH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree Disagree ____ If disagree, please indicate why. Please refer to 1. g |
IGH has created successful strategies to advance KT through its RFAs and in conjunction with the KT Portfolio on CIHR-wide KT initiatives. IGH has promoted the research/policy interface relevant to Gender, Sex and Health and to Reducing Health Disparities
* See also the Directions (long-term strategic plan for 2006-2010) in the Reflections and Directions report |
| 4. Contribution to Transformative Vision - While encouraged to sustain its efforts in contributing to the transformative vision of CIHR through its inter-disciplinary work, the Institute should endeavour to better communicate the benefits of the transformative vision to researchers. | Agree | IGH will continue to be a leader in building interdisciplinary research teams and in demonstrating the transformative and timely vision of CIHR IGH will profile its contributions to the transformative vision through flyers such as "CIHR Institute of Gender and Health Reflections 2001-2005" and "Reducing Health Disparities Initiative 2001-2005 led by CIHR's IGH" IGH will work with the communications portfolio to ensure that its transformative activities and accomplishments arte profiled widely within and external to CIHR * See also the Directions (long-term strategic plan for 2006-2010) in the Reflections and Directions report |
| 5. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree | IGH will continue to monitor its progress against its research priorities, objectives and performance targets as outlined in its annual strategic and operational plans IGH will continue to monitor its contribution to advancing the Blueprint priority directions of CIHR expected outcomes against performance indicators IGH fully supports the need for increased and enhanced in-house evaluation capacity at CIHR. This will be extremely helpful in the capture and analysis of appropriate and timely data on the scope of gender-relevant research in open competitions and strategic initiatives. * See also the Directions (long-term strategic plan for 2006-2010) in the Reflections and Directions report |
1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
2 A Request for Applications (RFA) is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.