Mid-Term Evaluation of the Institute of Aging (IA)

Executive Summary

December 2005



Table of Contents

1. Executive Summary
    1.1 Overview of CIHR
    1.2 Overview of the Institute of Aging (IA)
          1.2.1 Mandate
          1.2.2 Strategic Orientations and Objectives
          1.2.3 Research Priorities
    1.3 Evaluation Objectives and Issues
          1.3.1 Methodology
2. Evaluation Results
    2.1 Relevance
    2.2 Effectiveness
    2.3 Delivery
3. Recommendations
4. Management Response


1. Executive Summary


1.1 Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

CIHR was created under The CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

  • outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
  • outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
  • transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

  • effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
  • organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.

CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).

Top of page

1.2 Overview of the Institute of Aging (IA)

1.2.1 Mandate

The fundamental goal of the Institute of Aging (IA) is the advancement of knowledge in the field of aging to improve the quality of life and the health of older Canadians. The Institute supports research to promote healthy and successful aging and to address the causes, prevention, screening, diagnosis, treatment, support systems, and palliation for a wide range of conditions associated with aging. Research supported by the IA pertains to topics such as:

  • Long-term health promotion policies and strategies (individual, community and population levels);
  • Identification of health advantage and health risk factors related to the interaction of environments over a lifetime (cultural, social psychological, behavioural, physical, genetic);
  • Demography and epidemiology: identification of determinants of health status of a population later in life and potential impacts on health services for forward planning;
  • Factors which enable the elderly to remain independent: e.g., community characteristics, housing, transportation, and volunteers;
  • Biological mechanisms underlying the aging process itself and conditions associated with aging;
  • Geriatrics research: best practice research on care of the elderly (e.g., service delivery, co-morbidity, medication use and misuse, iatrogenic problems, rehabilitation, palliation); and
  • Aging-related disorders and disability (e.g., neurodegenerative disorders, cognitive decline, reduced mobility).

1.2.2 Strategic Orientations and Objectives

The IA has five strategic orientations (each with its own specific objectives):

  • Leadership in setting strategic research directions in the field of aging in Canada;
  • Development and support of capacity-building research initiatives and programs in the field of aging;
  • Development and support of strategic research initiatives, programs and projects on aging;
  • Promotion of the importance of research on aging and of the needs of the research community in aging; and
  • Facilitating the dissemination, transfer and translation of knowledge from research findings into potential applications through policies, interventions, services and products.

1.2.3 Research Priorities

On the basis of consultations with its IAB, researchers and various stakeholders, the IA has identified five research priorities in the field of aging:

  • Healthy and successful aging;
  • Biological mechanisms of aging;
  • Aging and maintenance of functional autonomy;
  • Cognitive impairment in aging; and
  • Health services and policy relating to older people.

Top of page

1.3 Evaluation Objectives and Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:

  • to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
  • to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.

The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:

  • Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
  • Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
  • Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
  • Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

1.3.1 Methodology

The evaluation consisted of four main lines of evidence:

  • a review of documents and administrative data relating to the IA;
  • 44 key informant interviews with IA staff and IAB members, researchers and students, stakeholders and partners;
  • case studies of three IA initiatives; and
  • a telephone survey of 72 funded and 36 non-funded researchers affiliated with the IA.1

Top of page

2. Evaluation Results


2.1 Relevance

The evaluation evidence indicates that the mandate and priorities of the IA are appropriate and relevant. A majority of key informants in all respondent groups regard the Institute's mandate, strategic orientations and research priorities as appropriate, though a minority feels the scope is too broad (e.g., in light of limited resources). In addition, in the survey four-fifths of funded IA researchers and almost two-thirds of non-funded researchers see a close fit of their research interests with the Institute's mandate.

IA is generally viewed as an appropriate mechanism to make a difference in its research domain of aging. Furthermore, researchers surveyed believe there is a great need for the IA to support research excellence, capacity development and funding of strategic priorities. No alternatives to the IA mechanism were identified, although some improvements were suggested (e.g., increased dialogue and partnerships to facilitate knowledge translation). The CIHR model of virtual Institutes is generally regarded as appropriate, with both advantages and disadvantages. For example, it is advantageous that Institutes are located across the country and that the model allows Scientific Directors to remain at their home institution, but on the other hand Institutes may feel disconnected from CIHR central and the transition of SDs and staff can be disruptive and result in a loss of corporate memory for an Institute.

Top of page

2.2 Effectiveness

Overall, IA has performed well and has made good progress in contributing to CIHR's overall objective and in fulfilling the Institute's mandate, strategic orientations and research priorities.

Through RFAs2 (Requests for Applications), the funding of research and major initiatives such as the CLSA, NET grants and CIA Partnership, IA is making progress in creating new knowledge in the field of aging. Most funded IA researchers believe that the Institute has contributed in advancing health research to respond to opportunities/priorities and create health knowledge. Most key informants also agree that progress is being made toward knowledge creation, however, a minority feels that it is premature to conclude how much new knowledge will truly be created as a direct result of IA's efforts/funding.

IA has funded or implemented a number of KT-related activities, including a presentation to the Romanow Commission, conducting Regional Seniors' Workshops on Research, supporting the Canadian Journal on Aging and building a KT component into RFAs. Most funded IA researchers surveyed believe that the Institute's activities support KT to at least some extent and researchers are highly involved in disseminating their research findings, in particular, to other researchers and health care professionals. Key informants generally agree that IA has implemented numerous activities related to KT (e.g., involving the dissemination of knowledge to policy makers and service providers). A minority of stakeholders note, however, that the true uptake of knowledge to result in improved policies or healthcare practices for older Canadians has not yet occurred. Perceived challenges for KT include a lack of direction from CIHR central and a need for more innovative approaches to result in true knowledge uptake.

IA has contributed significantly toward capacity building through a range of initiatives to attract and retain researchers in aging and through investments in NETs, STIHRs and Training Awards, which have been increasing annually. Key informants generally agree that IA has made a strong contribution to capacity development. Similarly, the majority of funded IA researchers in the survey feel that the Institute has contributed, to at least some extent, to the development of people and the research environment. Some key informants caution, however, that it is too early and there is a lack of quantitative data to assess conclusively the IA's impact on capacity in the field of research on aging.

Key informants agree that IA is making a major contribution to the transformative vision of CIHR, in particular, due to the Institute's emphasis on interdisciplinary research in aging and key initiatives such as the CLSA, RSWRs and CIA Partnership. Moreover, in the survey, funded IA researchers report that an average of 68% of their projects are interdisciplinary. Similarly, CIHR administrative data indicate that funding for research related to the IA mandate spans all four Themes, suggesting an excellent contribution to CIHR's transformative vision.

The Institute has also made a contribution to the CIHR ethics mandate, for example, by contributing financially to joint strategic initiatives with some relation to ethics issues, by addressing ethical issues around informed consent as part of the CLSA, and by having an ethics specialist and ELSI representative on the IAB. There is, however, low awareness of the ethics mandate and the Institute's contribution in this regard among many key informants.

Key informants and researchers surveyed generally believe, and documentation confirms, that IA is progressing well relative to its mandate and priorities, in particular, by supporting the advancement of knowledge in these areas. Similarly, the majority of funded IA researchers perceive that the mandate and strategic priorities have been achieved to at least some extent. CIHR investments in research related to aging have increased steadily over the past five years, further indicating that IA is influencing research under its mandate.

Key informant interviews and the document review indicate that IA has collaborated extensively and developed numerous partnerships, linkages and alliances with a broad range of stakeholders from numerous sectors, including provincial and federal government departments/agencies, health charities, non-profit NGOs and seniors organizations, and professional associations. IA has also worked effectively with other CIHR Institutes (e.g., as partners/contributors in joint RFAs and major Institute initiatives). Perceived barriers and challenges in developing partnerships include the time/effort required to form and nurture partnerships, some partners' lack of resources/capacity, and a limited appreciation of research on the part of some partners.

IA has demonstrated leadership and has had some influence on research agendas in aging. Most key informants agree that the Institute has demonstrated leadership and influence through RFAs/funding of research in its priority areas (e.g., Alzheimer's disease, dementia) and by encouraging collaborative interdisciplinary research in aging, for instance, through New Emerging Teams. Most funded IA researchers in the survey also indicated that the Institute has influenced the research agenda in aging to at least some extent, with over half saying to a large extent. In addition, leadership and a contribution to a more coordinated research agenda are illustrated by innovative initiatives such as the CLSA and Cognitive Impairment in Aging Partnership.

Top of page

2.3 Delivery

IA has used planning mechanisms effectively. The initial Strategic and Action Plan and selection of research priorities were based on extensive consultations (i.e., interviews and focus groups with a range of stakeholders from across the country) and reviewed by the IAB. Key informants generally believe that these mechanisms have been effective and that input from the community is accurately reflected in the plan and research priorities. Moreover, most researchers surveyed feel that the IA has successfully identified emerging priorities within its mandate.

The IAB has been operating effectively and contributing to the achievements of the Institute (e.g., strategic planning and key initiatives such as the Cognitive Impairment in Aging Partnership). Staff and Board members are generally satisfied with the operation, membership and effectiveness of the Board, though some minor concerns were raised about the current composition and degree of engagement of some members.

IA has consulted extensively with a wide range of stakeholders using both formal and informal methods. Key informants and researchers surveyed feel that the IA has been responsive to input and feedback and has considered this in its plans and activities. In addition, the Institute has utilized a wide range of communications approaches/vehicles, however, some key informants feel that the communications could be improved.

Factors that exert some influence on the effectiveness of IA include central CIHR functions and funding levels. Key informants expressed a range of positive and negative views on the degree of support and responsiveness of central CIHR functions; there was little consensus. Areas of concern for some interviewees include a lack of leadership on knowledge translation and ethics, operational support, and evaluation. Most key informants view the level of funding ($1M for the Institute support grant and approximately $6.5M for strategic funding) as inadequate to make a difference in all areas of IA's broad mandate, though views are variable on each type of funding. Although mixed views are expressed on the 70:30 split between funding for investigator-initiated and strategic research, many key informants regard this balance as appropriate. Still, some staff favour a higher proportion of strategic research funding whereas some researchers would like to see more emphasis on funding for investigator-initiated research.

Top of page

3. Recommendations

Following are recommendations that emerge from this mid-term evaluation of IA. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.

Recommendation 1:

The Institute has been found to be doing well and is encouraged to continue the following:

  1. support research excellence, capacity development and funding of strategic priorities in the field of aging;
  2. maintain the current structure and operation of the IAB as it is functioning effectively;
  3. sustain efforts in consulting with researchers and a diversity of stakeholders using range of formal and informal methods;
  4. maintain its efforts directed toward the creation of knowledge in the field of aging, utilizing its RFAs/funding vehicles and major initiatives such the CLSA, CIA Partnership and Mobility in Aging; and
  5. maintain its efforts in contributing to the transformative vision of CIHR through its emphasis on interdisciplinary research covering all four Themes as well as its initiatives that support this focus as well as knowledge translation (e.g., the CLSA, RSWRs and CIA Partnership).

The following areas are ones in which it is recommended the Institute take action to improve:

Recommendation 2:

Strategic Priorities - The Institute is encouraged to continue to implement their strategic research priorities as they remain appropriate but it should also consider periodic refinements to its focus on particular priorities, based on its previous investments and successes, to ensure that all priority areas receive attention.

It is recommended that IA clarify with CIHR and other Institutes which diseases common in old age it can feasibly focus on, given its level of resources.

Recommendation 3:

Communications and Consultations - Overall, the Institute is encouraged to maintain communications activities, using a variety of approaches/vehicles. If feasible, the Institute should also consider some improvements raised by key informants in this evaluation - for example, devote more resources to communications, formulate an updated communications action plan, communicate information in understandable and practical terms to reach a broad audience including seniors beyond those that participate in workshops, and clearly communicate the Institute's progress and achievements relative to its mandate and priorities to the researcher and stakeholder community.

Recommendation 4:

Collaborations and Partnerships - While encouraged overall to maintain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should also explore opportunities to broaden its partnerships to include, for example, more healthcare practitioner organizations, health charities, federal research funding organizations, provincial ministries and pharmaceutical companies.

Recommendation 5:

Ethics - The Institute is encouraged to further develop its efforts related to CIHR's ethics mandate, and to inform its researcher and stakeholder community about these efforts and any achievements.

Recommendation 6:

IAB Functioning - The current composition of the IAB should be examined to ensure that there is adequate representation of the NGO/practitioner community as well as both social and biomedical/bench scientists and that these members are fully engaged.

Recommendation 7:

Knowledge Translation - The IA should review its efforts in support of KT. The efforts expended by the IA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.

Recommendation 8:

Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.

Top of page

4. Management Response

Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:

Recommendation Response Action Plan

1. Strategic Priorities - The Institute is encouraged to continue to implement their strategic research priorities as they remain appropriate but it should also consider periodic refinements to its focus on particular priorities, based on its previous investments and successes, to ensure that all priority areas receive attention.

It is recommended that IA clarify with CIHR and other Institutes which diseases common in old age it can feasibly focus on, given its level of resources.

Agree

Current Actions

  • Graduated roll-out of new Mobility in Aging strategic priority commenced in June, 2005. Full strategy will be produced once consultations completed in Spring, 2006.
  • In May 2005, the Institute of Aging initiated a process to revise and renew its strategic plan.

Next Steps

  • Stage 1 of strategic plan renewal:
    • assessment of success in meeting objectives established in the inaugural strategic
    • analysis of administrative data and data from consultations across a range of stakeholders over the past 2 years.
  • Stage 2 will be determined by the IAB on review of the above.
  • The renewed strategic plan will outline proposed objectives of addressing each priority area along with resource commitments and time course.

Re: IA focus diseases Current Actions

  • The IAB has recently helped to establish criteria for relevancy to the Institute of Aging's strategic priorities

Next Steps

  • We propose to extend this exercise by assigning an IAB Work Group the task of preparing a statement with further criteria to expressly identify IA's distinct dominion.
2. Communications and Consultations - Overall, the Institute is encouraged to maintain communications activities, using a variety of approaches/vehicles. If feasible, the Institute should also consider some improvements raised by key informants in this evaluation - for example, devote more resources to communications, formulate an updated communications action plan, communicate information in understandable and practical terms to reach a broad audience including seniors beyond those that participate in workshops, and clearly communicate the Institute's progress and achievements relative to its mandate and priorities to the researcher and stakeholder community. Agree

Current Actions

  • With transition the IA was not sufficiently resourced to produce an Annual Report for 03-04. We are currently producing a Biennial Report and henceforth will be reporting annually.
  • IA will continue to distribute its quarterly electronic e-bulletin

Next Steps

  • These recommendations will be taken to the IAB for the purposes of strategic planning. A communications plan will be embedded within the overall strategic plan.
  • The Institute of Aging completed its transition by securing a full complement of permanent staff in November 2005. The Institute is now able to enhance activities relating to Communications, Knowledge Translation, and Partnership development, which have been of lower priority than research support over the transition period.
3. Collaborations and Partnerships - While encouraged overall to maintain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should also explore opportunities to broaden its partnerships to include, for example, more healthcare practitioner organizations, health charities, federal research funding organizations, provincial ministries and pharmaceutical companies. Agree

Current Actions

  • IA is negotiating a partnership agreement with the Canadian Geriatric Society that will promote the building of research capacity in geriatrics and disseminate Canadian knowledge on clinical care.
  • IA had meetings with Seniors Policy Division of Social Development Canada for knowledge exchange to inform policy and research.
  • Through the RSWR, IA has worked closely with provincial and local organizations with mandates associated with aging
  • IA is gathering information on new networks on aging at the provincial levels with whom partnerships can be forged
  • SD is holding workshops in Japan to enhance participation in CIHR - Japan Society for the Promotion of Science partnerships
  • NIA collaboration on supporting PDF is being finalized.

Next Steps

  • Develop partnerships with provincial organizations and ministries
  • A partnership strategy is being created for the IA's new Mobility in Aging strategic priority. This will include partners that are non-traditional to CIHR
  • Talks with NSERC re: partnership on Mobility in Aging will continue
  • Enhance participation with international interest groups of Gerontological Society of America
  • Explore European partnerships in context of EU FP7 theme
4. Ethics - The Institute is encouraged to further develop its efforts related to CIHR's ethics mandate, and to inform its researcher and stakeholder community about these efforts and any achievements. Agree

Current Actions

  • Partnership with CIHR Ethics office to support workshops relevant to Ethics and Aging research was forged in December, 2005.
  • Draft of IA position statement on Ethics in Aging research is being completed for publication on IA Web site.

Next Steps

  • IA IAB membership has and will continue to include ethics expertise; this individual serves as the IA link to the ELSI
  • Ethics issues will be topics of focus in newly-developed (newsletter) and revised (Web site) IA communications vehicles.
  • IA will ensure findings of Phase 1 development studies of CLSA related to ethics are communicated to relevant stakeholders.
5. IAB Functioning - The current composition of the IAB should be examined to ensure that there is adequate representation of the NGO/practitioner community as well as both social and biomedical/bench scientists and that these members are fully engaged Agree

Current and Future Actions

  • New IAB members are carefully selected from the list of nominees by the Scientific Director, Chair, and Past-Chair to maintain a balance among representatives of the different stakeholder, disciplines and pillars. One of IA's two NGO members has been unable to participate in most meetings, so the true participation of this sector may not be apparent.
  • CIHR recognizes the challenges associated with lay membership on Institute Advisory Boards. The Institute is working hard to raise the comfort and participation of its lay members by engaging them in committee work, reducing its use of research acronyms in communications and at meetings, explaining processes, and improving new member orientation.
6. Knowledge Translation - The IA should review its efforts in support of KT. The efforts expended by the IA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. Agree

Current Actions

  • Current KT focus has been the RSWR. On completion of the first phase of this program in the summer of 2006, resources will be available for analysis, planning, and implementation of the second phase.
  • IA is launching an annual KT award in aging, in memory of a pioneer in this field.
  • IA has recently created a KT work group within its IAB

Next Steps

  • Engage IAB KT Work Group and Corporate KT personnel to develop an IA KT strategy that will be an element of the IA Strategic Plan
  • KT will be a specific element of the strategic initiative on Mobility in Aging
  • Work with Corporate KT personnel to clarify role and Institute responsibilities in carrying out KT mandate.
7. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. Agree

Current Actions

  • IA's inaugural StrategicPlan (SP) included a series of Main Actions, which have been evaluated as achieved. The new SP will be similarly structured to facilitate evaluation. Each Action will be linked to its associated Common Performance Indicators to ensure alignment with CIHR evaluation framework.
  • Common Performance Indicators now linked to IA's annual operational planning and reporting
  • Analyses and reports from EIS are being regularized and standardized.
  • Processes for data management and sharing are being developed in tandem with IT enhancement to connect the Ottawa and Vancouver offices.

Next Steps

  • IA will ensure performance monitoring and reporting is a feature of new strategic plan.


1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
2 An RFA is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.