CIHR Dementia Research Strategy – Evaluation Report

May 2021

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by Ference & Company Consulting Ltd. in consultation with the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation. Also, thank you to those who supported the evaluation: Sarah Boorman, Meredith Haaf, Don Ference (Ference & Company), the Institute Teams and Scientific Co-Leads from the Institutes of Aging (IA) Dr. Jane Rylett, and Neurosciences, Mental Health and Addiction (INMHA) Dr. Samuel Weiss, and the former Institute of Aging Scientific Director, Yves Joanette, who completed his term in July 2019.

The Dementia Research Strategy (DRS) Evaluation Team

Ian Raskin, Michael Goodyer, Jean-Christian Maillet, Jonathan Gilbert, Ellie Fletcher, Daniela Corno, Carmelle Goldberg, Alison Croke, Jenny Larkin, Sabrina Jassemi, Alexandra Leguerrier.

For more information and to obtain copies, please contact: Evaluation@cihr-irsc.gc.ca.

Table of Contents

• List of Figures
• List of Acronyms
• Executive Summary
  • Program Overview
  • Evaluation Overview
  • Key Findings
  • Recommendations
• Program Profile
  • Program Overview
  • Components
• Description of Evaluation
  • Purpose, Scope and Context
  • Methodology
• Evaluation Findings
  • Relevance
  • Design and Delivery
  • Performance
• Conclusions and Recommendations
• Appendices
  • Appendix A: Report Figures
  • Appendix B: Methodology
• Endnotes

List of Figures

• Figure 1: CIHR's Annual Investment in Dementia Research, 2009-11 – 2017-18
• Figure 2: Percentage of DRS investments compared to all CIHR dementia related investments
• Figure 3: Number of Canadian publications in each priority areas of DRS
• Figure 4: Number of publications in all 3 priority areas of DRS by countries
• Figure 5: Specialized Index (SI) of Publications in all 3 priority areas of DRS by countries
• Figure 6: Average of Relative Impact Factors (ARIF) of Publications in all 3 priority areas of DRS by countries
• Figure 7: Average of Relative Citations (ARC) of Publications in all 3 priority areas of DRS by countries
• Figure 8: CCNA Governance Structure (2015)
• Figure 9: Partnership contributions
• Figure 10: Cost-Efficiency Table
• Figure 11: DRS researchers funding trajectory
• Figure 12: Collaborations Developed by CCNA Research Teams, by Theme
• Figure 13: Results of Acknowledgement Analysis of CCNA publications (as of October 2019)
• Figure 14: Integration of CCNA Research into Other Academic Research

List of Acronyms

Executive Summary

Program Overview

The Dementia Research Strategy (DRS) is led by the Institute of Aging (IA) in partnership with the Institute of Gender and Health (IGH), the Institute of Indigenous Peoples' Health (IIPH), the Institute of Circulatory and Respiratory Health (ICRH), and the Institute of Neurosciences, Mental Health and Addiction (INMHA). DRS supports research on the latest preventive, diagnostic, and treatment approaches to Alzheimer's disease and other neurodegenerative diseases causing dementia. The DRS is comprised of an international and national component that enables Canadian researchers to lead and participate in a number of new national and international initiatives. The ultimate goal of the DRS is to contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025.

Evaluation Overview

The objective of this evaluation is to provide management with timely, actionable evidence that will inform CIHR decision-making and planning regarding future activities and investments in dementia research. The DRS program evaluation assesses the relevance, design and delivery, and performance of the initiative during the period from 2009-10 to 2017-18, using a range of methods and data sources to triangulate evaluation findings.

The evaluation covers activities and investments of the international component (e.g., France-Québec-Canada, China-Canada Initiative on Alzheimer's Disease and Related Disorders, Centres of Excellence on Neurodegeneration [CoEN], Alzheimer's Disease Neuroimaging Initiative [ADNI], and Joint Programme - Neurodegenerative Disease Research [JPND]) and national component (e.g., Canadian Consortium on Neurodegeneration in Aging [CCNA]). The DRS activities under the New Directions in Dementia Research suite of programs were not assessed by the evaluation given that these activities were still novel at the time of this evaluation.

Key Findings

Relevance

The social and economic burden of dementia is increasing in Canada and around the world, creating a strong need for additional research on prevention, treatment, and quality of life improvements for those living with the disease as well as their caregivers. Stakeholders believe that the DRS meets this need to a moderate extent and generally complements other dementia research funding opportunities and supports within the Canadian and international research landscapes.

The federal government has a clear role in providing support for researchers to engage in innovative national and international collaboration and leadership initiatives that support dementia research. Federal agreements, legislation, and strategy documents – including the National Dementia Strategy released in 2019 – underscore that supporting dementia research is a key ongoing role and priority of the federal government.

CIHR's role in supporting dementia research directly aligns with its stated objective to contribute to scientific excellence in health and health care systems research as outlined in the CIHR Act (2000) and the CIHR strategic plans in place during the period under review. CIHR's role in providing support for dementia research through the DRS is directly aligned with the CIHR's objective stated in the Act to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system." CIHR's past two strategic plans: Roadmap (2009-10 – 2013-14) and Roadmap II (2014-15 – 2018-19) emphasize the need for the Agency to promote innovation, national and international collaboration, and research focused on improving quality of life for individuals living with chronic conditions such as dementia.

It is worth noting that researchers, partners, CIHR Senior and Program Management, and the Senate Standing Committee on Social Affairs, Science and Technology all voiced concerns that the current level of funding of the DRS is insufficient to meet federal priorities and policy commitments. More research, including patient-centered research, is needed to address knowledge gaps (e.g., around understanding dementia onset and the range and extent of its impacts) to help decision-makers develop appropriate research and funding strategies.

Design and Delivery

Overall, the design and implementation of the DRS have been effective in supporting progress toward DRS objectives. In particular, the national and international components of the DRS share the same research themes and overall objectives. However, the oversight and the governance structures differ between both components and there is a lack of documentation supporting the relationship between the two components. There is a need to better integrate the national and international components of the DRS.

The international component supports DRS objectives by facilitating Canada's participation in key international research initiatives and establishing linkages between the international research community and key stakeholders. The CCNA is an innovative model for supporting collaborative dementia research partnerships within Canada, with structures and processes such as cross-cutting activities that support research teams and ongoing communication that increase engagement, facilitate knowledge exchange, and build capacity in the dementia research enterprise.

The evaluation identified the following challenges and areas for improvement regarding the design and delivery of the DRS:

• Address limitations around the insufficient level of funding and lack of strategic funding inhibiting CCNA's ability to carry out its objectives;
• Enhance communication, clarity, and transparency around the CCNA to improve stakeholders' understanding of its objectives, structures, timelines, and performance measurement; and,
• CCNA to establish structures and processes to support more stakeholder engagement and improve partnership and collaboration.

Performance

The DRS has contributed to increasing research capacity in the field of dementia. This is evidenced by the fact that DRS researchers received more CIHR funding in the field of dementia research following the launch of the national and international components. DRS also supported trainees through funding and development opportunities. However, there may be room for improvement in communication and/or processes for awarding trainee funding given conflicting views on the need for more funding and reports of issues with the unsuccessful allocation of partner funds.

The DRS is making progress on a number of expected immediate outcomes. The CCNA supported researchers to form inter-institutional, interdisciplinary, Canada-wide collaborations and share data and resources, including biological tools, new techniques and protocols, technology and equipment, data for developing joint publications, and jointly supervise graduate students and postdoctoral fellows. Key informants also emphasized how the size, reach, and synergistic approach of the CCNA network and the collaboration and community involvement it supports further strengthen innovation in Canadian dementia research.

While it is too early to see the full reach and impact of overall DRS-supported research beyond academia, the DRS is making progress toward some of its expected intermediate outcomes. Evidence suggests some DRS funded research supported knowledge translation and dissemination through peer-reviewed publications, presentations, and methodologies.

The DRS, mainly through CCNA, is also improving consideration of equity, diversity, and inclusion (EDI) factors in dementia research as well as promoting the inclusion and growth of Early Career Investigators and geographical diversity. Improved consideration of sex and gender in dementia research is supported by the Institute of Gender and Health's involvement in the DRS and having a cross-cutting CCNA theme on "Women, Sex and Gender in Dementia" dedicated to the topic. Support for Indigenous-specific dementia research is incorporated into the CCNA, with a research team focusing on issues in dementia care for rural and Indigenous populations.

The DRS supported increased international recognition of Canada as a collaborative dementia research hub by supporting Canadian participation in international events and partnerships, promoting visibility of Canadian research, and providing opportunities for relationship building with international stakeholders.

Recommendations

The evaluation makes three recommendations to improve the performance of the DRS to achieve its expected results.

Recommendation 1:

CIHR should continue to invest strategically in the DRS and re-assess the nature and extent of funding to meet its expected outcomes as well as the needs of the National Dementia Strategy.

Recommendation 2:

CIHR should ensure that the appropriate objectives, resources and structures are in place to better integrate the national and international components of the DRS to work cohesively and synergistically to achieve its overall objectives.

Recommendation 3:

CIHR should examine approaches to enhance communication with, and engagement of, stakeholders, particularly DRS partners, knowledge users, people living with dementia, and their caregivers.

The management response to the evaluation recommendations can be found in the Management Action Plan section of the report.

Program Profile

Program Overview

CIHR's Dementia Research Strategy (DRS) supports research on the latest preventive, diagnostic, and treatment approaches to Alzheimer's disease and other neurodegenerative diseases causing dementia and enables Canadian researchers to lead and participate in a new wave of national and international initiatives. Ultimately, its goal is to contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025.

The DRS is under the scientific leadership of the Institute of Aging (IA) and is delivered in partnership with the Institute of Gender and Health (IGH), the Institute of Indigenous Peoples' Health (IIPH), the Institute of Circulatory and Respiratory Health (ICRH) and the Institute of Neurosciences, Mental Health and Addition (INMHA)^{Footnote 1}. The ongoing management of the DRS occurs through monthly DRS Integrated CIHR Working Group meetings and weekly meetings between the DRS IA's Scientific Director (accompanied by Institute employees), and Initiative Management and Institute Support (IMIS) staff.

The oversight of the national and international components of the DRS differ from one another. For instance, the Canadian Consortium on Neurodegeneration in Aging (CCNA), as a major initiative within CIHR and key initiative of the national component, has an oversight committee, composed of three CIHR representatives (Director General, Scientific Director leads) and six partnership representatives. Whereas, each initiative currently funded within the international component has its own oversight structure.

DRS objectives over the period of 2014-15 to 2018-19 were to:

• Create new scientific knowledge and enable its translation into improved health and wellness of people living with dementia, their families, and their caregivers; and
• Ensure collaborative Canadian participation and leadership toward world-class research that is aligned with a coordinated and global agenda in order to address the complex challenges of dementia.

In pursuit of these objectives, the DRS supports Canadian researchers to lead and participate in a new wave of national and international initiatives under the following three themes:

• Primary Prevention – Preventing the disease from occurring through the identification of the mechanisms and/or conditions responsible for the neurodegenerative processes that lead to Alzheimer's disease and other neurodegenerative diseases causing dementia.
• Secondary Prevention – Delaying/slowing the clinical progression of an already developing disease through better understanding of the mechanisms, diagnosis, and early intervention.
• Quality of Life – Improving the quality of life of those living with the disease or who support those having the disease as well as improving access to quality care and enabling the health care system to deal more efficiently with the rising number of individuals with dementia.

Components

The DRS is composed of an international component and national component that were launched at different points in time and are intended to complement each other and capitalize on the recognized excellence of Canadian research.

International Component

CIHR launched the International Collaborative Research Strategy for Alzheimer's Disease (ICRSAD) in March 2009. ICRSAD was renamed as the international component of DRS in 2015. This international component supports collaboration at the international level by creating a global network for generating and sharing knowledge on dementia prevention, therapies, care, quality of life and a cure.

The objectives of the international component include strengthening excellence, optimizing Canadian synergy, with a vision to position Canada as a world leader in research on prevention, early diagnosis, and treatment of Alzheimer's disease and other neurodegenerative diseases causing dementia.

This component facilitates Canada's participation in several key international initiatives through annual calls and alignment in research priorities. It supports Canadian researchers to participate in key international initiatives such as:

• Alzheimer's Disease Neuroimaging Initiative (ADNI) unites researchers with study data as they work to define the progression of Alzheimer's disease. The vision of ADNI is to create an open-access database from the data gathered from all participating researchers, using a harmonized protocol. ADNI is a partnership with the United States (US) that includes four Canadian Centres in Toronto, London, Montréal, and Vancouver;
• Network of Centres of Excellence on Neurodegeneration (CoEN) aims to encourage collaborative research between recognized national centres of excellence in neurodegeneration in order to accelerate progress in understanding the mechanisms of disease as well as the identification of new therapeutic approaches. CoEN is an initiative with 8 partner countries in Europe with CIHR being a founding partner; and
• European Union (EU) Joint Programme - Neurodegenerative Disease Research (JPND) is the largest global research initiative aimed at tackling the challenge of neurodegenerative diseases. JPND aims to increase coordinated investments between participating countries in research aimed at finding causes, developing cures, and identifying appropriate ways to care for those with neurodegenerative diseases. JPND is an EU initiative with 27 partner countries of which Canada was the first non-European country to join in 2012. CIHR has been a full member since 2014.

Past initiatives include:

• Canada-China Initiative on Alzheimer's Disease and Related Disorders, a joint program with the National Natural Science Foundation of China;
• France-Québec-Canada, a partnership with France;
• The Wellcome Trust-Medical Research Council partnership, a partnership between the University of Cambridge, University of Bristol, University of Hamburg and University of Toronto; and
• Global Alliance for Chronic Disease, a collection of the world's biggest public research funding agencies supporting joint programs into lifestyle-related or chronic diseases (cardiovascular diseases, diabetes, certain cancers, lung diseases and mental health).

National Component

In 2014-15, CIHR launched the CCNA, a ring-fenced initiative, to be the centerpiece of the national component of the DRS.

• The CCNA is designed to operate as an evolving consortium that is open, responsive, and adaptable to the dynamic research landscape. It intends to promote high impact, innovative, and interdisciplinary collaborations through a strategic pan-Canadian approach to position Canadian investigators as leaders in an era of converging national and international efforts. The specific objectives of the CCNA are to:
  • Strengthen and synergize innovation and collaborative Canadian research in neurodegenerative disease affecting cognition;
  • Introduce and support pan-Canadian shared research platforms needed to pool expertise and data, allowing the Canadian research community to be more innovative (e.g., cross-disease comparisons) and impactful in the science, prevention, treatment and care of Alzheimer's disease and other neurodegenerative diseases causing dementia;
  • Reinforce the international positioning, competitiveness and impact of Canadian research in neurodegenerative diseases affecting cognition; and,
  • Ultimately impact the quality of life and the quality of services for those having to live with the effects of neurodegenerative diseases affecting cognition and their caregivers.
• The additional New Directions in Dementia funding opportunities that were launched in 2015 and 2016 include:
  • Big Data and Dementia: To support the generation and enrichment of Canadian big datasets on neurodegenerative diseases and support the access, linkage, analysis and dissemination of big data on dementia in Canada and internationally.
  • Social Inclusion of Individuals with Dementia and Carers: To support research projects focused on: improving the social inclusion of those living with dementia and/or their carers to help both in "living better with dementia"; and, to identify and evaluate scalable relevant interventions having an impact on social inclusion at the individual or population levels.
  • Challenge of Dementia in Indigenous Populations: To build capacity among Indigenous and non-Indigenous scholars or students in the area of dementia research.
  • Wilfred and Joyce Posluns Chair in Women's Brain Health and Aging: To support a leading researcher based at an Ontario institution to develop and implement the Chair with the goal of impacting and enhancing research on sex and gender differences in brain health and aging.

Resources

CIHR investments in the DRS which are within the period of the evaluation represent a total of $37.7 million between 2009-10 and 2017-18^{Footnote 2}. Of this total investment, $21.0 million has been allocated to supporting the national component since the launch of the expression of interest in 2013-14 (average of $4.2 million per year) followed by launch of the CCNA in 2014-15 and $16.8 million has been allocated to supporting the international component since its launch in 2009-10 (average of $1.9 million per year).

During the same period (2009-10 to 2017-18), CIHR invested a total of $290.6 million in dementia related research, excluding the DRS, averaging $32.3 million per year. Figure 1 shows the funding allocated by CIHR to dementia related research and the DRS, by fiscal year.

Investments in the DRS represent a small proportion of CIHR's total investment in dementia related research. In fact, between 2009-10 and 2017-18 DRS accounted for only 11% of CIHR's total investments in dementia. This proportion increased to 18% with the launch of the CCNA as part of the national component in 2014-15. Figure 2 shows the percentage of investments in DRS compared to the investments in related research.

Description of Evaluation

Purpose, Scope and Context

The purpose of this evaluation is to provide CIHR senior management with valid, insightful and actionable findings regarding the following:

• Needs addressed by the DRS and alignment with CIHR and the Government of Canada priorities;
• Effectiveness of the design and delivery of the program in supporting the achievement of intended outputs and outcomes; and
• Achievement of the program's expected outputs, and immediate and intermediate outcomes.

This evaluation covers the period from 2009-10 to 2017-18 and is the first evaluation of the DRS since its inception. The following initiatives were in scope for the evaluation:

• International component:
  • France-Québec-Canada
  • China-Canada Initiative on Alzheimer's Disease and Related Disorders
  • CoEN
  • ADNI
  • JPND
• National component:
  • CCNA

The New Directions in Dementia Research suite of programs were out of scope for this evaluation due to their early stage of implementation.

The evaluation of DRS, committed to as part of CIHR's 2018-19 Evaluation Plan, was designed to help inform CIHR's decision-making and planning regarding the activities and investments for future phases of the DRS. The evaluation meets CIHR's requirements to the Treasury Board Secretariat (TBS) of Canada under the Policy on Results and the Financial Administration Act.

It is important to note that the DRS initiative continued to progress in parallel with the ongoing program evaluation. Most notably the CCNA was renewed in 2019 and although the timing of a final evaluation report did not align with the renewal, preliminary evaluation findings informed decision-making related to the funding renewal and continuous improvements to program elements.

The COVID-19 Pandemic

The ongoing COVID-19 pandemic has unprecedented impacts on individuals living with dementia, their carers, and the availability of key dementia supports and services, as well as on the DRS. While ongoing research is being conducted to explore the nature and extent of these impacts evidence shows that "people with dementia are more likely to be negatively impacted by social isolation, anxiety and confusion from worrisome news in the media and be at greater risk of behavioral changes, confusion and delirium during this period – putting them at risk of being hospitalized and hence further exposed to COVID-19"^{Footnote 3}. The CCNA has developed a collaborative task force with the Alzheimer Society of Canada (ASC) to roll out new research endeavors aimed at addressing the social, ethical and biomedical issues surrounding the impact of COVID-19 on individuals living with dementia^{Footnote 4}. More specifically, CCNA's Team 19 is conducting a number of new research projects within this task force, including an evaluation of the use of healthcare services and the analysis of deaths due to COVID-19 among people suffering from dementia and the organization of healthcare for Canada's elderly in the far North^{Footnote 5}.

Methodology

The evaluation addresses the following core issues of the DRS in accordance with the TBS Policy on Results (2016) and information needs of senior and program management:

• Relevance – continued need and alignment with government priorities;
• Design and delivery – effectiveness and efficiency of the implementation of the program; and
• Performance – achievement of expected outputs and outcomes.

It also considers equity analyses in accordance with CIHR's commitment to use Gender-Based Analysis plus (GBA+) to develop effective policies and programs.

Evaluation Questions

The evaluation addresses the following evaluation issues and questions:

Relevance:

1. To what extent is DRS meeting the need to support world class research on dementia?
   • 1.1. To what extent have DRS components been aligned with federal government and CIHR priorities?
   • 1.2. To what extent have DRS components been aligned with federal roles and responsibilities?
   • 1.3. To what extent has DRS continued to address a demonstrated need?

Design and Delivery:

2. To what extent have the design and implementation of DRS been effective and efficient?
   • 2.1. To what extent has DRS been an appropriate approach to supporting research on neurodegeneration in aging?
   • 2.2. To what extent are the DRS components being delivered in a cost-efficient manner?
   • 2.3. What improvements, if any, can be made to the design and delivery of DRS?

Performance:

3. To what extent has DRS achieved expected objectives?^{Footnote 6}
   • 3.1. To what extent has DRS made progress toward the achievement of expected immediate outcomes?
   • 3.2. To what extent has the DRS made progress toward the achievement of expected intermediate outcomes?

Evaluation Approach

Consistent with TBS guidance and best practices in evaluation, multiple lines of evidence were used to synthesize and triangulate evaluation findings, including both qualitative and quantitative data as well as primary and secondary data sources.

The evaluation employed the following lines of evidence:

• Document and data review: For the document review, CIHR's Evaluation Unit reviewed DRS, CIHR, Government of Canada, and International documents. For the data review, CIHR's Evaluation Unit analyzed financial and grant data from CIHR's Electronic Information System (EIS) and Research Reporting System (RRS). Both the document and data review provided context around the dementia research landscape and contributed to assessment of the initiative's progress and performance, including progress toward expected outcomes.
• Key informant interviews: The CIHR Evaluation Unit worked in collaboration with Prairie Research Associates (PRA) to conduct 44 key informant interviews with Program and Senior Management, DRS partners, and members of the research community. The information gathered was used to provide an in-depth understanding of views, perceptions, and opinions of the DRS and CCNA. The number and type of respondents who participated in these key informant interviews is presented in Appendix B.
• Bibliometric analysis of scientific publications: CIHR contracted Observatoire des sciences et des technologies to conduct a bibliometric analysis of scientific papers in dementia overall and the three priority areas targeted by DRS (e.g., prevention and control, quality of life, and therapeutics) for Canada and nine other leading research countries. The analysis included papers indexed in Web of Science (Clarivate) and PubMed (US National Library of Medicine).

Additional details about the methodology are provided in the Appendix B: Methodology.

Limitations of the Evaluation

It is common for evaluations to face limitations that can influence the validity, reliability, and utility of findings. Key limitations associated with the evaluation are outlined below.

• An overemphasis on the CCNA in the evaluation scope and results, given it is the largest ongoing investment in the DRS and subject to evaluation requirements as part of its Treasury Board submission.
• Funding opportunities that were part of the other national component of the DRS (New Directions in Dementia Research) were out of scope of the evaluation as it was too early in their trajectory to be able to evaluate the results of the research initiatives.
• Small sample sizes among some subgroups of key informant interviews.
• Gaps in performance measurement data for the international component of the DRS.
• Time elapsed since initial data collection activities and changes in the dementia context were beyond the scope of the evaluation (e.g., launch of the National Dementia Strategy in 2019).

These limitations were mitigated through the tri-angulation of results across data sources as well as consideration of the relative strengths and weaknesses of each data source. The mitigation strategies employed throughout the evaluation help ensure that the evaluation results can be used with confidence to inform program decision making. The limitations and mitigation strategies are discussed in more detail in Appendix B.

Evaluation Findings

Relevance

Key Findings:

The DRS addresses a demonstrated need to support dementia research and builds on Canada's international competitiveness in dementia research.

Dementia poses a significant and growing social and economic burden both in Canada as well as around the world. The World Health Organization (WHO) projects that the global number of individuals living with dementia will increase threefold, reaching 145 million by 2050^{Footnote 7}. Meanwhile, the burden on the Canadian economy is expected to increase from $33 billion annually in 2015 to $293 billion annually by 2040 when both direct medical and indirect costs are combined^{Footnote 8Footnote 9}.

As of 2015-16, there were already more than 419,000 Canadians aged 65 or older living with diagnosed dementia, approximately two-thirds (65%) of whom were women^{Footnote 10Footnote 11}. Evidence also indicates that Indigenous populations (e.g., First Nations, Inuit, and Métis) are at a higher risk of dementia compared to the rest of the Canadian population due to the increased prevalence of risk factors associated with its development, many of which stem directly from the impacts of colonization (e.g., physical inactivity, low educational attainment, and chronic conditions such as diabetes, obesity, and hypertension)^{Footnote 12}.

In light of the current and projected burden of dementia on individuals and society, the majority of key informants agreed that the DRS meets the need for dementia research in Canada to at least a moderate extent. To address this need, the DRS is able to leverage Canada's international competitiveness across its priority areas, which include: Prevention and Control of Dementia; Therapeutics of Dementia and Quality of life. To this end a bibliometric analysis^{Footnote 13} was conducted to help assess Canada's competitiveness in dementia research.

Overall, the findings of the bibliometrics analysis indicated that the annual number of publications by Canadian researchers has increased in each priority area corresponding to the inception of DRS in 2009. Specifically, the number of publications in the priority areas of Therapeutics of Dementia and Quality of life increased steadily after 2010 from 15 to 35 and from 3 to 15 respectively. The number of publications in the priority area of Prevention and Control of Dementia increased from 25 to 51 between 2010 and 2016 (Figure 3). Over this same time period (2000-2016), Canada is ranked 4th among the top 10 most productive nations in the number of publications^{Footnote 14} (Figure 4). The majority of countries in the top 10 most productive nations are partners with Canada on initiatives that are part of the DRS. Additionally, the bibliometric analysis indicates that Canada is more specialized across the three priority areas as whole, when compared to the world average as measured by the Specialization Index (SI)^{Footnote 15}. In fact, Canada ranks 3rd among the top 10 most productive countries for the SI of publications (Figure 5).

Finally, as an assessment of research quality, the findings from the bibliometric analysis indicate that Canada ranked 3rd for the Average Relative Impact Factor^{Footnote 16} (ARIF) which indicates that Canadian researchers are publishing in journals with a high Impact Factor (Figure 6). This finding aligns with data indicating that the number of citations received by Canadian publications is also well above the world average, with Canada ranking 5th for the Average Relative Citations^{Footnote 17} (ARC) (Figure 7).

The DRS is aligned with federal government and CIHR priorities as well as federal roles and responsibilities.

All key informants agreed that the federal government has a necessary role in supporting dementia research, with interviewees citing federal government agreements, legislation, and strategy documents (outlined below) that demonstrate alignment of the DRS with federal roles and responsibilities and showcase the federal government's ongoing priority of supporting dementia research. Specifically, key areas of alignment included the focus on increasing investment in dementia research, supporting innovative national and international research coordination and collaboration, and promoting Canadian leadership in the field, as highlighted in the following:

• G8 Dementia Summit (2013) – At the Summit, G8 health ministers agreed to set an ambition to identify a cure or disease-modifying therapy for dementia by 2025 and to increase spending on dementia research. They also agreed to enhance coordination through efforts such as developing an international action plan for research^{Footnote 18}. Legacy events were organized following this Summit.
• Second Global Dementia Legacy Event (2014) – This Canada-France joint event aimed to develop an action framework to address current challenges and barriers for enhanced collaboration between researchers and industry^{Footnote 19}.
• Economic Action Plan (2014) – Budget 2014 implemented a 2013 Speech from the Throne commitment to renew investments to tackle the growing onset of dementia by allocating new funding to create the CCNA^{Footnote 20}. This provided additional support for dementia research in line with the joint action agreed at the G8 Dementia Summit.
• The National Dementia Research and Prevention Plan (2014) – This plan reiterated the federal government's commitment to international leadership and partnership in dementia research through the international component of the DRS^{Footnote 21}.
• Bill C-233 (2017) – Bill C-233 (An Act respecting a national strategy for Alzheimer's disease and other dementias) was passed in 2017, encouraging greater investment and increased international coordination in dementia research^{Footnote 22}.
• World Health Organization Global action plan on the public health response to dementia 2017-2025 (2017) – Action areas include dementia diagnosis, treatment, care, and support – including for caregivers – as well as research innovation, all of which align with DRS objectives^{Footnote 23}.
• The National Dementia Strategy (2019) – The three themes of the DRS are linked to the objectives of the National Dementia Strategy: prevent dementia; advance therapies and find a cure; and improve the quality of life of people living with dementia and their caregivers^{Footnote 24}. Budget 2019 allocated $50 million to Public Health Agency of Canada (PHAC) over five years to support implementation of this strategy but not the research component associated with the Strategy^{Footnote 25}.

Further, CIHR's role in providing support for dementia research through the DRS is directly aligned with the CIHR Act (2000), specifically the objective to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system"^{Footnote 26}. In particular, the DRS addresses this objective by providing support for researchers to engage in innovative national and international collaborations and leadership initiatives that support dementia research on primary and secondary prevention, improve the quality of life and the efficiency of the health care system.

CIHR strategic plans for Health Research Roadmap: Creating innovative research for better health and health care (2009-10 to 2013-14) and Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians (2014-15 to 2018-19) outline the need for the Agency to promote innovation, national and international collaboration, and research focused on improving quality of life for individuals living with chronic conditions such as dementia^{Footnote 27, Footnote 28}.

DRS funding is insufficient to meet all of its objectives as well as the CIHR and federal government priority commitments.

There is a clear continued need for dementia research in Canada, as evidenced by the increasing number of people affected and the projected social and economic burden of the disease on Canadians. Despite the clear alignment of the DRS with federal priorities and evidence that the DRS is addressing dementia research needs to some extent, there are concerns that the current funding level for the initiative is insufficient to meet all of its stated commitments and achieve progress on expected outcomes as intended.

The majority of CIHR Senior and Program Management key informants reported that DRS funding was "not sufficient to meet CIHR and federal government priority commitments". Key informants also mentioned that more funding is required to achieve progress toward finding a cure or disease-modifying treatment to improve the quality of life for individuals living with dementia and their caregivers by 2025.

To complement key informant statements regarding the insufficient level of funding, since the launch of the national component in 2014-15, CIHR investments in DRS represent an average of $7.3 million annually which is part of the CIHR investments in dementia research averaging $40 million per year. Based on the 2013 G8 Dementia Summit Declaration and the recommendation from the Senate Standing Committee on Social Affairs, Science and Technology (SOCI), research investments in dementia research should represent 1% of current dementia care costs, which for Canada represents approximately $100 million toward dementia research each year^{Footnote 29}.

There is also a need to actively involve people with lived experience of dementia and their caregivers in setting research priorities and identifying knowledge gaps. For instance, several key informants identified knowledge gaps around the understanding of dementia onset and impacts (e.g., dementia types, prevalence, and symptoms), underscoring the challenge of determining the extent of the burden and efficiently targeting resources for research and care in response. Some key informants also noted a lack of involvement of people with lived experience of dementia in the prioritization of research topics. They suggested that this would be addressed through adoption of a more patient-centred approach with better linkages between researchers and individuals with lived experiences. Key informants reported that these unmet needs were limiting factors in developing appropriate research and funding strategies.

DRS generally complements other funding opportunities and initiatives that support dementia research.

Approximately half of key informants indicated that the DRS is complementary to support provided by not-for-profit and private sector organizations, other provincial and federal initiatives, and international funding opportunities. For instance, the DRS complements other initiatives that in some way support dementia research, by establishing and strengthening partnerships that enable coordination and synergism across the Canadian research landscape. For example, the CCNA and CLSA partnered on a CIHR Team Grant exploring the impact of big data on dementia. The project entitled Broad and Deep Analysis in Neurodegeneration (BRAIN) will track the longitudinal trajectory of dementia across several datasets to develop algorithms for delineating cognitive decline in the CLSA. Furthermore, the majority of DRS researchers interviewed reported that they had also applied for additional funding from both CIHR and the Alzheimer's Society of Canada.

Some key informants also noted overlaps between the DRS and other funding opportunities while specifying that it may be justifiable given the need for added funding and attention in certain research areas.

Design and Delivery

Key Findings:

The two components of DRS support the overall objectives, but better integration of the components would enhance cohesiveness and synergy.

The DRS was launched in 2014 when CCNA, the first initiative within the national component, was integrated with CIHR's foundational investment in ICRSAD. The strategy and its components share a common overall objective namely, "to support world-class research on dementia that will contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025" and same three themes (Primary Prevention, Secondary Prevention and Quality of Life).

Despite having common objectives and themes, a few key informants indicated that they were unaware of, or unclear on the linkages between the two components. These perceptions are further supported by program information which shows that each component has distinct governance and a different approach to structuring funding opportunities.

For instance, the international component is a suite of independent funding opportunities (e.g. CoEN, JPND, ADNI) which are overseen by their own governance structures and focus on international collaborations. Specifically, CIHR, via IA, plays an active role as a full voting member in the governance bodies of JPND and CoEN. CIHR was represented by Canadian ADNI site researchers on the ADNI Private Partner Scientific Board. Conversely, CCNA is comprised of a unified funding structure which allocates funds to teams, themes and programs. CCNA also has a well-defined governance structure which consists of three core components: Research Executive Committee, Strategic Advisory Council, and International Scientific Advisory Board (Figure 8). External oversight of the CCNA is provided by a CIHR CCNA Oversight Committee, which is comprised of three CIHR representatives (Director General and Scientific Director leads) and six partnership representatives involved in CCNA. The CCNA Oversight Committee raised concerns regarding the under-utilization of the International Scientific Advisory Board and the Strategic Advisory Committee. In the case of the Strategic Advisory Committee, it is reported that they have only met together once, although members are consulted on an ad hoc basis.

An overarching governance and oversight structure between the national and international components of the strategy would allow for a more integrated approach between the two components. This would better support the sharing of skills and expertise between national and international researchers; thereby, enhancing capacity building.

Overall, the evaluation found that there was a lack of documentation on the international component of the DRS, as well as on the relationship between the national and international components. In fact, documentation, key informants, financial and grant data for this evaluation were predominately CCNA oriented (given it is the centerpiece of the national component), despite attempts to seek additional information to better understand all the initiatives of DRS. There is a need to better integrate the two components and document how the two components work together to achieve the overall objectives of the DRS.

The international component facilitates Canada's participation in key international research partnerships.

The international component of the DRS facilitates dementia research partnerships between Canadian and international researchers and organizations, through a series a funding opportunities across several different initiatives (see Table 1).

Several key informants stressed the importance of the continuity of the international component of the DRS, highlighting its fundamental role in facilitating partnerships between national and international stakeholders in the field of dementia research. In addition to facilitating partnerships between countries, CoEN demonstrates a focus on establishing linkages between the international research community and industry partners to further accelerate research toward novel therapies.

Table 1: Initiatives Supported by the International Component of the DRS

CCNA's model is supporting innovative, collaborative research and research capacity, although there are challenges related to its complexity.

At the time of its inception, the CCNA was a novel and complex structure to facilitate and support successful dementia research partnerships across Canada, as acknowledged by its Oversight Committee which characterized the model as a "very new way of doing research"^{Footnote 30}. In its first phase (2014-2019), the CCNA consisted of five core elements, including four cross-cutting Programs, three research Themes, twenty research Teams, and eight Platforms to support teams, as well as an underlying central governance structure. Through these different elements, CCNA has embedded collaboration within its design. For instance, each of the 20 topic-based research team was required to have researchers from more than one university, centre, or province, with an average of 20 members per research team. The Oversight Committee also noted that the model purposefully included "co-leadership and built-in redundancies to the Teams and Themes to ensure corporate knowledge and sustainability"^{Footnote 31} – elements that support continued functioning (e.g., collaboration) as an initiative evolves.

Key informants and the CCNA Oversight Committee indicated that the structure of the CCNA supported collaborative research as intended. Many key informants expressed that CCNA fostered collaboration through structures and processes such as the cross-cutting activities and ongoing communication required between inter-institutional, interdisciplinary research teams, which served to increase engagement, facilitate knowledge exchange, and build capacity. Program data also reveal that two of the cross-cutting activities were found to be particularly supportive for all of CCNA's research teams. These included the Knowledge Translation and Exchange Program (KTEP) and Women, Gender, Sex, and Dementia (WGSD) cross-cutting components, the latter of which effectively helped to emphasize the importance of sex and gender in dementia research across all CCNA research. This approach represents a unique Canadian contribution to the overall dementia research landscape. The Oversight Committee also reported being pleased with overall progress made by the CCNA throughout its first phase and recommended continuation of funding as per its Terms of Reference^{Footnote 32}. Committee members found the CCNA model to be appropriate and effective in supporting the achieving the objectives of the national component of the DRS (e.g., increasing interdisciplinary collaborations, building capacity in dementia research and enhancing the national and international presence of Canadian researchers).

Despite these findings, program documents and key informants identified some challenges with the structure's model. The CCNA Oversight Committee raised various concerns regarding the decision making criteria and the need for clarity around the process to assess CCNA's leadership. Several key informants cited the need for greater clarity around the roles and responsibilities of the CCNA's leadership, as well as more transparency around CCNA's functions (e.g., allocation of funds, structure of teams) as key areas for improvement.

Other key informants questioned the appropriateness of CCNA's staffing structure and centralized leadership model. A couple of key informants stated that the CCNA model was too complex with its three themes, 20 different programs and eight platforms that are attempting to tackle too many elements; this has ultimately led to insufficient resources to meet all stated objectives. In its fifth Progress Report, CCNA stated that it "has never been a 'consortium' in anything but name. It is, in fact, a funded network."^{Footnote 33} The proposed structure was and remains complex with: 20 topic based Teams; six Platforms; four Cross-Cutting Programs; three Themes and one Central Administration^{Footnote 34}. In response, the CCNA Oversight Committee acknowledged the complexity of the CCNA and the inherent challenges in managing this large and multifaceted entity^{Footnote 35}. In response to concerns regarding its structure, the CCNA staffing structure was adjusted in 2017 to meet the needs of the consortium with a new research administrator, administrative coordinator, clinical research assistant, and communication officer. The CCNA also decentralized the leadership of the CCNA by creating Associate Director positions^{Footnote 36} to be filled by researchers in order to ensure transparency in decision-making processes in the lead up to the next phase of funding^{Footnote 37}. A couple of CIHR Senior Management key informants emphasized how CCNA demonstrated adaptability in responding to identified challenges. However, there remained uncertainty about how Associate Directors would be included in the decision-making going forward.

There were mixed views about the extent to which the DRS supports research collaboration.

The majority of partner and researcher key informants agreed that the DRS brought together stakeholders from different disciplines to foster interdisciplinary collaborations – particularly through the CCNA. Approximately half of interviewees felt that research partnerships had improved through an increase in funding and involvement in CCNA. Several key informants identified improvements around ease of interaction and commitment between partners, the frequency and quality of information provided to knowledge users and partners, and organizational support, particularly as a result of DRS meetings and events such as the Partners Forum and annual CCNA meetings. Together, these findings indicate that the DRS supports the development of research collaborations. Furthermore, both components of the DRS were successful in leveraging contributions from CIHR partners. As shown in Figure 9, the CCNA had a total of $8.1 million in contributions from CIHR partners such as the ASC and a provincial research foundation. Whereas, the international component has leveraged a total of $23 million from international research organizations via 17 funding opportunities.

There were mixed views regarding the extent to which the DRS has developed a sustainable model for the formation of collaboration and partnerships as a longer-term indicator of success. For instance, some key informants conveyed that it was too early to tell if the DRS has supported the formation of sustainable partnerships and whether the partnerships have had an impact on the health of Canadians.

The CIHR Oversight Committee identified challenges around CCNA's limited success in generating successful and sustainable partnerships with the wider stakeholder community, particularly industry partners^{Footnote 38}. Among their concerns were CCNA's inability to generate new partnerships and mitigate the withdrawal of existing partners (e.g., Sanofi, a French multinational pharmaceutical company)^{Footnote 39} due to low allocation of resources and underutilization of advisory bodies to support partnership development^{Footnote 40}. In line with these concerns, program data from a 2017 Partners' Forum demonstrated a significant decrease in partner participation (down from 31 partners attending in 2015 to 19 in 2017 or nearly 40%), with partner feedback highlighting a lack of engagement during both planning and the event itself as well as a limited understanding of the role of partners in the CCNA and its progress overall.

DRS was delivered in a cost-efficient manner since 2012-13.

The percentage of program administrative costs to total program expenditures speaks to how efficiently the DRS program is being run. In the case of DRS, due to some limitations in data availability, particularly in the early years of the initiative, the cost efficiency analysis only compares CIHR direct administrative costs^{Footnote 41} of DRS against total program investments for the fiscal years 2009-10 through 2017-18^{Footnote 42}. For the period of the evaluation, the ratio of DRS total direct administrative costs to the DRS program expenditures was 6.3%, which is higher than the ratio of CIHR total operating costs to the CIHR total program expenditures of 5.1%, for the same period under review.

The percentage for DRS is characterized by a ramp up period in the first years whereby administrative costs were high and grants and awards expenditures were low. In fact, the percentage of total CIHR direct administrative costs to total program expenditures peaked at 83.7% in 2009-10. This was largely attributable to the positions of an executive director and assistant director for the international component at CIHR from 2009-10 to 2011-12. Both positions were full-time from 2009-10 to 2010-11, before becoming part-time in 2011-12. In 2012-13, the responsibilities associated to the position of the executive director and the assistant director for the international component were transferred to the Institute of Aging^{Footnote 43}. Between 2012-13 and 2017-18, the ratio of DRS total direct administrative costs to the DRS program expenditures remained was 3.7%, representing an average direct operating ratio of 3.7 cents for each dollar of grant funds awarded, while the ratio of CIHR total operating costs to the CIHR total program expenditures was 4.9%. Figure 10 includes the comprehensive cost-efficiency data.

Improvements can be made to the design and delivery of the DRS.

Based on key informants and documentation, some improvements can be made to the design and delivery of the DRS. As outlined below, there is an opportunity to address limitations around the insufficient level of funding and lack of strategic funding inhibiting CCNA's ability to carry out its objectives; there is a need to enhance communication, clarity, and transparency around the CCNA and finally, there is an opportunity to establish structures and processes to support more stakeholder engagement and improve partnership and collaboration.

There is an opportunity to address the level and allocation of CCNA funding to better carry out its objectives.

Findings from an internal evaluation conducted by CCNA report that Research Executive Committee members, theme leaders and team leaders expressed concerns around insufficient funding to achieve all of CCNA's objectives. These concerns stem from initial discussions prior to the funding opportunity which lead to a perception that CCNA was to receive double the amount of funding^{Footnote 44} to what was made available in the funding opportunity. These unrealistic expectations resulted in disappointment when this of level of funding did not materialize. It is important to note that the peer review of CCNA's application and budget was deemed feasible with respect to the proposed objectives and total funding amount available.

Although the CCNA's level of funding was deemed appropriate, in its 2018 report, the CCNA Oversight Committee did identified a projected shortfall of almost $150,000 at the end of Phase 1. This raised concerns by CCNA Leadership, funders and the CCNA Oversight Committee regarding the sustainability of CCNA, highlighting the need to consider "alternative avenues through which funding can be sought."^{Footnote 45} Many key informants cited lack of funding and non-strategic allocation of funding as reasons for gaps in CCNA's design and implementation. For instance, some felt that funding was not strategically allocated across CCNA themes, which resulted in a mismatch between resources and expertise. Program data support the fact that limited funding allocated to CCNA inhibited the advancement of research. As such, "insufficient funding" was cited as the primary roadblock to collaboration in order to fully achieve teams' original goals by CCNA researchers^{Footnote 46}. CCNA also "noted that as synergy and interactions occur and new ideas are pursued, it is becoming obvious that [CCNA is] hampered by a level of funding which is clearly not commensurate with [its] capabilities. Simply put, the urgent need to find the causes of the neurodegenerative disorders requires a higher level of funding and that would be translated into more rapid progress in each team. Searching for additional funding sources will become a major focus of the CCNA Administration in the coming years."^{Footnote 47}

There is a need to enhance communication, clarity, and transparency around the CCNA.

Several key informants identified a need for improved communication and enhanced transparency with partners regarding the CCNA, particularly around its objectives, structures, timelines, and performance measurement. Some noted that higher quality, more frequent communication processes – for instance, regular conferences, meetings, face-to-face discussions, and quarterly status reports – could increase transparency and improve partners' experiences with the DRS. The need for added clarity in these areas was also reported by CCNA's Oversight Committee, which requested clarification on aspects of the CCNA model, including governance and decision making criteria (discussed above) as well as the reporting requirement of the "internal synergies within CCNA to be able to demonstrate the value add of the consortium model that could not take place otherwise."^{Footnote 48}

There is an opportunity for CCNA to improve stakeholder engagement, partnership and collaboration.

In addition to enhancing communication with stakeholders, some key informants also identified a need to engage knowledge users and partners earlier and more often to facilitate interaction between stakeholders and improve partnerships and collaboration. This area for improvement was echoed by the CCNA's Oversight Committee, which felt that advisory bodies had "ample, untapped support and guidance" to offer the CCNA and encouraged the CCNA to further engage and leverage its advisory bodies and partners through more frequent meetings and interactions^{Footnote 49}. One related suggestion from the Oversight Committee was to establish a role at CCNA to devote more attention to partnership development and alleviate burden on researchers^{Footnote 50}.

Performance

Key Findings:

DRS has contributed to increasing research capacity in the field of dementia by funding researchers and supporting trainees.

The analysis of the funding trajectory of DRS researchers demonstrates that both the national and the international components were successful in increasing research capacity in the field of dementia. The researchers involved in CCNA received a total of $48 million in funding from CIHR in the field of dementia research in the five years before the launch of the CCNA (2009-10 to 2013-14). Since the launch of the CCNA, those researchers received a total of $71 million in CIHR dementia funding (an increase of 48%), of which $29 million is directly attributable to the DRS. Funding for the international component revealed a similar trajectory, with international component researchers receiving $17 million in CIHR dementia research in the five years prior to the DRS. Following the launch of the DRS, the international component researchers received $38 million CIHR in dementia funding (an increase of 123%), of which $17 million is directly attributable to the DRS (see Figure 11).

The DRS focused on building capacity in the research enterprise by providing funding and development opportunities for trainees and early career investigators. For instance, program data show that there are 214 CCNA-affiliated trainees, more than 90 of which are either partially or fully funded by the CCNA. The breakdown of funded trainees is relatively equal across training levels (Masters, Doctoral and Postdoctoral) thereby highlighting the diverse opportunities for development. In addition, program data indicates that 17 of 40 CCNA (43%)^{Footnote 51} Team, Theme and Program lead and co-lead positions are filled by early career investigators which demonstrates that CCNA provides developmental opportunities for future leaders in Canadian dementia research. Consistent with program data, many key informants believe that the DRS has improved research capacity, they pointed to the exposure new researchers can gain through the opportunity, including access to new knowledge and ways of doing research through shared platforms, as well as developing experience and a reputation in the field by publishing research. Also, CCNA in partnership with the ASC has co-funded the CCNA Training and Capacity Building Program. However, there were some issues leading to an unsuccessful allocation of $1 million dollars in funding from the ASC, which contributed to the CCNA not being able to fully support trainees and which required the CCNA to rely on research groups of investigators to support trainees.

The CCNA has supported the formation of inter-institutional, interdisciplinary, and Canada-wide collaborations.

The constitution of CCNA research teams demonstrates the national component's ability to support the formation of inter-institutional, interdisciplinary, and Canada-wide collaborations. Among the 20 teams, program data show that there was an average of four provinces and nine institutions represented by team members. Further, all but one team included members from at least two Canadian regions, and nearly one third of teams had pan-Canadian representation, reflecting the national scope and extent of collaboration of research supported through the CCNA.

All CCNA research teams reported undertaking at least four collaborative activities, demonstrating that collaboration is a common element underpinning all CCNA-supported research. Many key informants also emphasized the size, reach, and synergistic approach of the CCNA network and its support for collaboration and community involvement as key contributors to the ability of the CCNA and DRS to strengthen innovation in Canadian dementia research.

Program data also highlights considerable collaboration between CCNA's research teams. Approximately 60% of teams reported exploring collaborations with other teams either within and/or across CCNA's research themes of Prevention (Theme 1), Treatment (Theme 2), and Quality of Life (Theme 3). Teams in the Treatment theme tended to explore collaborations with teams in other themes, while teams in the Quality of Life theme tended to explore within-theme collaborations.

Sharing data and resources was the most common type of cultivated collaboration between CCNA's teams across the three themes, as shown in Figure 12. This included sharing biological tools (e.g., specialized cell lines), new techniques or protocols, technology or equipment, data for the development of joint publications, and jointly supervised graduate students and postdoctoral fellows. The collaboration between CCNA's teams in the Treatment theme also reflected a focus on harmonizing outcome measures, such as through joint development of updated protocols or the validation/adoption of a new technique by multiple teams in order to allow comparisons between various experiments.

The DRS has increased capacity for knowledge translation and dissemination through peer-reviewed publications, presentations, and new methodologies.

Many key informants reported that the DRS had a positive impact on knowledge translation. The most commonly cited examples included the number of scientific journal publications and new tools (e.g., common data sharing platforms) generated through the DRS. More than two-thirds of researchers who were interviewed identified publications and presentations as a main academic achievement resulting from their project, while more than one-fifth identified new methodologies as a main achievement.

The interview findings are supported by research outputs documented on CCNA's website. As of October 2019, the CCNA research teams reported producing 123 scientific publications. Of these publications, 89% (109 publications) acknowledged CIHR and/or CCNA. Of these 109 publications, 61% (66 publications) acknowledged both CIHR and CCNA, 22% (24 publications) acknowledged only CCNA, and 17% (19 publications) acknowledged only CIHR (see Figure 13). Given that the majority (89%) of CCNA publications acknowledge CIHR and/or CCNA, the findings of the acknowledgement analysis appear to validate the accuracy of the publications listed on CCNA's website.

In addition to the acknowledgement analysis, a citation analysis of CCNA's publications was conducted using Clarivate's Web of Science Database. Of CCNA's 123 publications, 112 were available on Web of Science (91% of the total). The bibliometric analysis also revealed that a significant portion of this research has since been cited by other researchers. For instance, of the 112 scientific research publications resulting from the CCNA that were available on Web of Science, 66% were cited at least six times and five publications met the criteria for "highly cited papers"^{Footnote 52} on Web of Science.

Figure 14 shows the full distribution of the number of citations of CCNA-funded research which illustrates that other researchers have considered the implications of, and/or built upon, the CCNA research in their own research.

All 20 CCNA research teams contributed to knowledge translation and dissemination via publications, presentations and knowledge translation activities. CCNA-supported research has been disseminated through knowledge products, such as: public commentary (including publicity, pamphlets, guides, lay language summaries of work, invited talks - other than at scientific conferences, or interviews), interaction with media (e.g., interviews, podcasts), and collaboration with CCNA's partner organizations. In fact, over 600 knowledge translation products (e.g., presentations, academic and non-academic publications, and knowledge translation activities) have been developed on CCNA-supported research.

Both program documents and key informants indicated that it is too early to assess the impact of DRS supported peer-reviewed publications beyond academia. However, a previous analysis of CIHR-supported dementia research revealed that many peer-reviewed publications demonstrated evidence of knowledge translation and impact by informing downstream policy documents and patents, as measured by CIHR's Observable Influence Beyond Academia (OIBA) approach. Based on these findings it is expected that DRS-supported publications would show similar levels of knowledge translation and impact compared to CIHR-supported dementia research publications overall. As such, it would be important that CIHR be in a position to continue tracking knowledge translation activities and impact beyond academia of DRS-funded research.

Key informants highlighted a number of key impacts of DRS funded research including increased participation of knowledge users (particularly among clinicians) in research activities as well as the moderate reach of research findings into the public sphere through media coverage and incorporation in the development of policy and practice (e.g., treatment guidelines).

The CCNA has improved consideration of sex and gender research questions and supported Indigenous-specific research.

As per program data and key informants, the CCNA, notably through its Women, Sex, Gender, and Dementia program improved consideration of sex and gender research questions as well as other EDI factors, although there was some variation in the nature and extent of consideration each received. The majority of key informants believed that the level of priority communicated by the Institute of Gender and Health's involvement in the DRS, as well as the CCNA's theme dedicated to the topic, improved consideration of sex and gender in dementia research. In fact, they considered this to be one of the very important successes of the CCNA. Program data also lend further support to this assertion, showing that 90% of research teams reported factoring issues of sex and gender into their experiments and/or analyses.

Despite these indications of improvement, many CCNA teams – particularly those performing basic science research – appeared to consider sex and gender factors in follow-up analyses rather than during the development of their research questions. Further, some key informants exhibited skepticism that the DRS itself has contributed to improved consideration of sex and gender research questions, believing that this improvement would still have been possible through separately funded research projects not contained within the umbrellas of the CCNA or DRS.

Support for Indigenous-specific research is another way in which the CCNA considers EDI factors in dementia research. Such consideration is evident in the national component, under which one of CCNA's research teams has a specific focus on issues in dementia care for rural and Indigenous populations and four out of seven teams within the Quality of Life research theme reported conducting dementia research related to Indigenous health. This included one research team that formed direct partnerships with Indigenous organizations (e.g., Chiefs of Ontario).

The DRS has contributed to increased international recognition of Canada as a collaborative dementia research hub.

The success of efforts to showcase and increase the international profile of collaborative Canadian dementia research is exemplified by Canadian participation in global collaborative research initiatives and the formation of new international research collaborations.

More than half of key informants indicated that Canada was recognized as a hub for international research collaboration in neurodegeneration in aging, characterizing this status as being at least in part due to support provided through the DRS. Key informants reported that the Scientific Directors of CIHR's Institutes have internationally recognized knowledge and expertise and are well embedded in the research community, contributing to the recognition of Canada as a collaborative dementia research hub. As per program data, the IA, as the lead Institute of the DRS, took several key actions which have contributed to increasing the international recognition of Canada as a collaborative dementia research hub. The IA Scientific Director for period from 2011 to 2019 sat on many international advisory and management boards (e.g., for the EU JPND Research, Global Council on Brain Health), contributed to key international initiatives such as the WHO's Global Dementia Observatory Reference Guide and Canada's contribution to the Global Action Plan on Dementia. This type of participation and leadership in global dementia research enhances Canada's ability to align with global collaborative research plans and priorities and be recognized as a global research leader. Other DRS initiatives, such as hosting a "Canadian Pavilion" at international dementia research events, further support Canada's acknowledgement as a collaborative dementia research hub by promoting visibility of Canadian research and researchers and providing opportunities for relationship building with potential international collaborators.

Beyond raising Canada's research profile on the international stage, the DRS also facilitated the formation of new international collaborations. For instance, CCNA teams have established 42 new international collaborations since the initiative was established, with an average of approximately two new international collaborations per CCNA team. Some key informants added that these types of collaborative research projects of international scope further demonstrate Canada's appeal as an international research partner and its recognition as a hub for dementia research.

Conclusions and Recommendations

Relevance

The social and economic burden of dementia is increasing in Canada and around the world, creating a strong need for additional research on prevention, treatment, and quality of life improvements for those living with the disease as well as their caregivers. The majority of key informants agreed that the DRS meets this need to at least a moderate extent and generally complements other dementia research funding opportunities and supports within the Canadian and international research landscapes.

The evaluation also concludes that the federal government has a clear role in providing support for researchers to engage in innovative national and international collaboration and leadership initiatives that support dementia research – a demonstrated federal priority area. Specifically, there are a number of federal agreements, legislation, and strategy documents from recent years that underscore the federal government's clear and ongoing priority and role related to supporting dementia research, including agreements made at the 2013 G8 Dementia Summit, the National Dementia Strategy released in 2019, and action areas from the WHO Global action plan on the public health response to dementia 2017-2025, among others. CIHR's role in providing support also aligns directly with its stated objective to support scientific excellence in health and health care systems research as outlined in the CIHR Act (2000) and strategic plans.

However, researchers, partners, CIHR Senior and Program Management, and the Senate Standing Committee on Social Affairs, Science and Technology all voiced concerns that the current level of funding of the DRS is insufficient to meet federal priorities and policy commitments.

Design and Delivery

Both the national and international components of the DRS share common elements such as the research themes and overall objectives. However, there are also some challenges related to the integration of the components, which would allow for the strategy to be more cohesive and synergistic. More specifically, the two components of DRS are generally presented as distinct entities, with the common element being the overall theme of dementia research. The oversight structure differs between both components and there is no oversight structure that oversees the DRS as a whole. There is also a lack of documentation regarding the international component and the relationship between the two components.

The international component supports the DRS' overall objectives by facilitating Canada's participation in key international research partnerships and establishing linkages between the international research community and industry partners. As the central initiative of the national component, the CCNA is as an innovative model for supporting collaborative dementia research partnerships within Canada. CCNA structures and processes such as cross-cutting activities and ongoing communication between research teams also increase engagement, facilitate knowledge exchange, and build capacity in the dementia research enterprise.

Despite these effective design elements, the evaluation identifies several challenges and areas for improvement with the DRS:

• Insufficient level of funding and lack of strategic funding inhibiting CCNA's ability to carry out its objectives;
• Complexity and lack of clarity around the governance of the CCNA model, which made it challenging to achieve all stated objectives;
• Limited success in generating successful and sustainable partnerships with the wider stakeholder community, particularly industry partners made it difficult to assess the extent to which the DRS and more specifically CCNA supports successful longer-term partnership;
• The need to enhance communication, clarity, and transparency around the CCNA to improve stakeholders' understanding of its objectives, structures, timelines, and performance measurement;
• The need for CCNA to establish structures and processes to support more stakeholder engagement and improve partnership and collaboration; and
• The need for earlier and more frequent engagement of knowledge users (including people with lived experience and patient partners) and partners to facilitate interaction and improve partnerships and collaboration.

Performance

The evaluation findings indicate that the DRS is achieving or making progress toward the following expected outcomes:

• Research capacity is being supported through collaboration and knowledge exchange between DRS stakeholders, particularly within and between CCNA teams in the form of shared data and resources. The DRS also builds capacity in the research enterprise by supporting trainees and early career investigators with funding, development opportunities, and exposure.
• Both key informants and program data highlight the DRS' positive impact on knowledge translation. Notably, over 600 knowledge translation products (e.g., presentations, academic and non-academic publications, and knowledge translation activities) have been developed on CCNA-supported research.

Program data and key informant feedback also highlight that the DRS, specifically through CCNA's overall structure, is improving consideration of EDI factors in dementia research and conducting research aligned with national priorities, including sex and gender, indigenous populations and rural and remote communities. In fact, increased consideration of sex and gender research questions was viewed as a very important success of the CCNA, supported by the level of priority communicated by the IGH involvement in the DRS as well as the CCNA's cross-cutting theme dedicated to the topic. Support for research specific to Indigenous, rural and remote populations is also incorporated into the CCNA, with a team dedicated exclusively to supporting rural capacity building in dementia care.

While it is too early to see the full reach and impact of DRS-supported research beyond academia, DRS knowledge products are being disseminated and applied beyond academia at a higher rate than CIHR supported knowledge products as a whole. Further, ongoing Canadian participation in global initiatives, supported by the international component, showcase Canadian researchers as leaders on the world stage, facilitate the formation of new international partnerships, and enhance Canada's image as a hub for international research collaboration.

Recommendations

The evaluation makes three recommendations to improve the performance of the program to achieve its expected results.

Recommendation 1:

CIHR should continue to invest strategically in the DRS and re-assess the nature and extent of funding to meet its expected outcomes as well as the needs of the National Dementia Strategy.

Recommendation 2:

CIHR should ensure that the appropriate objectives, resources and structures are in place to better integrate the national and international components of the DRS to work cohesively and synergistically to achieve its overall objectives.

Recommendation 3:

CIHR should examine approaches to enhance communication with, and engagement of, stakeholders, particularly DRS partners, knowledge users, people living with dementia, and their caregivers.

Appendices

Appendix A: Report Figures

Figure 1: CIHR's Annual Investment in Dementia Research, 2009-11 – 2017-18

Source: CIHR EIS data.

Figure 2: Percentage of DRS Investments Compared to All CIHR Dementia Related Investments

Source: CIHR EIS data.

Figure 3: Number of Canadian Publications in Each Priority Areas of DRS

Source: Bibliometric analysis conducted by OST.

Figure 4: Number of Publication in All 3 Priority Areas of DRS by Countries

Source: Bibliometric analysis conducted by OST.

Figure 5: Specialized Index (SI) of Publications in All 3 Priority Areas of DRS by Countries

Source: Bibliometric analysis conducted by OST.

Figure 6: Average of Relative Impact Factors (ARIF) of Publications in All 3 Priority Areas of DRS by Countries

Source: Bibliometric analysis conducted by OST.

Figure 7: Average of Relative Citations (ARC) of Publications in All 3 Priority Areas of DRS by Countries

Source: Bibliometric analysis conducted by OST.

Figure 8: CCNA Governance Structure (2015)

Source: Canadian Consortium for Neurodegeneration in Aging (CCNA) Response to Oversight Committee Report 5, 2015, CCNA. Legend: Biological Sample Access Committee (BSAC); COMPASS-ND Protocol Implementation Team (PIT); Conference Program Planning Committee (CPPC); Database Development Committee (DDC); Diagnostic Adjudication Committee (DAC); Ethical, Legal and Social Issues (ELSI); Knowledge Transfer and Exchange (KTE); Membership Committee (MC); Publication and Data Access Committee (PDAC); Research Executive Committee (REC); Training and Capacity Building (TCB); Women, Gender, Sex and Dementia (WGSD).

Figure 9: Partnership Contributions

Source: CIHR EIS data.

Figure 10: Cost-Efficiency Table

Figure 11: DRS Researchers Funding Trajectory

Source: CIHR EIS data.

Figure 12: Collaborations Developed by CCNA Research Teams, by Theme

Source: Dementia Research Strategy (DRS) Document Review Report, 2018, CIHR.

Figure 13: Results of Acknowledgement Analysis of CCNA publications (as of October 2019)

Source: CCNA website; Web of Science.

Figure 14: Integration of CCNA Research into Other Academic Research

Source: Bibliometric Analysis conducted by CIHR Evaluation Unit.

Appendix B: Methodology

This section presents additional details about the methodology used to conduct the evaluation of the DRS. The table below provides the evaluation matrix, which shows the issues and questions to be addressed by the evaluation, as well as the indicators and data sources used to inform findings for each.

Table 2: Evaluation Matrix

Source: Adapted from Dementia Research Strategy (DRS) Evaluation Design Report (EDR), 2017, CIHR.

As seen in the evaluation matrix, the evaluation of the DRS employed multiple lines of evidence, including a document and data review, key informant interviews, and bibliometric analysis. The following sub-sections outline key details about data collection for each line of evidence, followed by a description of how evaluation findings were triangulated and reported.

Document and Data Review

CIHR's Evaluation Unit reviewed DRS, CIHR, Government of Canada, and international documents. For example, the document review included relevant legislation, government documents such as departmental plans, CCNA progress and oversight reports, and related strategy and evaluation reports. The main sources of data included CIHR's EIS and RRS. Information provided context around the dementia research landscape and contributed to assessment of the initiative's progress and performance, including toward expected outcomes. In addition, the review was also used to assess annual progress reporting and to inform planning – for instance, to support CIHR decision-making in the design of CCNA's Phase 2 Funding Opportunity, to create targeted key informant interview guides, and to develop intermediate outcome indicators associated with strengthening collaborative and innovative research in Canada.

Key Informant Interviews

CIHR Evaluation Unit worked in collaborations Prairie Research Associates to conduct 44 key informant interviews with Program and Senior Management, DRS partners, and members of the research community using interview guides prepared by CIHR's Evaluation Unit. Interviews were 30 to 60 minutes long, conducted over the telephone, and digitally recorded and transcribed with participants' consent. The information gathered through the interviews was used to provide an in-depth understanding of views, perceptions, and opinions of the DRS and CCNA.

The following table outlines the number and type of respondents who participated in key informant interviews. Interviewees were selected for their implication in the international or national component.

Table 3: Number of Key Informants by Respondent Group

Source: Dementia Research Strategy (DRS) Key Informant Interview Technical Report, 2019.

Bibliometric Analysis

CIHR contracted Observatoire des sciences et des technologies (OST) to conduct a bibliometric analysis of scientific papers in dementia overall as well as in three priority areas targeted by DRS (e.g., prevention and control, quality of life, and therapeutics) for Canada and nine other leading research countries, including Australia, China, France, Germany, Italy, Japan, Spain, UK, and USA. The analysis included papers indexed in World of Science (Clarivate) and PubMed (US National Library of Medicine) that were identified using relevant MeSH terms. In total, 13,818 relevant papers were identified in the three DRS priority areas and 73,744 papers were identified in dementia research overall. Information on the following was extracted for each:

• Number of publications;
• SI;
• ARIF;
• ARC; and
• International Collaboration.

Results were summarized quantitatively by country for each indicator and were used to inform findings about the relevance of dementia research to the international scientific community, and by extension, the DRS.

In addition, CIHR's RIU also conducted a supplementary analysis of the 123 publications listed on the CCNA website to estimate researcher compliance with the CIHR requirement to acknowledge CIHR as a source of support. Additionally, CIHR collected and identified all relevant DRS mandate related "Knowledge products", identified through Web of Science that were published between 2008-2010^{Footnote 53} and included author acknowledgement of CIHR, and ran a matching process with OIBA database which is maintained by CIHR's RIU. Results were used to inform findings about the DRS' performance with respect to achievement of expected intermediate outcomes.

Limitations and mitigations

The following describes the key limitations associated with the evaluation, which were mitigated through the tri-angulation of results across data sources as well as consideration of the relative strengths and weaknesses of each data source. The mitigation strategies employed throughout the evaluation help ensure that the evaluation results can be used with confidence to inform program decision making.

Table 4: Limitations and mitigation strategies

Analysis & Reporting

Following data collection, CIHR Evaluation Unit worked in collaboration with Ference & Company Consulting Ltd. to synthesize data, triangulate findings, and develop the final evaluation report.

Findings from each line of evidence were organized by questions and indicators using a structured result matrix to ensure all findings were based on sufficient evidence, identify areas for any additional data collection or analysis to address evidence gaps, and ultimately serve as a clear roadmap as to how findings were derived. In developing conclusions related to each evaluation question and issue, findings were cross-validated across lines of evidence and the relative strength of each line of evidence was considered, such that greater priority was placed on data considered to be more reliable and/or more relevant to a given evaluation question.

Following data collection, CIHR Evaluation Unit worked in collaboration with Ference & Company Consulting Ltd. to synthesize the data, triangulate findings, and develop the final evaluation report which includes the evaluation findings and the conclusions and recommendations. CIHR's Evaluation Unit presented the preliminary findings and the recommendations to the Scientific Director of IA and CIHR President.

Date modified:

2024-03-15