The International Oversight Committee for CLSA October 2010 Report to CIHR
- Contex
- Introduction
- Milestones Achieved and use of Funds
3.1. The Funding Profile
3.2. Scientific and Administrative Milestones
3.3. Operational Milestones - Governance and Management
4.1. Administrative Management
4.2.Scientific Governance
4.3. Partnerships - Monitoring Ongoing Excellence
5.1. Overall Quality/effectiveness of Platform Implementation
5.2. Data and Biological Sample Access Policies
5.3. Ethical, Legal, Social Issues (ELSI) - Policies and Procedures
- Responsiveness to the Research Community
- Appendix 1 - Membership of the CIHR International Oversight Committee for the CLSA
The following are highlights of the full evaluation report provided to the Canadian Institutes of Health Research (CIHR) by the International Oversight Committee (IOC) for the Canadian Longitudinal Study on Aging (CLSA) in October 2010. This report is the first in a series of annual evaluation reports to CIHR.
1. Contex
The IOC was established by CIHR to provide expert advice on the progress and forward planning related to the implementation of the CLSA Protocol. The existence of an ongoing oversight committee recognizes the large investment by CIHR (and the Canada Foundation for Innovation - CFI 1) in a key research platform. It provides an opportunity for external peer reviewers to provide ongoing advice to CIHR during the process of platform implementation (as per the CLSA February 2010 Protocol) based on a deeper cumulative knowledge of the project than would be the case with periodic ad hoc external reviews. The membership of the IOC is reproduced in Appendix 1; all members maintain an arms-length distance from the CLSA.
The assessment is based on a review of the CLSA Annual Report on Progress and Future Plans of May 2010, a September 2010 update, and discussions with the CLSA Scientific Management Team (SMT), representatives of the CIHR ELSI Advisory Committee for the CLSA and CIHR staff. To date, the IOC has held three meetings:
March 2, 2010, June 24, 2010, and September 10, 2010.
While many years of planning preceded the launch of the CLSA research platform and related aspects of recruitment, data collection, and data use, the CLSA is at an early stage in the implementation phase of its lifecycle. Finalization of the CFI funding agreement is expected by December 2010. Statistics Canada approved the Memoranda of Understanding (MOUs) that were part of the institutional funding sub-agreements to allow the four Computer Assisted Telephone Interview (CATI) sites access to contact information from consenting participants from the Canadian Community Health Survey in Spring 2010, after which the recruitment of the Tracking Cohort commenced. As such, it is particularly timely to provide a formative assessment of issues and priorities to position CLSA as effectively as possible for optimizing the research returns on the very significant financial investment by CIHR and CFI and the intellectual investments of the CLSA scientific team. Observations and recommendations for action should be interpreted in this context.
2. Introduction
The IOC is impressed by the vision, scope and scale of the CLSA and is cognizant of the time and effort that it takes to implement this type of endeavour. The leadership of the Senior Management Team (SMT) is to be commended on the evidence of significant progress and forward movement.
The IOC applauds the vision of the study and has great respect for its leadership. However, the Committee identified a number of issues that it believes need to be addressed rapidly to enhance the positioning of CLSA for success. These issues reflect primarily the fact that this initiative is a huge enterprise: as a major national research platform. Project governance, management and outreach and communication with the research community are thus issues of particular importance in order to optimize scientific outcomes for such a monumental research platform.
3. Milestones Achieved and Use of Funds
3.1. The Funding Profile
As is evident from the re-profiling of CIHR’s funding for the CLSA (see Figure 1), a number of the initial milestones have been revised, although the larger recruitment targets remain as per the February 2010 Protocol.
Figure 1. Cumulative funding projected for CLSA – 2010
Action: In future reports, the CLSA SMT to provide clear statements of forecast expenditures and actual expenditures in tabular and graphical form, along with comments on the variances.
3.2. Scientific and Administrative Milestones
Key scientific deliverables involve the implementation of both the Tracking Cohort of 20,000 and the Comprehensive Cohort of 30,000. Some key milestones have been met:
- All Computer Assisted Telephone Interview (CATI) sites are now operational;
- REB approval for the Tracking Cohort has been received – a significant milestone for a multisite initiative;
- The protocol for the Comprehensive Cohort has been submitted to the REBs of the relevant institutions.
Of other scientific milestones, a number remain “works in progress”; the most critical in the view of the IOC are as follows:
- Completing the data and sample access policy (even formulated as principles), in order that it not hold up the critical REB approval;
- Finalization of all sub-agreements with institutions such that funding can flow as required for recruitment and data collection in both cohorts;
- Revision of the protocol for the comprehensive cohort;
-
Finalization of the CFI funding agreement.
Action: The CLSA SMT to report on progress on these scientific milestones at the February 2011 meeting of the IOC.
Action: In future Annual Reports (during the recruitment phase), the CLSA SMT to provide a chart showing the cumulative number of recruitments forecast for each cohort, the number actually recruited to date against these forecasts, and comments on variances, with two data points per year (May1 and November1).
3.3. Operational Milestones
The Funding Agreement identifies a number of milestones, including full implementation of a number of key committees:
- May 30, 2010 - The Operations Committee – done 2
- May 30, 2010 - The Primary and Secondary Data Access Committee – interim committee in place
- November 30, 2010 - The Advisory Council – not yet in place; see section on governance and management below.
The IOC looks forward to seeing progress on the last two milestones at its next meeting.
4. Governance and Management
The scientific development of the instruments/platform, and the management of the ethics and sample access regimes, have understandably been the primary focus of the three CLSA PIs. However, the scope of this multi-site data collection, even in its start-up phase, requires management of multiple aspects of project implementation. The IOC judged that both project management and scientific governance require particular attention over the next few months to position the CLSA platform for optimal research return.
4.1. Administrative Management
The strategic and operational management of the project is, at present, unduly lean. Putting a business manager in place is an urgent first step to relieve the CLSA PIs of the burden of project administration and management, so as to maintain their focus in leading the scientific dimensions of the project.
Recommendation: The CLSA SMT to give immediate priority to the hiring of a National Business Manager to assume the business leadership and direction for all CLSA operations and to ensure control and accountability on a day-to-day basis.
Action: The CLSA SMT to report on the allocation of roles and responsibilities among major administrative players in its next report to the IOC.
4.2. Scientific Governance
While CLSA governance structures have been under discussion for some time, the IOC noted that an external Scientific Advisory Board has not yet been established. Such a Scientific Advisory Board would serve a dual role:
- a source of external advice on the multitude of substantive scientific issues and related procedures that comprise the CLSA;
- a means of ensuring broader stakeholder input to the directions and decisions that are made on behalf of the larger community – a form of quality assurance and public accountability.
Recommendation: That the CLSA SMT put in place a Scientific Advisory Board and make the terms of reference and membership available to the IOC for its February 2011 meeting.
4.3. Partnerships
The expectation of partnerships reflects the commitment of the CLSA team to seek matching funding, but equally the value of such partnerships in optimizing utilization of the CLSA data. Indeed the two should be mutually reinforcing.
5. Monitoring Ongoing Excellence
5.1. Overall Quality/effectiveness of Platform Implementation
To date the IOC has focused primarily on data access, but not as intensively on the instrument and the process of data collection. The Committee explored a number of aspects of the process of data collection with the CLSA PIs, but recognized that it needed a more comprehensive understanding of how the data collection is proceeding to make a full assessment.
Action: The CLSA SMT to provide prior to the February 2011 IOC meeting any additional comments needed for the IOC to understand the evolution and current status of the questionnaires.3
Recommendation: In order to promote broader community engagement, that the CLSA SMT keep the Protocol up to date and make it available on the website. A revised version should be provided to the IOC when there are significant revisions.
In response to the CLSA PIs report on their recent work on harmonization, the IOC was pleased to see consideration of comparability with other studies and recognized two aspects of harmonization – retrospective (as through the use of DataShaper) and prospective. A central challenge is to address whether adopting others’ approaches is truly state-of-the-art. In any case, it was unclear to the IOC whether harmonization represents a critical current task for CLSA, particularly given the numerous priorities detailed in this report.
5.2. Data and Biological Sample Access Policies
The IOC devoted much of its June 2010 meeting to a review of the draft data and biological sample access policy. Comments were provided to the CLSA PIs and relayed by CIHR to the CIHR ELSI Advisory Committee in early July 2010. The SMT reported that these comments have been taken into consideration in revisions of the document, but until the IOC has had the opportunity to see the revised version of the Data and Biological Sample Access Policy, it reserves further comment on the specifics of the policy.
Action: The CLSA SMT to provide a revised Data and Biological Sample Access Policy to the IOC for review at its February 2011 meeting.
The IOC considers data access to be a strategic issue for CLSA and recognizes the major ELSI element. If the original strategy is not well crafted, then much potential value could be lost. Three specific cautions emerged in discussion:
- The data access policy should not unduly compromise the ability to access the data for future and unforeseen research questions, providing normal ethics approval is obtained;
- There should be some caution exercised with respect to early release of the data, such that dirty or inadequate data are not released prematurely, and then only after they are cleaned and tested for problems first;
- There needs to be additional work on the data access policy to ensure that recruitment, data collection and data access proceed smoothly for both cohorts. The use of “principles” language should allow progress, with a drill down to more specific language over time.
5.3. Ethical, Legal, Social Issues (ELSI)
The CIHR ELSI Advisory Committee for CLSA has been very active and has delivered valuable and practical advice to the CLSA team. It was agreed that the IOC and the ELSI look at issues from a different perspective, and that there is benefit for the CLSA PIs in seeing multiple perspectives as has been done with the comments on the data access policy.
Action: CIHR and the IOC Chair to discuss with the ELSI Advisory Committee how best to formalise interactions between the two bodies.
6. Monitoring Ongoing Excellence
The IOC recognizes that the CIHR is in somewhat uncharted territory with respect to the implementation of a platform project of this type. In this context it urges flexibility by CIHR in application of procedures and rules designed for conventional operating grants which may not be directly transferable to large and complex infrastructure projects of this type.
Recommendation – That CIHR exercise flexibility in the application of procedures and rules designed for conventional grants which may not be directly transferable to the CLSA.
Both the CLSA SMT and the IOC agreed that the format for the Annual Report was inadequate as a communication device and that a revised approach should be developed for subsequent reports.
Action: Revise the reporting format to provide a more effective vehicle for communication through a consultative process led by the IOC Chair and in collaboration with the CLSA SMT and CIHR staff.
7. Responsiveness to the Research Community
There was little reference to the engagement of CLSA researchers outside the CLSA SMT in the May 2010 and September 2010 progress reports. While the IOC is aware of the existence of institutional site leaders, it recommends putting mechanisms in place to facilitate current and future engagement of the larger stakeholder community.
Action: A limited number of CLSA participants (e.g. Chairs of Working Groups, institutional site leaders, and the new chair of the Operations Committee) to be invited to participate along with the CLSA SMT in the February 2011 face-to-face meeting of the IOC.
The IOC noted the importance of regular updates to the website 4 and the production of frequent Newsletters to inform the activities of existing and potential CLSA researchers across Canada.
Appendix 1 - Membership of the CIHR International Oversight Committee for the CLSA
Chair:
Ms. Janet HALLIWELL
Salt Spring Island
British Columbia
Members:
Prof. Laurence BRANCH
Department of Health Policy and Management
USF College of Public Health
Tampa, Florida, United States
Prof. Paul BURTON
University of Leicester
Department of Health Sciences
Leicester, United Kingdom
Prof. Carol RYFF
Director, Institute on Aging
Madison, Wisconsin, United States
Prof. Cyrus COOPER (resigned September 2010)
University of Southampton
Southampton General Hospital
United Kingdom
1 The Canada Foundation for Innovation grant awarded to the CLSA supports renovations and the construction of key facilities for data and biological sample collection and storage, including statistical and genetic analyses. CFI funding is matched by the provinces and academic institutions where the infrastructure is located.
2 Following the September 10, 2010 meeting, and in response to comments from the IOC regarding the need for evidence of the engagement of the larger research community, the CLSA team appointed a non-SMT member of the CLSA research team as Chair of the Operations Committee - Dr. Andrew Wister from Simon Fraser University.
3 Such information could include the most recent version of the data collection protocol and the questionnaires being employed.
4 As of October 10, the website was outdated, with no information on the project progress, the full protocol, and approaches to data access, despite the September forecast that the website would be revised by September 20.