Management response to the fourth year review of the Canada graduate scholarships (CGS) program

May 12, 2009

Other format


PDF version
(74 KB)

Table of Contents

Introduction
Context
Evaluation methodology
Recommendations and responses by evaluation issue
Summary of response
Appendix 1 ─ Summary of Recommendations, Responses and Follow-up Actions

Introduction

An evaluation study of the Canada Graduate Scholarships (CGS) program was carried out in 2008 in keeping with the CGS Terms and Conditions. These stipulate a review of the CGS in its fourth year to identify areas for improvement in anticipation of the renewal of the Terms and Conditions which will expire on May 31, 2009.

The evaluation study produced eight (8) recommendations and a number of suggestions to improve the program. On the basis of these, the management of CIHR, NSERC and SSHRC have agreed to implement certain enhancements to the CGS program and to undertake further analysis of a number of options for enhancements that also have relevance to the agency-specific doctoral award programs.

Context

In the February 2003 Budget, the Government of Canada announced the introduction of the CGS. This program was introduced within a broader context of federal investments, initiated in 1997, aimed at increasing Canada's capacity for innovation. A national innovation strategy provided the framework for such investments as the Canada Research Chairs, the Canada Foundation for Innovation, CGS and others. The strategy sought to position Canada among the world's most innovative countries in terms of research and development.

Budget 2003 and subsequent investments in the CGS program through Budgets 2007 and 2009 provided increasing funds for the CGS program. The program therefore now funds on an ongoing basis up to 2,500 Master's and 2,500 doctoral scholarship holders at any one time with an investment, when fully ramped up, of $131.25 million annually, split between the three research granting agencies. Most recently, Budget 2009 committed funds to award a further 2,000 additional Master's awards and 500 additional doctoral awards over the next 3 years with a one-time investment increase of $87.5 million during this timeframe.

It is important to note that although all three research granting agencies already had agency-specific doctoral research awards prior to the creation of the CGS program; these awards were set at values lower than the CGS doctoral awards. Also, neither CIHR nor SSHRC offered scholarship support at the Master's level prior to the introduction of the CGS. As a result, the CGS represented a significant injection of funds in support of new talent for both Master's and doctoral research levels.

Conceptually, the CGS complements the Canada Research Chairs introduced in 2000. Both programs aim to attract and retain talent of a high-calibre. The chairs, on the one hand, support 2,000 research professorships designed to attract and retain some of the most accomplished and promising minds in the world. The CGS, on the other hand, when fully ramped up following Budget 2007 investments, will make available, on an ongoing basis, 5,000 research-focused graduate scholarships for students who demonstrate a high standard of achievement, plus the short-term addition of 2,500 more scholarships over three years based on Budget 2009 investments. One expected outcome of the CGS is the increased ability to attract and retain experienced researchers. Its perceived benefit for researchers resides in the financial incentives for students to pursue research-focused graduate studies and their reduced reliance on financial support from their academic supervisors who would otherwise need to fund these students through their research grants or from other sources. In other words, CGS can help to attract and retain experienced researchers because it provides increased funding to attract top students and an increased number of well-funded students available for the research enterprise.

In 2003, the government set the target of a five per cent annual increase in graduate student enrolment. As a priority for achieving this target, the rationale for the CGS was linked to the objective of doubling the number of federal government graduate scholarships, to almost 10,000 in total. When CGS ongoing investments will be completely ramped up and these are added to current ongoing investment levels for the agency-specific Master's and doctoral scholarships, the total annual number of supported graduate scholarships will be at least 9,000. With the short-term investment in Budget 2009, the three granting agencies will have surpassed the stated goal of 10,000 scholarships with an estimated increase to at least 11,500 graduate scholarship awards per year.

Overall, there has been a significant (57%) increase in full-time graduate studies enrolment (Master's and PhD) from 1996-2006 (Association of Universities and Colleges of Canada [AUCC], 2007). According to the Canadian Association of Graduate Studies (CAGS 2006b), student enrolment at the Master's level was stable from 1992 to 1998 and has been increasing slightly every year since 1998. However, the increase has been more pronounced since 2000. At the doctoral level, student enrolment has been relatively stable from 1992 to 2000 and has been increasing slightly since then. Factors associated with enrolment growth in graduate studies include job requirement inflation (AUCC, 2005), increased research support from federal and provincial governments and increased university operating budgets from the provincial governments (AUCC, 2002) and, more generally, government investments in education. Also, it should be noted that, among CGS award applicants, enrolment levels were high: 93 per cent of those applying for an award actually enrolled in graduate studies.

More recent developments in federal policy provide necessary context for this management response and for the agencies' follow-up action plans. Specifically, in Budget 2006, the government introduced an income-tax exemption for all scholarship, fellowship and bursary income. In the February 2008 Budget, the Government of Canada introduced the Vanier Canada Graduate Scholarships as well as the Canada Graduate Scholarships – Michael Smith Foreign Study Supplements; both programs are administered by CIHR, NSERC and SSHRC. The Vanier Scholarships introduced an annual award value of $50,000 for doctoral studies, versus the $35,000 doctoral CGS annual award value, representing a new level of prestige for merit-based student awards from the agencies. Budget 2008 also announced the Canada Student Grant Program to replace the Canada Millennium Scholarship Foundation, thereby redirecting roughly $350 million annually in student grants towards low and middle-income students and away from any past academic and merit-based investments at the Foundation. These and other developments provide important context for the consideration of the relevance and niche of the CGS as well as the determination of award values in the future.

With the creation of the CGS in 2003 and with additional funding brought about in 2007 and 2008, the Government of Canada has demonstrated that it makes the funding of graduate studies an important component of its strategy for science, technology and innovation. In this regard, the CGS is one component of the government's commitment to maintain Canada's G-8 leadership in public-sector R&D performance, identified in the federal science and technology strategy of 2007.

Since its inception, the CGS Program has contributed to maintaining excellence through the most talented, skilled and creative students in Canada. The award holders have demonstrated continued satisfactory progress and benefited from this program in pursuing various knowledge-intensive careers within academia, industry, government and other sectors of economy. The CGS Program still attracts the best and brightest young people who will become the next generation of researchers. The program has also put in place a review process that meets the highest standards in terms of review panel and selection criteria. In addition, the CGS program supports Canada's ability to prosper in today's global, innovation-driven economy which ultimately depends on the skills, knowledge and creativity of Canadians.

Evaluation methodology

Although these statements in no way seek to call into question the validity or adequacy of the findings of the CGS Evaluation study, it is prudent to call attention to the following aspects of the evaluation methodology as contextual considerations when reading the report and its recommendations:

  1. The evaluation findings focus primarily on the comparative and incremental impacts of CGS awards relative to agency-specific awards or no award impacts, rather than absolute outcomes and impacts of the program.
  2. An evaluation undertaken only 5 years after implementation of a program that intends to incent Masters and PhD degree completion, which in turn usually requires 3 to 8 years cannot document longer-term or ultimate outcomes and the evaluation findings should be interpreted in that light.
  3. As noted in the evaluation report, the evaluation study's conclusions must be regarded with some prudence with respect to the evidence concerning evaluation issues of program relevance and design. Evidence relative to these evaluation issues is considered to be "softer" and cannot be used to conclude the program is in fact relevant or not – more research would be beneficial, though the contextual information noted above seems to clearly indicate a continuing interest and need for the program or at least for similar investments.
  4. For the CGS expected outcome of increased ability to attract and retain experienced researchers, the evaluation study included relatively limited input from experienced researchers, for example, holders of Canada Research Chairs, among the 33 key informants interviewed.

Recommendations and responses by evaluation issue

Relevance

Recommendation 1: The Agencies should maintain student award programs.

Response:

Agreed. The CGS is a key instrument within a program continuum aimed at financially supporting and encouraging the development of highly skilled research professionals in Canada. The suite of programs for students and trainees supports undergraduate studies, through to PhDs and on to fellowships and early career research. Funding students through award programs is one of a number of measures to support the training of highly qualified personnel (HQP). One outcome of the introduction of the CGS program, and its subsequent funding increases, has been to reduce reliance on agency awards thereby providing flexibility to introduce other strategies and programs for HQP training, such as, for example, the CIHR Strategic Training Initiative in Health Research (STIHR) and NSERC'S Collaborative Research and Training Experience Program (CREATE). Diversified and balanced tools to invest at all levels of the five to ten year process by which students become researchers is of vital importance to produce the desired research talent for Canada.

Program Success

Recommendation 2: The logic of the CGS program should be rethought based on the information offered by this evaluation and other studies conducted since the inception of CGS.

Response:

Agreed. The Agencies will review the logic model of the CGS to better reflect the intent and added value of the program. Clarification of the CGS program intent and logic model will be informed by additional analysis and consultation. Such consultation could be undertaken during the summer and the fall of 2009 in order to inform government decision processes about program changes and associated impacts. The CGS program Terms and Conditions will be renewed until May 2010 in order to ensure continuing delivery of the program until such analysis, consultations and possible program change decisions are completed.

Design and Delivery

Recommendation 3: The Agencies should consider the possibility of reducing the gap in value between CGS and regular awards at the doctorate level.

Response:

Agreed. Recent federal policy revisions, such as the income-tax exemption granted to scholarship income and the introduction of the Vanier Scholarships, have created a need to review the value of the CGS relative to other doctoral awards. A key consideration for the Agencies in reviewing the gap in value between doctoral awards will be the resulting impact on the overall number of awards. Another important consideration will be the effect of the level of the doctoral awards relative to the awards available to post-doctoral fellows. Further analysis is required by the Agencies to assess the pros and cons of harmonizing the CGS and the agency-specific doctoral awards. This harmonization could address not only the gap in award values, but also include other parameters of award design and delivery.

Recommendation 4: The Agencies should consider the possibility of extending the duration of a Master's award to two years and that of a doctoral award to four years.

Response:

Further analysis is required. Assuming no additional funds were to be invested in the CGS program, extending the duration of awards would lead to a reduction in the number of award recipients over time. Decisions on the duration of the awards should be made on the basis of the intent of the program. It is not clear that the intent of the program is to support students through to completion of studies. Other funding sources and options may exist beyond the award period, though the availability of alternate funding does vary considerably across disciplines.

The nature of training experience also differs across disciplines and is a key determinant of the time to completion of degree. Further analysis is required to determine whether application of a uniform duration of CGS awards across the Agencies' respective student populations would be appropriate.

One of the currently stated intentions of the CGS program is to encourage timely completion of degrees. Extending the duration of non-repayable funding could be seen as an incentive for students to take longer to complete their degree. Ultimately, the duration of awards must be established in relation to the CGS program objectives. Consequently, the Agencies will consider the duration of awards in the context of their review of the program's logic model (see response to Recommendation No. 2).

Recommendation 5: The award programs should not restrict the international mobility of students.

Response:

Agreed. CIHR, NSERC and SSHRC have articulated international frameworks and strategies that support the international mobility of students. Moreover, the agencies strongly support the concept of "brain circulation" in reference to the international mobility of Canadian students and the reciprocal mobility of international students into the Canadian research environment.

The CGS requires that recipients pursue their degree at a Canadian university. This differs from some of the agency-specific awards which allow recipients to complete their training abroad. As the Agencies further analyze the potential to harmonize aspects of the CGS and agency-specific doctoral awards, they will consider the provisions for the international mobility of students.

When promoting the international mobility of students with public funds, an important consideration is the capacity to encourage the repatriation of a reasonable number of Canadian students and the retention in Canada of some international students. This consideration will require further analysis before any decisions are made about how or whether the CGS might better address this international dimension.

Currently, the CGS – Michael Smith Foreign Study Supplements introduced in 2008 will provide financial support to a number of CGS recipients, for a period of up to six months, to study and conduct research abroad, while pursuing a degree at a Canadian university. While CGS recipients have always had the possibility to participate in short-term international mobility, the supplements help to address the incremental costs.

Recommendation 6: The Agencies should develop an integrated external communication plan for CGS to contribute to its branding as an exceptional award.

Response:

Agreed. While certain aspects of the branding can be improved, the large number of highly meritorious applicants demonstrates that CGS is well recognized within postsecondary institutions in Canada as a prestigious award. Surveys of recipients confirmed that the federal role in providing support is also well recognized by students. An integrated communication plan will clarify the objective of more extensive external communication in order to target the appropriate audiences.

An integrated external communications plan will address the fact that three distinct names for the CGS awards (CIHR - Frederick Banting and Charles Best CGS; NSERC – Alexander Graham Bell CGS; SSHRC – Joseph-Armand Bombardier CGS) were announced in Budget 2007, and added to the CGS brand, for CGS awards offered by the three Agencies.

Through the communication plan, there is strong potential to brand the prestige associated with support from the Agencies, whose own well-recognized hallmarks include excellence and national competitions. Recognition by students and their potential employers of the prestige associated with the Agencies stands to be as important as branding the program's name.

The Agencies will give further consideration to an integrated external communication plan in the light of plans to harmonize the CGS and the agency doctoral awards in the future.

Recommendation 7: The Agencies should collaborate to develop a workable data collection mechanism for performance information.

Response:

Agreed. This is already underway and, for example, the best practice of conducting exit surveys of award recipients for performance information will be adopted by the three Agencies, among other data collection mechanisms.

Recommendation 8: The performance monitoring plan for CGS and related programs should be revisited with a view to make it more pertinent to program managers and to better delineate performance monitoring from evaluation assessment.

Response:

Agreed. The Agencies will revisit the performance monitoring plan of the CGS following the review of its logic model (see response to Recommendation 2).

Summary of response

Overall, the three research granting agencies feel that this evaluation exercise has been effective and useful. The recommendations provided do call into question important aspects of the CGS program design but it is felt that these challenges and the analysis and consultations that will result from these will have positive effects on the fine-tuning and continuing relevance and impact of this key investment program in graduate studies, research and the innovation system in Canada.

Appendix 1 provides an at-a-glance summary of the evaluation recommendations, the management response and the general timelines and considerations associated with delivery of the proposed actions. An important next step is the temporary one-year renewal of the CGS Terms and Conditions that would otherwise expire on May 31st, 2009. This one-year extension will provide the CGS management groups with sufficient time to fully analyse and consult on possible improvements to the program and then to secure required approvals as needed. Any program changes would therefore not be implemented before the 2010 competitions and may need to be extended further depending on the nature and operational feasibility of changes that are proposed.

Appendix 1 ─ Summary of Recommendations, Responses and Follow-up Actions

Recommendation Response Follow-up Action Priority/Timeline
1. The Agencies should maintain student award programs Agreed. N/a
2. The logic of the CGS program should be rethought based on the information offered by this evaluation and other studies conducted since the inception of CGS. Agreed, but further analysis is required. The Agencies will review the logic model of the CGS program to better attribute its impacts and distinguish its added-value. Renewal of Terms and Conditions in May 2009 for one year while undertaking consultations and considering program changes.
3. The Agencies should consider the possibility of reducing the gap in value between CGS and regular awards at the doctorate level. Agreed. The Agencies will analyze the potential to harmonize the CGS and the agency-specific doctoral awards in the future, including award values. Renewal of Terms and Conditions in May 2009 for one year. Implement any changes for 2010 or 2011, as appropriate.
4. The Agencies should consider the possibility of extending the duration of a Master's award to two years and that of a doctoral award to four years. Further analysis is required. The Agencies will consider the duration of awards in the context of their review of the program's logic model. Renewal of Terms and Conditions in May 2009 for one year and implement for 2010 CGS competition, as appropriate.
5. The award programs should not restrict the international mobility of students. Agreed. As the Agencies analyze the potential to harmonize the CGS and their regular doctoral awards, they will consider the provisions for the international mobility of students. Renewal of Terms and Conditions in May 2009 for one year and implement for 2010 CGS competition, as appropriate.
6. The Agencies should develop an integrated external communication plan for CGS to contribute to its branding as an exceptional award. Agreed. An integrated communication plan will be given further consideration in the light of plans to harmonize the CGS and agency-specific doctoral awards, following consultation on possible program changes For 2010-2011 fiscal year, following decisions on any program changes.
7. The Agencies should collaborate to develop a workable data collection mechanism for performance information. Agreed. Related actions are already underway. For 2010-2011 fiscal year, following decisions on any program changes.
8. The performance monitoring plan for CGS and related programs should be revisited with a view to make it more pertinent to program managers and to better delineate performance monitoring from evaluation assessment. Agreed. The Agencies will revisit the performance monitoring plan of the CGS following the review of its logic model. For 2010-2011 fiscal year, following decisions on any program changes.