Canada Graduate Scholarships (CGS) Program and Related Programs Review - Executive Summary
Evaluation & Analysis Branch
Senior evaluator and author:
Circum Network Inc.
R.A. Malatest & Associates Ltd.
R.A. Malatest & Associates Ltd.
R.A. Malatest & Associates Ltd.
Scientific authority and project managers:
Canadian Institutes of Health Research
Additional agency representatives:
Jocelyne Manseau Mandeville
Natural Sciences and Engineering Research Council of Canada
Social Sciences and Humanities Research Council of Canada
The present evaluation study of the Canada Graduate Scholarships (CGS) program was conducted in preparation for the renewal of its Terms and Conditions. It was conducted for the Presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC). It was managed by the Interagency Evaluation Steering Committee which is comprised of program and evaluation representatives from all three Agencies as well as Industry Canada.
In the February 2003 budget, the Government of Canada announced substantial funding for the three federal Agencies to introduce a new program: the Canada Graduate Scholarships (CGS); additional funding was extended in the 2007 and 2008 federal budgets. The program benefit is an award of $17,500 at the Master's level and $35,000 at the doctorate level. In comparison, NSERC Master's award have a similar value ($17,300) and agency specific doctoral awards carry values of $20,000 (SSHRC), $21,000 (NSERC) or $22,000 (CIHR).
The CGS has been incorporated into the selection processes in place for the CIHR Doctoral Research Awards, the NSERC Postgraduate Scholarships, and the SSHRC Doctoral Fellowships. Scholarships for Master's students were not available through CIHR and SSHRC prior to CGS. Upon full implementation, and at annual cost of $105 million, the Program was going to support 2,000 Doctoral students and 2,000 Master's students each year. CGS scholarships are awarded for three years for Doctoral students and for one year for Master's students.
An evaluation framework was produced in March 2007. It identified the following evaluation issues.
Relevance: Is there a continuing need for the CGS Program? Does the Program continue to be consistent with agency and government wide priorities?
Design and delivery: To what extent is the Program appropriately designed to achieve its objectives? What changes to the CGS Program design would make it more relevant and effective? Is the allocation of the scholarships among the Agencies appropriate given the current distribution of full time graduate students by group of disciplines? Is the mix of direct and indirect sources of support for graduate students optimal in each agency? To what extent has the Program been delivered by Agencies and universities as intended? To what extent is the CGS Program on track to meet its allotted number of funded Master's and Doctoral students, by agency? Should a portion of CGS and agency scholarships be allocated to certain disciplines or should budgets for disciplines be determined by the number of applications received?
Success: To what extent has the Program achieved its intended outcomes? What are the overall incremental program impacts? To what extent can outcomes be attributed in whole or in part to the CGS Program and/or other scholarship programs? What are the comparative impacts for CGS recipients, graduate students funded through agency specific scholarship programs and students who rely on other means of support? To what degree have the branding and communications of the Program to relevant stakeholders been successful in distinguishing the CGS from granting agency scholarships? Is the Program's performance monitoring (of outputs and outcomes) appropriate and adequate? Have the Program's activities had any unintended impacts (positive or negative)?
Program cost effectiveness: Is the Program delivered in a cost effective manner? Are there more cost effective ways to deliver the Program under the existing model? Are there alternative, more cost effective programs/models that could achieve the same objectives?
This evaluation is based on a mixture of qualitative evidence (e.g., program documentation, key informant interviews, and a review of other programs) and quantitative evidence (i.e., administrative data and a large survey of program applicants) where the latter was given most attention, and on the comparison of relevant results obtained by three groups of students, some of whom were involved in the programs and some who were not.
Available documentation was analysed and in-depth interviews were conducted with some 33 individuals to factor into aspects of this evaluation that could not be captured in the student survey.
The student survey benefits from a large sample size of 9,109 respondents and a reasonably good response rate, considering the groups that were targeted. Respondents were shown to be comparable to non-respondents; respondents from the sub-sample subjected to telephone follow-ups were also shown to be comparable to respondents who were not subjected to this type of follow-up.
A large portion of the analysis is based on a comparison of recipients of CGS awards, recipients of regular agency awards and students who applied for an award but did not receive one. These three groups of students are not strictly equivalent: one was considered worthy of the highest honour (a CGS scholarship); another one was identified as highly deserving (and received another agency scholarship); and the third group, while of high caliber (otherwise, universities would not have selected them for the competition), were not attributed a scholarship by the selection committees. However, they all emerge from the same group of "best students"; in fact, at NSERC and SSHRC, only students pre-qualified by universities are allowed to apply for graduate awards. This is a great advantage to this evaluation: because students in the three groups are similar, the difference among them is primarily whether they received an award and which award they received; therefore, differences in how they proceed through graduate studies can be more readily attributed to this key difference. Since there could possibly be other differences among the three groups of students, we implemented multivariate statistical control in order to focus the comparison on the impacts of the scholarships. This way, we controlled for other variables that could possibly explain differences observed in study progress among groups; after these statistical controls, if a difference persists among groups, it can probably be attributed to the effects of the programs.
Notwithstanding the strength of the design and of the data collection, there were some limitations to the available data. Administrative data were produced by three independent organizations, each with their own systems and procedures. While we strived to produce data on the same bases, some of the information produced might not be entirely comparable.
Some of the documentation is dated, even though the environment is characterised by rapid change; this is particularly true of macroscopic information relative to the academic and industrial environments. Also, the in-depth interviews did not necessarily allow to collect evidence that can be cross-referenced against hard facts; in fact, such interviews are often needed where empirical evidence is not available.
The survey of students targeted the first beneficiaries of CGS awards (in 2004, 2005 and 2006) and agency awards winners and applicants from the same years. The relatively short time elapsed between these years and the survey period in 2008 may not have allowed for the materialization of some outcomes, such as degree completion — although the relative brevity of the delay was the same for all three treatment groups.
This evaluation is based on the premise that CGS was designed to impact the behaviour of the best students, to increase the likelihood that they will enrol in graduate studies, that they would complete these graduate studies and that they would study (and later work) in Canada. Therefore, to assess the performance of the program, this evaluation puts this logic to the test and compares study progress for students in receipt of program benefits to those not in receipt. Others have suggested that this is not the appropriate test to perform because, in their view, CGS was created to impact the whole of the graduate student population.
Results and Recommendations
This evaluation study has reached a number of conclusions. Those concerning program effects are methodologically strong, thanks to the reliance on a quasi-experimental approach and on multivariate modelling. The evidence concerning issues dealing with program relevance, and design and delivery is softer and must be regarded with more prudence.
This chapter recalls the key conclusions of the evaluation and proposes an interpretation of findings.
The evaluation generally supports the notion that there is a continuing need for CGS and related programs, although the evidence is not one-sided.
The first rationale argument is that HQP are in high demand in Canada and that purviews into the near future conclude that it will not decrease any time soon. Canada ranks sixth in a list of developed countries with regard to the proportion of the population in the HQP category (23%), ex aequo with Australia and Korea; this highlights the need for a continuous influx of new HQP. While some studies conducted a decade ago question the existence of "brain drain", this evaluation uncovered that one-quarter of doctoral award applicants who were not studying at the time they were surveyed resided abroad and that one-quarter of award applicants expected to move abroad to study or to start a career. Therefore, there is a risk of loss of highly qualified personnel to other countries but the extent of this risk is uncertain and it is possible that it is countered by influx of HQP from other countries.
The second element of the rationale for the awards programs is that there is a financial barrier to access to graduate studies. This evaluation has found that the debt load of unsuccessful applicants belonging to the program target group is lower ($17,100) than that of the general graduate student population (about $20,000). On that basis, we conclude that the award applicant debt load is not a major deterrent to graduate studies. Still on the financial side, Master's level awards were shown to increase total student revenue from all sources by about $2,500 compared to non-recipients (whereas the award value is approximately $17,500) while CGS-D increases total revenue by $9,400 (for an award of $35,000) and regular doctoral awards increase total revenue by $1,600 compared to non-recipients (for an award of about $21,000). Thus, the main income-related effect of awards was to modify sources of revenue away from earned income.
Award programs are associated with results that contribute the overall objectives of HQP supply and research excellence:
- awards represent an incentive to enrol in graduate studies according to the recipients' self-assessment;
- awards increase slightly actual enrolment in graduate studies;
- awards increase recipients' recognition of the federal government's financial support to research training;
- at the Doctorate level, awards increase recipients' involvement in core research activities;
- awards reduce recipients' reliance on paid income and recipients' study related debt;
- awards improve recipients' self-assessed prospects of getting a job in an area relevant to their studies.
With the creation of CGS in 2003 and additional funding brought about in 2007 and 2008, the Government of Canada has demonstrated that it makes the funding of graduate studies an important component of its innovation strategy.
All in all, the rationale for supporting access to graduate studies probably still exists. Whether the best approach is to support academic excellence or to award scholarships on the basis of student financial need is not a closed debate.
Recommendation 1. The Agencies should maintain student award programs.
The logic of the CGS program is based on a cascade of short term and longer term effects. The following assessment of program success is based on whether or not the evidence from this evaluation shows that these effects took place; this summary factors in varied indicators as well as statistical significance and substantive significance. Where available, multivariate statistical results are the preferred source of information because they provide the most rigorous determination of program effects. Because there is still a debate about whom CGS participants should be compared to, we have offered results comparing them to non-recipients at the Master's level (where only NSERC has specific programs) and to non-recipients and agency-specific award recipients at the doctorate level.
Expected outcome #1: Increased incentive for students to enrol in graduate studies in Canada
The impact of CGS on incentives to enrol in graduate studies was measured by asking students for their self-assessment of this impact. Therefore, only students in receipt of an award could be included in this validation.
Three-quarters of award recipients indicated that the possibility of receiving an award or actually receiving an award were incentives to enrol in graduate studies. One-half said the same about the prestige of the award. However, the results were the same for CGS-D recipients and for regular doctoral awards recipients, thereby demonstrating no incremental impact of CGS in this regard.
Expected outcome #2: Increased enrolment in graduate studies in Canada
After a decade of stagnation, enrolment in graduate studies has been increasing steadily since 2000 — that is, three years before the introduction of CGS. Also, it should be noted that, among award applicants, enrolment levels were high: 93% of those applying for an award actually enrolled in graduate studies.
Award recipients were about four percentage points more likely to enrol in graduate studies than non-recipients, at the Master's level and at the doctorate level. It was also observed that one-third of Master's applicants who finished their Master's studies continued on to the doctorate level; this was statistically the same for CGS recipients and for regular agency award recipients. Also the same for these two groups was the proportion of Master's students who plan to continue on to a doctorate (one-half).
While some of the findings showed statistically significant differences between recipients and non-recipients, the actual differences were not substantial enough to conclude to a positive effect.
Expected outcome #3: Increased incentives for scholarship recipients to complete studies within a specific time period
As with expected outcome number 1, increased incentives to complete studies within a certain time period were self-assessed and therefore available only from award recipients. While between six (Master's) and seven (doctorate) out of ten award recipients indicated that the award was important in setting the student's pace of study, only one (doctorate) to two (Master's) out of ten stated that awards were important in the time it took the student to complete their study program. Even more important, the answers were statistically the same for CGS recipients and for recipients of regular agency awards.
Expected outcome #4: Increased recognition by the research community of the federal government's financial support for research training
Note that only program applicants were systematically canvassed about their views of the federal government's support of research training; other components of the research community (in particular, the researchers themselves) were not part of this assessment. Among CGS recipients and regular agency award program recipients, about eight out of ten thought that the federal government made a significant contribution to supporting research training in Canada. The results were the same for CGS award recipients and recipients of regular awards. Non-recipients were much less likely to share this view (by about 20 percentage points at the Master's level and 40 points at the doctorate level).
Expected outcome #5: Increased numbers of students completing degrees and doing so in a timely manner
It must be recognized at the outset that many program applicants had not completed their study program at the time of their participation in this evaluation. This was particularly true of students in doctoral studies — which, on average, last longer than the duration of CGS since its inception.
This evaluation can state nonetheless that, among award program applicants, there was an equal probability of having completed the study program in all groups (CGS recipients, regular award recipients and non-recipients) and at both levels (Master's and doctorate). Moreover, for those who had indeed completed their degree, the time to completion was 25 months at the Master's level and 48 months at the doctorate level, and it was the same for CGS recipients, regular award recipients and non-recipients. For those who had not yet completed their degree, expectations were that they would complete in 30 months at the Master's level and in 50 months at the doctorate level — again, without differences among groups. Asked whether they were progressing through their study program at the pace they were expecting, one-half of Master's students indicated that they were on pace or progressing faster than anticipated (same for CGS recipients and non-recipients); at the doctorate level, six out of ten CGS recipients answered the same way, which is ten points more than among agency award recipients and 20 points more than among non-recipients.
All in all, the conclusion is that CGS did not significantly affect the likelihood of completing the study program or the time to completion.
Expected outcome #6: High quality research training, as well as increased ability to attract and retain experienced researchers
Graduate students proved to be generally satisfied with their research environment. CGS recipients and regular award recipients were equally satisfied with it and slightly more satisfied than non-recipients.
About one-half of all program applicants held teaching assistantship positions; this proportion was the same for CGS recipients, regular award recipients and non-recipients. In parallel, about one-third of applicants held research assistantship positions; recipients were somewhat less likely to hold such positions.
Master's students of all three groups were equally likely to have contributed to academic publications. Among doctorate students, CGS recipients and recipients of regular agency awards had the same number of publications and they had more, on average, than non-recipients.
Among Master's students who completed their degree and held employment, CGS recipients and regular agency award recipients were more likely than non-recipients to hold a job that required the graduate degree they sought. Results at the doctoral level barely reach statistical significance.
From this somewhat limited set of indicators, this evaluation concludes that CGS has had limited impact on the quality of research training offered to students and on the ability of the university system to attract and retain experienced researchers.
Expected outcome #7: Increased capacity to meet demand for HQP in the faculties of Canadian universities and in the public and private sectors
While analyses of these indicators at doctorate level were hampered by limited sample size, at the Master's level, CGS recipients were shown to be more likely to hold a highly-qualified job than non-recipients. By their own account, CGS recipients and recipients of regular agency awards were more influenced by their studies than non-recipients to pursue research or teaching as a profession.
Other indicators of effects on meeting the demand for HQP showed no differences among groups of respondents: they all shared the same sectors of interest with regard to employment; they indicated the same likelihood of continuing on with post-doctoral research; they were equally likely to hold a faculty position after finishing their doctorate program.
Expected outcome #8: Improved branding of Canada as a home of research excellence and Canadian universities as world class research centres
This evaluation offers limited evidence regarding the improvement of the branding of Canada as a place of research excellence.
The ultimate expected outcome from CGS is "to contribute to Canada's Innovation Strategy to make Canada one of the most innovative countries in the world helping reach the target of moving from 14th place to among the top 5". This evaluation is not in a position to assess whether Canada has progressed toward that goal or whether CGS contributed to progress in that area. A 2007 Conference Board of Canada report entitled How Canada Performs, A Report Card on Canada indicated that Canada held the 14th OECD place in Innovation; however, most of the data used in that analysis dated back to 2003 or years prior to the implementation of CGS.
The necessary conclusion from the review of the success of CGS with regard to its stated logic is that the program has had limited impact over and above the regular agency awards at the doctorate level (while providing more funding and an envious branding); at the Master's level, comparisons with non-recipients suggest that CGS has had some of the impacts it was expected to have but that the magnitude of these impacts has been limited.
There is little doubt that federal support to graduate studies is necessary to achieve the innovation objectives of the Government of Canada. This evaluation shows, however, that the theory used in building a case for the CGS program has not met the test of time. Some important questions are still unanswered, though: what is the optimal level of support extended to graduate students? Would need-based support be more effective than excellence-based awards?
Recommendation 2. The logic of the CGS program should be rethought based on the information offered by this evaluation and other studies conducted since the inception of CGS.
A variety of impacts of award programs other than those that comprised the CGS program logic were tested. At the Master's level, the evidence shows clearly that, compared to the absence of support, CGS has significant positive unintended impacts on student income and debt levels, and on the necessity to work for pay while studying. These CGS effects are at par with the effects found for NSERC's PGS-M, which has the same monetary value as CGS-M. Also, Master's award programs were found to have no impact on the research environment in which students work (i.e., the diversity of environments to which they are exposed, their involvement in core research activities, interactions with other researchers).
At the doctoral level, where award impacts could be documented, CGS and related awards produced the same unintended impacts in all areas except those associated with total income, debt and working for pay. Since the value of the CGS doctoral award is two-thirds higher than that of regular scholarships, it should come as no surprise that CGS impacts students' finances. At the doctoral level, as was the case at the Master's level, the branding of CGS as superior scholarship does not appear to produce the unintended impacts we studied.
Cost-effectiveness and alternatives
This study was limited in its ability to gather objective evidence on program cost-effectiveness. Overall, no significant case was assembled either way.
Agencies have limited levers they can use to improve the supply of HQP. Scholarships and research grants (which translate into indirect support to students) appear to be the two most direct available approaches. Indirect approaches such as general support to research, excellence and indirect costs could contribute to the objective, but in a way that is less obviously tied to the end result.
Indirect support through grants produces outcomes that are different from the objectives of CGS and from the impacts associated with awards. Indirect support generally has a more positive impact on the students' involvement in research, on the diversity of research environments to which the student is exposed, and on interactions with other researchers. While indirect support is associated with higher student income (than the absence of direct and indirect support), it doesn't reduce students' reliance on paid work the way awards do.
Indirect support has no measured impacts on incentives to enrol in graduate studies, recognition of the federal government's financial support for research training, high-quality research, and the branding of Canada as a home of research excellence. In a complex world, where diversity is an efficient strategy to address issues, Indirect support through grants has a role to play in parallel with award programs. Agencies would be warranted to continue studying how to best integrate direct and indirect support in their portfolios.
Design and delivery
The analysis of design and delivery issues has not uncovered major concerns. Positive features of CGS include: the coverage of the Master's level, the assessment criteria, the application review process and the efficiency of the management of the program.
The large value discrepancy between CGS-D and regular doctoral awards was identified as a bone of contention. One critic of CGS wrote: "The first and most striking problem with the CGS program is that the doctoral scholarships are far too rich [...]. In contrast, some CGS winners, when you add in their teaching assistantships and top ups provided by some universities, will earn more than $50,000 annually. This is more than post docs, sessional lecturers and many assistant professors make [...]." (Siler, 2004) Additionally, this study demonstrates that, at the doctorate level, providing 67% more funding (the difference between the $35,000 CGS award and a typical $21,000 regular agency award) produces limited incremental impacts.
Recommendation 3. The Agencies should consider the possibility of reducing the gap in value between CGS and regular awards at the doctorate level.
The duration of the awards is considered too short by many. In fact, recipients are more satisfied with the money value of the awards than with their duration. The average time to completion of a Master's degree is certainly longer than one year and that of a doctoral degree vastly exceeds three years (six years according to Gluszynski and Peters, 2005). To truly affect the duration of graduate studies, it is likely that a more sustained funding effort is required.
Recommendation 4. The Agencies should consider the possibility of extending the duration of a Master's award to two years and that of a doctoral award to four years.
Of course, doing so would reduce the number of individuals who could be funded. A proper balance should be found between reducing the value of CGS awards and lengthening the period of student support.
The final problem identified is the set of rules concerning use of CGS outside of Canada. Decisions announced in the 2008 Budget open the door to CGS recipients receiving additional funding for short foreign study stints. Still, the logic behind the existing rules is that those who study in Canada are more likely to initiate their career in Canada afterwards. While this may be true, the importance of international networks and of learning under world-renowned researchers should not be understated. Allowing some of the best Canadians students to pursue graduate training outside of Canada could also translate into some of the best foreign students wanting to study with world-class Canadian researchers. For reference, about one-half of graduate students surveyed for this study considered international mobility important to their graduate education.
Recommendation 5. The award programs should not restrict the international mobility of students.
The distribution of CGS funding among Agencies decided by Government is different from that of other types of funding to Agencies: CGS funds are distributed according to the number of graduate students in disciplines associated with each agency whereas other funding does not follow this logic. This is not to say that the logic is faulty; it is a simple observation that it is different.
External communications from the Agencies should be adjusted to reflect the branding of CGS as an exceptional scholarship. As it stands, readily available information on CGS is scattered and factual rather than centralized and inspiring of excellence. Since this is a single program with a single name and a single purpose (albeit delivered by each Agency to its own constituency), an integrated external promotional presence with a single Web point of access should be envisaged. This will require a collaborative effort from the three Agencies.
Recommendation 6. The Agencies should develop an integrated external communication plan for CGS to contribute to its branding as an exceptional award.
A performance monitoring plan exists for CGS. It comprises a set of indicators associated to outputs and short term outcomes. Existing data management systems collect information relative to a small number of these indicators; they are all related in one way or another to the number of scholarships funded. Some other indicators have received some attention to date from NSERC but little to none from CIHR and SSHRC; they relate to degree completion and professional achievements. These latter indicators require that measurement be made some time after degree completion, which adds to the difficulty of obtaining the information.
Recommendation 7. The Agencies should collaborate to develop a workable data collection mechanism for performance information.
While the indicators of performance found in the performance monitoring plan may have been the appropriate ones at the inception of the program, it is not clear that they are still the most useful pieces of information for program managers. Also, there is no definitive indication that existing performance information has been used in managing the CGS program. That could be because program managers now need a different type of performance information than they did at program inception. Finally, some of the proposed indicators relate to outcomes that are located so far in the future (e.g., program completion) as to make it questionable whether they are performance monitoring indicators or evaluation indicators.
Recommendation 8. The performance monitoring plan for CGS and related programs should be revisited with a view to make it more pertinent to program managers and to better delineate performance monitoring from evaluation assessment.
- Date modified: