Summative Evaluation of the Regional Partnerships Program (RPP) - Final Report: Part 3

Return to Table of Contents

RPP Usage
Views of the "Local" Programs
Within Region Impact, Strengths and Weaknesses
Building Research Capacity
Views On RPP Continuation

RPP Usage

RPP makes available to each RPP Advisory Committee an annual allotment within which authorized expenditures in a target year may be used for approved projects. The two key factors that will influence the number of projects in a given year are project size and project length. As the project size rises, the number of projects funded will decrease. As the project length increases, the number of projects will decrease. Two categories of projects are recognised: awards (generally these are training awards) and grants (generally, these are operational grants).

The rules of RPP do not permit the carry-forward of an unused portion of an allotment from one year to the next.¹⁷ It will be recalled that, as of 1999, the allotments to each Advisory Committee were either $200,000 per year (PEI, NB) or $1M per year (NL, NS, MN, SK).

Of the many different approaches that are available for the use of an allotment of $1M, two extreme examples will be described. One approach is to fund one project of $1M/year for three years. Another approach is to fund 100 projects at $10,000/year and for three years each. Both of these approaches will consume all of the allotment for three years. After the first year, and for the next two years, no additional researchers would be funded ("recruited and/or retained"). Another approach would be to restrict projects selected to those of small budget and one year duration. This third approach would maximize the numbers of researchers supported but would restrict support to small projects able to be completed in one year. There are, of course, many other variations of these approaches. Table 6, for awards, and Table 7, for grants, describe the results of the several approaches that were applied by the six Advisory Committees.

The following tables show the usage of the allotments each year for all six Advisory Committees combined. The maximum that may be spent each year, for awards and grants combined, is $4,400,000. The benefit in terms of health research expenditures will be at least double the CIHR contribution given the requirement of partner dollars to at least match CIHR dollars.

As Table 8 shows, by 2001-2002 the usage had begun to approach the maximum available. Table 8 also shows that 80% of the dollars were used for grants in 1999-2000 and this decreased gradually to 67% by 2004-2005.

Table 6: Number and Size of Awards by Year - All RPP Provinces

Awards	1999-2000	2000-2001	2001-2002	2002-2003	2003-2004	2004-2005
Total CIHR Award-$ in Year	$295,194	$820,074	$1,054,020	$1,159,189	$1,268,062	$1,297,169
# of Awards in Year	17	34	40	42	45	55
Avg. CIHR Contribution per Award per Year	$17,364	$24,120	$26,351	$27,600	$28,179	$23,585

Table 7: Number and Size of Grants by Year - All RPP Provinces

Grants	1999-2000	2000-2001	2001-2002	2002-2003	2003-2004	2004-2005
Total CIHR Grant-$ in Year	$1,182,958	$2,111,893	$2,652,779	$2,571,588	$2,282,594	$2,682,087
# of Grants in Year	56	83	86	93	74	81
Avg. CIHR Contribution per Grant per Year	$21,124	$25,444	$30,846	$27,651	$30,846	$33,112

Table 8: Number and Size of Projects by Year - All RPP Provinces

Awards + grants	1999-2000	2000-2001	2001-2002	2002-2003	2003-2004	2004-2005
Total CIHR Award Grant $ in Year	$1,478,152	$2,931,967	$3,706,799	$3,730,777	$3,550,656	$3,979,256
Percentage of $ Used for Grants	80%	72%	72%	69%	64%	67%

Tables 9, 10 and 11 present analyses of the amount of partner funding per project in relation to the amount of CIHR contributions of the same project. The column headings in these two tables use abbreviations for column labels and these have the meanings as shown in the following table:

P < C	P/c=1.00	P/c=1.01-.10	P/c=1.11-.25	P/c=1.26-2.	P/C>2.00
Partner < CIHR	Partner / CIHR = 1.00	Partner / CIHR = 1.01 - 1.10	Partner / CIHR = 1.11 - 1.25	Partner / CIHR = 1.26 - 2.00	Partner / CIHR > 2.00
The Partner funding is less than the CIHR contribution	The Partner funding divided by the CIHR contributions is equal to 1.00	The Partner funding divided by the CIHR contributions equals between 1.01 and 1.10	The Partner funding divided by the CIHR contributions equals between 1.11 and 1.25	The Partner funding divided by the CIHR contributions equals between 1.26 and 2.00	The Partner funding divided by the CIHR contributions is greater than 2.00

Table 9: Comparison of Partner and CIHR Funding of Awards - All RPP Provinces

Awards	P < C	P/c=1.00	P/c=1.01-.10	P/c=1.11-.25	P/c=1.26-2.	P/C>2.00	Total
Number	10	47	4	0	10	2	73
Per Cent	14%	64%	5%	0%	14%	3%	100%

Fully 22% of the awards received partner funding in amounts larger than the CIHR contributions.

Of the 10 awards in which the Partner funding is less than the CIHR contribution, one essentially had proper partner funding since the shortfall was only 1%; four were terminated due to partner shortfall and five are in progress with time remaining in which the partner may fulfil his obligation:

One project had a partner shortfall of only 1%

Insufficient partner funds

Two projects are complete; partners under-funded
One project ended in 2002-2003 with partner shortfall
One project (2000-2001 to 2003-2004) did not receive any partner funds

Partner funds lag CIHR contributions

Five projects started in 2004-2005; partners have time to contribute

Table 10: Comparison of Partner and CIHR Funding of Grants - All RPP Provinces

Grants	P < C	P/c=1.00	P/c=1.01-.10	P/c=1.11-.25	P/c=1.26-2.00	P/C>2.00	Total
Number	25	144	3	4	6	41	223
Per Cent	11%	65%	1%	2%	3%	18%	100%

Fully 24% of the grants received partner funding in amounts larger than the CIHR contributions.

Of the 25 grants in which the Partner funding is less than the CIHR contribution, 12 were terminated with insufficient partner funding and 13 are in progress with time remaining in which the partner may fulfil his obligation:

Insufficient partner funds

Two projects ended in 2000-2001 with partner shortfalls
Five projects ended in 2002-2003 with partner shortfalls
Five projects started in 2002-2003; each of the 5 have received approximately $5,000 from CIHR; none have received partner funds; all stopped after the year

Partner funds lag CIHR contributions

Two are projects that span 2002-2003 to 2004-2005; there remains time for partners to complete funding;
One project is for the years 2002-03 to 2005-06; the partner has lagged by one year with the CIHR contribution
10 projects started in 2004-2005; they are all within the time frame for partner funding

Table 11: Comparison of Partner and CIHR Funding of Projects - All RPP Provinces

Awards+ Grants	P<C	P/c=1.00	P/c=1.01-.10	P/c=1.11-.25	P/c=1.26-2.	P/C>2.00	Total
Number	35	191	7	4	16	43	296
Per Cent	12%	65%	2%	1%	5%	15%	100%

Only 16 projects (4 awards -Table 9 and 12 grants -Table 10), from a total of 296 projects, have a confirmed partner shortfall. Given the complexity of the arrangements for partner funding, this is not an unreasonable occurrence.

Views of the "Local" Programs

In each of the RPP regions, in-person discussions took place with Advisory Committee members, researchers, government officials and managers with experience directing, managing, administering and conducting research. Their views are brought together here under three headings: within region impact, strengths and weaknesses; building research capacity; and views on RPP continuation.

The presentations under these headings span the full program. There are differences from one region to another, both with respect to the characteristics of the regions and the views of the professionals in those regions. These may be examined in detail in the working paper that presents a profile for each of the six regions.¹⁸

Within Region Impact, Strengths And Weaknesses

The summary position from the regions is that RPP has been a success. In all regions it is viewed as contributing to the building of health research capacity. Many secondary benefits are identified as well as the direct impacts of helping researchers start or continue their work as excellent scientists in smaller-population provinces. The Program design is judged to be very good. Singled out as particular strengths are the peer-review system, the stimulus to multi-stakeholder collaboration and the flexibility to tailor the program to the province. The major disappointment is level of funding.

The nature of RPP is that of an "antibiotic," a corrective that should cease to be needed at a certain point. It is not a "nutritional supplement" that ought to be administered indefinitely. Nor is it a "narcotic" that creates a habit that cannot be broken. (The program criteria prevents researchers from holding successive RPP grants to insure against a 'narcotic' effect.)

RPP interviewees are very positive about RPP's impact. They say that:

RPP is a needed source of funding; in some cases it has saved labs and projects from disappearing.
RPP motivates new researchers to come to the province.
RPP helps newer researchers to build confidence that they can succeed in major competitions; some need to learn the 'CIHR culture'.
RPP motivates seasoned researchers to continue (it is a morale booster for them as well).
RPP allows young researchers to get started before they have substantial data and impressive CVs.
RPP allows multi-year projects to finish or to bridge to the next full-funding success.
RPP allows each region to pursue its own priorities for health research.
RPP supports a teaching climate that keeps medical students in the province, making it more likely for them to remain as practitioners in the province.
RPP motivates funding partners who take note of the CIHR investment.
RPP stimulates collaboration among health research and other stakeholders, including the provincial government and foundations.
RPP has spawned special offices and new research infrastructure in several provinces.
the numbers of health researchers in RPP provinces are growing.
the success rate in all CIHR competitions is rising, reflecting rising average scores from provincial researchers.

Perceived strengths of the RPP:

An essential feature of RPP is its utilisation of the same peer-review system as all other aspects of CIHR. While there are some inadequacies, RPP is the best possible system for estimating scientific excellence. This was seen to have two important consequences: RPP funds highly regarded science and RPP-supported researchers safeguard their reputations as excellent scientists.
The planning and other requirements of RPP have encouraged health research planning, collaboration and integration on the provincial level. A variety of stakeholders come to the table for collaborative discussions. They benefit from the synergy in other ways besides the focus on RPP.
There is no barrier to integrating RPP into the other provincial efforts to support health research.
The creation of provincial foundations has served to strengthen the links between the provincial government and health research activities.
The national meetings of Chairs are useful when they occur.
RPP processes have encouraged the development of services such as internal peer review, inclusion of RPP in faculty orientation programs, mentoring, proposal preparation assistance and other supports.
RPP flexibility is appreciated because it permits for funding, such as a lower-scored proposal to build capacity in a priority area or the setting of a cap on the funding for individual projects in order to draw more individuals into health research.

Perceived limitations of the RPP:

The level of RPP funds is inadequate. The funding is insufficient to support strategic projects, which are becoming increasingly important (especially for Pillars 3 and 4 research that is of particular interest to some provincial authorities).
Researchers who have to find matching funds themselves can find RPP very challenging, given that RPP provinces have less wealth available. It can take until the last possible moment to find the match, creating stress for investigators and disrupting employment plans.
There should be a way within budgetary rules to allow for flexibility across years - ability to move funds across years.
RPP province proposals suffer competitive disadvantages compared with those from elsewhere because they automatically lose points for "research environment." Some researchers also think they detect a decision bias.
The allegation is made that reviewers sometimes rank proposals from RPP provinces just below comparable proposals from the four non-RPP provinces on the basis of an estimate on where the cut-off for full funding will appear. Given two comparable proposals, the one from the non-RPP province proposals will be rated higher and get full funding leaving the RPP-province proposal to be funded through RPP plus partner money. This approach by reviewers is said to neglect two fundamental issues: the larger provinces have locally controlled and focussed alternative sources that are well endowed; and finding the matching funds in some RPP provinces has been difficult.

Building Research Capacity

RPP provinces see themselves at a general disadvantage because of smaller health research capacity than the four larger provinces. The two major factors that RPP has been ameliorating are (1) generous provincial funding sources in the larger provinces and (2) disadvantages that proposals by researchers in RPP provinces face in CIHR competitions. Interviewees appreciate the need to specify targets for what would constitute "sufficient health research capacity" - which, once attained would mean that RPP is no longer needed - and have suggestions for defining it.

None of the RPP provinces consider themselves to yet have sufficient and sustainable health research capacity. Both "sustainable capacity" and "critical mass" emerged as key concepts during the evaluation for justifying the program and characterizing its concrete goals.

Health research capacity includes the following features:

The number of researchers, age distribution of researchers and turnover are such that there is a stable research population. Actual numbers of researchers may be more important than percentages (e.g. proportion of researchers in the entire provincial population) because capacity is seen to be at risk where the numbers of researchers is small.
Sophisticated equipment and services in close proximity.
Stable funding is available from sources dedicated to that province.
Research groups comprised of three or more investigators (or groups of researchers) are working in the same subject area.
Development support to researchers such as mentoring and help in the preparation of proposals is available.
Relationships exist with colleagues within and beyond the province even though this puts a major strain on travel budgets.
Appropriate laboratories exist.
There is a provincial Health Research Foundation or other concrete provincial involvement.

The common perception is that researchers in the non-RPP provinces have recourse to very well endowed funding sources such as major provincial foundations and large hospital research foundations. Until the RPP provinces find themselves in the same position - which means a combination of provincial wealth accumulation and policy decisions (both public and private) to dedicate part of that wealth to stable funding of health research - researchers in the RPP provinces feel they require assistance to compensate for the low levels of in-province funding.

Interviewees in RPP provinces feel that their relatively lesser research capacity results in research proposals from the four non-RPP provinces receiving higher scores in peer reviews than do proposals of equal scientific merit from a RPP province. The following factors are cited:

points for research environment - having colleagues close by with whom to consult, having sophisticated equipment and services in close proximity;
familiarity - more reviewers come from larger provinces and they are more confident in the people and research environments with which they are familiar;
limited budgets - if CIHR fully-funded all excellent research, according to its own definitions of excellence, then excellent proposals from all locations would be funded and review panels would not have to make the sorts of fine distinctions that are required in the current circumstances.

Advisory Committees may opt to steer funding so as to favour capacity building explicitly, for instance, a small proportion of RPP set aside for doctoral and post-doctoral students, and a focus on research groups.

Views On RPP Continuation

The RPP should definitely continue. The RPP provinces depend on it to attract new researchers and get them started. However, the Advisory Committee Chairs wish to participate as true partners in deliberations about any changes. The key points to discuss are the goal, how to reach it, and how to recognize that the goal has been achieved. The major changes suggested, besides consultations, are increasing the funding allotment and improving communications.

CIHR is asked to look at RPP as a strategic tool towards fulfilling its country-wide mandate. Terminating RPP would be a major blow to new researchers who benefit from RPP when getting started and who lack the alternative funding sources that are available in the non-RPP provinces. If RPP was to end, the action would also cast doubt on CIHR's intention to support health and health research across the country. The following suggestions were offered during the visits to the RPP provinces:

Keep the basic features of RPP - the peer review system, the emphasis on excellence, the matching funding, and no successive RPP grants. Continue to allow for local flexibility within the general rules in order to respect regional differences.
The present 50:50 match is widely accepted with a minority view that CIHR provide full funding to RPP research projects.
Consult with local stakeholders when designing the next phase of the program. Several Chairs noted that the national RPP meeting in Charlottetown, March 2004, struck the right collaborative note.

There is agreement that the following issues should be included in the agenda for planning for a continuation of RPP:

The wording by which to define the central objective for RPP - currently understood to be the development of sustainable capacity for health research.
The selection and definition of indicators by which to measure progress toward the objective.
Means for assuring the engagement of provincial governments in support of the RPP objective.
The size of the allotment to each region. The general view is that doubling the budget would have a significant positive effect and can be matched locally. Health research dollars in a smaller province, with a smaller initial base, can have a much greater impact than in one of the four larger provinces. Since excellence requires support, there would be significant growth of excellence in these provinces. Among the justifications for increasing the RPP allotment were:
- the increasing costs of doing science generally;
- the move towards larger operating projects and larger research teams;
- the increases in other CIHR programs;
- the desire to include some strategic grants
Methods for improving communications:
- There was agreement that the annual meetings of the chairs of Advisory Committees and other key individuals should be re-instituted, to foster the sharing of experience.
- There was a desire to improve processes for communicating results and public announcements. CIHR should work more closely with its partners to coordinate communications. Announcement processes are too complicated or too driven by intentions to have specific people present rather than just represented or are not designed to give equal prominence to both the regional and the federal roles.

¹⁷ At the start of RPP, there had been an expectation that such carry forward would be permitted but a mechanism was not found by which to achieve this.
¹⁸ Available as a separate document from CIHR. Scheduled for completion in June 2005.

[Table of Contents]
[1] [2] [3] [4]
[Appendix A]
[Appendix B]