| Table 1: Evaluation matrix - CIHR HIV/AIDS CBR Program |
| Issues/questions |
Indicators |
Data sources/methods |
| Rationale/relevance |
1. Are the objectives of the HIV/AIDS CBR Program relevant to:
- The Federal Initiative (FI) to Address HIV/AIDS in Canada?
- The Strategic Plan for the CIHR HIV/AIDS Research Initiative?
|
- Alignment between the objectives of the FI and funded projects
- Alignment between the objectives the CIHR Strategic Plan and funded projects
- Key informant opinion about the alignment between the Program and the FI/CIHR Strategic Plan
|
- Document review
- File review
- Key informant interviews
|
| 2. Is there a continued need for the HIV/AIDS CBR Program and each of its funding streams and funding tools? |
- Communities and academia are submitting proposals for funding (by stream and funding tool)
- Trend over time in the number of proposals submitted (by stream and funding tool)
- Evidence of complete commitment of funding (by stream and funding tool)
- Key informant opinion about the consequences of funding not being available
|
- File review
- Key informant interviews
- Stakeholder consultation
|
| 3. What community-based research needs, if any, is the Program unable to meet? How could the Program be adapted to meet these needs? |
- Evidence of qualified proposals being denied funding due to budget limitations
- Key informant opinion about the appropriateness of the available suite of funding tools
- Key informant opinion about the appropriateness of the types of projects being funded/
- Key informant opinion on unmet research needs
- Key informant suggestions for modifications to the suite of available funding streams and tools
|
- File review
- Key informant interviews
- Stakeholder consultation
|
| Design/delivery |
| 4. How is the HIV/AIDS CBR Program promoted? Have the promotional activities been effective in reaching relevant stakeholder groups? What other methods could be used to promote the Program? |
- Identification of promotional activities undertaken
- Role and effectiveness of Community-Based Research Facilitators in promoting the program
- Key informant perceptions of awareness of relevant stakeholder groups (Aboriginal communities, academics, HIV/AIDS organizations)
- Key informant perceptions of effectiveness of promotional activities (by stream and funding tool)
- Key informant suggestions for other promotional activities
|
- Document review
- Key informant interviews
- Case studies
- Stakeholder consultation
|
| 5. How effective is the design of the HIV/AIDS CBR Program in promoting community involvement in all stages of the research process? |
- The level of community involvement in each stage of the research process
- Key informant perceptions of the barriers to community involvement
- Equality of partnerships between community and academia
|
- Document review
- Key informant interviews
- Case studies
- Stakeholder consultation
|
| 6. Is the application process accessible to communities and academia? What improvements, if any, should be made to the application process? |
- Definition of application processes
- Applicants' experience completing registration and application forms
- Ability of applicants to provide the required information (e.g., completion of CVs, provision of letters of community support, provision of letters from collaborators)
- Key informants' perceptions of the quality and completeness of applications
- Key informant suggestions for modifications to the application process
|
- Document review
- Key informant interviews
- File review
- Case studies
- Stakeholder consultation
|
| 7. How effective is the HIV/AIDS CBR Program review process? |
- Community participation in Merit Review Committees
- Provision of feedback to applicants
- Helpfulness of feedback
- Clarity of criteria/guidelines for judging proposals
- Achieving appropriate balance between scientific merit and potential impact
- Reasons for rejected proposals (by stream and funding tool)
|
- Document review
- Key informant interviews
- File review
- Case studies
- Stakeholder consultation
|
| 8. To what extent are approved applicants able to meet the conditions required for the release of program funds? What assistance, if any, do applicants need in meeting these requirements? |
- Definition of requirements for release of funds
- Applicants' experience fulfilling the requirements for release of funds (e.g., obtaining ethics approval, completing budget forms, submitting revenue information)
- Key informants' suggestions for additional assistance that could be provided to applicants
|
- Document review
- Key informant interviews
- File review
- Case studies
- Stakeholder consultation
|
| 9. Are applicants meeting the specified reporting requirements? To what extent do the reporting requirements facilitate performance monitoring? |
- Definition of reporting requirements
- Evidence of applicants submitting the required reports (i.e., interim reports, final reports, and financial reports)
- Quality of reports submitted
- Key informants' descriptions of how project reports are used to monitor performance
|
- Document review
- Key informant interviews
- File review
- Case studies
- Stakeholder consultation
|
| 10. How does the CIHR CBR funding approach compare to approaches taken by other CBR programs? |
- Review of evaluations of other CBR programs
- Stakeholder experience with other funding programs
|
- Document review
- Key informant interviews
- Stakeholder consultation
|
| Success/impacts |
| 11. What are the perceived outputs, outcomes, and impacts of the research that this program funds? |
- Stated outputs, outcomes, and impacts of individual research projects
- Key informant perceptions of program outputs, outcomes, and impacts
|
- File review
- Key informant interviews
|
| 12. What progress has the HIV/AIDS CBR Program made toward delivering immediate and intermediate outcomes? |
- Reported outcomes from funded projects (by stream and funding tool)
- Key informant perceptions of progress
|
- File review
- Key informant interviews
- Case studies
|
| 13. To what extent has the HIV/AIDS CBR Program increased stakeholder involvement in research? |
- Level of partner involvement in projects
- Key informants' assessment of the strength of relationships between communities and academia
- Description of how funded projects have enhanced community research capacity
|
- File review
- Key informant interviews
- Case studies
|
| 14. To what extent have the results of CBR research been shared with stakeholders? |
- Description of the steps funding recipients have taken to share research results with communities (by stream)
|
- File review
- Key informant interviews
- Case studies
|
| 15. To what extent have the results of CBR research enabled communities to address issues based on evidence? |
- Key informants' perceptions of how communities are using the results of CBR research to address issues
|
- Key informant interviews
- Case studies
|
| 16. To what extent would this research have been conducted without the HIV/AIDS CBR Program? |
- Key informant knowledge of alternate funding sources
|
- Key informant interviews
- Stakeholder consultation
|
| 17. What changes can be made to improve the Program's performance or likelihood for success? |
- Key informants' suggestions for program improvements (by stream and funding tool)
- Identification of factors leading to failure to deliver on expected results
|
- Key informant interviews
- Stakeholder consultation
|
