Executive Summary

[ Table of Contents ]

Objective of the evaluation

The Canadian Institutes of Health Research (CIHR) hired PRA Inc. to evaluate the Human Immunodeficiency Virus (HIV)/ Acquired Immunodeficiency Syndrome (AIDS) Community-Based Research (CBR) Program and its funding tools. The purpose of the evaluation is to assess the rationale for the program, the effectiveness of its design and delivery processes, and its successes. The evaluation draws on five lines of evidence: a document review, a file review, key informant interviews, case studies, and stakeholder consultations. The CBR Steering Committee and CIHR will use the results of the evaluation to strengthen the program and maximize its impact on Canada's response to the HIV/AIDS epidemic.

Findings

Relevance

In 2005, the Government of Canada launched the Federal Initiative to Address HIV/AIDS in Canada (FI), which works toward "a Canada free from HIV and AIDS and the underlying conditions that make Canadians vulnerable to the epidemic" (Minister of Public Works, 2004). It emphasizes community action to address the HIV/AIDS epidemic, which remains a major public health issue. CIHR's HIV/AIDS CBR Program is one of the FI's component programs.

Key informants believe that the HIV/AIDS CBR Program is a flagship program that incorporates the policy direction of the FI. It encourages partnerships between communities, academia, and policymakers; involves people living with HIV and their communities in responding to the epidemic; builds research capacity in communities; and gives communities the evidence they need to address the issues they face.

Program awareness and promotion

Key informants and participants in the stakeholder consultation believe that academic institutions and large, established AIDS service organizations (ASOs) are well aware of CIHR's HIV/AIDS CBR Program, but they also commented that the program's reach is limited. They pointed to a lack of awareness about the program among small and rural organizations; organizations whose primary mandate is not HIV/AIDS, but address some HIV/AIDS needs (e.g., organizations addressing co-infection and determinants of health); and academic departments engaged in cross-disciplines.

Key informants and participants in the stakeholder consultation support CIHR's current promotional activities, which include posting program information and Request For Applications (RFAs) on its website; emailing communiqués and RFAs to previous applicants and other stakeholders; and attending and making presentations at HIV/AIDS conferences. However, they also offered several suggestions for supplemental CIHR promotional activities, including having a presence at community meetings and small HIV/AIDS conferences; conducting proposal writing and research implementation workshops; reaching out to organizations that may not think of themselves as HIV researchers; and making plain-language (non-academic) information about the program available to communities. Regardless of the program promotion methods used, key informants stressed that it is imperative to have continuous and multiple exposure, as there is high staff turnover in community organizations.

Top of Page

Funding tools

This suite of funding tools supports a natural progression of funding from capacity- and partnership-building to proposal development to full research projects; however, there is minimal evidence suggesting that the majority of applicants navigate through the complete continuum of funding sources.

• Catalyst and operating grants are traditional funding tools, which are well-known, especially in academic circles. Key informants and participants in the stakeholder consultation did not have any concerns with these funding tools, aside from some comments that these tools seem to be directed to academia. They also mentioned a need to increase the funding caps.
• Funding for capacity-building workshops is available to community and academic researchers to increase their basic understanding of CBR principles and practices. While key informants and participants in the stakeholder consultation did not have any concerns about the design of this funding tool, some did note that it is not well subscribed to. They were not able to identify what factors may be contributing to low uptake levels.
• Masters and Doctoral Awards are intended to build CBR capacity in academic institutions, which will increase the availability of academic partners to work with community organizations on research projects. However, some key informants do not recognize this as an intended benefit of this funding tool; a few key informants referred to it as a scholarship program for students that has no benefits for or accountability to the community.
• CBRF's primary role is to promote CBR and support the preparation of applications to CIHR and other funding sources. Some key informants and participants in the stakeholder consultation have the perception that communities and academia have limited awareness of CBRFs and many key informants indicated that the effectiveness of CBRFs relates directly to the capacity of the individual in the position.

Key informants and participants in the stakeholder consultation reported that there has been a lot of turnover in CBRFs. They explained that this reduces the effectiveness of the funding tool because each time a new CBRF starts, they must gain the community's trust and rebuild connections with potential research partners. CBRFs suggested that the staff turnover results from structural problems with the funding tool. One problem they mentioned is that CBRFs cannot act as co-investigators on CIHR applications and, therefore, do not receive formal recognition for the work they do. Another problem they identified is that CBRFs do not have enough time to determine the needs of their community, develop a work strategy, and build relationships because the funding period is only two years.

Community and academic partnerships

Projects funded through the HIV/AIDS CBR Program must demonstrate community impact and scientific rigour. To balance these two aspects, the CBR teams receiving funding through this program involve partnerships between communities and academia. For a partnership to be successful, each party must benefit from and be committed to the relationship.

Key informants cautioned that it can be difficult for communities and academia to establish effective partnerships. One reason for this is that each party may bring different priorities to the relationship. Key informants explained that often the community wants to gather information at the community level and tends to want to conduct small projects. Conversely, academics tend to want to engage in publishable research and, therefore, are often interested in large-scale projects.

Key informants noted that the community can sometimes resent having to partner with an academic researcher. They can feel coerced to partner with an academic researcher because they need a Principal Investigator (PI) who meets CIHR criteria, they need the academic institution to hold CIHR funds (because the community finds the CIHR process to become an eligible institution difficult), and/or they need to access the institution's research ethics board (REB).

Top of Page

Administrative processes

Application Process

Applying to the HIV/AIDS CBR Program is a two-stage process. Applicants must first register with CIHR and then they must submit an application for funding. Generally, applicants, other key informants, and participants in the stakeholder consultation believe that the application process is difficult and onerous, especially for communities. The following are some of the comments made about the application process:

• The application forms are difficult to complete and the collection of the required signatures is logistically challenging.
• The web forms are difficult to navigate and require a high level of computer literacy.
• The CVs are too academically-oriented and ask for qualifications that communities do not have (e.g., degrees, publications, students supervised). These challenges still remain, despite CIHR's efforts to create a community CV.

Merit Review process

Key informants and participants in the stakeholder consultation appreciate that there is a separate Merit Review Committee for each funding stream. They also commended CIHR for including a mix of community and academic members on the committees. Overall, members of the Merit Review Committees and other key informants believe that the community has clear voice in the review of applications.

Some key informants and participants in the stakeholder consultation suggested that the Aboriginal Merit Review Committee should include Elders. They said that the role of Elders should be to listen, bring guidance and wisdom, and lead prayers. They also indicated that, following Aboriginal traditions, Elders should be presented with tobacco, cloth, and honoraria for their contributions.

Key informants and participants in the stakeholder consultation identified the following challenges with the Merit Review process:

• Although both community and academics members vote on scientific merit and potential impact on the community, it can be difficult for some reviewers to separate the two. Some people believe that if the research does not have strong scientific merit, it cannot have a potential impact on the community.
• It can be difficult for the community members to assess some applications. Community members may have difficulty reviewing applications that are outside of their area of experience (e.g., Doctoral Awards).
• Some committee members said that CIHR has not established clear guidelines on how to assess some of the information included in the applications (e.g., the number of publications that a proposed team member has).
• Key informants noted that it can be difficult to recruit members for the committees. They mentioned that there is a small pool of researchers to recruit from and said it can be especially challenging to find researchers who are of Aboriginal descent and can speak French. They also reported that it can be difficult to find community members who have CBR experience.

Ethics certification

Prior to the release of funding, applicants must obtain ethics certification for their proposed project. Most commonly, projects obtain ethics certification from a university-based REB. In most cases, for communities to access these REBs, they have to partner with a university. Key informants identified several concerns with using university-based REBs to review CBR projects.

• Not all REBs will accept applications for grants where the money is not held by the university.
• University-based REBs do not understand CBR. The ethics review forms are not suitable for CBR projects, the boards ask questions that are not relevant to CBR, and the boards do not understand the responses applicants give to their questions (or think they did not answer the question).
• University REBs focus on the individual, whereas CBR focuses on the community; therefore, applicants are being asked to demonstrate that not only will the research not harm individuals, but it also will not harm the community.
• REBs do not understand Aboriginal research approaches.

Key informants noted that few organizations outside of universities offer ethics certification; those who do, may only make it available to a limited client base. Some of these REBs are only available to applicants to a particular funding source or organizations located within the same province. While some REBs may make their services available to anyone, some of the ones identified through interviews operate on a volunteer basis and, therefore, cannot accommodate a large number of applications.

Eligibility to hold CIHR funds

Key informants noted that it is difficult for communities to meet the eligibility requirements to hold CIHR funds. If community-based organizations (CBOs) want to hold the funds, it needs to demonstrate to CIHR that it has structures in place to manage the funds and that it has a research mandate. Key informants reported that CBOs that have successfully gone through the process spent more than one year demonstrating they meet all of the requirements. They asserted that making it difficult for CBOs to hold funds contradicts CBR principles.

Top of Page

Success

Key informants indicated that communities have a sense of ownership over the program and feel empowered to make a meaningful contribution to HIV/AIDS CBR. They said that the existence of the program raises the profile of CBR, creates opportunities to conduct CBR, and encourages the formation of the relationships needed to conduct the research.

Key informants reported that the program has increased community literacy in research design, methodologies, and analysis, and has successfully engaged more communities in conducting research, whether internally, through CIHR, or through other funding sources. Key informants also said that because of this program, communities are becoming more comfortable working with researchers and, as a result, the program has fostered the development of strong relationships between communities and academia. From the file review, some applicants noted that their partnerships led to subsequent research projects.

For communities to use research results to address the HIV epidemic at the local level, they need to be aware of the results. Therefore, knowledge dissemination is an important aspect of CIHR's HIV/AIDS CBR Program. Examples of knowledge dissemination methods that funded applicants have used include preparing two versions of research reports (one for academia and one for communities); conducting presentations at conferences (including some international conferences); posting results on the Internet; publishing journal articles; preparing factsheets and newsletter articles; summarizing research results; and organizing community gatherings, roundtables, and forums. Other dissemination activities mentioned in the project files include conducting radio interviews and sharing results with international government departments. Although there is evidence that knowledge dissemination activities are occurring, some key informants question their effectiveness.

CIHR's HIV/AIDS CBR Program is helping communities address the issues they are facing. Key informants reported that projects funded through the program are raising awareness of HIV/AIDS in the community, helping reduce stigma around the disease, and stimulating discussions about HIV/AIDS. They also said that through participation in these research projects, communities' awareness of the specific problems they face is increasing, they are recognizing that they have the capacity to address these problems, and they are taking steps to address these problems. Communities are using research findings in proposals for program funding, to make changes to their programs and services, and to advocate for policy change. Applicants noted using research findings to develop intervention programs and programming for small groups; inform the development of new data collection instruments; define valid research methodologies in additional research projects; and create databases of contacts and potential partners.

Need for the program

Key informants believe this is a unique and essential program for the HIV/AIDS community. There is a long history of community action addressing the epidemic. The community is best positioned to identify the research questions, define appropriate approaches for reaching stakeholders, and use the research results to implement change.

Key informants emphasized that the research conducted through CIHR's HIV/AIDS CBR Program is valuable to communities and simply would not happen if the program did not exist. Part of the reason for this is that few other funding sources are available to communities and academia for HIV/AIDS CBR.

Key informants believe that the HIV/AIDS CBR Program keeps communities engaged in CBR. They said that without the program, it would be harder for communities to get involved in research. They also suggested that any research that communities undertook would be small in scale and scope and would not involve robust partnerships with academia. They also indicated that communities would depend more on academia than they currently do to conduct HIV/AIDS research.

Top of Page

Recommendations

Based on the evaluation findings, the feedback received at the consultation, and discussions with CIHR and the CIHR CBR Steering Committee, PRA prepared a set of recommendations for the program. The evaluation resulted in the following 14 recommendations:

1. Expand efforts to raise awareness of CBR and the HIV/AIDS CBR Program by identifying the full range of HIV/AIDS CBR stakeholders and tailoring promotion, education, and capacity-building activities to meet their specific needs.
2. Clarify program eligibility requirements. This includes defining the range of projects eligible for funding as well as eligible expenses.
3. Update, restructure, and expand the CIHR CBR website to include complete program information that users can easily access.
4. Simplify the language used in the application forms and improve the usability of web forms.
5. Create a CV form (or forms) that captures relevant academic and community experience.
6. Give applicants the opportunity to have their application pre-screened for missing or incomplete information.
7. Examine the merit review process for each funding tool to determine where the process can be simplified and streamlined. Additionally, opportunities for providing a more interactive process should be explored.
8. Create a mentorship program for the Merit Review Committees to assist in developing the capacity of current members and recruiting new members.
9. Ensure the Merit Review Committee for the Aboriginal stream comprises Aboriginal people as well as regionally-appropriate Elder representation.
10. Review and revise the suite of funding tools, including:
    • the structure of and processes associated with existing tools. This includes, but is not limited to, the application and review processes, the frequency of funding cycles, and the funding caps.
    • the range of funding tools available to applicants. This includes expanding the suite of tools to include opportunities for students at all stages of their education and increasing opportunities for end of grant knowledge translation.
11. Revise the structure of the CBRF funding tool to improve its effectiveness and reduce staff turnover. In particular, increase the length of the funding cycle; institute a memorandum of understanding (MOU) between CIHR and the host organization that outlines the roles and responsibilities of and the relationship between CBRF and the host organization; and increase the number of Aboriginal CBRFs.
12. Explore opportunities to create Review Ethics Boards (REBs) for CBR.
13. Simplify the conditions that CBOs have to meet for the release of funding and develop an MOU for community organizations to hold CIHR funds.
14. Develop a performance measurement strategy for HIV/AIDS CBR and determine how project final reports can be used to monitor the performance of the program.

[ Previous Section | Next Section ]