Outreach Evaluation - NASMHPD

1 Articles Evaluating community Outreach Efforts: A. Framework and Approach Based on a National Mental Health Demonstration Project Brenda M. Joly University of Southern Maine Martha Elbaum Williamson University of Southern Maine Kimberly Pukstas Bernard Multnomah County Oregon Prashant Mittal University of Southern Maine Jennifer Pratt University of Southern Maine Background: community health efforts often Intervention: community Outreach . include Outreach activities designed to increase awareness and ensure uptake of services or Research Design: Two existing published programs. Yet, few comprehensive Outreach frameworks are applied to assess community evaluations exist, particularly those designed to Outreach . The Evaluation design included three improve access to and use of mental health components: 1) process, 2) outcomes and 3).

2 Services. context. Purpose: This article summarizes the use of two Data Collection and Analysis: Qualitative data established Evaluation frameworks and details an were based on focus groups and key informant approach to assessing Outreach that may have interviews. Quantitative data were gathered broad appeal to administrators, social workers, through Evaluation surveys, tracking forms and health educators, community organizers, and other Outreach protocols. The findings focus on others interested in exploring the results of their lessons learned that may have applicability to efforts. others interested in evaluating community Outreach efforts in areas beyond mental health. Setting: Multi-site national study conducted in five states. Keywords: community Outreach ; Evaluation .

3 Evaluation framework; mental health Journal of MultiDisciplinary Evaluation , Volume 8, Number 17 46. ISSN 1556-8180. January 2012. Brenda M. Joly, Martha Elbaum Williamson, Kimberly Pukstas Bernard, Prashant Mittal, and Jennifer Pratt C ommunity Outreach is often defined as interactions between community members and individuals representing Outreach , if undertaken at all, is often subsumed under broader program Evaluation initiatives. Furthermore, there organizations within that community . is relatively little research on the effects of Outreach is a common strategy for Outreach programs in improving access to engaging participants in social service, and use of mental health services (Zin, public health and mental health efforts. Meng, Rajeev, Fones, & Pin, 2009). The Social workers, community organizers, purpose of this article is to provide a and health educators are frequently framework and approach for evaluating involved in planning and implementing community Outreach efforts based of a Outreach activities in a variety of areas national demonstration project focused including cardiovascular disease on early identification and treatment of prevention (Balcazar, Alvarado, Hollen, serious mental health conditions.)

4 The Gonzalez-Cruz, & Pedregon, 2005; Grigg- article focuses on the Evaluation design, Saito, Och, Liang, Toof, & Silka, 2008), questions and methods as well as the mental health treatment in the elderly lessons learned during the planning and (McGovern, Lee, Johnson, & Morton, implementation phases. Results of the 2008) , HIV/AIDS prevention (Dorabjee Outreach efforts in achieving program et. al., 2004), cancer and health goals are presented elsewhere. disparities (Meade, Menard, Martinez, &. Calva, 2007), childhood vaccination Description of the Program (Findley et al., 2008), breastfeeding (Mitra, Khoury, Carothers, & Foretich The Early Detection and Intervention for 2003), and asthma (Primomo, Johnston, the Prevention of Psychosis Program DiBiase, Nodolf, & Noren, 2006), to name (EDIPPP) is a national demonstration a few.

5 Outreach activities are often used to project funded by the Robert Wood increase awareness, provide access to Johnson Foundation that was designed to services, and recruit individuals to identify, intervene early, and provide participate in community activities, treatment for young people between the research projects or health promotion ages of 12 and 25 who are at risk for a interventions. Outreach programs have psychotic episode in order to prevent the also been used by non-traditional sectors development of a severe mental illness. In to encourage community engagement 2006, five sites across the country each (Vanclay, Lucas, Lane, & Wills, 2007). received a two million dollar grant to Federal demonstration projects have long participate in this four year research acknowledged and incorporated Outreach project.

6 The grantees included: as an official project element dating back to the mid-1960s (Leviton, & Schuh, Maine Medical Center (also the 1991). Yet, while the community Outreach National Program Office - NPO), model is not a new approach, there appear Portland, Maine to be few evaluations of Outreach efforts. Mid-Valley Behavioral Care Despite Outreach being a major strategy in Network, Salem, Oregon many community -level interventions, University of California, Davis, there are few published Evaluation Sacramento, California frameworks in this area. Evaluation of Journal of MultiDisciplinary Evaluation , Volume 8, Number 17 47. ISSN 1556-8180. January 2012. Brenda M. Joly, Martha Elbaum Williamson, Kimberly Pukstas Bernard, Prashant Mittal, and Jennifer Pratt Washtenaw community Health Evaluation theory work of Shadish, Cook Organization, Ypsilanti, Michigan and Leviton (1991).

7 This seminal work Zucker Hillside Hospital, Queens, included four theoretical Evaluation New York components: use, program, value and knowledge construction. Based on these The program included a clinical theoretical constructs, Ottoson and Green component with a specialized team used (2005) develop a series of four questions to assess referrals and provide treatment to guide the Evaluation of community - including multi-family group therapy. based Outreach efforts. community Outreach was the primary strategy to identify potential research 1. How will Evaluation findings be participants and generate timely referrals used and by whom? into the treatment program. Outreach 2. What is Outreach and what factors efforts targeted school personnel, social may affect influence Outreach ?

8 Workers, doctors, nurses, students, 3. How will the success or failure of parents, clergy, police officers, and others Outreach be determined? who interacted regularly with young 4. What methods will be used to people. assess the success of Outreach ? Brief Description of Outreach In addition to this theoretical Evaluation framework, we also A detailed description of the Outreach incorporated the public health program model has been reported elsewhere (Ruff, Evaluation framework developed by the McFarlane, Downing, Cook, &Woodberry, Centers for Disease Control and unpublished). In general, community Prevention (CDC, 1999). The CDC. Outreach included: 1) formal training framework informs Evaluation practice sessions for professionals ( , school and is composed of six steps that should personnel, healthcare providers) and the be taken in any public health program public on the early warning signs of Evaluation : 1) engage stakeholders, 2).

9 Psychosis, the EDIPPP initiative and how describe the program, 3) focus the to make a program referral, 2) informal Evaluation design, 4) gather credible communication about EDIPPP, 3) a evidence, 5) justify conclusions, and 6). mapping exercise designed to identify ensure use and share lessons learned. community partners, 4) the development Our Evaluation was informed by both of an Advisory Board, and 5) the frameworks. Table 1 provides a brief dissemination of EDIPPP material. description of these approaches and examples of activities undertaken for each Methods element within the frameworks. In the discussion below, we present several examples, organized by Ottoson and Application of Existing Evaluation Green's (2005) translation of Evaluation Frameworks theory based on the four components for evaluating community Outreach .

10 In the The framework for this Evaluation was narrative for each component, we also informed by Ottoson and Green (2005) discuss key elements of the CDC. who based their approach on the seminal Journal of MultiDisciplinary Evaluation , Volume 8, Number 17 48. ISSN 1556-8180. January 2012. Brenda M. Joly, Martha Elbaum Williamson, Kimberly Pukstas Bernard, Prashant Mittal, and Jennifer Pratt framework that were used in the Evaluation . Table 1. Application of Existing Evaluation Frameworks Ottoson and Green CDC Framework Example Evaluation Activities Framework (2005) (1999). How will findings be Engage stakeholders Determined priority audiences used and by whom? Justify conclusions Engaged in a participatory approach: Ensure use and Solicited input from stakeholders share lessons Established routine communication learned Generated buy-in for data collection Reached agreement on: Purpose of Evaluation Elements to be evaluated Used a mixed-methods design and triangulated data to develop conclusions and recommendations Developed user-friendly reports: Assured data would help fulfill reporting requirements of grantees What is Outreach Describe program Designed Evaluation to focus on three and what factors Focus the Evaluation components: may influence design Process, context, outcome Evaluation Outreach ?