Supporting Evidence-Based Home Visiting to Prevent Child ...

1 Supporting Evidence-Based home Visiting to Prevent Child maltreatment DECEMBER 2010 BRIEF 3. Replicating Evidence-Based home Visiting In 2008 the Children's Bureau (CB) within the Administration Models: a framework for assessing Fidelity for Children and Families by Deborah Daro (ACF) at the Department Chapin Hall at the University of Chicago of Health and Human Services funded 17 cooperative agree- The Maternal, Infant, and Early Childhood home Visiting Program, authorized by the ments to support building Patient Protection and Affordable Care Act of 2010, represents a major expansion infrastructure for the wide- for Evidence-Based home Visiting services. Over the next five years, the program will spread adoption, implemen- tation, and sustaining of provide $ billion to states to invest in selected home - based services to promote early Evidence-Based home visita- childhood health and development and, ultimately, improve outcomes and opportunities tion programs.

2 Grantees are for children and families. To maximize the return on this major public investment, the leveraging their grant funds legislation places particular emphasis on building states' capacity to assess the fidelity with other funding sources and quality of the replication and expansion of Evidence-Based home Visiting models. to implement programs with Fidelity includes adhering to a model's staff training, certification, and supervision fidelity to their evidence - based models. Grantees are requirements; delivering family-level services at the specified intensity (dosage); and also conducting local imple- covering the prescribed content. Quality refers to how effectively the content is conveyed mentation and outcome eval- to families; for example, whether the home visitor engages parents during the visit and uations.

3 CB/ACF has funded whether this engagement is evidence of a positive, trusting relationship between the Mathematica Policy Research home visitor and the parents. This brief presents a framework for monitoring fidelity and Chapin Hall at the Univer- sity of Chicago to conduct a to home Visiting program models developed as part of the Supporting Evidence-Based cross-site evaluation of the home Visiting to Prevent Child maltreatment (EBHV) initiative's cross-site evaluation. grantees' programs. This is Maintaining fidelity to a program's design is critical both for achieving effective the third in a series of briefs from the cross-site evaluation. outcomes and for taking initiatives to scale. Despite the benefits of implement- ing programs as designed, many social service models have been taken to scale For more information about without sufficient attention to fidelity.

4 Systematically monitoring implementation EBHV, including earlier evalu- ation briefs, go to: http://. can help maintain program consistency and quality and identify any need to adjust the model's protocols. Indeed, agencies often modify program standards and content to fit local participants' needs, organizational capacity, and commu- nity context. In some cases, agency staff identify changes needed to accommo- date the characteristics of their community and target population. In other cases, funding cuts or staff shortages drive the need for modifications. Although some model modifications can strengthen a program's effects, others, particularly un- planned changes, can have detrimental effects and may reduce the likelihood of achieving maximum impact. This brief reports on the fidelity monitoring system Mathematica Policy Research and Chapin Hall at the University of Chicago Replicating Evidence-Based home Visiting Models developed for the cross-site evaluation of the EBHV initiative (Boller et al.)

5 2010;. Koball et al. 2009). It provides a set of indicators state planners can use in craft- ing their own fidelity monitoring systems and assessing the implementation of home Visiting models across different communities. Defining Fidelity Researchers use several theoretical frameworks to define fidelity. In summarizing work in this area, Carroll and colleagues identified five elements of implementa- tion fidelity: (1) adherence to the service model as specified by the developer;. (2) exposure or dosage; (3) the quality or manner in which services are delivered;. (4) participants' response or engagement; and (5) the understanding of essential program elements that are not subject to adaptation or variation (Carroll et al. 2007). For the EBHV initiative, we adapted the following definition of fidelity: F idelit y i s t h e e x t e n t t o whic h a n i n t e r v e n t i o n is implemented as intended Fidelity is the extent to which an intervention is implemented as intended by its designers.

6 It refers not only to whether or not all the intervention components and activities were actually implemented, but also to whether they were imple- by its d e s i g n e r s . I t r e f e r s mented properly. The concept includes two components: not only t o w h e t h e r o r not all t h e i n t e r v e n t i o n 1. Structural aspects of the intervention that demonstrate adherence to compon e n t s a n d a c t i v i t i e s basic program elements such as reaching the target population, delivering w ere act u a lly im p lem e n t- the recommended dosage, maintaining low caseloads, and hiring and ed, bu t a lso t o whet h e r they we r e i m p l e m e n t e d retaining well-qualified staff. properly. 2. Dynamic aspects of the participant-provider interaction. It is important to consider both aspects of fidelity to determine whether a home Visiting model has been implemented as designed.

7 Moreover, Evidence-Based programs must maintain model fidelity to achieve their intended outcomes. Many program evaluations focus on documenting the service delivery process and opening the black box of the service experience (Chen 2005; Hebbler and Gerlach-Downie 2002; Lee et al. 2008; Paulsell et al. 2010). Understanding both the structural elements and the manner in which services are delivered is particu- larly important in relationship- based programs such as those being implemented by the 17 EBHV For example, the quality of the relationship between the home visitor and the parent may influence the effectiveness of home Visiting services and the extent and quality of parent engagement and involvement (Korfmacher et al. 2007; Korfmacher et al. 2008; Roggman et al. 2008). The home Visiting models the EBHV grantees are implementing include Healthy Families America (HFA), Nurse Family Partnership (NFP), Parents as Teachers (PAT), SafeCare, and Triple These models represent many of the core operat- ing principles that researchers have associated with more robust outcomes (Daro 2006).

8 These and other national models have articulated specific expectations with respect to service dosage and duration, qualifications for home visitors, training for home visitors and supervisors, supervisory standards, and core char- acteristics of a high-quality participant-provider relationship. In addition, the models set management and financial stability standards applicant organizations must fulfill. These existing requirements served as the foundation upon which Mathematica and Chapin Hall built the cross-site fidelity assessment system. 2 EBHV Cross-Site Evaluation Brief 3. Replicating Evidence-Based home Visiting Models framework for assessing Fidelity T he p r o p o s e d fi d e l i t y asse s s m e n t f r a m e w o r k includes i n d i c a t o r s t h a t The proposed fidelity assessment framework includes indicators that can be used to monitor fidelity to the program model, track program can be u s e d t o m o n i t o r improvement, and conduct evaluations.

9 Below fidelity t o t h e p r o g r a m model, t r a c k p r o g r a m we discuss the monitoring tools selected for the improve m e n t , a n d EBHV evaluation. con duct ev a lu a t ions. To ensure robust program implementation, states must determine the local sites' capacity to support the selected models and monitor their adherence to the program standards over time. National model developers play an important role in ensuring that efforts to implement their models are built on a strong foundation. However, sustaining the effort over time requires that states pay particular attention to how the models are implemented and the extent to which the programs result in a network of services that can achieve the targeted outcomes. Initial Implementation Although the different national model developers impose different standards on those seeking to replicate their models, all require applicants to demonstrate their capacity to successfully implement and sustain services as intended.

10 This early vetting fosters replication and scaling up by establishing a firm foundation for subsequent implementation efforts. Sites implementing any Evidence-Based home Visiting model typically must meet criteria such as the following: Readiness of the applicant organization to take on the task of delivering the model, including housing the service; managing the hiring, supervision, and payment of all personnel; and maintaining fiscal stability. Compliance with staff qualifications and training requirements for home visitors and supervisors, including education or experience, attendance at required training, and demonstration of specified key competencies. Capacity to identify and enroll participants in the model's target population, including (1) evidence that the proposed service area has enough families who meet the eligibility criteria and (2) identification of appropriate linkages for securing referrals to and from the program.