FINAL California Evaluation Framework - CALMAC

1 Project Number: K2033910. The California Evaluation Framework Prepared for the California Public Utilities Commission and the Project Advisory Group June 2004. TecMarket Works And the project Team Members: Megdal & Associates Architectural Energy Corporation RLW Analytics Resource Insight B & B Resources Ken Keating and Associates Ed Vine and Associates American Council for an Energy Efficient Economy Ralph Prahl and Associates Innovologie Project Number: K2033910. The California Evaluation Framework Prepared for Southern California Edison Company To conduct a joint study supported by Pacific Gas & Electric Company San Diego Gas & Electric Company Southern California Edison Company Southern California Gas Company As mandated by the California Public Utilities Commission June 2004.

2 Submitted by TecMarket Works Framework Team TecMarket Works 165 West Netherwood Road Second Floor, Suite A. Oregon, WI 53575. Voice: (608) 835-8855. Fax: (608) 835-9490. E-mail: TecMarket Works Framework Team Acknowledgements Acknowledgements The authors wish to acknowledge and express our appreciation to the many individuals who contributed to the development of the California Evaluation Framework . Without the support and assistance of these individuals this effort would not have been possible. The Project Advisory Group provided general and topic-specific project guidance and reviews of all chapters of the Framework .

3 These individuals and their affiliations are: x Marian Brown, Project Manager, Advisory Group Chairperson, Southern California Edison x Eli Kollman and Jay Luboff, Energy Division, California Public Utilities Commission x Don Schultz and Christine Tam, Office of Ratepayer Advocates, California Public Utilities Commission x Mike Messenger and Sylvia Bender, California Energy Commission x Valerie Richardson, Chris Ann Dickerson, and Kenneth James, Pacific Gas and Electric Company x Athena Besa and Rob Rubin, Sempra Energy In addition to the oversight and support of the Project Advisory Group, several individuals provided ongoing comments.

4 Support, and recommendations to the Framework as a whole and to individual chapters of the Framework . Over 90 individuals provided over 350 comments or recommendations to the Project Team. We wish to thank all those who provided comments. We have reviewed and discussed all of the comments received and have considered these comments in the development of the Framework . In many cases the comments were instrumental in guiding the development of the project's focus and the Framework chapters. Several comments required consultation with the Project Advisory Group to reach a consensus on the impact of the comment to the Framework .

5 We wish to especially thank the following individuals who provided valuable comments that helped frame some of the discussions and improved the focus and structure of the Framework . x Hayley Goodson, The Utility Reform Network x Adrienne Kandel, California Energy Commission x Patrick McCarthy and Bill Stiegelman, Aspen Systems x Maureen McNamara, United States Environmental Protection Agency x Robert Mowris, Robert Mowris and Associates x Monica Nevius, Consortium for Energy Efficiency x Dean Schiffman, Sempra Energy June 2004 i California Evaluation Framework TecMarket Works Framework Team Acknowledgements x Phil Sissons.

6 Sisson and Associates We also want to thank the TecMarket Works Project Team that collaborated to design and develop the new California Evaluation Framework under the direction and oversight of the CPUC and the Project Advisory Group. These individuals include: x Nick Hall, TecMarket Works x Lori Megdal, Megdal & Associates x Pete Jacobs and Stuart Waterbury, Architectural Energy Corporation x Roger Wright, RLW Analytics x Paul Chernick, Resource Insight Incorporated x Ken Keating, Ken Keating and Associates x Sharyn Barata, B&B Resources x Ed Vine, Edward Vine and Associates x Steve Nadel and Marty Kushler, American Council for an Energy Efficient Economy x Ralph Prahl, Ralph Prahl and Associates x John Reed.

7 Innovologie The Team and its authors would also like to thank Johna Roth of TecMarket Works for her editing and formatting work in preparing this document. June 2004 ii California Evaluation Framework TecMarket Works Framework Team Table of Contents Table of Contents ACKNOWLEDGEMENTS ..I. CHAPTER 1: INTRODUCTION AND PURPOSE OF Framework .. 1. PURPOSE OF Evaluation AND A STANDARD Evaluation Framework .. 1. PROJECT APPROACH .. 2. Evaluation TYPES AND CONSIDERATIONS COVERED BY THE Framework .. 3. Impact Evaluation , and Measurement and Verification Approaches (Chapters 6 and 7) .. 3. Process Evaluations (Chapter 8).

8 4. Information and Education Program Evaluation (Chapter 9).. 4. Market Transformation Program Evaluation (Chapter 10).. 5. Non-Energy Effects Evaluation (Chapter 11) .. 6. Uncertainty (Chapter 12).. 6. Sampling (Chapter 13).. 6. Evaluation and Cost-Effectiveness (Chapter 14) .. 7. Evaluation RESEARCH NOT COVERED BY THE Framework .. 7. THE USE OF Evaluation ROADMAPS .. 9. CHAPTER 2: STUDY METHODOLOGY AND PUBLIC INPUT PROCESS .. 11. STUDY METHODOLOGY .. 11. Development of Work Plan and Literature 12. Development of Principles and 17. Development of Draft and FINAL Project Reports .. 17. PUBLIC INPUT PROCESS.

9 18. CHAPTER 3: USE OF THE Evaluation Framework BY DIFFERENT. STAKEHOLDERS .. 21. CHAPTER 4: Evaluation OVERVIEW AND ISSUES .. 27. PREFACE .. 27. CORE PURPOSE OF Evaluation .. 27. THE TWO KEY FUNCTIONS OF Evaluation SUMMATIVE AND FORMATIVE. EVALUATIONS .. 28. THE TWO BASIC TYPES OF Evaluation .. 29. PROGRAM THEORY/LOGIC MODEL USE IN Evaluation .. 30. Developing a Program Theory .. 33. Reasons for an Evaluator to Use and Refine the Program 35. Program Theories are Especially Important to the Evaluation of Complex Programs .. 36. The Relationship Between Program Theory, Market Assessment, and Market Theory.

10 37. Evaluation ETHICS .. 38. HISTORY OF ENERGY EFFICIENCY PROGRAM Evaluation IN California .. 38. 1970s-1994: The Pre-Protocol Era .. 39. 1994-1997: The Protocol Period .. 39. 1998-2000: Energy Efficiency, Electric Industry Restructuring, and The CBEE Period .. 40. Post-2000: Transition Period and the Energy 41. POLICY USE OF Evaluation RESULTS .. 42. Policy Based on Evaluation 42. Interpreting and Adjusting Results for Summative versus Formative Purposes .. 43. June 2004 iii California Evaluation Framework TecMarket Works Framework Team Table of Contents Making the Framework a Living Document .. 44.