Unexpected Event Reporting

1 Department of the Interior Bureau of Reclamation Denver, Colorado August 2011 Facilities Instructions, Standards, and Techniques Volume 6-3 Unexpected Event Reporting REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public Reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302.

2 Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY)8- 11-2011 2. REPORT TYPEF inal 3. DATES COVERED (From - To)Published August 19, 2011 Effective September 19, 2011 until superseded FIST 6-3, Unexpected Event Reporting 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) Martin Bauer, Nick Bunting, Jeffery Hoffman, Rick Jackson, Terry Kent, Toni Linenberger, Steve Melavic, and Max Spiker, 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Bur eau of Recl amat i on D env er Feder al Cent er Box 25007 Denver CO 80225-0007 8.

3 PERFORMING ORGANIZATION REPORT NUMBER FIST 6-3 9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) Power Resources Office, Technical Resources Bureau of Reclamation Mail Code 86-61600 PO Box 25007 Denver CO 80225-0007 10. SPONSOR/MONITOR S ACRONYM(S) PRO 11. SPONSOR/MONITOR S REPORT NUMBER(S) 12. DISTRIBUTION / AVAILABILITY STATEMENT Available from the National Technical Information Service, Operations Division, 5285 Port Royal Road, Springfield, Virginia 22161 13. SUPPLEMENTARY NOTES 14. ABSTRACT The Bureau of Reclamation operates and maintains 53 hydroelectric powerplants and many switchyards, pumping plants, and associated facilities that are important to electric power and water delivery systems. These facilities house complex electrical and mechanical equipment; protective relays and associated circuits play an essential role in protecting this equipment as well as the electric power system.

4 Improving the reliability of this equipment is facilitated by analyzing the Event when that equipment operates and ensuring that the operation is correct t o prevent the recurring events . This document defines Reclamation practices for documenting Unexpected events at Reclamation power facilities. 15. SUBJECT TERMS Unexpected Event Reporting 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT U 18. NUMBER OF PAGES 19a. NAME OF RESPONSIBLE PERSONM artin Bauer a. REPORT U b. ABSTRACT U c. THIS PAGE U 19b. TELEPHONE NUMBER (include area code) 303-445-2901 Standard Form 298 (Rev. 8/98) Prescribed by ANSI Std. 239-18 6 Department of the Interior Bureau of Reclamation Denver, Colorado August 2011 Facilities Instructions, Standards, and Techniques Volume 6-3 Unexpected Event Reporting Prepared by: Power Resources Office Disclaimer This written material consists of general information for internal use only by Bureau of Reclamation operations and maintenance staff.

5 Information contained in this document regarding commercial products or firms may not be used for advertising or promotional purposes and is not to be construed as an endorsement or deprecation of any product or firm by the Bureau of Reclamation. iii Acronyms and Abbreviations AGC Automatic Generation Control CIO Chief Information Officer DOC Designers Operating Criteria FIST Facilities Instructions, Standards, and Techniques NERC North American Electric Reliability Corporation O&M Operation and Maintenance PEB Power Equipment Bulletin PDF Portable Document Format created by Adobe Systems PLC Programmable Logic Controller PRO Power Resources Office PO&M Power Operations and Maintenance RCSIRT Reclamation Computer Security Incident Response Team RDO Reclamation Duty Officer Reclamation Bureau of Reclamation REO Regional Emergency Official RESC Reclamation Enterprise Service Center RTU Remote Terminal Unit SCADA Supervisory Control and Data Acquisition SMIS Safety Management Information System SOP Standing Operating Procedure SSLE Security, Safety.

6 And Law Enforcement WECC Western Electricity Coordinating Council v Contents Page Acronyms and Abbreviations iii Contents v FIGURES vii 1. Introduction 1 Purpose and Scope 1 Effective Dates 2 Definitions 2 Cross References 5 2. Coordination with Other Investigations 6 3. Event Documentation 6 4. General Procedures 6 5. Event Numbering 15 6. Suspected Sabotage Reporting 16 7. Event Classification 16 8. Restoration Teams 17 9. Investigation Team 17 10. Event Data 18 11. Event Analysis and Findings 18 12. Documentation 19 Minor events 20 Significant events 20 Severe events 20 13. Compliance Review 21 14. Corrective Action Plan 22 Minor events 23 Significant events 23 Severe events 24 15. Unexpected Event Reporting Program Evaluation 24 Unexpected Event Reporting Improvement 24 Measurement of Compliance with FIST 25 Analysis of Compliance with FIST 25 Program Value 25 Program Adjustments 25 Appendix A Power O&M 172 27 Appendix B Example of an Investigation Team Charter 32 Version 33 Charter Approval 35 Sponsor 35 Investigation Team Members 36 Revision History 37 Purpose 39 Business Need 39 Background 39 Scope 39 Testing Data 40 Expenses 40 Communication 40 Roles and Responsibilities 41 Deliverables 41 Appendix C Specific Tasks and Timelines Associated with Unexpected Event Reporting 43 Specific Tasks and Timelines associated with Unexpected Event Reporting 45 Appendix D Incident Report Outline 47 Appendix E Investigation Processes 49 Investigation Process 51 I.

7 PURPOSE 51 II. INVESTIGATION PROCESS 51 III. INCIDENT REPORT AND DEVELOPING FINDINGS 57 Appendix F events and How They Are To Be Classified 59 Appendix G Suspected Sabotage Reporting Procedures 63 vii Purpose 65 Scope 65 Roles and Responsibilities 65 Background 66 Determination of Suspected Sabotage 66 Reporting of Suspected Sabotage 70 Power System Related Event or Activity 70 Electric System Coordination 70 Cyber System-Related Event or Activity (SCADA, EACSS, etc.) 71 Response 71 Restoration 71 Sabotage Reporting Followup Actions 72 Analysis of Information 72 Internal Notification 72 Notification of Federal Law Enforcement 72 External Notification of Sabotage 72 Attachment A Sabotage Precursors and Indicators 75 Physical Sabotage Reporting Guidelines 75 Supervisory Control and Data Acquisition (SCADA) Sabotage Reporting Guidelines 75 Electronic Access Control and Surveillance System Sabotage Reporting Guidelines 76 FIGURES Figure 1.

8 Unexpected Event Documentation Process. 8 Figure 2. Compliance Review. 9 Figure 3. Cyber, Security, and Sabotage Reporting Procedures. 10 Figure 4. Event Documentation Filling Out Part I of PO&M 172. 11 Figure 5. Event Documentation Filling Out Part II of PO&M 172. 12 Figure 6. Event Documentation Filling Out Part III of PO&M 172. 13 Figure 7. Event Documentation Filling Out Part IV of PO&M 172. 14 1 1. Introduction The Bureau of Reclamation (Reclamation) operates and maintains 53 hydroelectric powerplants and many switchyards, pumping plants, and associated facilities in the 17 Western States. These facilities are critical to the electric power and water delivery systems relied on by many. They contain complex electrical and mechanical equipment that must be kept operational. The analysis of events will increase availability of the facilities.

9 The analysis is only possible by documenting those events . With the documentation, the analysis can result in a meaningful corrective action plans to reduce Event recurrence. Text that appears bold and brackets in this Facilities Instructions, Standards, and Techniques (FIST) volume is considered a Reclamation standard practice. Reclamation standard practices reflect minimum operations and maintenance (O&M) activity levels. Variation from Reclamation standard practices and schedules is acceptable provided that local instructions exist to support the variation. Non Bold text is a suggested practice or explanation. Variation from suggested practices or explanation does not require Local Instructions. Purpose and Scope The purpose of this document is to provide practical guidance to Reclamation staff in evaluating and Reporting Unexpected events .

10 It is to be used by personnel in the facility, area, regional, and Denver offices to conduct the various levels of Unexpected Event evaluation and Reporting . [This FIST is applicable at facilities operated and maintained directly by Reclamation staff and can be applied at facilities that are owned by Reclamation but maintained by others. Each region will determine the degree to which the program can be applied to their respective facilities that meet this criteria.] The Unexpected Event Reporting Program is a structured approach to evaluating and documenting events . This is accomplished by a defined process of Event assessment that is tailored in scope and depth to the severity of the Event . Foremost, the Unexpected Event Reporting Program is intended to be a learning tool for the betterment of our operations and maintenance programs through self-examination and information sharing.