Managing The Unexpected - High reliability organization

1 Managing The UnexpectedJacksonville FloridaFebruary 28, 2005 Presenters:Karl WeickKathleen SutcliffeWhat is reliability ? reliability depends on the lack of unwanted, unanticipated, and unexplainable variance in performance -Eric Hollnagel, 1993, p. 512 reliability at Diablo Canyon (1) The major determinant of reliability in an organization is not how greatly it values reliability or safety per se over other organizational values, but rather how greatly it disvaluesthe mis-specification, mis-estimation, and misunderstanding of things;(2)All else being equal, the more things that more members of an organization care about mis-specifying, mis-estimating, and misunderstanding, the higher the level of reliability that organization can hope to attain -Paul Schulman, 19973 Examples of High reliability Organizations Nuclear power-generation plants Naval aircraft carriers Chemical production plants Offshore drilling rigs Air traffic control systems Incident command teams Wildland firefighting crews Hospital ER/Intensive care units Investment banks4 Basic Message Mindful updating is facilitated by processes that focus on failures, simplifications, operations, resiliencies, and expertise.

2 5 Lapses in reliability at the Union Pacific RailroadPreoccupation with failures: Inadequate reporting of slowdowns in to simplify: If you want to classify freight cars, then you do that in a classification to operations: Management team stays at headquarters, intimidates those who bring bad news. Commitment to resilience: Workarounds evolved by the Southern Pacific are labeled as incompetent; fire people and remove to expertise: Top down decision-making by the authorities, not the The past settles its ..the ability to deal with a crisis situation is largely dependent on the structures that have been developed before chaos arrives. The event can in some ways be considered as an abrupt and brutal audit: at a moment s notice, everything that was left unprepared becomes a complex problem, and every weakness comes rushing to the forefront. Preventing Chaos in a Crisis, Lagadec, p.

3 547F-1048F-104 Ejection Seat910 FoamDebrisFigure A shower of foam debris after the impact on Columbia s left wing. The event was not observed in real These are the results of a trajectory analysis that used a computational fluid dynamics approach in a program called CART-3D, a comprehensive (six-degree-of-freedom) computer simulation based on the laws of physics. This analysis used the aerodynamic and mass properties of bipod ramp foam, coupled with the complex flow field during ascent, to determine the likely position and velocity histories of the view was taken from Dallas.(Robert McCullough/ 2003 The Dallas Morning News)13 NASA Mission STS-107 as a HROP reoccupation with failure. NASA definition of accepted risk = known (mis-specified), understood (misunderstanding), tolerable (mis-estimation) threat. There is no failure in a can do culture. If not safe, say so says poster, and yet people are asked to prove shuttle is unsafe.

4 There is little room in this business for overconfidence, yet Mission Management team meets infrequently and this is interpreted as overconfidence. Rationale for continuing to launch was lousy, yet this was not treated as a sign of a system in poor health. NASA did not use Challenger disaster as a case to promote learning, but Navy did use Thrasher and Scorpion disasters to educate. If people were afraid of losing their jobs when they disagree, how would you know that? Managers had no to simplify When information gets filtered as it moves upward, top management winds up operating on a simpler view than does the bottom. Why would you want a photo of something that could be fixed after landing? The Crater model that was used to estimate effects is a simplification. To call the shuttle operational is simpler than to call it experimental. To call a problem in family is simpler than to call it out of family.

5 NASA needs to avoid oversimplification says CAIB (pg. 181). Multiple perspectives (conceptual slack) help you see more details and more ways to to operations Sources and reasons for imaging requests not sought out. People don t know proper channels for imaging request so can t follow them. Managers wait for dissent rather than seek it. Mission management means manage here and now, not next mission . Debris assessment team not treated as problem resolution team. Frontline not contacted before decision made not to seek external imaging. Meaning of no to image request is : Why assess debris if there is nothing we can do? After Apollo 13? Used only limited, handy resources to deal with Unexpected ( Crater but not film from astronauts) The viewgraph nothing has changed shifts attention from resilience. Minimal support for debris assessment team. Debris assessment team uses institutional channels not mission channels to get No one knew much about images and imaging ( detour over Hawaii).

6 Don t use Crater expertise at Huntington Beach. Attribute excess expertise to one supportive tile specialist. NASA is not a badgeless culture: who wants the images, not, whatare the merits of imaging. How does management know if technical staff need images?14 Rate Your Preoccupationwith Failure Regard close calls and near misses as a kind of failure that reveals potential danger rather than as evidence of our success and ability to avoid danger We treat near misses and errors as information about the health of our system and try to learn from them(1 = not at all, 2 = to some extent, 3 = a great deal) 15 MISSIONDATECOMMENTSSTS-1 April 12, 1981 Lots of debris damage. 300 tiles 18, 1983 First known left bipod ramp foam shedding 2, 1988 Debris knocks off tile; structural damage and near burn through results. STS-32 RJanuary 9, 1990 Second known left bipod ramp foam 2, 1990 First time NASA calls foam debris safety of flight issue, and re-use or turn-around issue.

7 STS-42 January 22, 1992 First mission after which the next mission (STS-45) launched without debris In-Flight Anomaly 24, 1992 Damage to wing RCC Panel 10-right. Unexplained Anomaly, most likely orbital debris. STS-50 June 25, 1992 Third known bipod ramp foam event. Hazard Report 37: an accepted risk. STS-52 October 22, 1992 Undetected bipod ramp foam loss (Fourth bipod event).STS-56 April 8, 1993 Acreage tile damage (large area). Called within experience base and consid-ered in family. STS-62 October 4, 1994 Undetected bipod ramp foam loss (Fifth bipod event).STS-87 November 19, 1997 Damage to Orbiter Thermal Protection System spurs NASA to begin 9 flight tests to resolve foam-shedding. Foam fix ineffective. In-Flight Anomaly eventually closed after STS-101 as accepted risk. STS-112 October 7, 2002 Sixth known left bipod ramp foam loss. First time major debris event not assigned an In-Flight Anomaly.

8 External Tank Project was assigned an Action. Not closed out until after STS-113 and 16, 2003 Columbia launch. Seventh known left bipod ramp foam loss The Board identified 14 flights that had significant Thermal Protection System damage or major foam moss. Two of the bipod foam loss events had not been detected by NASA prior to the Columbia Accident Investigation Board requesting a review of all launch images. 1617 Figure These two briefing slides are from the STS-113 Flight Readiness Review. The first and third bullets on the right-hand slide are incorrect since the design of the bipod ramp had changed several times since the flights listed on the Your Reluctance to Simplify People around here take nothing for granted People are encouraged to express different points of view(1 = not at all, 2 = to some extent, 3 = a great deal) 18 Figure The small cylinder at top illustrates the size of debris Crater was intended to analyze.

9 The larger cylinder was used for the STS-107 analysis; the block at right is the estimated sizeof the Your Sensitivity to Operations During an average day, people come into enough contact with each other to build a clear picture of the situation. People are familiar with operations beyond one s own job.(1 = not at all, 2 = to some extent, 3 = a great deal) 2021 MISSED Day 4. Rodney Rocha inquires if crew has been asked to inspect for damage. No Day 6. Mission Control fails to ask crew member David Brown to downlink video he took of External Tank separation, which may have revealed missing bipod Day 6. NASA and National Imagery and Mapping Agency personnel discuss possible request for imagery. No action Day 7. Wayne Hale phones Department of Defense representative, who begins identifying imaging assets, only to be stopped per Linda Ham s Day 7. Mike Card, a NASA Headquarters manager from the Safety and Mission Assurance Office, discusses imagery request with Mark Erminger, Johnson Space Center Safety and Mission Assurance.

10 No action Day 7. Mike Card discusses imagery request with Bryan O Connor, Associate Administrator for Safety and Mission Assurance. No action Day 8. Barbara Conte, after discussing imagery request with Rodney Rocha, calls LeRoyCain, the STS-107 ascent/entry Flight Director. Cain checks with Phil Engelauf, and then delivers a no Day 14. Michael Card, from NASA s Safety and Mission Assurance Office, discusses the imaging request with William Readdy, Associate Administrator for Space Flight. Readdydirects that imagery should only be gathered on a not-to-interfere basis. None was Your Commitment to Resilience There is a concern with building people s competence and response repertoires. People have a number of informal contacts that they sometimes use to solve problems.(1 = not at all, 2 = to some extent, 3 = a great deal) 222323 Flexibility in Trying Times In highly uncertain circumstances, when lives were immediately at risk, management failed to defer to its engineers and failed to recognize that different data standards qualitative, subjective, and intuitive and different processes democratic rather than protocol and chain of command were more appropriate.