HACMP Troubleshooting Guide - billsden.org

1 High Availability ClusterMulti-Processing for AIXT roubleshooting GuideVersion Edition (June 2002)Before using the information in this book, read the general information in Notices for HACMP Troubleshooting edition applies to HACMP for AIX, version and to all subsequent releases of this product until otherwise indicated in new editions. Copyright International Business Machines Corporation 1998, 2002. All rights to Government Users Restricted Rights--Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Guide 3 ContentsAbout This Guide7 Chapter 1:Diagnosing the Problem11 Troubleshooting an HACMP Cluster.

2 11 Becoming Aware of the Problem ..11 Application Services Are Not Available .. 12 Messages Displayed on System Console .. 12 Determining a Problem Source ..12 Examining Messages and Log Files .. 13 Investigating System Components .. 14 Tracing System Activity .. 16 Using the cldiag Utility to Perform Diagnostic Tasks .. 17 Using the Cluster Snapshot Utility to Check Cluster Configuration.. 17 Using SMIT Cluster Recovery Aids ..18 Correcting a Script Failure .. 18 Verifying Expected Behavior.

3 19 Chapter 2:Examining Cluster Log Files21 HACMP Messages and Cluster Log Files ..21 Types of Cluster Messages .. 21 Cluster Message Log Files .. 22 Understanding the File ..25 Format of Messages in the File .. 25 Viewing the File .. 25 Understanding the Log File ..27 Format of Messages in the Log File .. 28 Viewing the Log File .. 30 Changing the Location of the Log File .. 32 Resource Group Processing Messages in the File .. 32 Understanding the System Error Log ..33 Format of Messages in the System Error Log.

4 33 Viewing Cluster Messages in the System Error Log .. 33 Understanding the Cluster History Log File ..35 Format of Messages in the Cluster History Log File .. 35 Viewing the Cluster History Log File .. 36 Understanding the / File ..36 Viewing the / File .. 36 Contents4 Troubleshooting Guide Understanding the File ..37 Format of Messages in the File .. 37 Viewing the File .. 38 Understanding the / File ..39 Format of Messages in the / File .. 39 Viewing the / File .. 40 Chapter 3:Investigating System Components41 Overview.

5 41 Checking Highly Available Applications ..41 Checking the HACMP Layer ..42 Checking HACMP Components .. 42 Checking for Cluster Configuration Problems .. 46 Checking a Cluster Snapshot File .. 48 Checking the Logical Volume Manager ..53 Checking Volume Group Definitions .. 53 Checking Physical Volumes .. 55 Checking Logical Volumes .. 56 Checking Filesystems .. 57 Checking the TCP/IP Subsystem..59 Checking Point-to-Point Connectivity .. 60 Checking the IP Address and Netmask .. 61 Checking ATM Classic IP Hardware Addresses.

6 62 Checking the AIX Operating System ..63 Checking Physical Networks ..63 Checking Disks and Disk Adapters ..64 Recovering from PCI Hot Plug Network Adapter Failure .. 66 Checking System Hardware ..66 Chapter 4:Solving Common Problems67 HACMP Installation Issues ..67 Cannot Find Filesystem at Boot Time.. 67cl_convert Does Not Run Due to Failed Installation .. 68 Configuration Files Could Not Be Merged During Installation .. 68 System ID Licensing Issues .. 68 HACMP Startup Issues ..69 ODMPATH Environment Variable Not Set Correctly.

7 69 Cluster Manager Starts but then Hangs .. 69clinfo Daemon Exits After Starting.. 70 Node Powers Down; Cluster Manager Will Not Start .. 70configchk Command Returns an Unknown Host Message.. 71 Cluster Manager Hangs During Reconfiguration .. 71clsmuxpd Does Not Start or Exits After Starting .. 71 Pre- or Post-Event Does Not Exist on a Node After Upgrade .. 72 ContentsTroubleshooting Guide 5 Node Fails During Configuration with 869 LED Display .. 72 Node Cannot Rejoin the Cluster After Being Dynamically Removed.

8 73 Disk and File System Issues ..73 AIX Volume Group Commands Cause System Error Reports .. 74varyonvg Command Fails on Volume Group .. 74cl_nfskill Command Fails .. 75cl_scdiskreset Command Fails .. 75fsck Command Fails at Boot Time .. 75 System Cannot Mount Specified File Systems .. 76 Cluster Disk Replacement Process Fails .. 76 Network and Switch Issues ..76 Unexpected Adapter Failure in Switched Networks .. 77 Cluster Nodes Cannot Communicate .. 77 Distributed SMIT Causes Unpredictable Results.

9 77 Cluster Managers in a FDDI Dual Ring Fail to Communicate .. 77 Token-Ring Network Thrashes .. 78 System Crashes Reconnecting MAU Cables After a Network Failure.. 78 TMSCSI Will Not Properly Reintegrate when Reconnecting Bus .. 78 Lock Manager Communication on FDDI or SOCC Networks Is Slow .. 79 SOCC Network Not Configured after System Reboot .. 79 Unusual Cluster Events Occur in Non-Switched Environments.. 79 Cannot Communicate on ATM Classic IP Network .. 80 Cannot Communicate on ATM LAN Emulation Network.

10 81 HACMP Takeover Issues ..82varyonvg Command Fails during Takeover .. 83 Highly Available Applications Fail.. 83 Node Failure Detection Takes Too Long .. 84 Cluster Manager Sends a DGSP Message.. 84cfgmgr Command Causes Unwanted Behavior in Cluster .. 85 Deadman Switch Causes a Node Failure .. 86 Releasing Large Amounts of TCP Traffic Causes DMS Timeout .. 92A device busy Message Appears After node_up_local Fails .. 92 Adapters Swap Fails Due to an rmdev device busy Error .. 93 MAC Address Is Not Communicated to the Ethernet Switch.