ConnectX-5 Adapter Card User Manual

1 Mellanox ConnectX -5 Ex 100Gb/s VPI Single and Dual Adapter Cards User Manual P/N: MCX555A-ECAT, MCX556A-ECAT, MCX556A-EDATRev #: MLNX-15-513622 Mellanox TechnologiesMellanox Technologies350 Oakmead Parkway Suite 100 Sunnyvale, CA : (408) 970-3400 Fax: (408) 970-3403 Copyright 2018. Mellanox Technologies Ltd. All Rights , Mellanox logo, Accelio , BridgeX , CloudX logo, CompustorX , Connect-IB , ConnectX , CoolBox , CORE-Direct , EZchip , EZchip logo, EZappliance , EZdesign , EZdriver , EZsystem , GPUD irect , InfiniHost , InfiniBridge , InfiniScale , LinkX , Kotura , Kotura logo, Mellanox CloudRack , Mellanox CloudXMellanox , Mellanox Federal Systems , Mellanox HostDirect , Mellanox Multi-Host , Mellanox Open Ethernet , Mellanox OpenCloud , Mellanox OpenCloud Logo , Mellanox PeerDirect , Mellanox ScalableHPC , Mellanox StorageX , Mellanox TuneX , Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch , MetroDX , MetroX , MLNX-OS , NP-1c , NP-2 , NP-3 , NPS , Open Ethernet logo, PhyX , PlatformX , PSIPHY , SiPhy , StoreX , SwitchX , Tilera , Tilera logo, TestX , TuneX , The Generation of Open Ethernet logo, UFM , Unbreakable Link , Virtual Protocol Interconnect.

2 Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies, other trademarks are property of their respective the most updated list of Mellanox trademarks, visit :THIS HARDWARE , SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS . THE CUSTOMER 'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY . ANY EXPRESS OR IMPLIED WARRANTIES , INCLUDING, BUT NOT LIMITED TO , THE IMPLIED WARRANTIES OF MERCHANTABILITY , FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED.

3 IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT , INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO , PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES ; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION ) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY , WHETHER IN CONTRACT , STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE ) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH TechnologiesTable of Contents Table of Contents .. 1 List of Tables .. 4 List of Figures .. 5 Revision History .. 6 About This Manual .. 7 Intended Audience .. 7 Chapter 1 Introduction .. Product Overview.. Features and Benefits.. Operating Systems/Distributions .. Connectivity .. 12 Chapter 2 Interfaces .. InfiniBand Interface .. Ethernet QSFP28 Interface.

4 PCI Express Interface .. LED Interface.. 13 Chapter 3 Hardware Installation .. System Requirements .. Hardware.. Operating Systems/Distributions .. Software Stacks.. Safety Precautions .. Pre-Installation Checklist .. Bracket Installation Instructions .. Removing the Existing Bracket.. Installing the New Bracket .. card Installation Instructions.. Cables and Modules.. Cable Installation.. Adapter card Un-installation Instructions .. Safety Precautions .. card Un-installation .. Identify the card in Your System.. On Windows .. 19 Rev On Linux.. 20 Chapter 4 Driver Installation .. Linux.. Hardware and Software Requirements.. Downloading Mellanox OFED.. Installing Mellanox OFED Installation Script.. Installation Procedure.. Installation Results .. Installation Logging .. openibd Script .. Driver Load Upon System Boot.. mlnxofedinstall Return Codes.. Uninstalling Mellanox OFED.

5 Installing MLNX_OFED Using YUM.. Setting up MLNX_OFED YUM Repository .. Installing MLNX_OFED Using the YUM Tool .. Uninstalling Mellanox OFED Using the YUM Tool.. Installing MLNX_OFED Using apt-get Tool.. Setting up MLNX_OFED apt-get Repository .. Installing MLNX_OFED Using the apt-get Tool .. Uninstalling Mellanox OFED Using the apt-get Tool.. Updating Firmware After Installation .. Updating the Device Online .. Updating the Device Manually.. Updating the Device Firmware Automatically upon System Boot .. UEFI Secure Boot.. Enrolling Mellanox's Public Key On your Systems .. Removing Signature from kernel Modules .. Performance Tuning .. Windows Driver .. Hardware and Software Requirements.. Downloading Mellanox WinOF-2 Driver .. Installing Mellanox WinOF-2 Driver.. Attended Installation.. Unattended Installation .. Installation Results .. Extracting Files Without Running Installation.. Uninstalling Mellanox WinOF-2 Driver.

6 Attended Uninstallation .. Unattended Uninstallation.. Firmware Upgrade .. 49 Rev Deploying the Driver on a Nano Server.. Offline Installation.. Online Update .. 50 Chapter 5 Updating Adapter card Firmware.. Firmware Update Example .. 51 Chapter 6 Troubleshooting.. General .. Linux.. Windows .. 54 Chapter 7 Specifications .. MCX555A-ECAT Specifications .. MCX556A-ECAT Specifications .. MCX556A-EDAT Specifications .. MCX555A-ECAT Specifications .. MCX556A-ECAT Specifications .. MCX556A-EDAT Specifications .. Airflow Specifications .. Adapter card LED Operations .. Board Mechanical Drawing and Dimensions .. Bracket Mechanical Drawing.. 63 Appendix A Finding the GUID/MAC and Serial Number on the Adapter card 64 Appendix B Safety Warnings .. 66 Appendix C Avertissements de s curit d installation (Warnings in French) 68 Appendix D Sicherheitshinweise (Warnings in German) .. 70 Appendix E Advertencias de seguridad para la instalaci n (Warnings in Spanish) 72 Rev TechnologiesList of Tables Table 1:Revision History Table.

7 6 Table 2:Documents List..7 Table 3:Single and Dual-port VPI Adapter Cards ..9 Table 4:Features ..10 Table 5:Hardware and Software Requirements ..22 Table 6:Installation Results..27 Table 7:mlnxofedinstall Return Codes..29 Table 8:Hardware and Software Requirements ..38 Table 9:General Troubleshooting..52 Table 10:Linux Troubleshooting..53 Table 11:Windows Troubleshooting ..54 Table 12:MCX555A-ECAT Specification Table ..55 Table 13:MCX556A-ECAT Specification Table ..56 Table 14:MCX556A-EDAT Specification Table ..57 Table 15:MCX555A-ECAT Specification Table ..58 Table 16:MCX556A-ECAT Specification Table ..59 Table 17:MCX556A-EDAT Specification Table ..60 Table 18:Airflow Specifications ..61 Table 19:Physical and Logical Link Indications (Ethernet Mode)..61 Table 20:Physical and Logical Link Indications (InfiniBand Mode) ..61 Rev TechnologiesList of FiguresFigure 1:PCI Device (Example) ..20 Figure 2:Installation Results ..46 Figure 3:Mechanical Drawing of the Dual-port Adapter Cards.

8 62 Figure 4:Dual-port Tall Bracket ..63 Figure 5:MCX555A-ECAT Board Label (Example) ..64 Figure 6:MCX556A-ECAT Board Label (Example) ..64 Figure 7:MCX556A-EDAT Board Label (Example) ..656 Mellanox TechnologiesRevision HistoryThis document was printed on July 11, 1 - Revision History TableDateRevComments/ChangesJuly Updated Adapter card LED Operations on page 61 January Updated Adapter card LED Operations on page 61 Added a note to Hardware on page 14 November Updated Adapter card LED Operations on page 61 October Updated the following specification tables: MCX555A-ECAT Specification Table on page 55 MCX556A-ECAT Specification Table on page 56 MCX556A-EDAT Specification Table on page 57 Updated Linux on page 22 April Updated Product Overview on page 9 Updated card Installation Instructions on page 15 March releaseRev TechnologiesAbout This Manual This User Manual describes Mellanox Technologies ConnectX -5 Ex VPI Single and Dual QSFP28 port PCI Express x16 Adapter cards.

9 It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documenta-tion. Intended AudienceThis Manual is intended for the installer and user of these Manual assumes basic familiarity with InfiniBand and Ethernet network and architecture DocumentationTable 2 - Documents ListMellanox Firmware Tools (MFT) User ManualDocument no. 2204 UGUser Manual describing the set of MFT firmware management tools for a single node. See => Products => Software => Firmware ToolsMellanox Firmware Utility (mlxup) User Manual and Release NotesMellanox firmware update and query utility used to update the firmware. See => Products => Software => Firmware Tools => mlxup Firmware UtilityMellanox OFED for LinuxUser ManualDocument no. 2877 User Manual describing OFED features, performance, Band diagnostic, tools content and configuration. See => Products => Software => InfiniBand/VPI Drivers => Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)Mellanox OFED for Linux Release NotesDocument no.

10 2877 Release Notes for Mellanox OFED for Linux driver kit for Mel-lanox Adapter cards: See: =>Products => Software => InfiniBand/VPI Drivers => Linux SW/Drivers => Release Notes WinOF-2 for WindowsUser ManualDocument no. MLX-15-3280 User Manual describing WinOF-2 features, performance, Ether-net diagnostic, tools content and configuration. See => Products => Software => Windows SW/Drivers Mellanox OFED for Windows DriverRelease NotesRelease notes for Mellanox Technologies' MLNX_EN for Linux driver kit for Mellanox Adapter cards: See => Products => Software => Ethernet Drivers => Mellanox OFED for Windows => WinOF-2 Release NotesIBTA Specification Release Architecture Specification: TechnologiesDocument ConventionsWhen discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. IB is used in this document to mean this document PCIe is used to mean PCI SupportCustomers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods.