ConnectX-4 Lx Single and Dual 10 25 Gbs Ethernet …

1 ConnectX -4 Lx Single and Dual 10/25 Gb/s Ethernet sfp28 Port adapter card user manual P/N: MCX4111A-XCAT, MCX4121A-XCAT, MCX4111A-ACAT, MCX4121A-ACAT. Rev Mellanox Technologies NOTE: THIS HARDWARE , SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED. DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY. KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT. USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST. ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY. QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES. CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE. HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES , INCLUDING , BUT NOT LIMITED TO, THE. IMPLIED WARRANTIES OF MERCHANTABILITY , FITNESS FOR A PARTICULAR PURPOSE AND.

2 NONINFRINGEMENT ARE DISCLAIMED . IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR. ANY THIRD PARTIES FOR ANY DIRECT , INDIRECT, SPECIAL , EXEMPLARY, OR CONSEQUENTIAL. DAMAGES OF ANY KIND (INCLUDING , BUT NOT LIMITED TO , PAYMENT FOR PROCUREMENT OF. SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION ). HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE. PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH. DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100. Sunnyvale, CA 94085. Tel: (408) 970-3400. Fax: (408) 970-3403. Copyright 2018. Mellanox Technologies Ltd. All Rights Reserved. Mellanox , Mellanox logo, Accelio , BridgeX , CloudX logo, CompustorX , Connect-IB , ConnectX , CoolBox , CORE-Direct , EZchip , EZchip logo, EZappliance , EZdesign , EZdriver , EZsystem , GPUD irect , InfiniHost , InfiniBridge , InfiniScale , Kotura , Kotura logo, Mellanox CloudRack , Mellanox CloudXMellanox , Mellanox Federal Systems , Mellanox HostDirect , Mellanox Multi-Host , Mellanox Open Ethernet , Mellanox OpenCloud , Mellanox OpenCloud Logo , Mellanox PeerDirect , Mellanox ScalableHPC , Mellanox StorageX , Mellanox TuneX , Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch , MetroDX , MetroX , MLNX-OS , NP-1c , NP-2 , NP-3 , NPS , Open Ethernet logo, PhyX , PlatformX , PSIPHY , SiPhy , StoreX , SwitchX , Tilera , Tilera logo, TestX , TuneX , The Generation of Open Ethernet logo, UFM , Unbreakable Link.

3 Virtual Protocol Interconnect , Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. For the most updated list of Mellanox trademarks, visit Doc #: MLNX-15-5609 Mellanox Technologies 2. Table of Contents Revision History .. 8. About this manual .. 10. Intended Audience .. 10. Related Documentation .. 10. Document Conventions .. 11. Chapter 1 Introduction .. 12. Product Overview .. 12. Operating Systems/Distributions .. 13. Connectivity .. 13. Features and Benefits.. 14. Chapter 2 Interfaces .. 16. Ethernet sfp28 Interface .. 16. PCI Express Interface .. 16. LEDs Interface .. 16. Chapter 3 Hardware Installation .. 17. System Requirements .. 17. Hardware .. 17. Operating Systems/Distributions.. 17. Software Stacks .. 17. Safety Precautions .. 17. Pre-installation Checklist.

4 17. Bracket Installation Instructions .. 18. Removing the Existing Bracket.. 18. Installing the New Bracket .. 18. adapter card Installation Instructions .. 19. Cables and Modules .. 20. Cable Installation.. 20. adapter card Un-installation Instructions .. 21. Safety Precautions .. 21. card Un-installation .. 21. Identify the card in Your System.. 22. On Windows .. 22. On Linux .. 23. Chapter 4 Driver Installation .. 24. Linux Driver .. 24. Rev Mellanox Technologies 3. Hardware and Software Requirements.. 24. Downloading Mellanox OFED.. 24. Installing Mellanox OFED .. 25. Installation Procedure .. 26. Installation Results .. 29. Installation Logging .. 29. openibd Script .. 29. Driver Load Upon System Boot.. 30. mlnxofedinstall Return Codes.. 30. Uninstalling Mellanox OFED .. 30. Installing MLNX_OFED using YUM .. 30. Setting up MLNX_OFED YUM Repository.

5 30. Installing MLNX_OFED using the YUM Tool.. 32. Uninstalling Mellanox OFED using the YUM Tool .. 33. Installing MLNX_OFED using apt-get .. 33. Setting up MLNX_OFED apt-get Repository .. 33. Installing MLNX_OFED using the apt-get Tool.. 34. Uninstalling Mellanox OFED using the apt-get Tool .. 35. Updating Firmware After Installation .. 35. Updating the Device Online .. 35. Updating the Device Manually .. 35. Updating the Device Firmware Automatically upon System Boot .. 36. UEFI Secure Boot.. 36. Enrolling Mellanox's Public Key On your Systems .. 36. Removing Signature from Kernel Modules .. 37. Performance Tuning .. 38. Windows Driver .. 39. Hardware and Software Requirements.. 39. Downloading Mellanox WinOF-2 Driver .. 39. Installing Mellanox WinOF-2 Driver.. 40. Attended Installation.. 40. Unattended Installation .. 45. Installation Results.

6 45. Extracting Files Without Running Installation.. 46. Uninstalling Mellanox WinOF-2 Driver .. 49. Attended Uninstallation .. 49. Firmware Upgrade .. 49. Performance Tuning .. 49. VMware Driver .. 50. Hardware and Software Requirements.. 50. Installing VMware .. 50. Removing Previous Mellanox Driver .. 51. Rev Mellanox Technologies 4. Loading/Unloading Driver Kernel Modules.. 51. Firmware Programming .. 52. Chapter 5 Updating adapter card Firmware.. 53. Firmware Update (Example) .. 53. Chapter 6 Troubleshooting.. 54. General .. 54. Linux.. 55. Windows .. 56. Chapter 7 Specifications .. 57. MCX4111A-XCAT Specifications .. 57. MCX4121A-XCAT Specifications .. 57. MCX4111A-ACAT Specifications .. 59. MCX4121A-ACAT Specifications .. 60. Airflow Specifications .. 61. adapter LEDs Operations .. 62. Board Mechanical Drawing and Dimensions .. 62. Bracket Mechanical Drawing.

7 63. Appendix A Interface Connectors Pinout .. 65. sfp28 Connector Pinout .. 65. PCI Express x8 Connector Pinout .. 67. Appendix B Finding the MAC and Serial Number on the adapter card .. 68. Appendix C Safety Warnings .. 69. Appendix D Avertissements de s curit d'installation (Warnings in French) 71. Appendix E Sicherheitshinweise (Warnings in German) .. 73. Appendix F Advertencias de seguridad para la instalaci n (Warnings in Spanish) 75. Rev Mellanox Technologies 5. List of Tables Table 1: Revision History Tables .. 8. Table 2: Related Documents List.. 10. Table 3: Single and Dual-Port 10Gb/s Ethernet adapter Cards .. 12. Table 4: Single and Dual-Port 25Gb/s Ethernet adapter Cards .. 13. Table 5: Features .. 14. Table 6: Hardware and Software Requirements .. 24. Table 7: mlnxofedinstall Return Codes.. 30. Table 8: Hardware and Software Requirements.

8 39. Table 9: Hardware and Software Requirements.. 50. Table 10: MXC4111A-XCAT Specifications Table .. 57. Table 11: MCX4121A-XCAT Specifications Table .. 57. Table 12: MCX4111A-ACAT Specifications Table .. 59. Table 13: MCX4121A-ACAT Specifications Table .. 60. Table 14: Physical and Logical Link Indications .. 62. Table 15: sfp28 Connector Pinout .. 65. Rev Mellanox Technologies 6. List of Figures Figure 1: PCI Device (Example) .. 23. Figure 2: Installation Results (Example) .. 46. Figure 3: Mechanical Drawing of the MCX4111A-XCAT/MCX4111A-ACAT adapter card .. 62. Figure 4: Mechanical Drawing of the MCX4121A-ACAT/MCX4121A-XCAT adapter card .. 63. Figure 5: Single -port Bracket .. 63. Figure 6: Dual-port Bracket .. 64. Figure 7: Rear View of Module With Pin Placement .. 65. Figure 8: PCIe x8 Connector Pinout .. 67. Figure 9: MCX4431A-GCAN Board Label (Example).

9 68. Figure 10: MCX4431M-GCAN Board Label (Example) .. 68. Rev Mellanox Technologies 7. Revision History This document was printed on January 29, 2018. Table 1 - Revision History Tables Date Rev Comments/Changes November 2017 Updated Features on page 14. Added a note to System Requirements on page 17. July 2017 Updated adapter LEDs Operations on page 62. Updated Cable Installation on page 20. May 2017 Updated the following Specifications Tables: MXC4111A-XCAT Specifications Table on page 57. MCX4121A-XCAT Specifications Table on page 58. MCX4111A-ACAT Specifications Table on page 59. MCX4121A-ACAT Specifications on page 60. September 2016 Updated adapter card Installation Instructions on page 19. Added card Un-installation on page 21. Updated the following tables: MXC4111A-XCAT Specifications Table on page 57. MCX4121A-XCAT Specifications Table on page 58.

10 MCX4111A-ACAT Specifications Table on page 59. MCX4121A-ACAT Specifications on page 60. August 2016 Updated Cable Installation on page 20. Updated Physical and Logical Link Indications on page 62. Added Board Mechanical Drawing and Dimensions on page 62. June 2016 Added Mellanox logo "25 is the New 10 " in the cover page. Updated Safety Warnings on page 69. April 2016 Updated Physical and Logical Link Indications on page 62. March 2016 Removed I2C-compatible Interface from Interfaces on page 16. Updated Features and Benefits on page 14. February 2016 Updated Bracket Installation Instructions on page 18. February 2016 Updated Linux Driver on page 24. Updated Windows Driver on page 39. Updated VMware Driver on page 52. Added a note to Finding the MAC and Serial Number on the adapter card on page 68. December 2015 Updated Related Documentation on page 10.