Mellanox OFED Linux User’s Manual

1 Mellanox OFED for Linux user Manual Rev Software version Mellanox Technologies NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT (S) ) AND ITS RELATED. DOCUMENTATION ARE PROVIDED BY Mellanox TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY. KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT. USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST. ENVIRONMENT HAS NOT MET THE STANDARDS SET BY Mellanox TECHNOLOGIES TO FULLY. QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, Mellanox TECHNOLOGIES. CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE. HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE. IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND. NONINFRINGEMENT ARE DISCLAIMED.

2 IN NO EVENT SHALL Mellanox BE LIABLE TO CUSTOMER OR. ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL. DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF. SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION). HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT , STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE. PRODUCT (S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH. DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100. Sunnyvale, CA 94085. Tel: (408) 970-3400. Fax: (408) 970-3403. Copyright 2018. Mellanox Technologies Ltd . All Rights Reserved . Mellanox , Mellanox logo, Accelio , BridgeX , CloudX logo, CompustorX , Connect -IB , ConnectX , CoolBox , CORE-Direct , EZchip , EZchip logo, EZappliance , EZdesign , EZdriver , EZsystem , GPUD irect , InfiniHost , InfiniBridge , InfiniScale , LinkX , Kotura , Kotura logo, Mellanox CloudRack , Mellanox CloudXMellanox , Mellanox Federal Systems , Mellanox HostDirect , Mellanox Multi-Host , Mellanox Open Ethernet , Mellanox OpenCloud , Mellanox OpenCloud Logo , Mellanox PeerDirect , Mellanox ScalableHPC , Mellanox StorageX , Mellanox TuneX , Mellanox Connect Accelerate Outperform logo , Mellanox Virtual Modular Switch , MetroDX , MetroX , MLNX-OS , NP-1c , NP-2 , NP-3 , NPS , Open Ethernet logo, PhyX , PlatformX , PSIPHY , SiPhy , StoreX , SwitchX , Tilera , Tilera logo, TestX , TuneX.

3 The Generation of Open Ethernet logo, UFM , Unbreakable Link , Virtual Protocol Interconnect , Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies , Ltd. All other trademarks are property of their respective owners . For the most updated list of Mellanox trademarks, visit Document Number: 2877 Mellanox Technologies 2. Table of Contents Table of Contents .. 3. List of Tables .. 10. Chapter 1 Overview .. 19. Stack Architecture .. 20. mlx4 VPI Driver .. 20. mlx5 Driver .. 21. Mid-layer Core.. 22. Upper Layer Protocols (ULPs).. 22. IP over IB (IPoIB) .. 22. iSCSI Extensions for RDMA (iSER) .. 23. SCSI RDMA Protocol (SRP).. 23. user Direct Access Programming Library (uDAPL) .. 23. MPI .. 23. InfiniBand Subnet Manager .. 23. Diagnostic Utilities .. 24. Mellanox Firmware Tools .. 24. Mellanox OFED Package.

4 24. ISO Image.. 24. Software Components .. 24. Firmware .. 25. Directory Structure .. 25. Module Parameters .. 26. mlx4 Module Parameters .. 26. mlx4_ib Parameters.. 26. mlx4_core Parameters .. 27. mlx4_en Parameters .. 29. mlx5_core Module Parameters .. 29. ib_core Parameters.. 30. ib_ipoib Parameters .. 30. Device Capabilities .. 30. Chapter 2 Installation .. 31. Hardware and Software Requirements .. 31. Downloading Mellanox OFED .. 31. Installing Mellanox OFED .. 32. Installation Script .. 32. Installation Procedure.. 34. Installation Results .. 37. Rev Mellanox Technologies 3. Installation Logging .. 37. openibd Script .. 38. Driver Load Upon System Boot .. 38. mlnxofedinstall Return Codes .. 38. Installing MLNX_OFED with Innova IPsec Adapter Cards.. 39. Uninstalling Mellanox OFED .. 39. Installing MLNX_OFED Using YUM.

5 39. Setting up MLNX_OFED YUM Repository .. 39. Installing MLNX_OFED Using the YUM Tool .. 41. Uninstalling Mellanox OFED Using the YUM Tool .. 42. Installing MLNX_OFED Using apt-get .. 42. Setting up MLNX_OFED apt-get Repository .. 42. Installing MLNX_OFED Using the apt-get Tool .. 43. Uninstalling Mellanox OFED Using the apt-get Tool .. 43. Updating Firmware After Installation .. 44. Updating the Device Online .. 44. Updating Firmware and FPGA Image on Innova IPsec Cards .. 45. Updating the Device Manually.. 45. Updating the Device Firmware Automatically upon System Boot .. 46. UEFI Secure Boot .. 46. Enrolling Mellanox 's Public Key On your Systems .. 46. Removing Signature from kernel Modules .. 47. Performance Tuning .. 47. Installing Upstream rdma-core Libraries .. 48. Installation Using mlnxofedinstall Script.

6 48. Installation Using Repositories .. 48. Installing libdisni Package .. 48. Chapter 3 Features Overview and Configuration .. 50. Ethernet Network .. 50. Interface.. 50. ConnectX-3/ConnectX-3 Pro Port Type Management .. 50. ConnectX-4 and above Port Type Management/VPI Cards Configuration 51. Counters .. 51. Persistent Naming .. 52. Quality of Service (QoS) .. 53. Mapping Traffic to Traffic Classes .. 53. Plain Ethernet Quality of Service Mapping .. 53. RoCE Quality of Service Mapping .. 54. Map Priorities with set_egress_map .. 55. Quality of Service Properties .. 55. Quality of Service Tools.. 56. Rev Mellanox Technologies 4. Packet Pacing .. 62. Quantized Congestion Notification (QCN) .. 64. QCN Tool - mlnx_qcn.. 64. Setting QCN Configuration .. 66. Ethtool .. 67. Checksum Offload .. 71. Ignore Frame Check Sequence (FCS) Errors.

7 72. RDMA over Converged Ethernet (RoCE) .. 72. RoCE Modes .. 73. GID Table Population.. 75. RoCE Lossless Ethernet Configuration .. 78. Type Of Service (ToS).. 84. RoCE LAG (ConnectX-3/ConnectX-3 Pro).. 84. RoCE LAG (ConnectX-4/ConnectX-4 Lx/ConnectX-5) .. 86. Soft RoCE .. 87. Enabling/Disabling RoCE on VFs (ConnectX-4) .. 87. Force DSCP .. 87. Force Time to Live (TTL) .. 88. Flow Control .. 89. Priority Flow Control (PFC) .. 89. Dropless Receive Queue (RQ).. 93. Explicit Congestion Notification (ECN) .. 95. Enabling ECN on ConnectX-4/ConnectX-4 Lx/ConnectX-5 .. 95. RSS Support.. 96. RSS Hash Function .. 96. RSS Verbs Support for ConnectX-4 HCAs .. 97. Time-Stamping .. 98. Time-Stamping Service.. 98. RoCE Time Stamping .. 102. One Pulse Per Second (1 PPS) .. 104. Flow Steering .. 105. Enable/Disable Flow Steering.

8 105. Flow Steering Support .. 106. A0 Static Device Managed Flow Steering .. 107. Flow Domains and Priorities .. 107. Flow Steering Dump Tool .. 111. Wake-on-LAN (WoL).. 111. Hardware Accelerated VLAN (Q-in-Q Tunneling) .. 111. Requirements.. 112. VLAN Stripping in Linux Verbs .. 112. Offloaded Traffic Sniffer.. 112. Physical Address Memory Allocation .. 113. Dump Configuration .. 114. Rev Mellanox Technologies 5. InfiniBand Network.. 116. Interface.. 116. ConnectX-3/ConnectX-3 Pro Port Type Management .. 116. ConnectX-4 Port Type Management .. 116. RDMA Counters .. 116. OpenSM .. 116. opensm .. 116. osmtest.. 118. Partitions .. 118. Effect of Topology Changes .. 121. Routing Algorithms .. 121. Quality of Service Management in OpenSM .. 144. Enhanced QoS .. 154. QoS Configuration Examples .. 155. Congestion Control Manager.

9 164. DOS MAD Prevention .. 168. MAD Congestion Control .. 168. IB Router Support in OpenSM .. 169. OpenSM Activity Report .. 170. Quality of Service (QoS) .. 171. QoS Architecture .. 172. Supported Policy .. 173. CMA Features.. 174. Secure Host .. 175. Secure Mode Operation .. 175. IP over InfiniBand (IPoIB) .. 177. Upper Layer Protocol (ULP) .. 177. Enhanced IPoIB .. 182. IPoIB Mode Setting .. 182. Port Configuration .. 183. IPoIB Configuration .. 183. Subinterfaces .. 186. Verifying IPoIB Functionality.. 188. Bonding IPoIB.. 188. Dynamic PKey Change .. 189. Precision Time Protocol (PTP) over IPoIB .. 189. One Pulse Per Second (1 PPS) over IPoIB .. 190. Advanced Transport .. 190. Atomic Operations.. 190. Dynamically Connected Transport (DCT).. 192. MPI Tag Matching and Rendezvous Offloads .. 192. Optimized Memory Access.

10 192. Contiguous Pages.. 192. Memory Region Re-registration .. 193. Rev Mellanox Technologies 6. Memory Window.. 195. user -Mode Memory Registration (UMR) .. 196. On-Demand-Paging (ODP) .. 196. Inline-Receive.. 200. Coherent Accelerator Processor Interface (CAPI) .. 200. Tunneled Atomic .. 201. AS Notify.. 202. Mellanox PeerDirect .. 202. Mellanox PeerDirect Async .. 202. Mellanox Relaxed Ordering (RSYNC) .. 203. CPU Overhead Distribution .. 203. Resource Domain Experimental Verbs .. 203. Resource Domain Attributes.. 203. Query Interface Experimental Verbs.. 204. QP-burst Experimental Family .. 205. CQ Experimental Family .. 205. WQ Experimental Family.. 205. Out-of-Order (OOO) Data Placement Experimental Verbs .. 205. Overview .. 205. Usage .. 206. Notes.. 206. WQE Format in MLNX_OFED .. 207. IB Router .. 207. Storage Protocols.