Mellanox Adapters Programmer's Reference Manual (PRM)

1 Mellanox Adapters Programmer's Reference Manual (PRM). Supporting ConnectX -4 and ConnectX -4 Lx Rev NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED. DOCUMENTATION ARE PROVIDED BY Mellanox TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY. KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE. THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT. HAS NOT MET THE STANDARDS SET BY Mellanox TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S). AND/OR THE SYSTEM USING IT. THEREFORE, Mellanox TECHNOLOGIES CANNOT AND DOES NOT. GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

2 ANY. EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF. MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL Mellanox BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT. LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE). ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF.

3 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100. Sunnyvale, CA 94085. Tel: (408) 970-3400. Fax: (408) 970-3403. Copyright 2016. Mellanox Technologies. All Rights Reserved. Mellanox , Mellanox logo, BridgeX , CloudX logo, Connect-IB , ConnectX , CoolBox , CORE-Direct , GPUD irect , InfiniHost , InfiniScale , Kotura , Kotura logo, Mellanox Federal Systems , Mellanox Open Ethernet , Mellanox ScalableHPC , Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch , MetroDX , MetroX , MLNX-OS , Open Ethernet logo, PhyX , SwitchX , TestX , The Generation of Open Ethernet logo, UFM , Virtual Protocol Interconnect , Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies, Ltd.

4 Accelio , CyPU , FPGAD irect , HPC-X , InfiniBridge , LinkX , Mellanox Care , Mellanox CloudX , Mellanox Multi-Host , Mellanox NEO , Mellanox PeerDirect , Mellanox Socket Direct , Mellanox Spectrum , NVMeDirect , StPU , Spectrum logo, Switch-IB , Unbreakable-Link are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. 2 Mellanox Technologies Document Number: MLNX-15-4845. Table of Contents Table of Contents .. 3. List of Tables .. 9. List of Figures .. 22. Revision History .. 23. About this Manual .. 24. Intended Audience .. 24. Related Documentation .. 24. Documentation Conventions.

5 24. Chapter 1 Introduction.. 27. Major Features Summary .. 27. Hardware Interfaces.. 27. Host Interface - PCI Express .. 28. PCI Express.. 28. NC-SI (DMTF-Compliant Management Link).. 28. I2C-Compatible .. 28. Network Interface .. 29. Chapter 2 Operational Overview .. 30. Software Interface .. 30. HCA Configuration .. 30. HCA Operation .. 30. Work Queues .. 31. Completion Queues and Completion Events .. 31. Event Queues .. 31. Interrupts.. 32. HCA Memory Resources .. 32. Chapter 3 Networking and Stateless Offloads.. 34. Networking Transport Objects .. 34. Overview.. 34. Networking Services Usage.. 36. Ethernet.. 36. Stateless Offloads.

6 37. Checksum Offload .. 37. Large Send Offload (LSO) .. 38. Receive Side Scaling (RSS) .. 38. TIR Spreading Traffic Mechanism .. 39. Transmit Side Scaling.. 39. Interrupt Moderation.. 39. Large Receive Offload (LRO) .. 39. LRO Introduction and Device Capabilities .. 39. LRO Session Creation .. 40. LRO Session Termination .. 40. Merging New Segments Into Existing LRO Session .. 41. LRO Packet Scatter to Memory .. 42. LRO CQE Fields Summary .. 42. VLAN Insertion/Stripping .. 44. Packet Padding (TX).. 44. Start of Packet Padding (RX) .. 44. End of Packet Padding (RX).. 45. Self-Loopback Control Using Transport Domains .. 45. Mellanox Technologies 3.

7 Sniffer .. 45. Chapter 4 PCI Interface .. 47. PCIe Compliance.. 47. Capabilities Reporting .. 47. Initialization Segment .. 48. Data Interface .. 51. PCIe Attributes .. 51. Chapter 5 Memory Resources and Utilization .. 52. Interconnect Context Memory.. 52. Memory Resources .. 52. Address Translation Tables .. 52. HCA Control Objects (Contexts) .. 52. User Access Region.. 53. Chapter 6 Address Translation and Protection .. 54. Virtual to Physical Address Translation and Protection Checks.. 54. Address Translation Indirection .. 56. Zero-Based Virtual Address Regions and Windows.. 57. Reserved LKey .. 58. Address Translation Control Structures.

8 58. MKey Context.. 58. Memory Key Configuration (Creation).. 61. Chapter 7 Software Interface .. 62. ISSI - Interface Step Sequence ID.. 62. ISSI History .. 62. ISSI = 1 .. 62. User Access Region (UAR).. 63. UAR Sections .. 63. UAR Page Format.. 64. Blue Flame .. 65. Sharing UARs .. 66. NIC_Vport Context - NIC Virtual Port Context .. 66. Work Queues .. 69. Work Queues Structure and Access .. 69. Send Queue .. 70. Receive Queue .. 72. Doorbell Record .. 73. WQE Ownership.. 74. Posting a Work Request to Work Queue .. 74. Posting Work Request on Shared Receive Queue .. 74. Work Request (WQE) Formats.. 75. Send WQE Format .. 75. Send WQE Construction Summary.

9 82. Receive WQE Format .. 83. Shared Receive WQE Format.. 83. Transport Interface Receive (TIR) .. 84. Transport Interface Send (TIS) .. 89. Receive Queue (RQ) .. 90. RQ States Summary .. 96. RQ Error Semantics .. 97. Memory RQ with Inline Memory Queue .. 97. RQ Associated with RMP.. 97. RQ Table (RQT) .. 98. Send Queue (SQ) .. 99. SQ States Summary .. 103. SQ Error Semantics .. 103. Send WQE Inline Header .. 103. Mellanox Technologies 4. Inline Modes .. 104. Receive Memory Pool (RMP) .. 104. RMP States Summary.. 107. Flow Table.. 107. Position in Processing Flow .. 107. General Structure .. 107. Match Criteria .. 108. Processing Actions.

10 108. Flow Table Chaining .. 108. Multi-Processing Paths .. 108. Default Behavior .. 109. Invalid Flows .. 109. Packet Classification Ambiguities .. 109. Flow Statistics .. 109. Flow Tagging .. 109. Configuration Interface.. 110. Allocating a Flow Table.. 110. Selecting the Root Flow Table.. 110. Flow Table Level .. 110. Allocating a Match Criteria - Flow Groups .. 110. Allocating a Flow Counter.. 111. Adding a Flow .. 112. Redefining a Flow .. 112. Freeing Resources .. 113. Querying Device Database .. 113. Special Flows Definition .. 114. Characteristics of Flow Table Types .. 114. NIC Receive.. 114. NIC Transmit .. 114. Completion Queues.