RDMA Aware Networks Programming User Manual

1 RDMA Aware Networks Programming user ManualRev Technologies350 Oakmead Parkway Suite 100 Sunnyvale, CA : (408) 970-3400 Fax: (408) 970-3403 Copyright 2013. Mellanox Technologies. All Rights , Mellanox logo, BridgeX , ConnectX , CORE-Direct , InfiniBridge , InfiniHost , InfiniScale , PhyX , SwitchX , Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies, , FabricIT , MLNX-OS , MetroX , ScalableHPC , Unbreakable-Link , UFM and Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective :THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.

2 THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH Number:Rev Technologies2 Mellanox Technologies350 Oakmead Parkway Suite 100 Sunnyvale, CA : (408) 970-3400 Fax: (408) 970-3403 Copyright 2015.

3 Mellanox Technologies. All Rights , Mellanox logo, BridgeX , ConnectX , Connect-IB , CoolBox , CORE-Direct , GPUD irect , InfiniBridge , InfiniHost , InfiniScale , Kotura , Kotura logo, MetroX , MLNX-OS , PhyX , ScalableHPC , SwitchX , TestX , UFM , Virtual Protocol Interconnect , Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies, , ExtendX , FabricIT , FPGAD irect , HPC-X , Mellanox Care , Mellanox CloudX , Mellanox Open Ethernet , Mellanox PeerDirect , Mellanox Virtual Modular Switch , MetroDX , NVMeDirect , StPU , Switch-IB , Unbreakable-Link are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective :THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.

4 THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH Technologies3 Table of ContentsRevision History.

5 9 Glossary .. 10 Chapter 1 RDMA Architecture Overview .. InfiniBand .. Virtual Protocol Interconnect (VPI) .. RDMA over Converged Ethernet (RoCE).. Comparison of RDMA Technologies .. Key Components .. Support for Existing Applications and ULPs .. References .. 18 Chapter 2 RDMA- Aware Programming Overview .. Available Communication Operations .. Send/Send With Immediate.. Receive .. RDMA Read .. RDMA Write / RDMA Write With Immediate .. Atomic Fetch and Add / Atomic Compare and Swap .. Transport Modes .. Reliable Connection (RC) .. Unreliable Connection (UC) .. Unreliable Datagram (UD) .. Key Concepts.

6 Send Request (SR) .. Receive Request (RR) .. Completion Queue .. Memory Registration .. Memory Window .. Address Vector .. Global Routing Header (GRH) .. Protection Domain .. Asynchronous Events .. Scatter Gather .. Polling .. Typical Application .. 23 Chapter 3 VPI Verbs API .. Initialization .. ibv_fork_init .. Device Operations .. ibv_get_device_list .. ibv_free_device_list .. ibv_get_device_name .. 27 Rev ibv_get_device_guid .. ibv_open_device .. ibv_close_device .. ibv_node_type_str .. ibv_port_state_str .. Verb Context Operations .. ibv_query_device.

7 Ibv_query_port .. ibv_query_gid .. ibv_query_pkey.. ibv_alloc_pd .. ibv_dealloc_pd .. ibv_create_cq .. ibv_resize_cq.. ibv_destroy_cq .. ibv_create_comp_channel .. ibv_destroy_comp_channel.. Protection Domain Operations .. ibv_reg_mr .. ibv_dereg_mr .. ibv_create_qp .. ibv_destroy_qp .. ibv_create_srq .. ibv_modify_srq .. ibv_destroy_srq .. ibv_open_xrc_domain.. ibv_create_xrc_srq .. ibv_close_xrc_domain .. ibv_create_xrc_rcv_qp .. ibv_modify_xrc_rcv_qp .. ibv_reg_xrc_rcv_qp .. ibv_unreg_xrc_rcv_qp .. ibv_create_ah .. ibv_destroy_ah .. Queue Pair Bringup (ibv_modify_qp).

8 Ibv_modify_qp .. RESET to INIT .. INIT to RTR .. RTR to RTS.. Active Queue Pair Operations .. ibv_query_qp.. ibv_query_srq .. ibv_query_xrc_rcv_qp .. ibv_post_recv .. ibv_post_send .. ibv_post_srq_recv .. 60 Rev ibv_req_notify_cq .. ibv_get_cq_event .. ibv_ack_cq_events .. ibv_poll_cq .. ibv_init_ah_from_wc .. ibv_create_ah_from_wc .. ibv_attach_mcast .. ibv_detach_mcast .. Event Handling Operations .. ibv_get_async_event .. ib_ack_async_event .. ibv_event_type_str .. Experimental APIs .. ibv_exp_query_device .. ibv_exp_create_qp .. ibv_exp_post_send.

9 71 Chapter 4 RDMA_CM API .. Event Channel Operations .. rdma_create_event_channel .. rdma_destroy_event_channel .. Connection Manager (CM) ID Operations .. rdma_create_id .. rdma_destroy_id .. rdma_migrate_id .. rdma_set_option .. rdma_create_ep .. rdma_destroy_ep .. rdma_resolve_addr .. rdma_bind_addr .. rdma_resolve_route.. rdma_listen .. rdma_connect .. rdma_get_request .. rdma_accept .. rdma_reject .. rdma_notify .. rdma_disconnect .. rdma_get_src_port.. rdma_get_dst_port.. rdma_get_local_addr.. rdma_get_peer_addr .. rdma_get_devices .. rdma_free_devices .. rdma_getaddrinfo.

10 Rdma_freeaddrinfo .. rdma_create_qp .. 91 Rev rdma_destroy_qp .. rdma_join_multicast .. rdma_leave_multicast .. Event Handling Operations .. rdma_get_cm_event .. rdma_ack_cm_event .. rdma_event_str .. 97 Chapter 5 RDMA Verbs API .. Protection Domain Operations .. rdma_reg_msgs .. rdma_reg_read.. rdma_reg_write .. rdma_dereg_mr .. rdma_create_srq .. rdma_destroy_srq .. Active Queue Pair Operations .. rdma_post_recvv .. rdma_post_sendv.. rdma_post_readv .. rdma_post_writev .. rdma_post_recv .. rdma_post_send.. rdma_post_read .. rdma_post_write .. rdma_post_ud_send.