RDMA Verbs Specification 25 Apr 2003 - …

1 Hilland, et al. [Page 1] 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829 3031323334353637383940414243444546474849 5051 Jeff Hilland Hewlett-Packard Company Paul Culley Hewlett-Packard Company Jim Pinkerton Microsoft Corporation Renato Recio IBM Corporation April.

2 2003 RDMA Protocol Verbs Specification (Version ) 1 Status of this Memo This document is a Release Specification of the RDMA Consortium. Copies of this document and associated errata may be found at 2 Abstract This document describes an abstract interface to a RDMA enabled NIC (RNIC). This interface is implemented as a combination of the RNIC, its associated firmware, and host software. It provides access to the RNIC queuing and memory management resources, as well as the underlying networking layers. RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 2] 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829 3031323334353637383940414243444546474849 5051 Table of Contents 1 Status of this Memo.

3 1 2 Abstract ..1 3 Introduction ..8 4 Glossary ..10 Abbreviations ..20 5 RNIC Interface ..23 The RNIC ..24 RNIC Expected Creation Sequence ..25 Expected Destruction Sequence ..26 Opening an Query Closing an Protection Domains ..29 Allocating a Deallocating a Completion Queues ..31 Creating a Completion Querying Completion Queue Modifying Completion Queue Destroying a Completion 6 Queue Pairs ..34 Queue Pair Resource Handling ..35 Creating a Queue Querying Queue Pair Modifying Queue Pair Destroying a Queue Queue Pair Resource States ..42 Idle Idle to Idle ..45 Idle to RTS ..45 Idle to Error ..47 RTS (Ready to Send) RTS to RTS.

4 49 RTS to Closing ..50 RTS to Terminate ..50 RTS to Error ..51 Terminate Error Closing Shared Receive Queue ..63 Creating a Shared Receive Modifying a Shared Receive Destroying a Shared Receive Associating an S-RQ with a Shared Receive Queue Processing S-RQ Error S-RQ Resource RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 3] 1 2 3 4 5 6 7 8 9 S-RQ Limit Stopping QP processing and Sending the Terminate Message ..69 Outstanding RDMA Read Resource Management ..72 Example IRD/ORD Connection Management ..76 Connection Active Connection Initialization after LLP Startup.

5 77 Passive Connection Initialization after LLP Startup ..79 Connection Normal Close ..81 ULP Initiated Termination ..82 ULP Initiated Abortive Teardown ..83 Remote Termination ..84 Local Termination, Local Abortive Teardown and Remote Abortive 7 Memory Management ..88 Memory Management Overview ..88 Steering Tag (STag) ..89 STag of Summary of Memory Region STag Memory Registration ..94 Memory Memory Region Tagged Offset (TO) ..95 Memory Region Creation and Allocate Non-Shared Memory Region STag ..96 RI-Register Non-Shared Memory Region ..96 RI-Reregister Non-Shared Memory Region ..97 Register Shared Memory Region ..99 Fast-Register Non-Shared Memory Region.

6 100 Access to Registered Memory ..101 Local Access to Registered Remote Access to Registered Multiple Registrations of Memory Memory Access Control ..105 Local Access Remote Access Addressing ..107 Addressing Registered Addressing with VA based TO ..108 Addressing with Zero Based TO ..109 Physical Buffer Page Lists ..110 Block Lists ..111 Error Checking of Local and Remote Accesses to Querying Memory Regions ..112 Invalidating Memory Regions ..112 Deallocation of STag associated with a Memory Region ..115 Memory Allocating Memory Windows ..116 Binding Memory Windows to Memory Regions ..117 RDMA Verbs Specification 25 Apr 2003 Hilland, et al.

7 [Page 4] 1 2 3 4 5 6 7 8 9 Querying Memory Windows ..121 Invalidating or De-allocating Memory Windows ..121 Invalidating or De-allocating Active Windows ..122 Summary of Memory Window STag States ..122 Error Checking during Memory Window Operations ..123 Error Checking at Window Bind Time ..123 Error Checking at Window Access Time ..124 Error Checking at Window Invalidate Time ..124 8 Work Requests and the WR Processing Model ..126 Work Requests ..126 Creating Work Work Request Send/Receive ..126 RDMA ..127 Memory ..130 Work Request Signaled Completions ..131 Scatter/Gather List ..132 RDMA Data Source & Data Sink.

8 133 Work Request Processing Model ..134 Submitting Work Request to a Work Work Request Memory Management Operation Ordering ..138 Read Fence and Local Fence Indicators ..141 Completion Returning Completed Work Asynchronous Completion Error Handling ..148 Immediate Work Completion Asynchronous 9 RNIC Verbs ..158 Consumer Accessibility ..158 RNIC Resource Management ..159 Open RNIC ..159 Query RNIC ..160 Close RNIC ..162 Protection Allocate PD ..163 Deallocate PD ..164 Completion Create CQ ..164 Query CQ ..165 Modify CQ ..166 Destroy CQ ..167 Shared Receive Create S-RQ ..168 Query S-RQ ..169 Modify S-RQ.

9 170 RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 5] 1 2 3 4 5 6 7 8 9 Destroy S-RQ ..171 Queue Create QP ..171 Query QP ..175 Modify QP ..177 Destroy QP ..179 Memory Allocate Non-Shared Memory Region STag ..180 Register Non-Shared Memory Region (RI-Register) ..181 Query Memory Region ..183 Deallocate STag ..184 Reregister Non-Shared Memory Region (RI-Reregister) ..185 Register Shared Memory Region ..188 Allocate Memory Window ..189 Query Memory Window ..190 Work Request Processing ..191 QP PostSQ ..191 PostRQ ..198 CQ Poll for Completion (Poll CQ).

10 199 Request Completion Notification ..201 Event Handling ..201 Set Completion Event Set Asynchronous Event Result Types ..204 Immediate Status RNIC Management verb Status ..205 PD Management verb Status ..205 CQ Management verb Status ..206 S-RQ Management verb Status ..206 QP Management verb Status ..207 Memory Management verb Status ..208 Post verb Status ..209 Event Management verb Status ..210 Completion Status Asynchronous Event 10 Security Considerations ..218 RDMA Trust Trusted Peer Access Model ..222 Untrusted Peer Access Model ..222 Spoofing, Tampering, and Information Denial of DOS When Resources Are Dedicated Per QP.