Transcription of RDMA Verbs Specification 25 Apr 2003
1 Hilland, et al. [Page 1] 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829 3031323334353637383940414243444546474849 5051 Jeff Hilland Hewlett-Packard Company Paul Culley Hewlett-Packard Company Jim Pinkerton Microsoft Corporation
2 Renato Recio IBM Corporation April, 2003 RDMA Protocol Verbs Specification (Version ) 1 Status of this Memo This document is a Release Specification of the RDMA Consortium. Copies of this document and associated errata may be found at 2 Abstract This document describes an abstract interface to a RDMA enabled NIC (RNIC). This interface is implemented as a combination of the RNIC, its associated firmware, and host software.
3 It provides access to the RNIC queuing and memory management resources, as well as the underlying networking layers. RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 2] 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829 3031323334353637383940414243444546474849 5051 Table of Contents 1 Status of this Memo ..1 2 Abstract ..1 3 Introduction ..8 4 Glossary ..10 Abbreviations ..20 5 RNIC Interface ..23 The RNIC ..24 RNIC Expected Creation Sequence ..25 Expected Destruction Sequence.
4 26 Opening an Query Closing an Protection Domains ..29 Allocating a Deallocating a Completion Queues ..31 Creating a Completion Querying Completion Queue Modifying Completion Queue Destroying a Completion 6 Queue Pairs ..34 Queue Pair Resource Handling ..35 Creating a Queue Querying Queue Pair Modifying Queue Pair Destroying a Queue Queue Pair Resource States ..42 Idle Idle to Idle ..45 Idle to RTS ..45 Idle to Error ..47 RTS (Ready to Send) RTS to RTS ..49 RTS to Closing.
5 50 RTS to Terminate ..50 RTS to Error ..51 Terminate Error Closing Shared Receive Queue ..63 Creating a Shared Receive Modifying a Shared Receive Destroying a Shared Receive Associating an S-RQ with a Shared Receive Queue Processing S-RQ Error S-RQ Resource RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 3] 1 2 3 4 5 6 7 8 9 S-RQ Limit Stopping QP processing and Sending the Terminate Message ..69 Outstanding RDMA Read Resource Management.
6 72 Example IRD/ORD Connection Management ..76 Connection Active Connection Initialization after LLP Startup ..77 Passive Connection Initialization after LLP Startup ..79 Connection Normal Close ..81 ULP Initiated Termination ..82 ULP Initiated Abortive Teardown ..83 Remote Termination ..84 Local Termination, Local Abortive Teardown and Remote Abortive 7 Memory Management ..88 Memory Management Overview ..88 Steering Tag (STag) ..89 STag of Summary of Memory Region STag Memory Registration ..94 Memory Memory Region Tagged Offset (TO).
7 95 Memory Region Creation and Allocate Non-Shared Memory Region STag ..96 RI-Register Non-Shared Memory Region ..96 RI-Reregister Non-Shared Memory Region ..97 Register Shared Memory Region ..99 Fast-Register Non-Shared Memory Region ..100 Access to Registered Memory ..101 Local Access to Registered Remote Access to Registered Multiple Registrations of Memory Memory Access Control ..105 Local Access Remote Access Addressing ..107 Addressing Registered Addressing with VA based TO ..108 Addressing with Zero Based TO.
8 109 Physical Buffer Page Lists ..110 Block Lists ..111 Error Checking of Local and Remote Accesses to Querying Memory Regions ..112 Invalidating Memory Regions ..112 Deallocation of STag associated with a Memory Region ..115 Memory Allocating Memory Windows ..116 Binding Memory Windows to Memory Regions ..117 RDMA Verbs Specification 25 Apr 2003 Hilland, et al. [Page 4] 1 2 3 4 5 6 7 8 9 Querying Memory Windows ..121 Invalidating or De-allocating Memory Windows.
9 121 Invalidating or De-allocating Active Windows ..122 Summary of Memory Window STag States ..122 Error Checking during Memory Window Operations ..123 Error Checking at Window Bind Time ..123 Error Checking at Window Access Time ..124 Error Checking at Window Invalidate Time ..124 8 Work Requests and the WR Processing Model ..126 Work Requests ..126 Creating Work Work Request Send/Receive ..126 RDMA ..127 Memory ..130 Work Request Signaled Completions ..131 Scatter/Gather List ..132 RDMA Data Source & Data Sink.
10 133 Work Request Processing Model ..134 Submitting Work Request to a Work Work Request Memory Management Operation Ordering ..138 Read Fence and Local Fence Indicators ..141 Completion Returning Completed Work Asynchronous Completion Error Handling ..148 Immediate Work Completion Asynchronous 9 RNIC Verbs ..158 Consumer Accessibility ..158 RNIC Resource Management ..159 Open RNIC ..159 Query RNIC ..160 Close RNIC ..162 Protection Allocate PD ..163 Deallocate PD ..164 Completion Create CQ.
