InfiniBand& ManycoreDay - viops.jp

1 RDMA InfiniBand & Manycore Day 00 01 InfiniBand & Manycore Day 02 InfiniBand 03 RDMA Copyright 2010 NTT DATA CORPORATION103 RDMA 04 RDMA 1 SDP05 InfiniBand 1 vSMP06 07 00 ( ) NTT IT NTT BizXaaS ( ) Copyright 2010 NTT DATA CORPORATION2 OSS Open Cloud Campus OpenStack (JOSUG) JEUG VIOPS InterCloud SIG GICTF 00 Disclaimer InfiniBand Then, why me? InfiniBand Copyright 2010 NTT DATA CORPORATION3 InfiniBand InfiniBand HCA Linux SDP DAPL (Direct Access Provider Library) RDMA ICSC Socket Extension 01 InfiniBand & Manycore Day InfiniBand 1 Manycore Copyright 2010 NTT DATA CORPORATION4 RDMA InfiniBand RDMA RDMA InfiniBand SDP, vSMP 02 InfiniBand Specification Errata Copyright 2010 NTT DATA CORPORATION5 Volume 1 : 1727pp Volume 2.

2 834pp Volume 1+2 2500page L1 L4) 02 InfiniBand Volume1 Chapter 3 Architectural Overview 55page Addressing LID Local Identifier 16bit GID Global Identifier 128bit GID IPv6 InfiniBand Copyright 2010 NTT DATA CORPORATION602 InfiniBand RC, RD, UC, UD RDMA : Remote DMA Atomic Operation Congestion Control Slow Drain Copyright 2010 NTT DATA CORPORATION703 RDMA InfiniBand HCA IB spec. p96 Copyright 2010 NTT DATA CORPORATION803 RDMA HCA NICWR Work Request QP Queue Pair CQCompletion Queue QP WR Copyright 2010 NTT DATA CORPORATION9 CQCompletion Queue QP WR WR CQE PD Protection Domain MR Memory Region HCA MW Memory Window MR 02 InfiniBand InfiniBand 1.

3 HCA PD CQ 2. QP 3. RC 4. HCA 2/3 4 Copyright 2010 NTT DATA CORPORATION104. HCA 2/3 4 5. WR QP Send/Recv RDMA RDMA Send/Recv 6. CQ 03 RDMA I/O I/O DMA I/O Copyright 2010 NTT DATA CORPORATION11 RDMA SMP 03 RDMA RDMA (user/kernel space ) CPU CPU Remote Direct Memory Access RDMA Copyright 2010 NTT DATA CORPORATION1203 RDMA 1. socket 2.

4 RDMA (ULT) Copyright 2010 NTT DATA CORPORATION1303 RDMA Socket Send/Recv HCAI/O BridgeMain MemoryKernel BufferI/OBusCopyright 2010 NTT DATA CORPORATION14 System BusCPU #1cache User BufferCPU 03 RDMA RDMA (ULT) RDMA Write HCAI/O BridgeMain MemoryKernel BufferI/OBusCopyright 2010 NTT DATA CORPORATION15 System BusCPU #1cache User BufferCPU 03 RDMA 1. socket -> I/O -> I/O -> -> (kernel buffer) -> (user buffer)2. RDMA (ULT) -> I/O -> I/O -> -> Copyright 2010 NTT DATA CORPORATION16(userbuffer) RDMA SMP 03 RDMA I/O DMA READ/WRITE RDMA Read/Write RDMA Write HCA DMA DMA READ RDMA Write HCA DMA Copyright 2010 NTT DATA CORPORATION17 DMA WRITE READ WRITE 04 RDMA 1 SDPRDMA Copyright 2010 NTT DATA CORPORATION18 RDMA UCB FastSockets [1] [1] Steven , Thomas , and David High-performance local area communication with fast sockets.

5 USENIX, RDMA 1 SDPSDP TCP Socket Direct Protocol byte-stream Recv Posting Send/Recv Copyright 2010 NTT DATA CORPORATION19 RDMA socket LD_PRELOAD 04 RDMA 1 SDP 9 SDR SDP TCP RCCopyright 2010 NTT DATA CORPORATION20 SDP TCP RC 0-copy RC Virtual SMP05 InfiniBand 1 vSMPS caleMP IB SMP vSMP 8core 16GB 4 32core 64GB SSI Node#12QC/16 GBNode#22QC/16GB2port InfiniBandCopyright 2010 NTT

6 DATA CORPORATION212QC/16 GBNode#32QC/16 GBNode#42QC/16GB2QC/16GB2port InfiniBand HCAHCAHCAHCA05 InfiniBand 1 vSMP vSMP N IB N Subnet manager Subnet manager SSI Subnet Manager SSI Copyright 2010 NTT DATA CORPORATION22 Subnet Manager SSI vSMP OS SSI 05 InfiniBand 1 vSMP spinlock spinlock CAS InfiniBand Atomic Operation Copyright 2010 NTT DATA CORPORATION23 I/O I/O etc. ScaleMP 06 InfiniBand & Manycore Day InfiniBand RDMA NFS/RDMA Oracle RAC RDS : Reliable Datagram Socket DAPL RDMA Copyright 2010 NTT DATA CORPORATION RDMA Socket API 07 RDMA RDMA Copyright 2010 NTT DATA CORPORATION Copyright 2010 NTT DATA CORPORATION26 TM Copyright 2010 NTT DATA CORPORATION28 References InfiniBand OpenFabricsCopyright 2010 NTT DATA CORPORATION29 OpenFabrics InterConnect Software Consortium