InfiniBand Technology Overview - SNIA

1 Dror Goldenberg, Mellanox TechnologiesDror Goldenberg, Mellanox TechnologiesInfiniBand Technology Overview2 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights Legal NoticeThe material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions:Any slide or slides used must be reproduced without modificationThe SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentation is a project of the SNIA Education Technology Overview 2007 Storage Networking Industry Association.

2 All Rights Technology OverviewThe InfiniBand architecture brings fabric consolidation to the data center. Storage networking can concurrently run with clustering, communication and management fabrics over the same infrastructure, preserving the behavior of multiple fabrics. The tutorial provides an Overview of the InfiniBand architecture including discussion of High Speed Low Latency, Channel I/O, QoS scheduling, partitioning, high availability and protocol offload. InfiniBand based storage protocols, iSER (iSCSI RDMA Protocol), NFS over RDMA and SCSI RDMA Protocol (SRP), are introduced and compared with alternative storage protocols, such as iSCSI and FCP.

3 The tutorial further enumerates value-add features that the InfiniBand brings to clustered storage, such as atomic operations and end to end data Objectives: Understand the InfiniBand architecture and feature set. Understand the benefits of InfiniBand for networked storage. Understand the standard InfiniBand storage Technology Overview 2007 Storage Networking Industry Association. All Rights and General OverviewProtocol Stack Layers Storage Protocols over InfiniBandBenefits5 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights Need for Better I/ODatacenter trendsMulti-core CPUsBladed architectureFabric consolidationServer virtualization & consolidationIncreasing storage demandBetter I/O is requiredHigh capacityEfficient Low latencyCPU OffloadScalableVirtualization friendlyHigh availabilityPerformanceLow powerTCO reductionCompute NodeCPUCPUOSAppAppI/OCPUCoreCPUCoreOSI/O Compute NodeCPUCoreOSOSOSOSCPUCoreI/OCompute Node6 InfiniBand Technology Overview 2007 Storage Networking Industry Association.

4 All Rights InfiniBand ArchitectureIndustry standard defined by the InfiniBand Trade AssociationDefines System Area Network architectureComprehensive specification: from physical to applications Architecture supportsHost Channel Adapters (HCA)Target Channel Adapters (TCA)SwitchesRoutersFacilitated HW design for Low latency / high bandwidthTransport Manager7 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights Comparison of Fabric TechnologiesFeatures and PriceFibre ChannelStandard 10 GbEInfiniBandBandwidth4Gb/s (4 GFC)8Gb/s (8 GFC)10Gb/s20Gb/s (4x DDR)Raw Bandwidth (unidirectional)400MB/s (4 GFC)800MB/s (8 GFC)1,250 MB/s2,000 MB/s* (4x DDR)4,000 MB/s (4x QDR)Reliable ServiceYesNoYesFabric ConsolidationPractically noPractically partial**YesCopper Distance15m10 GBase-CX4 15m10 GBase-T 100mPassive SDR 20m/ DDR 10mActive DDR 25mOptical Distance100m10 GBase-SR 300m10 GBase-LRM 220m300m (SDR)150m (DDR)* 1,940 MB/s measuredCheck outSNIA Tutorial:Fibre Channelover EthernetCheck outSNIA Tutorial.

5 Fibre Channelover Ethernet**8 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights TopologiesExample topologies commonly usedArchitecture does not limit topologyModular switches are based on fat tree architectureBack to Back2 Level Fat TorusDual StarHybrid9 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights NodeULPN etworkLayerLinkLayerPhysicalLayerTranspo rtLayerApplicationInfiniBand NodeULPN etworkLayerLinkLayerPhysicalLayerTranspo rtLayerApplicationInfiniBand Protocol LayersPacket relayPHYPHYI nfiniBand RouterPacket relayPHYLinkPHYLinkInfiniBand Switch10 InfiniBand Technology Overview 2007 Storage Networking Industry Association.

6 All Rights LayerWidth (1X, 4X, 8X, 12X) including auto-negotiationSpeed (SDR/DDR/QDR) including auto-negotiation4X DDR HCAs are currently shippingPower managementPolling / SleepingConnectorBoard: MicroGiGaCNPluggable: QSFP8/10 encodingMaintain DC BalanceLimited run length of 0 s or 1 sControl symbols ( )Lane de-skew, auto negotiation, training, clock tolerance, framingLane Speed Link Width SDR ( )DDR (5 GHz)QDR (10 GHz) Speed (10 Link Speed (1099bit/sec)bit/sec)* MicroGiGaCN is a trademark of Fujitsu Components Limited11 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights Layer Cont d* currently deployedWidthSpeedConnectorTypeMinReachP owerMediaConverter300m/150mOptical Media4 XSDR/DDRM icro-GiGaCN12 strand MPO4 XDDRM icro-GiGaCN12 strand attachedFiber Optics*:Fiber Optics*:4X MicroGiGaCN WidthSpeedConnectorMinReachType / Power4 XSDR/DDRM icro-GiGaCN20 Micro-GiGaCNPassiveCopper Cables*:Copper Cables*:4X - MicroGiGaCNMPO Media Converter 12X 24 pair MicroGiGaCN 4X - MicroGiGaCNOptical Cable 12 InfiniBand Technology Overview 2007 Storage Networking Industry Association.

7 All Rights LayerAddressing and SwitchingLocal Identifier (LID) addressingUnicast LID - 48K addressesMulticast LID up to 16K addressesEfficient linear lookupCut through switching supported Multi-pathing support through LMCI ndependent Virtual LanesFlow control (lossless fabric)Service level VL arbitration for QoSCongestion controlForward / Backward Explicit Congestion Notification (FECN/BECN)Data IntegrityInvariant CRCV ariant CRCI ndependent Virtual Lanes (VLs)HighPriorityWRRLowPriorityWRRP rioritySelectPacketsto beTransmittedH/L Weighted Round Robin (WRR) VL ArbitrationEfficient FECN/BECN Based Congestion ControlSwitchthresholdFECNBECNBECNHCAHCA Per QP/VLinjection rate controlVL ARB13 InfiniBand Technology Overview 2007 Storage Networking Industry Association.

8 All Rights LayerGlobal Identifier (GID) addressingBased on IPv6 addressing schemeGID = {64 bit GID prefix, 64 bit GUID}GUID = Global Unique Identifier (64 bit EUI-64)GUID 0 assigned by the manufacturerGUID (N-1) assigned by the Subnet ManagerOptional for local subnet accessUsed for multicast distribution within end nodesEnables routing between IB subnetsStill under definition in IBTAWill leverage IPv6 routing algorithmsSubnet ASubnet BIB Router14 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights - Host Channel Adapter ModelAsynchronous interfaceConsumer posts work requestsHCA processesConsumer polls completionsTransport executed by HCAI/O channel exposed to the applicationTransport servicesReliable / UnreliableConnected / Datagram and RDMA Offload Technology Overview 2007 Storage Networking Industry Association.

9 All Rights LayerQueue Pair (QP) transport endpointAsynchronous interfaceSend Queue, Receive Queue, Completion QueueFull transport offloadSegmentation, reassembly, timers, retransmission, etcOperations supportedSend/Receive messaging semanticsRDMA Read/Write enable zero copy operationsAtomics remote Compare & Swap, Fetch & AddMemory management - Bind/Fast Register/InvalidateKernel bypassEnables low latency and CPU offloadEnabled through QPs, Completion Queues (CQs), Protection Domains (PD), Memory Regions (MRs)16 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights divide the fabric into isolated domainsPartial and full membership per partitionPartition filtering at switchesSimilar toFC VLANsHost AHost BInfiniBand fabricPartition 1 Inter-HostPartition 2private to host BPartition 3private to host APartition 4sharedI/O AI/O BI/O CI/O D17 InfiniBand Technology Overview 2007 Storage Networking Industry Association.

10 All Rights Packet HeaderPayload LenFlow LabelTClassHop LimSGID[127:96]SGID[95:64]SGID[63:32]SGI D[31:0]DGID[127:96]DGID[95:64]DGID[63:32 ]DGID[31:0]GRH (Optional)Partition KeyDestination QPTVerOpcodersvdPSNrsvdASMPadBTHI nfiniBand Data PacketExtended headers: Reliable Datagram ETH (4B) Datagram ETH (8B) RDMA ETH (16B) Atomic ETH (28B) ACK ETH (4B) Atomic ACK ETH (8B) Immediate Data ETH (4B) Invalidate ETH (4B)18 InfiniBand Technology Overview 2007 Storage Networking Industry Association. All Rights Data IntegrityHop by hopVCRC 16 bit CRCCRC16 0x100 BEnd to endICRC 32 bit CRCCRC32 0x04C11DB7 Same CRC as EthernetApplication levelT10/DIF Logical Block GuardPer block CRC16 bit CRC 0x8BB7 InfiniBand FabricFibreChannelSANVCRCVCRCVCRCICRCT10 /DIFT10/DIFVCRCVCRCVCRCICRCS witchSwitchGatewaySwitchInfiniBandBlock StorageFC Block StorageVCRC19 InfiniBand Technology Overview 2007 Storage Networking Industry Association.