Using SR-IOV offloads with Open-vSwitch and …

1 Using SR-IOV offloads with Open-vSwitch and similar applicationsRony Efraim, Or Gerlitz Mellanox Ra anana, use cases such as NFV pose high performance re-quirements (both in terms of bandwidth and real-time-ness)on hypervisors. When all VM traffic is handled by a hostswitching entity (such as when emulated or Para-Virtualguest network interfaces are used), a very high load on thelocal host CPU is implied. Thus, the traditional way of pro-viding network access to VMs doesn t meet the perfor-mance requirements. SR-IOV devices present opportunityto improve network performance. Up until recent versionsof the Linux kernel, SRIOV environment posed deep limi-tations on the ability of the hypervisor to manage the net-work when flow based approaches like open vswitch andTC or IP based tunnels were used. Recently, an approachthat facilitates SR-IOV performance while maintainingflow-based management Using the TC subsystem frame-work to program the SR-IOV e-switch was introduced inthe Linux kernel.

2 This allows to have hardware offloads formost traffic, and a software fallback for control traffic. Weshow how to use the new infrastructure in the kernel fromthe OVS daemon (or similar software switching applica-tions) for achieving HW offloaded data-path in SRIOV , SR-IOV , switchdev, TC, Flower, OVS,offload, flows, learning, tunnels IntroductionIn a virtual server environments, the most common way toprovide Virtual Machine (VM) switching connectivity isvia a Virtual Switch sitting in the hyper visor. The Virtualswitch is basically a software switch that acts similar to aLayer 2 hardware switch providing inbound/outbound andinter-VM communication. One of the dominant virtual switches is OVS ( open VirtualSwitch) which switches frames between local VMs on thehost (sometimes called east-west traffic) and between localVMs and remote VMs (sometimes called north-south traf-fic). One major difference between OVS and a regular IEEE Ethernet bridge is that OVS switches flows as op-pose to a regular Ethernet bridge which provides frame de-livery between VMs based on hypervisors expose emulated or para-virtual(PV) devices to guest virtual machines and multiplex theI/O requests of the guests onto the real hardware throughthe software switch residing in the host OS.

3 Each emulatedor PV device assigned to a guest has a TAP device associ-ated with it on the hypervisor where this interface isplugged to the virtual switch along with the uplink NICport. Emerging technologies such as NFV and other perfor-mance-sensitive cloud use cases, call for high performancerequirements (both in terms of bandwidth and real-time-ness) on data plane devices within network virtually all of the VMs traffic is first handled by thehost switching entity, this implies a very high load on thelocal host CPU implementing the virtual switch. Hence, thetraditional way of providing network access to VMsdoesn t meet the performance is a specification by PCI-SIG that allows a singlephysical device to expose multiple virtual devices. Thosevirtual devices can be safely assigned to guest virtual ma-chine giving them direct access to the hardware. Usinghardware directly reduces the CPU load on the hypervisorand usually results in better performance and lower way SR-IOV embedded switches are dealt with inLinux is limited in its expressiveness and flexibility, butthis is not necessarily due to hardware limitations.

4 The ker-nel software model for controlling the SR-IOV switch sim-ply did not allow the configuration of anything more com-plex than MAC/VLAN based forwarding. Hence the bene-fits brought by SRIOV come at a price of managementflexibility, when compared to software virtual switcheswhich are used in Para-Virtual (PV) schemes and allow im-plementing complex policies and virtual topologies. In our work we have created a change towards getting thebest of both worlds: the performance of SR-IOV with themanagement flexibility of software switches. This providesa richer model for controlling the SR-IOV e-switch forflow-based switching and tunneling. The code we veintegrated into the upstream Linux kernel provides theinfrastructure for Using virtual switches such as OVS inSRIOV to previous work The problem of HW offloading flow based SRIOV datapath has been previously discussed in [1].

5 Theimplementation which is described in [1] took an approachof directly offloading the flow rules from the Open-vSwitch (OVS) kernel software datapath through anextended kernel switchdev API exposed by VF representornet-devices. The solution described in the following sections which isnow merged into the upstream Linux kernel is slightlydifferent. It does involve VF representor net-devices as inthe earlier work. However, the offloading decision is donein user-space and is programmed to the HW driver throughthe TC subsystem. This difference provides a more genericmechanism that supports multiple SW switchingimplementations, not limited to OVS. It also allows forintegration with user-space software policy engine formaking the offloading decision. SRIOV switchdev offloads mode, VF Repre-sentor net-devices Up until recently, the way SR-IOV embedded switcheswere dealt with in Linux was limited in its expressivenessand flexibility.

6 The kernel software model for controllingthe SR-IOV switch did not allow the configuration of any-thing more complex than MAC/VLAN based refer to this model as the legacy kernel version , a new mode (switchdev) for SR-IOV switches was introduced. Similarly to the switchdevdrivers serving HW switch ASICs, a SW representationmodel is used (here for the SR-IOV E-Switch ports) whichin turn allows to offload the SW switch traffic rules to theHW e-switch. The new model is based on the introduction of per VFrepresentor host net-device. The VF representor plays thesame role as TAP devices in Para-Virtual (PV) setup. Apacket sent through the VF representor on the host arrivesto the VF, and a packet sent through the VF is received byits representor. The administrator can hook the representornet-device into a kernel switching component. Once theydo that, packets from the VF are subject to steering(matching and actions) of that software so indeed hurts the performance benefits of SRIOVas it forces all the traffic to go through the , this SW representation is what eventuallyallowed to introduce hybrid model, where some of theVF/VM traffic is offloaded to the HW while keeping otherVM traffic to go through the hypervisor.

7 Examples for thelatter are first packet of flows which are needed for SWswitches learning and/or matching against policy databaseor types of traffic for which offloading is not desired or notsupported by current e-switch setup of NIC Embedded switches is expected to bethrough a PCI device driver. As such, a devlink/pci basedscheme for setting the mode of the e-switch wasintroduced. The first upstream driver to implement the new SRIOV switchdev scheme is mlx5 which is serving the MellanoxConnectX4 NIC ASIC family. In the mlx5 driver, the VFrepresentors implement a functional subset of mlx5 Ethernet net-devices Using their own profile. This designbought us robust implementation with code reuse andsharing [8].The representors are created by the host PCI driver whenset in SRIOV and the e-switch is configured to switchdevmode. Currently, in mlx5 the e-switch management is donethrough the PF e-switch vport and hence the VFrepresentors along with the existing PF net-device whichrepresents the uplink share the PCI PF device mlx5, the initial setup of the e-switch to serve forsoftware based switching is provisioned with thefollowing: A single miss rule which matches all packets thatdo not match any other HW switching rule; andthe action is to forwards packets to the e-switchmanagement port for further processing.

8 Eventu-ally these packets show up at the representationport. Since all the VF reps run over the same e-switch port, we use more logic in the host PCIdriver to do HW steering of missed packets intothe HW queue opened by a the respective VFvport representor. send-to-vport rules which are higher priority thanother "normal" steering rules which present at thee-switch datapath. Such rules apply only for pack-ets that originate in the e-switch manager vport. Flow API Using TC, TC HW offloadsIn the recent years, few APIs were suggested in the Linuxnetworking kernel community to support flowprogramming from user-space. The term flow rule refers tocertain set of matchings on packet headers; an associatedset of actions is typically applied on the matching match. The TC architecture is described in [10].Two reccent proposals are: John Fastabend s Flow API [2]and Jiri Pirko s TC Flower classifier [3][4], with the latterbeing upstream since version of the kernel.

9 Flowerallows to match on L2/L3/L4 packet headers with wildcards (masking) as typically required in open -flow basedenvironments where the matching part is stated in amanner which is common for building block towards being able to actuallyoffload flow rules to HW was the introduction of aframework for offloading TC classifier rules/actions toNIC HW drivers. This was done through Amir Vadai s [5]and John Fastabend s [6] work to extend the setup_tcNDO (net-device-operation) which was previously usedonly for setting HW QoS, this extension was merged intoversion of the upstream kernel. John enabled the TCU32 classifier offloads onto the Intel 10 Gbs ixgbe NICdriver and Amir enabled TC Flower classifier offloading inthe Mellanox 100 Gbs mlx5 NIC driver. The initialoffloaded actions were HW drop (turn out that for bothhumans and HWs, the easiest thing is to say no.)

10 Inboth drivers and in mlx5 also packet marking based on HWmatching (referred to as skbedit in TC speak). HW basedmarking is a means to reduce CPU utilization in Para-Virtbased SW switching datapath, Using a breakdown of therule to two (1) HW: original matching mark and (2)SW: mark original action. Since that, more classifier offloads have been accepted orare proposed at time of writing. One of them is thematchall classifier by Yotam Gigi and Jiri Pirko which isused for offloading port mirroring in the mlxsw switchdevdriver which serves the Mellanox 100 Gbs switch ASICHW. More proposals deal with TC actions to handle IPtunneling, offloading e-BPF classifier, actions toimplement sFlow on switches, kernel TC offloading framework supports ruleaddition, replacement, deletion and statistics, where theoffloaded rule matching part is stated in a manner which isspecial to the classifier ( Flower, U32, eBPF) and theactions part are provided in their generic element which is critical when offloading SWbased switching to NIC and Switch HWs is the ability tospecify for a given flow if it should be programmed to theSW datapath, the HW datapath or both.