Using SR-IOV offloads with Open-vSwitch and …

1 Using SR-IOV offloads with Open-vSwitch and similar applicationsRony Efraim, Or Gerlitz Mellanox Ra anana, use cases such as NFV pose high performance re-quirements (both in terms of bandwidth and real-time-ness)on hypervisors. When all VM traffic is handled by a hostswitching entity (such as when emulated or Para-Virtualguest network interfaces are used), a very high load on thelocal host CPU is implied. Thus, the traditional way of pro-viding network access to VMs doesn t meet the perfor-mance requirements. SR-IOV devices present opportunityto improve network performance.

2 Up until recent versionsof the Linux kernel , SRIOV environment posed deep limi-tations on the ability of the hypervisor to manage the net-work when flow based approaches like Open vSwitch andTC or IP based tunnels were used. Recently, an approachthat facilitates SR-IOV performance while maintainingflow-based management Using the TC subsystem frame-work to program the SR-IOV e-switch was introduced inthe Linux kernel . This allows to have hardware offloads formost traffic, and a software fallback for control traffic. Weshow how to use the new infrastructure in the kernel fromthe OVS daemon (or similar software switching applica-tions) for achieving HW offloaded data-path in SRIOV , SR-IOV , switchdev, TC, Flower, OVS,offload, flows, learning, tunnels IntroductionIn a virtual server environments, the most common way toprovide virtual Machine (VM) switching connectivity isvia a virtual Switch sitting in the hyper visor.

3 The Virtualswitch is basically a software switch that acts similar to aLayer 2 hardware switch providing inbound/outbound andinter-VM communication. One of the dominant virtual switches is OVS (Open VirtualSwitch) which switches frames between local VMs on thehost (sometimes called east-west traffic) and between localVMs and remote VMs (sometimes called north-south traf-fic). One major difference between OVS and a regular IEEE Ethernet bridge is that OVS switches flows as op-pose to a regular Ethernet bridge which provides frame de-livery between VMs based on hypervisors expose emulated or para- virtual (PV) devices to guest virtual machines and multiplex theI/O requests of the guests onto the real hardware throughthe software switch residing in the host OS.

4 Each emulatedor PV device assigned to a guest has a TAP device associ-ated with it on the hypervisor where this interface isplugged to the virtual switch along with the uplink NICport. Emerging technologies such as NFV and other perfor-mance-sensitive cloud use cases, call for high performancerequirements (both in terms of bandwidth and real-time-ness) on data plane devices within network virtually all of the VMs traffic is first handled by thehost switching entity, this implies a very high load on thelocal host CPU implementing the virtual switch.

5 Hence, thetraditional way of providing network access to VMsdoesn t meet the performance is a specification by PCI-SIG that allows a singlephysical device to expose multiple virtual devices. Thosevirtual devices can be safely assigned to guest virtual ma-chine giving them direct access to the hardware. Usinghardware directly reduces the CPU load on the hypervisorand usually results in better performance and lower way SR-IOV embedded switches are dealt with inLinux is limited in its expressiveness and flexibility, butthis is not necessarily due to hardware limitations.

6 The ker-nel software model for controlling the SR-IOV switch sim-ply did not allow the configuration of anything more com-plex than MAC/VLAN based forwarding. Hence the bene-fits brought by SRIOV come at a price of managementflexibility, when compared to software virtual switcheswhich are used in Para- virtual (PV) schemes and allow im-plementing complex policies and virtual topologies. In our work we have created a change towards getting thebest of both worlds: the performance of SR-IOV with themanagement flexibility of software switches. This providesa richer model for controlling the SR-IOV e-switch forflow-based switching and tunneling.

7 The code we veintegrated into the upstream Linux kernel provides theinfrastructure for Using virtual switches such as OVS inSRIOV to previous work The problem of HW offloading flow based SRIOV datapath has been previously discussed in [1]. Theimplementation which is described in [1] took an approachof directly offloading the flow rules from the Open-vSwitch (OVS) kernel software datapath through anextended kernel switchdev API exposed by VF representornet-devices. The solution described in the following sections which isnow merged into the upstream Linux kernel is slightlydifferent.

8 It does involve VF representor net-devices as inthe earlier work. However, the offloading decision is donein user-space and is programmed to the HW driver throughthe TC subsystem. This difference provides a more genericmechanism that supports multiple SW switchingimplementations, not limited to OVS. It also allows forintegration with user-space software policy engine formaking the offloading decision. SRIOV switchdev offloads mode, VF Repre-sentor net-devices Up until recently, the way SR-IOV embedded switcheswere dealt with in Linux was limited in its expressivenessand flexibility.

9 The kernel software model for controllingthe SR-IOV switch did not allow the configuration of any-thing more complex than MAC/VLAN based refer to this model as the legacy kernel version , a new mode (switchdev) for SR-IOV switches was introduced. Similarly to the switchdevdrivers serving HW switch ASICs, a SW representationmodel is used (here for the SR-IOV E-Switch ports) whichin turn allows to offload the SW switch traffic rules to theHW e-switch. The new model is based on the introduction of per VFrepresentor host net-device. The VF representor plays thesame role as TAP devices in Para- virtual (PV) setup.

10 Apacket sent through the VF representor on the host arrivesto the VF, and a packet sent through the VF is received byits representor. The administrator can hook the representornet-device into a kernel switching component. Once theydo that, packets from the VF are subject to steering(matching and actions) of that software so indeed hurts the performance benefits of SRIOVas it forces all the traffic to go through the , this SW representation is what eventuallyallowed to introduce hybrid model, where some of theVF/VM traffic is offloaded to the HW while keeping otherVM traffic to go through the hypervisor.