Single Root IO Virtualization:
The whole point of SR-IOV is to reduce the latency of DMA, so the bandwidth performance of virtualized InfiniBand should be no different from the bandwidth available to non-virtualized HCAs.
SR-IOV technology reduces the overhead of I/O processing and I/O latencies by allowing each VM to access a portion of the physical I/O device directly. This reduces the need for software emulation of I/O devices in memory. The SR-IOV specification defines the architecture for a new generation of I/O devices that have multiple Virtual Functions(VFs), each of which shares common properties of a PCI physical function (PF). This approach allows each VM to see its own separate virtual device while multiple VMs share the actual physical device.
With SR-IOV, there is still a network adapter device driver that runs in VMM space. This driver is referred to as the PF driver, since its job is to manage the global functions for the SR-IOV devices and configure shared resources.
More importantly, each VM gets its own VF driver that can communicate directly with the VFs on the network adapter. Each of the VFs has its own transmit and receive buffers as well as interrupts and MAC address.
1. This arrangement reduces the need for the VMM to create “virtualized” network adapters in system memory, which saves system resources.
2. It also improves performance, since the VMM no longer performs the memory-to-memory copies between the virtualized network adapter and physical network adapter buffers.
High-Performance Virtualization: SR-IOV and InfiniBand
https://blogs.cisco.com/performance/why-is-sr-iov-relevant-in-the-hpc-world/
MPI Bandwidth
SR-IOV incurs less than 2% loss of bandwidth across the entire range of message sizes.
It is is capable of delivering over 95% of the theoretical line speed.
SRIOV relevance in HPC
1. An SR-IOV compliant device doesn’t require a hypervisor in order to be used. For instance, support for SR-IOV devices and drivers has long been available in the stock Linux kernel.
2. A VF device doesn’t require the virtual machine monitor software interposition in order to participate in I/O. Instead, the VF is a first class agent on the PCIe bus and, as such, it can originate or claim PCIe transactions autonomously.
3. A VF device doesn’t require a VM in order to be used. Instead, a VF device can be presented to any address space in the system. For example, the VF device can be presented to a usual unix-style process running in user space.
Performance Evaluation on SR-IOV
http://www.cse.iitb.ac.in/synerg/lib/exe/fetch.php?media=public:students:kallol:kallolreport.pdf
Dynamic network provisioning in SRIOV based network IO Virtualization
Experiments done with SRIOV:
1. Basic experiments with SRIOV has been done to verify its performance in virtual machines network bandwidth.
2. High level network bandwidth fairness among multiple virtual machines, connected with separate VFs in SRIOV, is verified.
3. The performance evaluation in case of inter VM communication is performed.
Experiments results:
1. SRIOV can give nearly line rate BW when there is only one VM in the system.
2. The basic fairness in between SRIOV virtual functions is proved.
3. The bandwidth in inter VM communication through virtual functions are three times larger than the line rate. This means there may be some direct channel for inter VM communication.
Citrix to introduce support for SR-IOV in Open vSwitch – UPDATED
http://virtualization.info/en/news/2010/09/citrix-to-introduce-support-for-sr-iov-in-open-vswitch.html
First of all, the next version of Citrix hypervisor will expand support for the Single Root I/O Virtualization (SR-IOV) technology, introduced in XenServer 5.6, so that its configuration will not require an interaction with the command-line interface.
Secondarily and more importantly, the next version of Open vSwitch will support SR-IOV so that virtual traffic won’t bypass the policy enforced at the switch level.
Adapter eSwitch Versus Hypervisor vSwitch
http://www.qlogic.com/OEMPartnerships/Dell/Documents/Converged/TB_Efficient_VM_to_VM_Communication.pdf
This capability enables VMs sharing the Dell-QLogic 10GbE adapter port to communicate with each other without the traffic ever exiting the server and without burdening the external network switch and associated infrastructure. No practical
limit exists for the number of VMs with network traffic that can be
switched via the eSwitch.
A hypervisor vSwitch (virtual switch) is a software program that is part of the hypervisor. A vSwitch allows one VM to communicate with another over ethernet.
Because the vSwitch is implemented in software, any traffic that flows through the vSwitch consumes CPU cycles.
Traffic cascades from a vSwitch into an eSwitch. The vSwitch is what connects the vnic in the VM through the hypervisor. The uplink for the vSwitch is a Qlogic 10gbe converged network Adapter nic partition. The advantage of the eSwitch is that it provides an efficient way of routing ethernet traffic between VMs that attach to nic partitions that share the same physical port.
ethernet traffic between such VMs need not traverse through the uplink to the external switch but is transparently and efficiently routed through the eSwitch.
SR-IOV Support in Intel-VTC
http://www.intel.in/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf
In summary, the key benefits of PCI-SIG SR-IOV on a virtualized
platform are:
1. A standard way of sharing the capacity of any given I/O device thus allowing for the most efficient use of that resource in a virtual system
2. Near native I/O performance for each virtual machine on a physical server
3. Data protection between the virtual machines on the same physical server
4. Smoother transition of a virtual machine between the physical servers thus enabling a dynamically provisioned I/O environment
High-Performance Virtualization: SR-IOV and InfiniBand
https://blogs.cisco.com/performance/why-is-sr-iov-relevant-in-the-hpc-world/
MPI Bandwidth
SR-IOV incurs less than 2% loss of bandwidth across the entire range of message sizes.
It is is capable of delivering over 95% of the theoretical line speed.
SRIOV relevance in HPC
1. An SR-IOV compliant device doesn’t require a hypervisor in order to be used. For instance, support for SR-IOV devices and drivers has long been available in the stock Linux kernel.
2. A VF device doesn’t require the virtual machine monitor software interposition in order to participate in I/O. Instead, the VF is a first class agent on the PCIe bus and, as such, it can originate or claim PCIe transactions autonomously.
3. A VF device doesn’t require a VM in order to be used. Instead, a VF device can be presented to any address space in the system. For example, the VF device can be presented to a usual unix-style process running in user space.
Performance Evaluation on SR-IOV
http://www.cse.iitb.ac.in/synerg/lib/exe/fetch.php?media=public:students:kallol:kallolreport.pdf
Dynamic network provisioning in SRIOV based network IO Virtualization
Experiments done with SRIOV:
1. Basic experiments with SRIOV has been done to verify its performance in virtual machines network bandwidth.
2. High level network bandwidth fairness among multiple virtual machines, connected with separate VFs in SRIOV, is verified.
3. The performance evaluation in case of inter VM communication is performed.
Experiments results:
1. SRIOV can give nearly line rate BW when there is only one VM in the system.
2. The basic fairness in between SRIOV virtual functions is proved.
3. The bandwidth in inter VM communication through virtual functions are three times larger than the line rate. This means there may be some direct channel for inter VM communication.
Citrix to introduce support for SR-IOV in Open vSwitch – UPDATED
http://virtualization.info/en/news/2010/09/citrix-to-introduce-support-for-sr-iov-in-open-vswitch.html
First of all, the next version of Citrix hypervisor will expand support for the Single Root I/O Virtualization (SR-IOV) technology, introduced in XenServer 5.6, so that its configuration will not require an interaction with the command-line interface.
Secondarily and more importantly, the next version of Open vSwitch will support SR-IOV so that virtual traffic won’t bypass the policy enforced at the switch level.
Adapter eSwitch Versus Hypervisor vSwitch
http://www.qlogic.com/OEMPartnerships/Dell/Documents/Converged/TB_Efficient_VM_to_VM_Communication.pdf
This capability enables VMs sharing the Dell-QLogic 10GbE adapter port to communicate with each other without the traffic ever exiting the server and without burdening the external network switch and associated infrastructure. No practical
limit exists for the number of VMs with network traffic that can be
switched via the eSwitch.
A hypervisor vSwitch (virtual switch) is a software program that is part of the hypervisor. A vSwitch allows one VM to communicate with another over ethernet.
Because the vSwitch is implemented in software, any traffic that flows through the vSwitch consumes CPU cycles.
Traffic cascades from a vSwitch into an eSwitch. The vSwitch is what connects the vnic in the VM through the hypervisor. The uplink for the vSwitch is a Qlogic 10gbe converged network Adapter nic partition. The advantage of the eSwitch is that it provides an efficient way of routing ethernet traffic between VMs that attach to nic partitions that share the same physical port.
ethernet traffic between such VMs need not traverse through the uplink to the external switch but is transparently and efficiently routed through the eSwitch.
SR-IOV Support in Intel-VTC
http://www.intel.in/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf
In summary, the key benefits of PCI-SIG SR-IOV on a virtualized
platform are:
1. A standard way of sharing the capacity of any given I/O device thus allowing for the most efficient use of that resource in a virtual system
2. Near native I/O performance for each virtual machine on a physical server
3. Data protection between the virtual machines on the same physical server
4. Smoother transition of a virtual machine between the physical servers thus enabling a dynamically provisioned I/O environment
