Comment 26 for bug 1541902

------- Comment From <email address hidden> 2016-08-30 14:40 EDT-------
(In reply to comment #24)
> Can you provide some details on how to test the GPU pass-through? I'm
> working on the SRU text for this and would like to include those details.

Sorry for the delay here, Ryan. GPU pass through is no different from any other PCI pass through.

On the host, the GPU appears as a regular PCI device. For instance:

$ lspci
...
0002:01:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40m] (rev a1)
...

To pass through this GPU to a VM, you'll need to use a libvirt XML such as the one in the following example:

<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0002' bus='0x01' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>

There is currently a limitation when you want to pass through two or more GPUs to the same VM. In these cases, you will need to increase the MMIO memory size of the VM. The MMIO memory size needs to be set to 32 GB * <number of GPUs being passed through to the VM>. For instance, if we are passing through two GPUs to the same VM, we will need to set the MMIO memory size to 64 GB (0x1000000000) by adding this to the QEMU command line:

-global spapr-pci-host-bridge.mem_win_size=0x1000000000

Or, equivalently, adding this to the libvirt domain XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
...
<qemu:commandline>
<qemu:arg value='-global'/>
<qemu:arg value='spapr-pci-host-bridge.mem_win_size=0x1000000000'/>
</qemu:commandline>
<domain>

If you are passing through 4 GPUs to the same VM, you will need to set the MMIO memory size to 128GB (0x2000000000) and so on.

Inside the VM the GPUs will appear as regular PCI devices.

Please, let me know if this is sufficient.