[SRU][Zesty] Support SMMU passthrough using the default domain

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1688158

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

ccing bug for the record on testing

On Wed, May 10, 2017 at 8:33 AM, Nate Watterson
<email address hidden> wrote:
> Ok, what is the physical device that ./fio.virtualfs is being created
> on? Is it and NVME or SATA (internal or PCI attached) disk?
>
> -----Original Message-----
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Wednesday, May 10, 2017 9:06 AM
> To: Nate Watterson <email address hidden>
> Cc: Manoj Iyer <email address hidden>; Timur Tabi
> <email address hidden>
> Subject: RE: how to test iommu.passthough=0/1 ??
>
> On Wed, 10 May 2017, Nate Watterson wrote:
>
>>
>> Manoj,
>>
>> What was the backing store for the file mounted as /dev/loop0 in
>> the results
>> you sent?
>
> dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120
> mkfs.ext4 ./fio.virtualfs
> sudo losetup /dev/loop0 ./fio.virtualfs
>
>
>>
>>
>>
>> For iperf, I typically only test the target device as the iperf
>> server and
>> directly connect a host PC to run the client threads. I test this
>> way
>> because it usually has the worst performance with SMMU enabled.
>>
>>
>>
>> Target CMD: iperf -s
>>
>> Client CMD: iperf -c 192.168.2.222 -fg -i10 -t60 -P8
>>
>>
>>
>> -Nate
>>
>>
>>
>> From: Manoj Iyer [mailto:<email address hidden>]
>> Sent: Tuesday, May 9, 2017 10:29 PM
>> To: Nate Watterson <email address hidden>
>> Cc: Timur Tabi <email address hidden>
>> Subject: RE: how to test iommu.passthough=0/1 ??
>>
>>
>>
>> Nate,
>>
>>
>>
>> Thanks for that info. I will give that a try. Did you see my iperf
>> results ?
>> those were kind of odd too.. I thought with the mlx card I might be
>> testing
>> with PCIe device .. but may I was hitting a limitation with iperf..
>>
>>
>>
>> Also, I tested the same kernel on Cavium system, and with
>> iommu.passthough=1/0 was I seeing a significant difference in the
>> iops with
>> a file mounted to /dev/loop0. Its attached here..
>>
>>
>>
>>
>> On Tue, May 9, 2017 at 8:52 PM, Nate Watterson
>> <email address hidden>
>> wrote:
>>
>> Hi Manoj,
>>
>> Sorry for the late reply. I finally found some time to test
>> passthrough mode and verified using the SMMU performance
>> counters that the ‘iommu.passthrough’ param is working as
>> I
>> would expect it to (1 == Passthrough / 0 == non-Passthrough).
>> I
>> am not sure why you’re seeing this strange behavior but
>> based on
>> the logs you’ve sent, I have a few comments:
>>
>>
>>
>> Your boot log is showing that only the PCIe SMMUs are enabled
>> (this is good, it means your FW is recent). Because of this,
>> you
>> will need to do your testing with a PCIe attached device
>> instead
>> of the SD card or SSD (assuming it was connected to the
>> onboard
>> SATA). I would recommend using an NVME drive as the impact of
>> enabling passthrough mode will only really be apparent with
>> devices capable of sustaining high IO throughput.
>>
>>
>>
>> I would recommend changing your FIO profile to access the
>> device
>> directly instead of going through the filesyst...

ccing bug for the record on testing.

On Wed, May 10, 2017 at 10:07 AM, Nate Watterson
<email address hidden> wrote:
> Unless this is a PCIe attached SATA disk, you will not see any impact
> from the iommu.passthrough setting since the SATA SMMUs are not even
> enabled! Additionally, the FIO results you sent last week I think
> showed rates beyond the physical capabilities of SATA so you should
> probably make sure that file system caching is disabled or just
> access the disk directly using a readonly profile. Also, the SMMU is
> sufficiently fast that you will typically not even see the SMMU
> impact on a relatively low BW peripheral like SATA. I would recommend
> getting an NVME drive.
> -Nate
>
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Wednesday, May 10, 2017 10:05 AM
> To: Nate Watterson <email address hidden>
> Cc: Timur Tabi <email address hidden>
> Subject: RE: how to test iommu.passthough=0/1 ??
>
>
>
> On Wed, May 10, 2017 at 8:33 AM, Nate Watterson
> <email address hidden> wrote:
>
>> Ok, what is the physical device that ./fio.virtualfs is being
>> created on? Is it and NVME or SATA (internal or PCI attached) disk?
>
> Ah! it was on the SATA drive. Sorry misunderstood your initial
> question.
>
>
>> -----Original Message----- From: Manoj Iyer
>> [mailto:<email address hidden>] Sent: Wednesday, May 10, 2017 9:06
>> AM To: Nate Watterson <email address hidden> Cc: Manoj Iyer
>> <email address hidden>; Timur Tabi <email address hidden>
>> Subject: RE: how to test iommu.passthough=0/1 ?? On Wed, 10 May
>> 2017, Nate Watterson wrote:
>>> Manoj, What was the backing store for the file mounted as
>>> /dev/loop0 in the results you sent?
>> dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 mkfs.ext4
>> ./fio.virtualfs sudo losetup /dev/loop0 ./fio.virtualfs
>>> For iperf, I typically only test the target device as the iperf
>>> server and directly connect a host PC to run the client threads. I
>>> test this way because it usually has the worst performance with
>>> SMMU enabled. Target CMD: iperf -s Client CMD: iperf -c
>>> 192.168.2.222 -fg -i10 -t60 -P8 -Nate From: Manoj Iyer
>>> [mailto:<email address hidden>] Sent: Tuesday, May 9, 2017 10:29
>>> PM To: Nate Watterson <email address hidden> Cc: Timur Tabi
>>> <email address hidden> Subject: RE: how to test
>>> iommu.passthough=0/1 ?? Nate, Thanks for that info. I will give
>>> that a try. Did you see my iperf results ? those were kind of odd
>>> too.. I thought with the mlx card I might be testing with PCIe
>>> device .. but may I was hitting a limitation with iperf.. Also,
>>> I tested the same kernel on Cavium system, and with
>>> iommu.passthough=1/0 was I seeing a significant difference in the
>>> iops with a file mounted to /dev/loop0. Its attached here.. On
>>> Tue, May 9, 2017 at 8:52 PM, Nate Watterson
>>> <email address hidden> wrote: Hi Manoj, Sorry for the late
>>> reply. I finally found some time to test passthrough mode and
>>> verified using the SMMU performance counters that the
>>> ‘iommu.passthrough’ param is working as I would expect it to (1
...

Jeff,

I have back-ported patches to address "Support SMMU passthrough using
the default domain". This adds a new kernel command line option
iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64.
http://bugs.launchpad.net/bugs/1688158

0 - Use IOMMU translation for DMA.
1 - Bypass the IOMMU for DMA.

The test case recommended by Nate using 'fio' is:

$ sudo apt update ; sudo apt install fio

$ sudo fio --name=global --readonly --group_reporting --direct=1
--ioengine=libaio --rw=read --eta-newline=1s --size=1T
--blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0
--filename=/dev/nvme0n1

You can find the kernel and the instructions to install the kernel in
the Centriq test PPA:
https://launchpad.net/~centriq-team/+archive/ubuntu/test/

Nate and I have gone back and forth on testing these patches and
finally came down to having this test run on an NVME device.
Unfortunately I cant seem to find any nvme HW at this time, we would
need to order these drive and could take time.

Could you please test the kernel using the test case mentioned above
and reply back to this email with test case out put for both cases.
iommu.passthrough=0 and iommu.passthrough=1. We should see better
performance (iops may be) with IOMMU bypassed for DMA (ie
iommu.passthrough=1).

The bug is cced on this email, so if you replay with a summary output
in both cases it will be automatically recorded in the bug. Once I get
your test results I will be able to make a case for SRUing these
patches.

Thanks a ton
Manoj Iyer

Just FYI, it will likely take me a few days to get to this.

Jeffrey Hugo
Senior Engineer
Qualcomm Datacenter Technologies, Inc.
1-303-247-5002

From: Manoj Iyer [mailto:<email address hidden>]
Sent: Thursday, May 11, 2017 9:23 PM
To: Jeff Hugo <email address hidden>; <email address hidden>
Cc: Nate Watterson <email address hidden>; Timur Tabi <email address hidden>
Subject: [Bug #1688158] Please help test kernel for iommu.passthrough=0/1

Jeff,

I have back-ported patches to address "Support SMMU passthrough using the default domain". This adds a new kernel command line option iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64. http://bugs.launchpad.net/bugs/1688158

0 - Use IOMMU translation for DMA.
1 - Bypass the IOMMU for DMA.

The test case recommended by Nate using 'fio' is:

$ sudo apt update ; sudo apt install fio

$ sudo fio --name=global --readonly --group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1

You can find the kernel and the instructions to install the kernel in the Centriq test PPA:
https://launchpad.net/~centriq-team/+archive/ubuntu/test/

Nate and I have gone back and forth on testing these patches and finally came down to having this test run on an NVME device. Unfortunately I cant seem to find any nvme HW at this time, we would need to order these drive and could take time.

Could you please test the kernel using the test case mentioned above and reply back to this email with test case out put for both cases. iommu.passthrough=0 and iommu.passthrough=1. We should see better performance (iops may be) with IOMMU bypassed for DMA (ie iommu.passthrough=1).

The bug is cced on this email, so if you replay with a summary output in both cases it will be automatically recorded in the bug. Once I get your test results I will be able to make a case for SRUing these patches.

Thanks a ton
Manoj Iyer

On Fri, May 12, 2017 at 9:06 AM, Jeff Hugo <email address hidden>
wrote:
> Just FYI, it will likely take me a few days to get to this.

Thanks Jeff, no problem.

>
> Jeffrey Hugo
> Senior Engineer
> Qualcomm Datacenter Technologies, Inc.
> 1-303-247-5002
>
> From: Manoj Iyer [mailto:<email address hidden>]
> Sent: Thursday, May 11, 2017 9:23 PM
> To: Jeff Hugo <email address hidden>; <email address hidden>
> Cc: Nate Watterson <email address hidden>; Timur Tabi
> <email address hidden>
> Subject: [Bug #1688158] Please help test kernel for
> iommu.passthrough=0/1
>
> Jeff,
>
> I have back-ported patches to address "Support SMMU passthrough using
> the default domain". This adds a new kernel command line option
> iommu.passthrough=0/1 to Configure DMA to bypass the IOMMU for ARM64.
> http://bugs.launchpad.net/bugs/1688158
>
> 0 - Use IOMMU translation for DMA.
> 1 - Bypass the IOMMU for DMA.
>
> The test case recommended by Nate using 'fio' is:
>
> $ sudo apt update ; sudo apt install fio
>
> $ sudo fio --name=global --readonly --group_reporting --direct=1
> --ioengine=libaio --rw=read --eta-newline=1s --size=1T
> --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_0 --filename=/dev/nvme0n1
>
> You can find the kernel and the instructions to install the kernel in
> the Centriq test PPA:
> https://launchpad.net/~centriq-team/+archive/ubuntu/test/
>
> Nate and I have gone back and forth on testing these patches and
> finally came down to having this test run on an NVME device.
> Unfortunately I cant seem to find any nvme HW at this time, we would
> need to order these drive and could take time.
>
> Could you please test the kernel using the test case mentioned above
> and reply back to this email with test case out put for both cases.
> iommu.passthrough=0 and iommu.passthrough=1. We should see better
> performance (iops may be) with IOMMU bypassed for DMA (ie
> iommu.passthrough=1).
>
> The bug is cced on this email, so if you replay with a summary output
> in both cases it will be automatically recorded in the bug. Once I
> get your test results I will be able to make a case for SRUing these
> patches.
>
> Thanks a ton
> Manoj Iyer

Boot tested on Thunder-X no regressions.

Loading Linux 4.10.0-20-generic ...
Loading initial ramdisk ...
Ds1337GetDatetime Error Getting TimeEFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.10.0-20-generic (buildd@bos01-arm64-006) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) ) #23~lp1688158+sru3-Ubuntu SMP Thu May 11 20:54:26 UTC 2017 (Ubuntu 4.10.0-20.23~lp1688158+sru3-generic 4.10.8)
[ 0.000000] Boot CPU: AArch64 Processor [430f0a11]
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: EFI v2.40 by American Megatrends
[ 0.000000] efi: ESRT=0x1ffce3ce18 SMBIOS 3.0=0x1ffce3cb18 ACPI 2.0=0x1ffeafd000
[ 0.000000] esrt: Reserving ESRT space from 0x0000001ffce3ce18 to 0x0000001ffce3ce50.

$ uname -a
Linux kosberg 4.10.0-20-generic #23~lp1688158+sru3-Ubuntu SMP Thu May 11 20:54:26 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.10.0-20-generic root=UUID=872f9ad6-19e8-48d7-b109-0024974284b6 ro iommu.passthrough=1

It’s working as expected. I see improvement with passthrough enabled (see below). Nate also confirmed the SMMU setup via JTAG.

Passthrough=0 -

root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly --group_reporting --direct=1 --ioengine=libaio --rw=read --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 00m:05s]
Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:03s]
Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s] [5080/0/0 iops] [eta 00m:01s]
Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0 iops] [eta 00m:00s]
nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26 2017
  read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
    slat (usec): min=32, max=440, avg=191.74, stdev=157.14
    clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
     lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
    clat percentiles (usec):
     | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6112],
     | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[ 6176],
     | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[ 6432],
     | 99.99th=[ 7008]
    bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84, stdev= 9.44
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
  cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
  IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s, maxb=2537.5MB/s, mint=10001msec, maxt=10001msec

Disk stats (read/write):
  nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828, util=91.79%

passthrough=1 –

root@null-8cfdf006971f:/home/ubuntu#
ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [R(1)] [36.4% done] [3238MB/0KB/0KB /s] [6476/0/0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [R(1)] [4...

Thanks Jeff/Nate I will sru these patches shortly.

On Mon, May 15, 2017 at 2:55 PM, Jeff Hugo <email address hidden>
wrote:
> It’s working as expected. I see improvement with passthrough
> enabled (see below). Nate also confirmed the SMMU setup via JTAG.
>
> Passthrough=0 -
>
> root@null-8cfdf006971f:/home/ubuntu# fio --name=global --readonly
> --group_reporting --direct=1 --ioengine=libaio --rw=read
> --eta-newline=1s --size=1T --blocksize=512k --iodepth=32 --numjobs=1
> --runtime=10s --name=nvme_0 --filename=/dev/nvme0n1
> nvme_0: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K,
> ioengine=libaio, iodepth=32
> fio-2.2.10
> Starting 1 process
> Jobs: 1 (f=1): [R(1)] [36.4% done] [2540MB/0KB/0KB /s] [5080/0/0
> iops] [eta 00m:07s]
> Jobs: 1 (f=1): [R(1)] [45.5% done] [2539MB/0KB/0KB /s] [5077/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [54.5% done] [2540MB/0KB/0KB /s]
> [5079/0/0 iops] [eta 00m:05s]
> Jobs: 1 (f=1): [R(1)] [63.6% done] [2540MB/0KB/0KB /s] [5079/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [72.7% done] [2539MB/0KB/0KB /s]
> [5078/0/0 iops] [eta 00m:03s]
> Jobs: 1 (f=1): [R(1)] [81.8% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:Jobs: 1 (f=1): [R(1)] [90.9% done] [2540MB/0KB/0KB /s]
> [5080/0/0 iops] [eta 00m:01s]
> Jobs: 1 (f=1): [R(1)] [100.0% done] [2539MB/0KB/0KB /s] [5078/0/0
> iops] [eta 00m:00s]
> nvme_0: (groupid=0, jobs=1): err= 0: pid=36481: Mon May 15 17:10:26
> 2017
> read : io=25377MB, bw=2537.5MB/s, iops=5074, runt= 10001msec
> slat (usec): min=32, max=440, avg=191.74, stdev=157.14
> clat (usec): min=153, max=7641, avg=6106.54, stdev=107.40
> lat (usec): min=442, max=7904, avg=6298.45, stdev=174.49
> clat percentiles (usec):
> | 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 6048], 20.00th=[
> 6048],
> | 30.00th=[ 6112], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[
> 6112],
> | 70.00th=[ 6112], 80.00th=[ 6112], 90.00th=[ 6176], 95.00th=[
> 6176],
> | 99.00th=[ 6304], 99.50th=[ 6368], 99.90th=[ 6432], 99.95th=[
> 6432],
> | 99.99th=[ 7008]
> bw (MB /s): min= 2498, max= 2540, per=99.98%, avg=2536.84,
> stdev= 9.44
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
> lat (msec) : 2=0.01%, 4=0.02%, 10=99.96%
> cpu : usr=3.84%, sys=96.12%, ctx=67, majf=0, minf=528
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=50754/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=25377MB, aggrb=2537.5MB/s, minb=2537.5MB/s,
> maxb=2537.5MB/s, mint=10001msec, maxt=10001msec
>
> Disk stats (read/write):
> nvme0n1: ios=200550/0, merge=0/0, ticks=23848/0, in_queue=23828,
> util=91.79%
>
>
>
> passthrough=1 –
>
> root@null-8cfdf006971f:/home/ubuntu#
> ing --direct=1 --ioengine=libaio --rw=read --eta-newline=1s
> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 --runtime=10s
> --name=nvme_...

Tested on Power8
ubuntu@manjo-srutest:~$ uname -a
Linux manjo-srutest 4.10.0-22-generic #24~sru4+test.1-Ubuntu SMP Wed May 24 18:42:19 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@manjo-srutest:~$

I get a kernel panic on AMD64 system .. investigating.

This is fixed in upstream by patch, will cherry-pick this patch and test on intel system to fix this regression, and resubmit the sru.

commit 0ef7b235aeddf9b2f78889c7823a24745ebf0cf7
Author: Andy Shevchenko <email address hidden>
Date: Wed Feb 15 16:42:21 2017 +0200

    iommu/vt-d: Fix crash on boot when DMAR is disabled

    By default CONFIG_INTEL_IOMMU_DEFAULT_ON is not set and thus
    dmar_disabled variable is set.

    Intel IOMMU driver based on above doesn't set intel_iommu_enabled
    variable.

    The commit b0119e870837 ("iommu: Introduce new 'struct iommu_device'")
    mistakenly assumes it never happens and tries to unregister not ever
    registered resources, which crashes the kernel at boot time:

This patch was applied on top of the patch series to implement smmu pass though on zesty kernel and tested on Dell PowerEdge T710 and it fixes the regression.
ubuntu@adib:~$ uname -a
Linux adib 4.10.0-21-generic #23~sru4.test5 SMP Tue May 30 18:25:49 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@adib:~$

Re-tested boot on ARM Centriq platform from the PPA.
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.10.0-21-generic #23~sru4.test4-Ubuntu SMP Tue May 30 21:04:32 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ubuntu:~$

Re-test boot on Power8 from PPA.
ubuntu@manjo-srutest:~$ uname -a
Linux manjo-srutest 4.10.0-21-generic #23~sru4.test4-Ubuntu SMP Tue May 30 21:03:42 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@manjo-srutest:~$

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-zesty' to 'verification-done-zesty'. If the problem still exists, change the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Jeff,

Could someone from QTI please help test
http://launchpad.net/bugs/1688158 zesty-proposed kernel for SMMU
passthough? We won't have an NVME drive (yet) installed in time for
this SRU verification. I have boot tested this kernel and it boots fine
on the QDF2400.

To enable proposed add the following line to your /etc/apt/sources.list

deb http://us.ports.ubuntu.com/ubuntu-ports/ zesty-proposed main
restricted

Then install the kernel and fio tool.
$ sudo apt update

$ sudo apt install linux-image-4.10.0-23-generic
linux-image-extra-4.10.0-23-generic

$ sudo apt install fio

For kernel parameter cases 0/1 run the tests.
iommu.passthough=0/1

$ sudo fio --name=global --readonly --group_reporting --direct=1
--ioengine=libaio --rw=read --eta-newline=1s --size=1T --blocksize=512k
--iodepth=32 --numjobs=1 --runtime=10s --name=nvme_0
--filename=/dev/nvme0n1

After you are done posting the results to the bug. Please update the
tag "verification-needed-zesty" to "verification-done-zesty"

Thanks
Manoj Iyer

Sorry for the delays. The nvme fio test didn't stress the system enough with this build, so I setup two systems with a direct wired, back to back, high performance network link and ran iperf. With the client in pass through, and the server in passthough/nonpassthrough, the data rate was slightly less than half in the nonpassthrough case.

This is verified

This bug was fixed in the package linux - 4.10.0-26.30

---------------
linux (4.10.0-26.30) zesty; urgency=low

  * linux: 4.10.0-26.30 -proposed tracker (LP: #1700528)

  * CVE-2017-1000364
    - Revert "UBUNTU: SAUCE: mm: Only expand stack if guard area is hit"
    - Revert "mm: do not collapse stack gap into THP"
    - Revert "mm: enlarge stack guard gap"
    - mm: larger stack guard gap, between vmas
    - mm: fix new crash in unmapped_area_topdown()
    - Allow stack to grow up to address space limit

linux (4.10.0-25.29) zesty; urgency=low

  * linux: 4.10.0-25.29 -proposed tracker (LP: #1699028)

  * CVE-2017-1000364
    - SAUCE: mm: Only expand stack if guard area is hit

  * CVE-2017-9074
    - ipv6: Prevent overrun when parsing v6 header options
    - ipv6: Check ip6_find_1stfragopt() return value properly.

  * [Zesty] QDF2400 ARM64 server - NMI watchdog: BUG: soft lockup - CPU#8 stuck
    for 22s! (LP: #1680549)
    - iommu/dma: Stop getting dma_32bit_pfn wrong
    - iommu/dma: Implement PCI allocation optimisation
    - iommu/dma: Convert to address-based allocation
    - iommu/dma: Clean up MSI IOVA allocation
    - iommu/dma: Plumb in the per-CPU IOVA caches
    - iommu/iova: Fix underflow bug in __alloc_and_insert_iova_range

  * Zesty update to 4.10.17 stable release (LP: #1692898)
    - xen: adjust early dom0 p2m handling to xen hypervisor behavior
    - target: Fix compare_and_write_callback handling for non GOOD status
    - target/fileio: Fix zero-length READ and WRITE handling
    - iscsi-target: Set session_fall_back_to_erl0 when forcing reinstatement
    - usb: xhci: bInterval quirk for TI TUSB73x0
    - usb: host: xhci: print correct command ring address
    - USB: serial: ftdi_sio: add device ID for Microsemi/Arrow SF2PLUS Dev Kit
    - USB: Proper handling of Race Condition when two USB class drivers try to
      call init_usb_class simultaneously
    - USB: Revert "cdc-wdm: fix "out-of-sync" due to missing notifications"
    - staging: vt6656: use off stack for in buffer USB transfers.
    - staging: vt6656: use off stack for out buffer USB transfers.
    - staging: gdm724x: gdm_mux: fix use-after-free on module unload
    - staging: wilc1000: Fix problem with wrong vif index
    - staging: comedi: jr3_pci: fix possible null pointer dereference
    - staging: comedi: jr3_pci: cope with jiffies wraparound
    - usb: misc: add missing continue in switch
    - usb: gadget: legacy gadgets are optional
    - usb: Make sure usb/phy/of gets built-in
    - usb: hub: Fix error loop seen after hub communication errors
    - usb: hub: Do not attempt to autosuspend disconnected devices
    - x86/boot: Fix BSS corruption/overwrite bug in early x86 kernel startup
    - selftests/x86/ldt_gdt_32: Work around a glibc sigaction() bug
    - x86, pmem: Fix cache flushing for iovec write < 8 bytes
    - um: Fix PTRACE_POKEUSER on x86_64
    - perf/x86: Fix Broadwell-EP DRAM RAPL events
    - KVM: x86: fix user triggerable warning in kvm_apic_accept_events()
    - KVM: arm/arm64: fix races in kvm_psci_vcpu_on
    - arm64: KVM: Fix decoding of Rt/Rt2 when trapping AArch32 CP accesses
    - block: fix blk_integrity_register to use templ...