QEMU throws failure msg while booting guest with SRIOV VF

Bug #1630554 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Tim Gardner
Yakkety
Fix Released
Undecided
Tim Gardner
Zesty
Fix Released
Undecided
Unassigned

Bug Description

== Comment: #10 - VIPIN K. PARASHAR <email address hidden> - 2016-10-04 02:39:30 ==
(In reply to comment #9)
> I took a quick look to this one and also noticed these 2 links:
> https://bugzilla.redhat.com/show_bug.cgi?id=1237034
> http://marc.info/?l=linux-kernel&m=143737274332400&w=2
> and it looks like to avoid these message maybe just need to have this in
> config file:
> CONFIG_KVM_VFIO=y
> I think we have that in powerkvm but not in Ubuntu KVM.

On x86 running Ubuntu 16.04
==================
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
$ head /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 61
model name : Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
stepping : 4
microcode : 0x22
cpu MHz : 2300.179
cache size : 3072 KB
physical id : 0
$ uname -a
Linux Workstation 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ grep CONFIG_KVM_VFIO /boot/config-4.4.0-38-generic
CONFIG_KVM_VFIO=y
$

On PowerPC
========
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Yakkety Yak (development branch)
Release: 16.10
Codename: yakkety
# head /proc/cpuinfo
processor : 0
cpu : POWER8E (raw), altivec supported
clock : 2061.000000MHz
revision : 2.1 (pvr 004b 0201)

processor : 1
cpu : POWER8E (raw), altivec supported
clock : 2061.000000MHz
revision : 2.1 (pvr 004b 0201)

# uname -a
Linux c158f2u09os 4.8.0-17-generic #19 SMP Thu Sep 29 12:03:21 CDT 2016 ppc64le ppc64le ppc64le GNU/Linux
# grep CONFIG_KVM_VFIO /boot/config-4.8.0-17-generic
#

I see that CONFIG_KVM_VFIO is enabled on x86 running Ubuntu 16.04
while ppc64 running 16.10 doesn't have it enabled.

== Comment: #9 - Carol L. Soto <email address hidden> - 2016-09-30 16:25:32 ==
I took a quick look to this one and also noticed these 2 links:
https://bugzilla.redhat.com/show_bug.cgi?id=1237034
http://marc.info/?l=linux-kernel&m=143737274332400&w=2
and it looks like to avoid these message maybe just need to have this in config file:
CONFIG_KVM_VFIO=y
I think we have that in powerkvm but not in Ubuntu KVM.

== Comment: #6 - VIPIN K. PARASHAR <email address hidden> - 2016-09-23 06:50:22 ==
$ git log 178a787502123b0 -1
commit 178a787502123b01499c5a4617b94bb69ad49dd5
Author: David Gibson <email address hidden>
Date: Mon Feb 1 11:14:15 2016 +1100

    vfio: Enable VFIO device for powerpc

    ec53500f "kvm: Add VFIO device" added a special KVM pseudo-device which is
    used to handle any necessary interactions between KVM and VFIO.

    Currently that device is built on x86 and ARM, but not powerpc, although
    powerpc does support both KVM and VFIO. This makes things awkward in
    userspace

    Currently qemu prints an alarming error message if you attempt to use VFIO
    and it can't initialize the KVM VFIO device. We don't want to remove the
    warning, because lack of the KVM VFIO device could mean coherency problems
    on x86. On powerpc, however, the error is harmless but looks disturbing,
    and a test based on host architecture in qemu would be ugly, and break if
    we do need the KVM VFIO device for something important in future.

    There's nothing preventing the KVM VFIO device from being built for
    powerpc, so this patch turns it on. It won't actually do anything, since
    we don't define any of the arch_*() hooks, but it will make qemu happy and
    we can extend it in future if we need to.

    Signed-off-by: David Gibson <email address hidden>
    Reviewed-by: Eric Auger <email address hidden>
    Signed-off-by: Paul Mackerras <email address hidden>
$

$ git describe --contains 178a787502123b
v4.6-rc1~141^2~49^2~20
$

Commit (178a7875) seems to have enabled VFIO devices for powerpc.
But this commit should already be available since 4.6 kernel and ubuntu is
using 4.7 kernel here, so this should be already in.

== Comment: #0 - SRIKANTH B. AITHAL <email address hidden> - 2016-09-15 12:31:49 ==
---Problem Description---
QEMU throwing false failure message while booting guest with SRIOV VF device

---uname output---
Linux c158f2u09os 4.7.0unofficial #5 SMP Mon Sep 5 08:53:38 EDT 2016 ppc64le ppc64le ppc64le GNU/Linux

---Additional Hardware Info---
Mellanox CX4

Machine Type = 8247-22L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1. Edit guest ubuntu1610_srik and add below VF device [cold plug]
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0005' bus='0x01' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </hostdev>

2. Start the guest: virsh start ubuntu1610_srik

3. In the /var/log/libvirt/qemu/ubuntu1610_srik.log we will notice below failure message.

"-device vfio-pci,host=0005:01:00.3,id=hostdev0,bus=pci.0,addr=0x1: Failed to create KVM VFIO device: No such device"

But VF would have been passthroughed and probed succesfully inside the guest.. I could configure it and transfer data over it.. and VF got reprobed succesfully in host after I took down 'ubuntu1610_srik'.

This bug is to see why QEMU is printing false failure message.

=====================
Environment details:
=====================

Host :
 Ubuntu 16.10
 kernel version:
 Linux c158f2u09os 4.7.0unofficial #5 SMP Mon Sep 5 08:53:38 EDT 2016 ppc64le ppc64le ppc64le GNU/Linux
Guest:
1. ubuntu1610_srik
 kernel version: 4.7.0unofficial
2. ubuntu160401_srik
 kernel version: 4.4.0-38-generic

MOFED versions:
 MOFED version in Host as well Guest: 3.4-OFED.3.4.0.1.0.1
 CX4 Firmware version: 12.17.0222

CVE References

Revision history for this message
bugproxy (bugproxy) wrote : sosreport

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-146368 severity-medium targetmilestone-inin1610
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-10-05 14:30 EDT-------
Found below patch submitted upstream to enable VFIO KVM device
for powerpc

Subject: KVM: PPC: Finish enabling VFIO KVM device on POWER

Link: https://patchwork.ozlabs.org/patch/655305/

Not accepted upstream yet.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-10-06 03:23 EDT-------
The message means that KVM cannot accelerate TCE operations on one of DMA windows, in this particular case - the huge DMA window. Such support is not present in the HostOS as it is not in upstream kernel so there is a message.

Commit 178a7875 tried adding VFIO KVM device but it did not on PPC64 book3s KVM due to a mistake, This - https://git.kernel.org/cgit/linux/kernel/git/paulus/powerpc.git/commit/?h=kvm-ppc-next&id=4b3d173d0440d37534906b6d93c02dfb577c68ce - does add it but it is not in upstream yet.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-10-06 04:46 EDT-------
(In reply to comment #15)

> Commit 178a7875 tried adding VFIO KVM device but it did not on PPC64 book3s
> KVM due to a mistake, This -
> https://git.kernel.org/cgit/linux/kernel/git/paulus/powerpc.git/commit/
> ?h=kvm-ppc-next&id=4b3d173d0440d37534906b6d93c02dfb577c68ce - does add it
> but it is not in upstream yet.

Which is below on patchwork - KVM mailing list.

https://patchwork.kernel.org/patch/9289069/

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-10-27 09:42 EDT-------
This patch is currently upstream since 4.9-rc1.

Revision history for this message
Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu Xenial):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → In Progress
Revision history for this message
Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu Yakkety):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → In Progress
Changed in linux (Ubuntu Zesty):
assignee: Taco Screen team (taco-screen-team) → nobody
status: New → Fix Released
Luis Henriques (henrix)
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Yakkety):
status: In Progress → Fix Committed
Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-yakkety
bugproxy (bugproxy)
tags: removed: verification-needed-xenial verification-needed-yakkety
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-11-16 23:51 EDT-------
I have tested the patch on 4.8.0-28-generic.. now I do not see any errors while booting guest with VFIO device..

Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-yakkety
Tim Gardner (timg-tpi)
tags: added: verification-done-yakkety
removed: verification-needed-yakkety
bugproxy (bugproxy)
tags: removed: verification-done-yakkety verification-needed-xenial
Luis Henriques (henrix)
tags: added: verification-done-yakkety verification-needed-xenial
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for linux-lts-xenial has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (22.5 KiB)

This bug was fixed in the package linux - 4.4.0-51.72

---------------
linux (4.4.0-51.72) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1644611

  * 4.4.0-1037-snapdragon #41: kernel panic on boot (LP: #1644596)
    - Revert "dma-mapping: introduce the DMA_ATTR_NO_WARN attribute"
    - Revert "powerpc: implement the DMA_ATTR_NO_WARN attribute"
    - Revert "nvme: use the DMA_ATTR_NO_WARN attribute"

linux (4.4.0-50.71) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1644169

  * xenial 4.4.0-49.70 kernel breaks LXD userspace (LP: #1644165)
    - Revert "UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts by
      default"
    - Revert "UBUNTU: SAUCE: (namespace) fs: Don't remove suid for CAP_FSETID for
      userns root"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Don't remove suid for
      CAP_FSETID in s_user_ns""
    - Revert "UBUNTU: SAUCE: (namespace) fs: Allow superblock owner to change
      ownership of inodes"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Allow superblock owner to
      change ownership of inodes with unmappable ids""
    - Revert "UBUNTU: SAUCE: (namespace) security/integrity: Harden against
      malformed xattrs"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: ima/evm: Allow root in s_user_ns
      to set xattrs""
    - Revert "(namespace) dquot: For now explicitly don't support filesystems
      outside of init_user_ns"
    - Revert "(namespace) quota: Handle quota data stored in s_user_ns in
      quota_setxquota"
    - Revert "(namespace) quota: Ensure qids map to the filesystem"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: quota: Convert ids relative to
      s_user_ns""
    - Revert "(namespace) Revert "UBUNTU: SAUCE: quota: Require that qids passed
      to dqget() be valid and map into s_user_ns""
    - Revert "(namespace) vfs: Don't create inodes with a uid or gid unknown to
      the vfs"
    - Revert "(namespace) vfs: Don't modify inodes with a uid or gid unknown to
      the vfs"
    - Revert "UBUNTU: SAUCE: (namespace) fuse: Translate ids in posix acl xattrs"
    - Revert "UBUNTU: SAUCE: (namespace) posix_acl: Export
      posix_acl_fix_xattr_userns() to modules"
    - Revert "(namespace) vfs: Verify acls are valid within superblock's
      s_user_ns."
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Update posix_acl support to
      handle user namespace mounts""
    - Revert "(namespace) fs: Refuse uid/gid changes which don't map into
      s_user_ns"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Refuse uid/gid changes which
      don't map into s_user_ns""
    - Revert "(namespace) mnt: Move the FS_USERNS_MOUNT check into sget_userns"

linux (4.4.0-49.70) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1640921

  * Infiniband driver (kernel module) needed for Azure (LP: #1641139)
    - SAUCE: RDMA Infiniband for Windows Azure
    - [Config] CONFIG_HYPERV_INFINIBAND_ND=m
    - SAUCE: Makefile RDMA infiniband driver for Windows Azure
    - [Config] Add hv_network_direct.ko to generic inclusion list
    - SAUCE: RDMA Infiniband for Windows Azure is dependent on amd64...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (26.6 KiB)

This bug was fixed in the package linux - 4.8.0-28.30

---------------
linux (4.8.0-28.30) yakkety; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1641083

  * lxc-attach to malicious container allows access to host (LP: #1639345)
    - Revert "UBUNTU: SAUCE: (noup) ptrace: being capable wrt a process requires
      mapped uids/gids"
    - (upstream) mm: Add a user_ns owner to mm_struct and fix ptrace permission
      checks

  * [Feature] AVX-512 new instruction sets (avx512_4vnniw, avx512_4fmaps)
    (LP: #1637526)
    - x86/cpufeature: Add AVX512_4VNNIW and AVX512_4FMAPS features

  * zfs: importing zpool with vdev on zvol hangs kernel (LP: #1636517)
    - SAUCE: (noup) Update zfs to 0.6.5.8-0ubuntu4.1

  * Move some device drivers build from kernel built-in to modules
    (LP: #1637303)
    - [Config] CONFIG_TIGON3=m for all arches
    - [Config] CONFIG_VIRTIO_BLK=m, CONFIG_VIRTIO_NET=m

  * I2C touchpad does not work on AMD platform (LP: #1612006)
    - pinctrl/amd: Configure GPIO register using BIOS settings

  * guest experiencing Transmit Timeouts on CX4 (LP: #1636330)
    - powerpc/64: Re-fix race condition between going idle and entering guest
    - powerpc/64: Fix race condition in setting lock bit in idle/wakeup code

  * QEMU throws failure msg while booting guest with SRIOV VF (LP: #1630554)
    - KVM: PPC: Always select KVM_VFIO, plus Makefile cleanup

  * [Feature] KBL - New device ID for Kabypoint(KbP) (LP: #1591618)
    - SAUCE: mfd: lpss: Fix Intel Kaby Lake PCH-H properties

  * hio: SSD data corruption under stress test (LP: #1638700)
    - SAUCE: hio: set bi_error field to signal an I/O error on a BIO
    - SAUCE: hio: splitting bio in the entry of .make_request_fn

  * cleanup primary tree for linux-hwe layering issues (LP: #1637473)
    - [Config] switch Vcs-Git: to yakkety repository
    - [Packaging] handle both linux-lts* and linux-hwe* as backports
    - [Config] linux-tools-common and linux-cloud-tools-common are one per series
    - [Config] linux-source-* is in the primary linux namespace
    - [Config] linux-tools -- always suggest the base package

  * SRU: sync zfsutils-linux and spl-linux changes to linux (LP: #1635656)
    - SAUCE: (noup) Update spl to 0.6.5.8-2, zfs to 0.6.5.8-0ubuntu4 (LP:
      #1635656)

  * [Feature] SKX: perf uncore PMU support (LP: #1591810)
    - perf/x86/intel/uncore: Add Skylake server uncore support
    - perf/x86/intel/uncore: Remove hard-coded implementation for Node ID mapping
      location
    - perf/x86/intel/uncore: Handle non-standard counter offset

  * [Feature] Purley: Memory Protection Keys (LP: #1591804)
    - x86/pkeys: Add fault handling for PF_PK page fault bit
    - mm: Implement new pkey_mprotect() system call
    - x86/pkeys: Make mprotect_key() mask off additional vm_flags
    - x86/pkeys: Allocation/free syscalls
    - x86: Wire up protection keys system calls
    - generic syscalls: Wire up memory protection keys syscalls
    - pkeys: Add details of system call use to Documentation/
    - x86/pkeys: Default to a restrictive init PKRU
    - x86/pkeys: Allow configuration of init_pkru
    - x86/pkeys: Add self-tests

  * kernel invalid ...

Changed in linux (Ubuntu Yakkety):
status: Fix Committed → Fix Released
bugproxy (bugproxy)
tags: added: verification-done-xenial
removed: verification-needed-xenial
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.