svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.15 (tsc_adjust)

Bug #1821903 reported by Po-Hsu Lin
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
New
Undecided
Unassigned
linux (Ubuntu)
Incomplete
Undecided
Unassigned
Trusty
Incomplete
Undecided
Unassigned
Xenial
Fix Committed
Undecided
Unassigned
Cosmic
Won't Fix
Undecided
Unassigned
linux-signed-lts-xenial (Ubuntu)
Trusty
Fix Committed
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Cosmic
Invalid
Undecided
Unassigned

Bug Description

This svm test timeouted on T-3.13
failed on T-4.4 with:

root@gonzo:/home/ubuntu/autotest/client/tmp/ubuntu_kvm_unit_tests/src/kvm-unit-tests# TESTNAME=svm TIMEOUT=90s ACCEL= ./x86/run x86/svm.flat -smp 2 -cpu qemu64,+svm
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel x86/svm.flat -smp 2 -cpu qemu64,+svm # -initrd /tmp/tmp.PDi0UKyCcl

enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 466000
cr4 = 20
PASS: null
PASS: vmrun
PASS: ioio
PASS: vmrun intercept check
PASS: cr3 read intercept
PASS: cr3 read nointercept
PASS: cr3 read intercept emulate
PASS: dr intercept check
PASS: msr intercept check
PASS: mode_switch
PASS: asid_zero
PASS: sel_cr0_bug
FAIL: tsc_adjust
    Latency VMRUN : max: 80562 min: 19763 avg: 20374
    Latency VMEXIT: max: 83146 min: 20193 avg: 20834
PASS: latency_run_exit
    Latency VMLOAD: max: 38298 min: 4619 avg: 4838
    Latency VMSAVE: max: 32870 min: 4637 avg: 4809
    Latency STGI: max: 55478 min: 3846 avg: 3923
    Latency CLGI: max: 67473 min: 3379 avg: 3437
PASS: latency_svm_insn
SUMMARY: 15 tests, 1 unexpected failures

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-4.4.0-143-generic 4.4.0-143.169~14.04.2
ProcVersionSignature: User Name 4.4.0-143.169~14.04.2-generic 4.4.170
Uname: Linux 4.4.0-143-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.29
Architecture: amd64
Date: Wed Mar 27 11:00:42 2019
SourcePackage: linux-signed-lts-xenial
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
summary: - svm in ubuntu_kvm_unit_test failed on T-3.13 / T-4.4
+ svm in ubuntu_kvm_unit_test failed on 3.13 4.4
summary: - svm in ubuntu_kvm_unit_test failed on 3.13 4.4
+ svm in ubuntu_kvm_unit_test failed on 3.13 / 4.4
tags: added: xenial
Revision history for this message
Po-Hsu Lin (cypressyew) wrote : Re: svm in ubuntu_kvm_unit_test failed on 3.13 / 4.4

Could be found on X-4.4 as well.

no longer affects: ubuntu-kernel-tests
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1821903

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Trusty):
status: New → Incomplete
Changed in linux (Ubuntu Xenial):
status: New → Incomplete
Po-Hsu Lin (cypressyew)
no longer affects: linux-signed-lts-xenial (Ubuntu)
Revision history for this message
Thadeu Lima de Souza Cascardo (cascardo) wrote : Re: svm in ubuntu_kvm_unit_test failed on 3.13 / 4.4

I fixed the issue with tsc-adjust on kvm-unit-tests itself, sent upstream, fixed on our disco branch. That should fix it for 4.4 kernels as well, though I only tested on 5.0 kernels for now. Trusty likely needs further review.

Revision history for this message
Thadeu Lima de Souza Cascardo (cascardo) wrote :
Changed in linux (Ubuntu Cosmic):
status: New → Fix Committed
Changed in linux (Ubuntu Xenial):
status: Incomplete → Fix Committed
Changed in linux-signed-lts-xenial (Ubuntu Trusty):
status: New → Fix Committed
Changed in linux-signed-lts-xenial (Ubuntu Xenial):
status: New → Invalid
Changed in linux-signed-lts-xenial (Ubuntu Cosmic):
status: New → Invalid
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Saw this one passed with Bionic 5.0 kernel (AMD64)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Issue still exist with Cosmic amd64 (node gonzo):
'/home/ubuntu/autotest/client/tmp/ubuntu_kvm_unit_tests/src/kvm-unit-tests/tests/svm'
 BUILD_HEAD=b5d7d7f1
 timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.VnkI4knNSX -smp 2 -cpu qemu64,+svm,+tsc-adjust # -initrd /tmp/tmp.Ool7CJM5vy
 qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
 qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
 enabling apic
 enabling apic
 paging enabled
 cr0 = 80010011
 cr3 = 467000
 cr4 = 20
 PASS: null
 PASS: vmrun
 PASS: ioio
 PASS: vmrun intercept check
 PASS: cr3 read intercept
 PASS: cr3 read nointercept
 PASS: cr3 read intercept emulate
 PASS: dr intercept check
 PASS: msr intercept check
 PASS: mode_switch
 PASS: asid_zero
 PASS: sel_cr0_bug
 FAIL: tsc_adjust
 Latency VMRUN : max: 169094 min: 17443 avg: 17769
 Latency VMEXIT: max: 127855 min: 17870 avg: 18294
 PASS: latency_run_exit
 Latency VMLOAD: max: 61609 min: 4802 avg: 4958
 Latency VMSAVE: max: 36274 min: 4676 avg: 4869
 Latency STGI: max: 72350 min: 3969 avg: 4070
 Latency CLGI: max: 69159 min: 3511 avg: 3534
 PASS: latency_svm_insn
 SUMMARY: 15 tests, 1 unexpected failures
 FAIL svm (15 tests, 1 unexpected failures)

tags: added: cosmic
tags: added: sru-20190603
tags: added: ubuntu-kvm-unit-tests
summary: - svm in ubuntu_kvm_unit_test failed on 3.13 / 4.4
+ svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.18
Changed in linux (Ubuntu Cosmic):
status: Fix Committed → Confirmed
Revision history for this message
Po-Hsu Lin (cypressyew) wrote : Re: svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.18

For node gonzo with 4.15, it's:
  WARNING: vmx not supported, add '-cpu host'
  SUMMARY: 0 tests
  SKIP vmx (0 tests)

That might explain why this has passed with Disco.
(The svm test is still failing on some specific node (bug 1847947)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Found on node gonzo with T-4.4 (4.4.0-204.236~14.04.1)

FAIL: tsc_adjust
SUMMARY: 15 tests, 1 unexpected failures

This didn't fail on other amd64 nodes like fozzie / onibi before.

tags: added: sru-20210222
Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu Cosmic):
status: Confirmed → Won't Fix
summary: - svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.18
+ svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.15
Revision history for this message
Po-Hsu Lin (cypressyew) wrote : Re: svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.15
Download full text (3.2 KiB)

The svm test will now fail early with node gonzo + 4.15.0-147

BUILD_HEAD=0fb180cf
timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.2L6tnYwNVP -smp 2 -cpu max,+svm -m 4g # -initrd /tmp/tmp.MBp5T3NwS4
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 10bf000
cr4 = 20
NPT detected - running all tests with NPT enabled
PASS: null
PASS: vmrun
PASS: ioio
PASS: vmrun intercept check
Unhandled exception 6 #UD at ip 0000000000400fc0
error_code=0000 rflags=00010002 cs=00000008
rax=0000000000000000 rcx=0000000000000000 rdx=00000000ffffffff rbx=0000000000000000
rbp=0000000000434de8 rsi=0000000000000000 rdi=000000000041a9a0
 r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000
r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000
cr0=0000000080010011 cr2=0000000000000000 cr3=00000000010bf000 cr4=0000000000000020
cr8=0000000000000000
 STACK: @400fc0 40041d
FAIL svm

$ git bisect log
git bisect start
# bad: [0fb180cfbdf236559cd10675a85b9e6d78deafc4] UBUNTU: SAUCE: vmx: split tests that fail on bionic
git bisect bad 0fb180cfbdf236559cd10675a85b9e6d78deafc4
# good: [764aa0b88d9556520457f13b38d5cea21600545b] arm: powerpc: comment halt(code)
git bisect good 764aa0b88d9556520457f13b38d5cea21600545b
# bad: [c88b7beaafd711e0ca98529a56c9d35eedc60b5f] lib/alloc.c: fix missing include
git bisect bad c88b7beaafd711e0ca98529a56c9d35eedc60b5f
# good: [d5b60621b0e6863fd62e4657b7a2ec70bf7eee34] lib: Add WRITE_ONCE and READ_ONCE implementations in compiler.h
git bisect good d5b60621b0e6863fd62e4657b7a2ec70bf7eee34
# good: [e44a257a89ef821802b21cf4fd0ee6f124278e15] arm/arm64: gic: Move gic_state enumeration to asm/gic.h
git bisect good e44a257a89ef821802b21cf4fd0ee6f124278e15
# good: [fc67b07a4b5183a336f254bc886f9032f603bd5b] s390x: smp: Test stop and store status on a running and stopped cpu
git bisect good fc67b07a4b5183a336f254bc886f9032f603bd5b
# bad: [7edd698ed003e3c4d64868208c30d33fd97168b7] scripts/arch-run: Support testing of hvf accel
git bisect bad 7edd698ed003e3c4d64868208c30d33fd97168b7
# good: [7e7d93578f02bba2e484ba49a08fa6b2ebbdc4a8] svm: Fix nmi hlt test to fail test correctly
git bisect good 7e7d93578f02bba2e484ba49a08fa6b2ebbdc4a8
# good: [a3307eb5dbdba6b657a0d9b19eb9bc0c05d04add] Merge tag 's390x-2020-04-30' of https://github.com/davidhildenbrand/kvm-unit-tests
git bisect good a3307eb5dbdba6b657a0d9b19eb9bc0c05d04add
# good: [9c8389546e1fbb6e25cc5bc22bc2f2765e9dbfbe] svm: Test V_IRQ injection
git bisect good 9c8389546e1fbb6e25cc5bc22bc2f2765e9dbfbe
# bad: [401299a581d7b255a615de482c3a9e97e090fd8a] svm_tests: add RSM intercept test

Caused by this test case:
https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/commit/401299a581d7b255a615de482c3a9e97e090fd8a
git bisect bad 401299a581d7b255a615de482c3a9e97e090fd8a
# good: [b49a1a6d4e234a60211805b7cb0db06c9b293209] x86: VMX: Add a VMX-preemption timer expiration test
git bisect good b49a1a6d4e234a60211805b7cb0db06c9b293209
# first bad commit: [401299a...

Read more...

tags: added: sru-20210621
tags: added: bionic
removed: verification-needed
Revision history for this message
Krzysztof Kozlowski (krzk) wrote :

Found on bionic/fips 4.15.0-1068.77 (and previous: 4.15.0-1067.76)

tags: added: fips-test sru-20210719
Revision history for this message
Krzysztof Kozlowski (krzk) wrote :

Found on bionic/ibm-gt 4.15.0-1098.108

Po-Hsu Lin (cypressyew)
summary: - svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.15
+ svm in ubuntu_kvm_unit_tests failed on 3.13 / 4.4 / 4.15 (tsc_adjust)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.