ISST-LTE: LPM on Ubuntu15.04 lpar hangs.

Bug #1428351 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Leann Ogasawara

Bug Description

Defect Description:
---------------------------

LPM on Ubuntu15.04 LE lpar hangs at 0%. I tried it twice and waited for more then 1 hour but there was not progress.

Details:
----------
1. LPM stuck at 0%
2. No work load tests are running on lpar.
3. Lpar is accessible though out on source CEC side.
4. Destination CEC shows lpar reference code as: "End powering on VIO slots".

Machine version details:
-------------------

root@highlp3:~# uname -a
Linux highlp3 3.18.0-13-generic #14-Ubuntu SMP Fri Feb 6 09:57:41 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
root@highlp3:~#

root@highlp3:~# cat /etc/issue
Ubuntu Vivid Vervet (development branch) \n \l

VIOS version: 2.2.3.3
Source firmware: b0211p_1507.811
Destination firmware: b0211p_1507.811

Steps to recreate:
-----------------------
1. Install Ubuntu15.04 LE on a lpar.
2. Setup LPM environment (install RMC and RSCT packages).
3. Execute LPM operation on HMC.

First, there is a kernel fix needed to make sure the stream id is passed in cpu endian during hcall to check VASI state. The following in the dmesg log confirms that this issue is present:

[ 666.978230] rtas_ibm_suspend_me: vasi_state returned -4

This is the relevant upstream commit:

commit 3df76a9dcc74d5f012b94ea01ed6e7aaf8362c5a
Author: Cyril Bur <email address hidden>
Date: Wed Jan 21 13:32:00 2015 +1100

    powerpc/pseries: Fix endian problems with LE migration

Just found that this commit took part of kernel 3.19, which is going to be the default kernel for 15.04, so, we don't need to worry about asking Canonical to integrated it.

# git checkout v3.19-rc7
# git show 3df76a9dcc74d5f012b94ea01ed6e7aaf8362c5a

I have updated the kernel in lpar.

root@highlp3:~# dpkg -l | grep linux | grep 3.19.0-6
ii linux-image-3.19.0-6-generic 3.19.0-6.6 ppc64el Linux kernel image for version 3.19.0 on PowerPC 64el SMP
ii linux-image-extra-3.19.0-6-generic 3.19.0-6.6 ppc64el Linux kernel extra modules for version 3.19.0 on PowerPC 64el SMP

Still LPM is getting hang. The difference is last time it got hung at 0% this time it got hung at 99%.

I can't seem to get the console on the desintation even after a rmvterm. I checked the cpus from the phyp debug console and all of the cpus seem to be stuck at the same place.

NIA C0000000000809AC MSR 8000000100009033 LR C00000000087D7B4

Can you please recover the lpar and look up the following addresses using addr2line:

addr2line -e <path to vmlinuz> C0000000000809AC
addr2line -e <path to vmlinuz> C00000000087D7B4

And please install the kernel source as well so we can cross reference those lines within the kernel.

Revision history for this message
bugproxy (bugproxy) wrote : dmesg

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-121922 severity-critical targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : /var/log/syslog

Default Comment by Bridge

Luciano Chavez (lnx1138)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-03-09 17:33 EDT-------
I just looked at the patch, and it seems it was accepted in 4.0 and not in 3.19 (as I said before), so, we need canonical to accept this patch.

Canonical, can you include the following commit in 15.04 kernel?
- 3df76a9dcc74d5f012b94ea01ed6e7aaf8362c5a

tags: added: targetmilestone-inin1504
removed: targetmilestone-inin---
Changed in linux (Ubuntu):
assignee: nobody → Leann Ogasawara (leannogasawara)
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 3.19.0-10.10

---------------
linux (3.19.0-10.10) vivid; urgency=low

  [ Andy Whitcroft ]

  * [Packaging] control -- make element ordering deterministic
  * [Config] allow dracult to support initramfs as well
    - LP: #1109029
  * [Packaging] generate live watchdog blacklists
    - LP: #1432837

  [ Leann Ogasawara ]

  * [Config] CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
    - LP: #1397860
  * rebase to v3.19.2

  [ Upstream Kernel Changes ]

  * thinkpad_acpi: support new BIOS version string pattern
    - LP: #1417915
  * arm64: Invalidate the TLB corresponding to intermediate page table
    levels
    - LP: #1432546
  * perf tools: Support parsing parameterized events
    - LP: #1430341
  * perf tools: Extend format_alias() to include event parameters
    - LP: #1430341
  * perf Documentation: Add event parameters
    - LP: #1430341
  * perf tools: Document parameterized and symbolic events
    - LP: #1430341
  * perf: provide sysfs_show for struct perf_pmu_events_attr
    - LP: #1430341
  * perf: add PMU_EVENT_ATTR_STRING() helper
    - LP: #1430341
  * perf: define EVENT_DEFINE_RANGE_FORMAT_LITE helper
    - LP: #1430341
  * powerpc/perf/hv-24x7: parse catalog and populate sysfs with events
    - LP: #1430341
  * powerpc/perf/{hv-gpci, hv-common}: generate requests with counters
    annotated
    - LP: #1430341
  * powerpc/perf/hv-gpci: add the remaining gpci requests
    - LP: #1430341
  * powerpc/perf/hv-24x7: Document sysfs event description entries
    - LP: #1430341
  * powerpc/iommu: Remove IOMMU device references via bus notifier
    - LP: #1425202
  * powerpc/pseries: Fix endian problems with LE migration
    - LP: #1428351
  * intel_idle: support additional Broadwell model
    - LP: #1400970
  * tools/power turbostat: support additional Broadwell model
    - LP: #1400970
  * KVM: x86: flush TLB when D bit is manually changed.
    - LP: #1397860
  * Optimize TLB flush in kvm_mmu_slot_remove_write_access.
    - LP: #1397860
  * KVM: Add generic support for dirty page logging
    - LP: #1397860
  * KVM: x86: switch to kvm_get_dirty_log_protect
    - LP: #1397860
  * KVM: Rename kvm_arch_mmu_write_protect_pt_masked to be more generic for
    log dirty
    - LP: #1397860
  * KVM: MMU: Add mmu help functions to support PML
    - LP: #1397860
  * KVM: MMU: Explicitly set D-bit for writable spte.
    - LP: #1397860
  * KVM: x86: Change parameter of kvm_mmu_slot_remove_write_access
    - LP: #1397860
  * KVM: x86: Add new dirty logging kvm_x86_ops for PML
    - LP: #1397860
  * KVM: VMX: Add PML support in VMX
    - LP: #1397860
  * HID: multitouch: add support of clickpads
  * HID: multitouch: Add support for button type usage

  [ Upstream Kernel Changes ]

  * rebase to v3.19.2
    - LP: #1428947
 -- Andy Whitcroft <email address hidden> Mon, 23 Mar 2015 15:28:16 +0000

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-04-06 16:58 EDT-------
*** Bug 123541 has been marked as a duplicate of this bug. ***

------- Comment From <email address hidden> 2015-04-06 16:58 EDT-------
*** Bug 123531 has been marked as a duplicate of this bug. ***

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.