[LTCTest][Opal][FW860.20] HMI recoverable errors failed to recover and system goes to dump state.

Bug #1684054 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
High
Canonical Kernel Team
linux (Ubuntu)
Fix Released
High
Manoj Iyer
Zesty
Fix Released
High
Unassigned

Bug Description

== Comment: #0 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-17 06:08:41 ==
---Problem Description---
HMI Recoverable error injection tests leads to system checkstop followed by system dump with ubuntu 17.04 os and kernel 4.10.0-19-generic ppc64le

Contact Information = <email address hidden>

---uname output---
#21-Ubuntu SMP Thu Apr 6 17:03:05 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = PowerNV 8284-22A

---System Hang---
 System is in dumping state. after dump finishes system will IPL to OS again.

---Debugger---
A debugger is not configured

== Comment: #3 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-17 06:12:51 ==
# uname -a
#21-Ubuntu SMP Thu Apr 6 17:03:05 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
# cat /etc/os-release
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty
root@p8wookie:~#

== Comment: #4 - Kevin W. Rudd <email address hidden> - 2017-04-17 11:10:22 ==

== Comment: #5 - MAHESH J. SALGAONKAR <email address hidden> - 2017-04-17 13:34:03 ==
it looks like below commit is a culprit:

=======================================
commit 2337d207288f163e10bd8d4d7eeb0c1c75046a0c
Author: Nicholas Piggin <email address hidden>
Date: Fri Jan 27 14:24:33 2017 +1000

    powerpc/64: CONFIG_RELOCATABLE support for hmi interrupts

    The branch from hmi_exception_early to hmi_exception_realmode must use
    a "relocatable-style" branch, because it is branching from unrelocated
    exception code to beyond __end_interrupts.

    Signed-off-by: Nicholas Piggin <email address hidden>
    Signed-off-by: Michael Ellerman <email address hidden>
=======================================

With the above commit changes now hmi_exception_realmode() is called using bctrl which ends up messing up TOC (r2) value and further access using new r2 results into unpredictable behaviour.

----------------------------------------
c000000000025f50 <hmi_exception_realmode>:
c000000000025f50: 3a 01 4c 3c addis r2,r12,314
c000000000025f54: b0 01 42 38 addi r2,r2,432
c000000000025f58: a6 02 08 7c mflr r0
-----------------------------------------

With above commit the hmi_exception_early() code jumps to c000000000025f50 (hmi_exception_realmode+0x0) which then sets up new value for r2.

If we revert above commit the code jumps to c000000000025f58 (hmi_exception_realmode+0x8) and hmi handler works fine.

After reverting above patch I don't see this issue anymore. I have rebuilt the ubuntu kernel after reverting above patch and you can find the kernel rpm at:

Can you please retry your tests with above kernel and see if issue still persists.

== Comment: #6 - MAHESH J. SALGAONKAR <email address hidden> - 2017-04-17 23:02:31 ==
Spoke to Michael Ellerman this morning. He helped me to identify the root cause and a fix patch beow:

diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 857bf7c5b946..7cfeb8768587 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -982,7 +982,7 @@ TRAMP_REAL_BEGIN(hmi_exception_early)
  EXCEPTION_PROLOG_COMMON_2(PACA_EXGEN)
  EXCEPTION_PROLOG_COMMON_3(0xe60)
  addi r3,r1,STACK_FRAME_OVERHEAD
- BRANCH_LINK_TO_FAR(r4, hmi_exception_realmode)
+ BRANCH_LINK_TO_FAR(r12, hmi_exception_realmode)
  /* Windup the stack. */
  /* Move original HSRR0 and HSRR1 into the respective regs */
  ld r9,_MSR(r1)

== Comment: #7 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-18 01:52:03 ==

== Comment: #8 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-18 01:53:57 ==
Hi Mahesh
Tested all the HMI Recoverable errors on the below patched kernel, attached the corresponding executing logs. All tests are working fine.

#21 SMP Mon Apr 17 12:58:30 EDT 2017 ppc64le ppc64le ppc64le GNU/Linux

Thanks

== Comment: #9 - MAHESH J. SALGAONKAR <email address hidden> - 2017-04-18 06:07:56 ==
(In reply to comment #8)
> Hi Mahesh
> Tested all the HMI Recoverable errors on the below patched kernel, attached
> the corresponding executing logs. All tests are working fine.
>
> Linux p8wookie 4.10.0-19.bz153487-generic #21 SMP Mon Apr 17 12:58:30 EDT
> 2017 ppc64le ppc64le ppc64le GNU/Linux
>
>
> Thanks

Thanks. Michael has posted fix for this upstream.

http://patchwork.ozlabs.org/patch/751647/

I will rebuild the new ubuntu kernel with above patch.

== Comment: #12 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-18 09:27:59 ==
(In reply to comment #11)
> >
> > https://git.kernel.org/powerpc/c/be5c5e843c4afa1c8397cb740b6032
>
> I have built new kernel with above patch and you can find it below path
>
>:/home2/mahesh/u2/bz153487v2/linux-image-4.10.0-19.bz153487v2-
> generic_4.10.0-19.bz153487v2.21_ppc64el.deb

Tested with this new patched kernel, all tests are working fine.

Linux p8wookie 4.10.0-19.bz153487v2-generic #21 SMP Tue Apr 18 07:43:13 EDT 2017 ppc64le ppc64le ppc64le GNU/Linux

Will attach is full the execution logs here.

== Comment: #13 - Pridhiviraj Paidipeddi <email address hidden> - 2017-04-18 09:29:43 ==

== Comment: #14 - MAHESH J. SALGAONKAR <email address hidden> - 2017-04-19 03:52:18 ==
(In reply to comment #12)
> (In reply to comment #11)
> > >
> > > https://git.kernel.org/powerpc/c/be5c5e843c4afa1c8397cb740b6032
> >

Thanks for testing. We need to mirror this to ubuntu for fix patch inclusion

>
> Linux p8wookie 4.10.0-19.bz153487v2-generic #21 SMP Tue Apr 18 07:43:13 EDT
> 2017 ppc64le ppc64le ppc64le GNU/Linux
>
> Will attach is full the execution logs here.

Revision history for this message
bugproxy (bugproxy) wrote : sosreport

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-153487 severity-critical targetmilestone-inin1704
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → kernel-package (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-05-05 01:34 EDT-------
*** Bug 153729 has been marked as a duplicate of this bug. ***

Revision history for this message
Alastair D'Silva (deece) wrote :

Have the same problem when resetting CAPI cards. I've confirmed that Ubuntu-4.10.0-21.23 + be5c5e843c4afa1c8397cb740b6032bd4142f32d from upstream fixes the problem.

Manoj Iyer (manjo)
Changed in kernel-package (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Manoj Iyer (manjo)
Revision history for this message
Manoj Iyer (manjo) wrote :

Can you please test the kernel in https://launchpad.net/~ubuntu-power-triage/+archive/ubuntu/test/ and report back here?

Changed in kernel-package (Ubuntu):
importance: Undecided → High
status: New → Incomplete
Changed in ubuntu-power-systems:
status: New → Incomplete
Revision history for this message
bugproxy (bugproxy) wrote : HMI_TFMR_ERRORS

------- Comment (attachment only) From <email address hidden> 2017-05-15 05:31 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : HMI_TOD_ERRORS

------- Comment (attachment only) From <email address hidden> 2017-05-15 05:33 EDT-------

Manoj Iyer (manjo)
tags: added: ubuntu-17.04
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (4.6 KiB)

------- Comment From <email address hidden> 2017-05-15 05:28 EDT-------
Tested with below kernel level, all the HMI Recoverable errors are working fine as expected.
root@p8wookie:~# uname -a
Linux p8wookie 4.10.0-20-generic #22~sru+test.1-Ubuntu SMP Tue May 9 19:39:52 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@p8wookie:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty
root@p8wookie:~#

uname -a
uname -a
Linux p8wookie 4.10.0-20-generic #22~sru+test.1-Ubuntu SMP Tue May 9 19:39:52 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#cat /etc/os-release
cat /etc/os-release
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#lscpu
lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 8
Core(s) per socket: 4
Socket(s): 4
NUMA node(s): 4
Model: 2.1 (pvr 004b 0201)
Model name: POWER8E (raw), altivec supported
CPU max MHz: 4322.0000
CPU min MHz: 2061.0000
L1d cache: 64K
L1i cache: 32K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-31
NUMA node1 CPU(s): 32-63
NUMA node16 CPU(s): 64-95
NUMA node17 CPU(s): 96-127
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#dmesg -D
dmesg -D
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#dmesg -C
dmesg -C
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#PATH=/usr/local/sbin:$PATH putscom -c 00000000 1c013281 0003080000000000
0000000000local/sbin:$PATH putscom -c 00000000 1c013281 0003080
0000080000000000
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#dmesg
dmesg
[ 596.778554] Severe Hypervisor Maintenance interrupt [Recovered]
[ 596.778561] Error detail: Timer facility experienced an error
[ 596.778564] HMER: 0840000000000000
[ 596.778567] TFMR: 4d12000980a44000
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#Timer facility experienced an error and got recovered
dmesg -C
dmesg -C
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#PATH=/usr/local/sbin:$PATH putscom -c 00000000 1d013281 0003080000000000
0000000000local/sbin:$PATH putscom -c 00000000 1d013281 0003080
0000080000000000
[console-pexpect]#echo $?
echo $?
0
[console-pexpect]#dmesg
dmesg
[ 599.792816] Severe Hypervisor Maintenance interrupt [Recovered]
[ 599.792820] Error detail: Timer facility experienced an error
[ 599.792823] HME...

Read more...

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-06-29 04:28 EDT-------
(In reply to comment #19)
> Have the same problem when resetting CAPI cards. I've confirmed that
> Ubuntu-4.10.0-21.23 + be5c5e843c4afa1c8397cb740b6032bd4142f32d from upstream
> fixes the problem.

Is this been included now in ubuntu 17.04 ?

Manoj Iyer (manjo)
Changed in ubuntu-power-systems:
importance: Undecided → High
Revision history for this message
Manoj Iyer (manjo) wrote :

SRU submitted.

Changed in kernel-package (Ubuntu):
status: Incomplete → In Progress
Changed in ubuntu-power-systems:
status: Incomplete → In Progress
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Stefan Bader (smb)
affects: kernel-package (Ubuntu) → linux (Ubuntu)
tags: added: kernel-da-key
Seth Forshee (sforshee)
Changed in linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-07-27 18:33 EDT-------
I just opened another BZ 157097 for the same issue. I was referred to this bug and I see that it addresses the same issue I was debugging. But we need the upstream commit be5c5e843c4afa1c8397cb740b6032bd4142f32d pulled into Xenial 16.04.3 HWE v4.10 kernel also.

Bad commit 2337d207288f163e10bd8d4d7eeb0c1c75046a0c is included in 16.04.3 HWE v4.10 kernel, so we need the fixing upstream commit in Xenial (16.04.3) also if possible. I know we are cutting close to 16.04.3 release date but this is a regression, so it would be good to have the fixing commit if possible.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-07-31 08:16 EDT-------
*** Bug 157097 has been marked as a duplicate of this bug. ***

Manoj Iyer (manjo)
tags: added: triage-g
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-08-01 14:58 EDT-------
Can you please pull the fixing commit be5c5e843c4afa1c8397cb740b6032bd4142f32d into 16.04.3 HWE v4.10 kernel also ? If its too late, can you please pull this into 16.04.3 SRU atleast ?

Changed in linux (Ubuntu Zesty):
status: New → In Progress
Changed in linux (Ubuntu Zesty):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Zesty):
importance: Undecided → High
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-zesty' to 'verification-done-zesty'. If the problem still exists, change the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-zesty
Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (4.4 KiB)

------- Comment From <email address hidden> 2017-08-17 04:48 EDT-------
(In reply to comment #30)
> This bug is awaiting verification that the kernel in -proposed solves the
> problem. Please test the kernel and update this bug with the results. If the
> problem is solved, change the tag 'verification-needed-zesty' to
> 'verification-done-zesty'. If the problem still exists, change the tag
> 'verification-needed-zesty' to 'verification-failed-zesty'.
>
> If verification is not done by 5 working days from today, this fix will be
> dropped from the source code, and this bug will be closed.
>
> See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Thank you!

Tested with zesty-proposed kernel (Selected version '4.10.0.33.33' (Ubuntu:17.04/zesty-proposed [ppc64el]) for 'linux-generic') and verified that the problem is solved. Below is the test o/p results:

-----------------------------------Test Results-------------------------------------------------
p8wookie login:
Ubuntu 17.04 p8wookie hvc0

p8wookie login: root
Password:
Last login: Thu Aug 17 03:30:30 CDT 2017 from 9.84.221.193 on pts/0
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-33-generic ppc64le)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
root@p8wookie:~#
root@p8wookie:~# uname -a
Linux p8wookie 4.10.0-33-generic #37-Ubuntu SMP Fri Aug 11 10:53:58 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@p8wookie:~#
root@p8wookie:~# dmesg -C
root@p8wookie:~# dmesg
root@p8wookie:~# cd skiboot/external/xscom-utils/
root@p8wookie:~/skiboot/external/xscom-utils# for i in /sys/devices/system/cpu/cpu*/cpuidle/state1/disable; do echo 1 > $i; done
root@p8wookie:~/skiboot/external/xscom-utils# for i in /sys/devices/system/cpu/cpu*/cpuidle/state2/disable; do echo 1 > $i; done
root@p8wookie:~/skiboot/external/xscom-utils# ./putscom -c 00000000 1c013100 0000000000100000
0000000000100008
root@p8wookie:~/skiboot/external/xscom-utils#
root@p8wookie:~/skiboot/external/xscom-utils# dmesg
[ 419.675063] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675066] Error detail: Processor Recovery done
[ 419.675069] HMER: 2040000000000000
[ 419.675072] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675074] Error detail: Processor Recovery done
[ 419.675077] HMER: 2040000000000000
[ 419.675080] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675082] Error detail: Processor Recovery done
[ 419.675084] HMER: 2040000000000000
[ 419.675087] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675089] Error detail: Processor Recovery done
[ 419.675092] HMER: 2040000000000000
[ 419.675094] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675096] Error detail: Processor Recovery done
[ 419.675098] HMER: 2040000000000000
[ 419.675101] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675103] Error detail: Processor Recovery done
[ 419.675105] HMER: 2040000000000000
[ 419.675107] Harmless Hypervisor Maintenance interrupt [Recovered]
[ 419.675109] Error detail: Processor Recovery done
[ 4...

Read more...

tags: added: verification-done-zesty
removed: verification-needed-zesty
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.5 KiB)

This bug was fixed in the package linux - 4.10.0-33.37

---------------
linux (4.10.0-33.37) zesty; urgency=low

  * linux: 4.10.0-33.37 -proposed tracker (LP: #1709303)

  * CVE-2017-1000112
    - Revert "udp: consistently apply ufo or fragmentation"
    - udp: consistently apply ufo or fragmentation

  * CVE-2017-1000111
    - Revert "net-packet: fix race in packet_set_ring on PACKET_RESERVE"
    - packet: fix tp_reserve race in packet_set_ring

  * ThunderX: soft lockup on 4.8+ kernels when running qemu-efi with vhost=on
    (LP: #1673564)
    - irqchip/gic-v3: Add missing system register definitions
    - arm64: KVM: Do not use stack-protector to compile EL2 code
    - KVM: arm/arm64: vgic-v3: Use PREbits to infer the number of ICH_APxRn_EL2
      registers
    - KVM: arm/arm64: vgic-v3: Fix nr_pre_bits bitfield extraction
    - arm64: Add a facility to turn an ESR syndrome into a sysreg encoding
    - KVM: arm/arm64: vgic-v3: Add accessors for the ICH_APxRn_EL2 registers
    - KVM: arm64: Make kvm_condition_valid32() accessible from EL2
    - KVM: arm64: vgic-v3: Add hook to handle guest GICv3 sysreg accesses at EL2
    - KVM: arm64: vgic-v3: Add ICV_BPR1_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_IGRPEN1_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_IAR1_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_EOIR1_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_AP1Rn_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_HPPIR1_EL1 handler
    - KVM: arm64: vgic-v3: Enable trapping of Group-1 system registers
    - KVM: arm64: Enable GICv3 Group-1 sysreg trapping via command-line
    - KVM: arm64: vgic-v3: Add ICV_BPR0_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_IGNREN0_EL1 handler
    - KVM: arm64: vgic-v3: Add misc Group-0 handlers
    - KVM: arm64: vgic-v3: Enable trapping of Group-0 system registers
    - KVM: arm64: Enable GICv3 Group-0 sysreg trapping via command-line
    - arm64: Add MIDR values for Cavium cn83XX SoCs
    - [Config] CONFIG_CAVIUM_ERRATUM_30115=y
    - arm64: Add workaround for Cavium Thunder erratum 30115
    - KVM: arm64: vgic-v3: Add ICV_DIR_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_RPR_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_CTLR_EL1 handler
    - KVM: arm64: vgic-v3: Add ICV_PMR_EL1 handler
    - KVM: arm64: Enable GICv3 common sysreg trapping via command-line
    - KVM: arm64: vgic-v3: Log which GICv3 system registers are trapped
    - arm64: KVM: Make unexpected reads from WO registers inject an undef
    - KVM: arm64: Log an error if trapping a read-from-write-only GICv3 access
    - KVM: arm64: Log an error if trapping a write-to-read-only GICv3 access

  * ibmvscsis: Do not send aborted task response (LP: #1689365)
    - target: Fix unknown fabric callback queue-full errors
    - ibmvscsis: Do not send aborted task response
    - ibmvscsis: Clear left-over abort_cmd pointers
    - ibmvscsis: Fix the incorrect req_lim_delta

  * hisi_sas performance improvements (LP: #1708734)
    - scsi: hisi_sas: define hisi_sas_device.device_id as int
    - scsi: hisi_sas: optimise the usage of hisi_hba.lock
    - scsi: hisi_sas: relocate sata_done_v2_hw()
    - scsi: hisi_sas: optimise DMA slot memory

  * hisi_sas...

Read more...

Changed in linux (Ubuntu Zesty):
status: Fix Committed → Fix Released
Changed in ubuntu-power-systems:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.