Oops pages show wrong time window in JournalErrors

Bug #1962454 reported by Daniel van Vugt
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apport
Fix Released
Medium
Benjamin Drung
apport (Ubuntu)
Fix Released
Medium
Benjamin Drung
Focal
Fix Released
Medium
Unassigned
Jammy
Fix Released
Medium
Unassigned

Bug Description

[Impact]

Oops pages show wrong time window in JournalErrors. For example, I just experienced a crash at:

  Feb 28 17:31:12

And the JournalErrors entries are:

  Feb 28 17:31:30 - Feb 28 17:31:41

So don't relate to the crash.

[Test Plan]

1. Find a recent crash report from a relevant Ubuntu release on https://errors.ubuntu.com/ If you're looking at https://errors.ubuntu.com/problem/SOMETHING then scroll down and pick a relevant instance from the Occurrences list.

2. Now you're on a page starting with https://errors.ubuntu.com/oops/ look at ApportVersion and verify that apport is a recent enough version to contain the proposed fix. If not then go to step 1.

3. Verify the Date field falls within the time range of the JournalErrors entries.

Step 3 might fail in the case of there being no system log entries from around the time of the crash. That does not necessarily mean the test plan has failed. Go to step 1 and pick a different crash.

Prior to the fix you would almost never find oops pages that would pass the test. After the fix you should find many/most oops pages do pass the test.

[Where problems could occur]

Worst case - in any part of the bug reporting/collection procedure, since that is what's changing.

[Other Info]

Due to the huge amount of broken autopkgtest tests, the diff for the SRUs are bigger than desired. The individual commits in https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/apport/ are probably easier to review.

* jammy SRU: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/apport/log/?h=1fa042cc27714c407494b3d6dfd0730bb984f3eb
* focal SRU: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/apport/log/?h=eaa92037c7dfba621719c6f81fd75f6a09e90881

Revision history for this message
Brian Murray (brian-murray) wrote :

This could be due to the complete data collection happening after the crash, not immediately at the time of the crash. One thing apport could do is look at the time stamp of the crash and gather the journal slightly before that.

tags: added: rls-jj-incoming
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Yes for gnome-shell we want the journal in the seconds after the crash to see the Javascript stack dump (if any). But for some other crashes we need to also see what happened just before the crash.

Maybe collect 10 seconds before to 10 seconds after the crash?

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

And in this example, JournalErrors is from the wrong day:

https://errors.ubuntu.com/oops/779ee4de-8891-11ec-8fa0-fa163e983629

Changed in apport (Ubuntu):
importance: Undecided → Low
Changed in errors:
status: New → Invalid
Changed in apport (Ubuntu):
status: New → Triaged
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Bumping priority because this is preventing us from being able to see the JavaScript stack traces in gnome-shell crash reports.

Changed in apport (Ubuntu):
importance: Low → Medium
tags: added: fr-2321
tags: added: jammy
removed: fr-2321 rls-jj-incoming
tags: added: fr-2321
Changed in apport (Ubuntu Jammy):
status: New → Triaged
importance: Undecided → Medium
Changed in apport:
assignee: nobody → Brian Murray (brian-murray)
status: New → In Progress
Benjamin Drung (bdrung)
Changed in apport:
status: In Progress → Fix Committed
milestone: none → 2.21.0
Benjamin Drung (bdrung)
Changed in apport:
status: Fix Committed → Fix Released
importance: Undecided → Medium
Benjamin Drung (bdrung)
Changed in apport:
assignee: Brian Murray (brian-murray) → Benjamin Drung (bdrung)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.21.0-0ubuntu2

---------------
apport (2.21.0-0ubuntu2) kinetic; urgency=medium

  * tests: Use unlimited core ulimit for SIGQUIT test
  * data/apport: Fix log file writing if chown/chmod fails

 -- Benjamin Drung <email address hidden> Fri, 10 Jun 2022 15:39:32 +0200

Changed in apport (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Even though the change is probably small, I would like a clarified test-case and regression potential analysis on this bug. Thanks!

Changed in apport (Ubuntu Jammy):
status: Triaged → Incomplete
Revision history for this message
Benjamin Drung (bdrung) wrote :

@vanvugt, can you come up with a test case?

description: updated
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Done, although the test plan is still a little vague requiring you to wait some time _after_ the update is released to verify. An alternate test plan would involve intentionally crashing something on the local machine.

And sorry for the delay, I was on vacation.

description: updated
no longer affects: errors
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

In the past I've used 'xeyes &; pkill xeyes' to generate crash report for SRU verification.

Revision history for this message
Steve Langasek (vorlon) wrote : Proposed package upload rejected

An upload of apport to jammy-proposed has been rejected from the upload queue for the following reason: "contains bugfixes without bug references or test cases, does not meet SRU policy".

tags: added: foundations-todo
Benjamin Drung (bdrung)
description: updated
Changed in apport (Ubuntu Jammy):
status: Incomplete → Confirmed
Simon Chopin (schopin)
Changed in apport (Ubuntu Focal):
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Daniel, or anyone else affected,

Accepted apport into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu82.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Daniel, or anyone else affected,

Accepted apport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu27.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Focal):
status: Confirmed → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Benjamin Drung (bdrung) wrote :

Since I did not find a crash report with the updated apport, I created one myself.

```
sudo sh -c "echo 'some messages' > /dev/kmsg"
sudo sh -c "echo 'more messages' > /dev/kmsg"
divide-by-zero
sudo sh -c "echo 'message after crash' > /dev/kmsg"
```

I waited over 10 second to actual act on the the crash report window. Verified 2.20.11-0ubuntu82.2 for jammy: https://errors.ubuntu.com/oops/0d64a788-6a64-11ed-a976-fa163e55efd0

tags: added: verification-done-jammy
removed: verification-needed-jammy
Changed in apport (Ubuntu):
assignee: nobody → Benjamin Drung (bdrung)
Revision history for this message
Benjamin Drung (bdrung) wrote :

Successfully retried verification for jammy: https://errors.ubuntu.com/oops/b0c797d4-6a6c-11ed-b539-fa163ef35206

Revision history for this message
Benjamin Drung (bdrung) wrote :

Tested on focal. Test script:

```
#!/bin/sh
set -u

log() {
 local msg="$1"
 echo "I: $msg"
 sudo sh -c "echo '$msg' > /dev/kmsg"
}

sudo rm /var/crash/*

date
log "some messages"
sleep 1
log "more messages"
sleep 3600 &
killall -11 sleep
log 'message after crash'
sleep 5
log 'message 5 seconds after crash'
sleep 10
log 'message 10 seconds after crash'
```

Steps:
* Run the test script
* Close the Apport crash window
* Wait > 10 seconds
* `ubuntu-bug /var/crash/_usr_bin_sleep.1000.crash`

Verified 2.20.11-0ubuntu27.25 for focal: https://errors.ubuntu.com/oops/b6e647e1-6a79-11ed-a977-fa163e55efd0

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu27.25

---------------
apport (2.20.11-0ubuntu27.25) focal; urgency=medium

  * Point Vcs-* URIs to git
  * whoopsie-upload-all: Catch FileNotFoundError during process_report
    (LP: #1867204)
  * Grab a slice of JournalErrors around the crash time (LP: #1962454)
  * data/apport:
    - Initialize error log as first step (LP: #1989467)
    - Fix PermissionError for setuid programs inside container (LP: #1982487)
    - Fix reading from stdin inside containers (LP: #1982555)
  * Fix autopkgtest test case failures (LP: #1989467):
    - Mark autopkgtest with isolation-container restriction
    - Fix failure if kernel module isofs is not installed
    - Do not check recommended dependencies
    - Skip UI test if kernel thread is not found
    - Fix race in test_crash_system_slice
    - Fix check for not running test executable
    - Use shadow in *_different_binary_source
    - Mock kernel package version in UI test
    - Fix test_kerneloops_nodetails if kernel is not installed
    - Drop broken test_crash_setuid_drop_and_kill
    - Expect linux-signed on arm64/s390x as well
    - Skip SegvAnalysis for non x86 architectures
    - Use unlimited core ulimit for SIGQUIT test
    - Fix race with progress window in GTK UI tests
    - Use sleep instead of yes for tests
    - Fix test_add_gdb_info_script on armhf
    - Fix wrong Ubuntu archive URI on ports
    - Fix KeyError in test_install_packages_unversioned
    - Depend on python3-systemd for container tests
    - Depend on psmisc for killall binary
    - Replace missing oxideqt-codecs
    - Drop broken test_install_packages_from_launchpad
    - Fix test_install_packages_permanent_sandbox* for s390x

 -- Benjamin Drung <email address hidden> Thu, 15 Sep 2022 14:43:39 +0200

Changed in apport (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu82.2

---------------
apport (2.20.11-0ubuntu82.2) jammy; urgency=medium

  * Point Vcs-* URIs to git
  * Grab a slice of JournalErrors around the crash time (LP: #1962454)
  * data/apport:
    - Initialize error log as first step (LP: #1989467)
    - Fix PermissionError for setuid programs inside container (LP: #1982487)
    - Fix reading from stdin inside containers (LP: #1982555)
  * Fix autopkgtest test case failures (LP: #1989467):
    - Mark autopkgtest with isolation-container restriction
    - Fix failure if kernel module isofs is not installed
    - Do not check recommended dependencies
    - Skip UI test if kernel thread is not found
    - Fix race in test_crash_system_slice
    - Fix check for not running test executable
    - Use shadow in *_different_binary_source
    - Mock kernel package version in UI test
    - Fix test_kerneloops_nodetails if kernel is not installed
    - Drop broken test_crash_setuid_drop_and_kill
    - Expect linux-signed on arm64/s390x as well
    - Skip SegvAnalysis for non x86 architectures
    - Use unlimited core ulimit for SIGQUIT test
    - Fix race with progress window in GTK UI tests
    - Use sleep instead of yes for tests
    - Fix test_add_gdb_info_script on armhf
    - Fix wrong Ubuntu archive URI on ports
    - Fix KeyError in test_install_packages_unversioned
    - Depend on python3-systemd for container tests
    - Depend on psmisc for killall binary
    - Replace missing oxideqt-codecs
    - Drop broken test_install_packages_from_launchpad
    - Fix test_install_packages_permanent_sandbox* for s390x

 -- Benjamin Drung <email address hidden> Wed, 14 Sep 2022 18:28:26 +0200

Changed in apport (Ubuntu Jammy):
status: Fix Committed → Fix Released
Benjamin Drung (bdrung)
tags: removed: foundations-todo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.