console-setup failure due to race with systemd-tmpfiles

Bug #1824227 reported by Robert C Jennings
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
console-setup (Ubuntu)
Fix Released
High
Steve Langasek
Xenial
Fix Released
High
Steve Langasek
Bionic
Fix Released
High
Steve Langasek
Cosmic
Fix Released
High
Steve Langasek
Disco
Fix Released
High
Steve Langasek

Bug Description

[SRU Justification]
I'm seeing a console-setup.service failure quite regularly in testing
where the temp file that should have been created can't be found.
This is a regular xenial cloud image.

  19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
  ...
  19:51:13 systemd[1]: Starting Set console font and keymap...
  19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file
  19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE
  19:51:15 systemd[1]: Failed to start Set console font and keymap.
  19:51:15 systemd[1]: console-setup.service: Unit entered failed state.
  19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'.
        ...

/bin/setupcon has a lovely tempfile function that looks like:
  if \
    TMPFILE=`mktemp /tmp/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /run/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /dev/.tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XXXXXX 2>/dev/null` \
      || TMPFILE=`mktemp 2>/dev/null`

Here's our edited IRC conversation on the bug:
<@vorlon> so I do think you're being hit by the tmp cleaner
<@vorlon> also this seems like bad pathological default behavior for
          the tmp cleaner, to delete files that have just been created
<@vorlon> but we should fix console-setup to not rely on /tmp
<@vorlon> and I prefer that we do that instead of trying to fiddle with
          the ordering of the systemd units on startup
<@vorlon> i.e. console-setup has an undeclared dependency
          on systemd-tmpfiles-clean; let's remove the dependency
          instead of declaring it

<@vorlon> are you failing the race more often now than in the past?
<@rcj> vorlon: it feels like it's failing more often but I don't have
          data to answer that.

<@vorlon> are we shipping an image with a dirty rootfs?
<@vorlon> dirty in the sense that e2fsck doesn't take one look at it,
          say "yep, nothing to do here" and exit
<@vorlon> in the sense that this is what would make dev-sda1.device slow
          to complete AIUI
<@rcj> would filesystem resize on first boot mark it dirty? Because
          that will happen
<@vorlon> huh good question
<@vorlon> it might
<xnox> rcj, unclean shutdown?
<@rcj> xnox: first boot

[Test case]
1. Install console-setup from -proposed
2. Reboot
3. Verify that `systemctl status console-setup` shows that the service has completed successfully.

Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE.

[Regression potential]
None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: console-setup 1.108ubuntu15.4
ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18
Uname: Linux 4.15.0-1029-gcp x86_64
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
Date: Wed Apr 10 19:24:12 2019
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: console-setup
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Robert C Jennings (rcj) wrote :
Revision history for this message
Robert C Jennings (rcj) wrote :
Steve Langasek (vorlon)
Changed in console-setup (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in console-setup (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
Changed in console-setup (Ubuntu Cosmic):
status: New → Triaged
importance: Undecided → High
Steve Langasek (vorlon)
Changed in console-setup (Ubuntu Disco):
status: Triaged → Fix Committed
assignee: nobody → Steve Langasek (vorlon)
Steve Langasek (vorlon)
description: updated
description: updated
Steve Langasek (vorlon)
description: updated
Steve Langasek (vorlon)
Changed in console-setup (Ubuntu Cosmic):
status: Triaged → In Progress
Changed in console-setup (Ubuntu Bionic):
status: Triaged → In Progress
Changed in console-setup (Ubuntu Xenial):
status: New → In Progress
Changed in console-setup (Ubuntu Cosmic):
assignee: nobody → Steve Langasek (vorlon)
Changed in console-setup (Ubuntu Bionic):
assignee: nobody → Steve Langasek (vorlon)
Changed in console-setup (Ubuntu Xenial):
assignee: nobody → Steve Langasek (vorlon)
importance: Undecided → High
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Robert, or anyone else affected,

Accepted console-setup into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/console-setup/1.178ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in console-setup (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in console-setup (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Robert, or anyone else affected,

Accepted console-setup into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/console-setup/1.178ubuntu2.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in console-setup (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Robert, or anyone else affected,

Accepted console-setup into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/console-setup/1.108ubuntu15.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package console-setup - 1.178ubuntu12

---------------
console-setup (1.178ubuntu12) disco; urgency=medium

  * setupcon: use /run for tempfiles (and dump the various unnecessary
    fallback paths), since /run is always mountable rw at least as early as
    /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep
    /tmp as a fallback in case we have access to write to /tmp and to a
    console, but not to /run. LP: #1824227.

 -- Steve Langasek <email address hidden> Wed, 10 Apr 2019 12:51:56 -0700

Changed in console-setup (Ubuntu Disco):
status: Fix Committed → Fix Released
tags: added: id-5caba54405de185b64adc2db
tags: added: id-5cae47869ef68677a65c1e71
Revision history for this message
Robert C Jennings (rcj) wrote :

I have built cosmic, bionic, and xenial images in GCE with the packages from -proposed and tested that console setup is functioning correctly with the change. Thanks. I will be marking each of the releases as tested.

tags: added: verification-done verification-done-bionic verification-done-cosmic verification-done-xenial
removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package console-setup - 1.178ubuntu9.1

---------------
console-setup (1.178ubuntu9.1) cosmic; urgency=medium

  * setupcon: use /run for tempfiles (and dump the various unnecessary
    fallback paths), since /run is always mountable rw at least as early as
    /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep
    /tmp as a fallback in case we have access to write to /tmp and to a
    console, but not to /run. LP: #1824227.

 -- Steve Langasek <email address hidden> Wed, 10 Apr 2019 13:02:51 -0700

Changed in console-setup (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for console-setup has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package console-setup - 1.178ubuntu2.8

---------------
console-setup (1.178ubuntu2.8) bionic; urgency=medium

  * setupcon: use /run for tempfiles (and dump the various unnecessary
    fallback paths), since /run is always mountable rw at least as early as
    /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep
    /tmp as a fallback in case we have access to write to /tmp and to a
    console, but not to /run. LP: #1824227.

 -- Steve Langasek <email address hidden> Wed, 10 Apr 2019 13:13:34 -0700

Changed in console-setup (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package console-setup - 1.108ubuntu15.5

---------------
console-setup (1.108ubuntu15.5) xenial; urgency=medium

  * setupcon: use /run for tempfiles (and dump the various unnecessary
    fallback paths), since /run is always mountable rw at least as early as
    /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep
    /tmp as a fallback in case we have access to write to /tmp and to a
    console, but not to /run. LP: #1824227.

 -- Steve Langasek <email address hidden> Wed, 10 Apr 2019 13:16:29 -0700

Changed in console-setup (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.