Unlocking the second crypto disk (/home) echos password on console

Assigning to cryptsetup for now

Please execute the following command, as it will automatically gather debugging information, in a terminal:
apport-collect 876626
When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

apport information

apport information

Status changed to 'Confirmed' because the bug affects multiple users.

This also affects the first prompt, and there are further issues - the "press s to skip mounting or m for manual recovery" message appears on top of the password prompt. You can get around it with pressing esc several times, but you end up with the same issue of it echoing the passphrase.

See http://leaf.dragonflybsd.org/~alexh/ubuntu_crypttab.jpg for the echoing problem.

See http://leaf.dragonflybsd.org/~alexh/ubuntu_plymouth.jpg for the second issue with the "s or m" message.

The priority of this should be the highest available, as it is a major security flaw. A fix to this will stop all echoing, not just echo stars.

Cheers,
Alex

FWIW, this issue seems to be with plymouth, as expected, not with cryptsetup. It shouldn't be too hard to find the root cause with plymouth tracing enabled.

Cheers,
Alex

I have been testing this in VirtualBox.
If graphical boot is on and working, a box hiding your password with dots shows. But if graphical boot messes up (which seems to happen frequently) or graphical boot is disabled, password echoes.

I tried doing a selective upgrade from 11.04. And surprisingly upgrading plymouth did not trigger the bug. After upgrading these packages it appeared:
man-db 2.5.9-4
debianutils 4.0.2
libblkid1 2.19.1-2ubuntu3
libc-bin 2.13-0ubuntu13
doc-base 0.9.5ubuntu2
man-db 2.5.9-4
ureadahead 0.100.0-11
initramfs-tools-bin 0.99ubuntu8
initramfs-tools 0.99ubuntu8
mountall 2.31
initscripts 2.88dsf-13.10ubuntu4
upstart 1.3-0ubuntu11
friendly-recovery 0.2.18
ifupdown 0.7~alpha5.1ubuntu5
initramfs-tools 0.99ubuntu8

Downgrading upstart from oneiric 1.3-0ubuntu11 to natty 0.9.7-1 fixes this for me.

Cybjit, thanks for your investigation into this issue. I'm not sure if this is a bug on the plymouth or upstart side. James, do you know what has changed in upstart that would account for this misbehavior?

Still a problem on precise, still fixed by downgrading to natty upstart.

No change with upstart 1.4.

is this huge gaping security hole going to be fixed anytime soon?

On Tue, Feb 07, 2012 at 10:54:08PM -0000, bwalex wrote:
> is this huge gaping security hole going to be fixed anytime soon?

We have not yet isolated the cause.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Yeah, it's not like this should be a priority, so taking four months without progress is completely ok.

@ilf: I would suggest a bit less sarcasm and a bit more encouragement for hard-working volunteer programmers!

Is it possible that some other process is receiving the keyboard inputs and echoing them back? I'm thinking this because typing in a password for the swap partition only echoes *s -- but later inputs will trigger the echoing of the actual characters.

Well, apparently if nobody comments on this issue, it just keeps getting ignored. So I can fully understand ilf's sarcasm. It's just annoying that Ubuntu focuses on making "pretty" GUIs (which look IMHO like disgustingly cheap Mac OSX clones) and a security vulnerability just sits there for MONTHS literally with at least 7 differents reports of the same issue. Way to tell your users what your focus is on. Then again, the choice of distro is ours. And judging from that Unity nonsense and security prioritization my decision is to move away from Ubuntu anyways. I'll keep it installed as long as it seems to make sense, but will migrate with new installations. No hard feelings.

@Benjamin: I am grateful for general FLOSS development, but I am totally pissed at the way Ubuntu is handling passphrase input for crypto disks at boot.

It started almost two years ago with bug 566818.
Then it became a security issue last year with bug 778659.
Now this on has been idling for four months.

I would be able to look over this, if there were clear responsibilities and a working process, fixing it in a reasonable time frame.

But it's "Importance: Undecided" for plymouth and Oneiric?
It's only assigned to someone in Precise upstart, too?
Why has the assignee not commented here once?
If plymouth may be responsible, has upstream been contacted? (Nothing in their Bugtracker).

So far natty upstart still works with precise, without echoing, I can only hope it continues to work while this is unresolved.
I keep the 'upgrade' out with this in /etc/apt/preferences:
Package: upstart
Pin: version 0.0
Pin-Priority: -1

I attempted a bisection of upstart a while ago. But between getting the correct repository history, and getting it to build, I failed. Any pointers?

This bug still happens with 12.04 beta2

IMO, this is a showstopper for LTS upgrade.

Still present in current Beta 2, three days until FinalFreeze. Way to go for LTS!

Quite true, ilf. My indifferentiometer reads over 30 kiloyawns/sec for this bug. Guess it's more important to have pretty CSS and alpha-channeld menus in Unity than to have an actual usable system. Thumbs up for eyecandy!

The workaround is getting harder to pull off since udev has a dependency on upstart version. Natty upstart and Oneiric udev now.

Are you shitting me? Half a year and one release later, you're delaying this again? You're shipping this on a production LTS?

@ilf -please be assured that we are still working on this issue. There are a lot of variables to consider here and the problem is providing difficult to isolate. Please can I ask those affected by this issue to be patient.

As an update on the issue itself, we believe the problem is not actually related to crypto: it is simply manifested for users with crypto devices since they are prompted by plymouth for the passphrase and that is where the problem is observed.

As we learn more, we will update this bug and quite possibly ask those affected for further details to help us resolve this issue.

ilf, this bug is a corner case. It only affects users who are using encryption on a volume *other* than the one containing their root filesystem, only when these filesystems are mounted at boot time, and only when they are opting not to use the default plymouth splash interface. I have been using encrypted filesystems on my laptop continuously since at least 2007 and like most users of cryptsetup am entirely unaffected by this bug.

So while it's important for us to fix this, it is not a release blocker by any means, and has been given a corresponding priority within the development team.

If you disagree with the timeline, you can try to persuade a developer in the community to do this work, or you can contract someone to do the work. But whether or not you elect to do this, please stop harranguing the developers. That's not what the bug tracker is here for.

Note to self: Encryption of data volumes on a headless system is a "corner case".

"Ubuntu is designed with security in mind." https://wiki.ubuntu.com/LTS

@Steve Langesek: An encrypted data partition on an Ubuntu server meets all your criteria, but is by no means a "corner case".

Thank you for targeting this for Precise. I am happy to see this is being addressed. Hope the fix makes it in SRU sooner than later.

This issue does appear to have been exposed by a change to Upstart: Upstart now resets the terminal attributes for /dev/console to ensure a sane environment for Upstart itself to operate in. It does this (and *should* do this) since it cannot know what state the initramfs left the console in (in fact consider the scenario if there *is* no initramfs on systems like ARM).

However, in resetting the console, Upstart has exposed a bug in Plymouth which is only disabling echoing once (when it first opens a terminal device).

Here is what's happening for the crypto scenario:

1) plymouthd is started from the initramfs.
2) plymouthd opens /dev/console and puts the terminal into "raw" (no-echo) mode such that if passwords are prompted for, they will not be displayed.
3) The passphrase for the (1st) root partition is prompted for by the plymouth client. Crucially, this happens from the initramfs. This correctly obscures the entered passphrase and displays asterisks as the user types both in graphical and text mode (using the plymouth "details" plugin).
4) The initramfs finishes executing and hands control to Upstart.
5) Upstart resets the terminal attributes on /dev/console since it is not aware plymouthd is connected to it too, but unfortunately, plymouthd is not aware of Upstart resetting the attributes and still believes (incorrectly) that echoing is disabled.
6) The passphrase for further crypto volumes is now prompted for, but this time from Upstart jobs
    (/etc/init/cryptdisks-enable.conf, /etc/init/cryptdisks-udev.conf). The user is prompted to enter further passphrases which are now echoed to the terminal due to the reset performed by Upstart.

The real problem here is plymouth: when prompting for a password, it is unsafe to assume the terminal it is connected to is still in the state it was put into when the device was first opened. The fix is to set the terminal to raw mode immediately prior to prompting for any password. I will send a fix to Plymouth upstream to accomplish this.

A temporary workaround to the problem would be to modify the Upstart jobs /etc/init/cryptdisks-enable.conf and/etc/init/cryptdisks-udev.conf to manually disable then re-enable terminal echoing. Something like this:

script

    stty -echo -icanon

    # << main part of scripts >>

    stty echo icanon

end script

Hi James,

On Mon, Apr 23, 2012 at 07:42:08PM -0000, James Hunt wrote:
> The real problem here is plymouth: when prompting for a password, it is
> unsafe to assume the terminal it is connected to is still in the state
> it was put into when the device was first opened. The fix is to set the
> terminal to raw mode immediately prior to prompting for any password. I
> will send a fix to Plymouth upstream to accomplish this.

I'm not convinced that we should consider this a plymouth bug. I think
plymouth is right to assume that its console settings will remain
persistent, and it's upstart that's in the wrong here for changing the
settings out from underneath it. Why does upstart care about the echo flag
at all? Couldn't it simply read the existing echo flag value, and OR that
in with the rest of its preferred settings?

> A temporary workaround to the problem would be to modify the Upstart
> jobs /etc/init/cryptdisks-enable.conf and/etc/init/cryptdisks-udev.conf
> to manually disable then re-enable terminal echoing. Something like
> this:

> script

> stty -echo -icanon

> # << main part of scripts >>

> stty echo icanon

> end script

I don't think the latter part is right, because plymouth is still running at
the end of the job and still owns the console, so its preferred console
settings should still apply. (Which is part of why I think this is not a
plymouth bug.) Also this job has no 'console' line, so the stty command
would have to have its stdin attached to the console somehow... so it's
really not worth trying to deploy a quick fix here.

Looking back at the upstart history, I see this:

revno: 1266
committer: Scott James Remnant <email address hidden>
branch nick: upstart
timestamp: Wed 2010-03-17 22:34:37 +0000
message:
  * init/main.c:
    - Don't change the settings of the foreground console, this is often
      owned by plymouth and not supposed to be in Canonical Mode; all other
      paths have stty sane settings anyway (which these are not), so there
      really isn't need for init to do this. LP: #540256.

And I can't find anywhere in the history where this decision was consciously
reversed: it appears to have been a casualty of the upstream 1.3 merge onto
the Ubuntu branch.

Please consider whether we should restore the pre-1.3 Ubuntu upstart
behavior of not changing the foreground console settings, and whether this
change should be included upstream - I don't know why Scott never made this
change upstream.

Hi Steve,

Since Plymouth is a 'long-running process' wrt the boot and since it is attaching to the console (a shared device), I really don't think it should be making any assumptions -- particularly before requesting a password: it should set a 'secure' terminal environment immediately before prompting. On my test system, over 1500 processes were created between plymouthd starting in the initramfs and my test plymouth client Upstart job running which gives huge scope for a process to tweak the
terminal settings after plymouth initially sets them.

We could make Upstart perform more intelligent bit-twiddling but there are a _lot_ of options (atleast 47) and Upstart cannot really know which bits it should be twiddling in all scenarios as -- for example -- /dev/console might be attached to a serial line.

Upstart cannot make assumptions about the state the console is left in since either something running in the initramfs may have disturbed sane defaults, or as mentioned, there may be no initramfs so a reset generally makes perfect sense I think. The reset it is doing is similar to "stty sane" (which also includes 'ECHO') which gives a 'normal' console setup.

Details of when Scott changed the console handling in Upstart (both in Upstream and Ubuntu) are:

------------------------------
revno: 1326
committer: Scott James Remnant <email address hidden>
branch nick: upstart
timestamp: Thu 2011-08-11 13:30:15 -0700
message:
  * init/main.c: Deal with failure to setup the system console by
  falling back to /dev/null, so we don't end up without default fds
  and castrate the process.
------------------------------

We need input from Scott as to the precise retionale for this change since simply undoing it might cause equally bad problems in other parts of the system. I wonder if he implemented it for bug 880049 and just forgot to update that bug?

> I don't think the latter part is right, because plymouth is still running at
> the end of the job and still owns the console

I'm not sure any process owns the console. If we impose policy to state that plymouth does, which plymouth do we mean? The one that runs from the initramfs or the one started as an Upstart job? What about PID 1? What about 'console owner' Upstart jobs? I think the point here is that plymouthd doesn't have awareness of whether it is running in the initramfs or via an Upstart job and Upstart doesn't have an awareness of whether it was started directly or from an initramfs, neither does it
know about the existing plymouthd process if that was started from the initramfs.

> Also this job has no 'console' line, so the stty command
> would have to have its stdin attached to the console somehow... so it's
> really not worth trying to deploy a quick fix here.

The workaround I outlined wasn't necessarily intended to be included in Ubuntu directly, but was more for users affected by this issue since they could add those two commands relatively easily to restore expected behaviour. I appreciate there is no 'console' stanza, but it doesn't need one - the job runs the plymouth client that asks plymouthd to request a password from the user, and plymouthd is connected to /dev/console.

I've che...

-c1326 doesn't change the console handling, it merely changed the error case:

- if (system_setup_console (CONSOLE_OUTPUT, (! restart)) < 0)
+ if (system_setup_console (CONSOLE_OUTPUT, (! restart)) < 0) {

The Plymouth-safe console behavior was never committed upstream, but was part of the Ubuntu delta applied to 0.6.5-5, this patch has apparently been dropped:

http://launchpadlibrarian.net/41179455/upstart_0.6.5-4_0.6.5-5.diff.gz

- /* Set the standard file descriptors to the ordinary console device,
- * resetting it to sane defaults unless we're inheriting from another
- * init process which we know left it in a sane state.
- */
- if (system_setup_console (CONSOLE_OUTPUT, (! restart)) < 0)
- nih_free (nih_error_get ());

This was never merged upstream because I'm not Lennart, and I don't go around saying things like "all relevant distributions use Plymouth to set up the system console" :p Something has to set up the console correctly and other Upstart users (e.g. Chrome OS) don't even have an initramfs or initrd so need init to do that, as the first thing being run.

The correct upstream fix obviously would be to conditionally set up the system console in some manner, dependent on whether or not it's already "owned" by a running process or whether init is truly the first process to be run.

On Tue, Apr 24, 2012 at 06:11:48PM -0000, Scott James Remnant wrote:
> The correct upstream fix obviously would be to conditionally set up the
> system console in some manner, dependent on whether or not it's already
> "owned" by a running process or whether init is truly the first process
> to be run.

Sounds like the correct fix would indeed be to check whether the console is
locked as James suggested, and do the setup only if it's not.

And for an Ubuntu-specific fix, it seems it would still be correct to treat
the console as owned by plymouth.

Thanks for the insight, Scott.

Thanks for finally looking into this.

While on this, please also look at #778659.
I do not agree with Comment #33 that "[obscuring] the entered passphrase and [displaying] asterisks" is "correct". It should not display any input, as cryptsetup(8), cryptdisks_start(8), passwd(1), gpg(1) and all others do.

Hello Holger, or anyone else affected,

Accepted upstart into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I do not have any oneiric environment left, but using upstart 1.3-0ubuntu12 and udev 173-0ubuntu4.2 on precise does not echo for me.

Hello Holger, or anyone else affected,

Accepted upstart into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package upstart - 1.5-0ubuntu7

---------------
upstart (1.5-0ubuntu7) precise-proposed; urgency=low

  * Correct a build failure from the previous upload.

upstart (1.5-0ubuntu6) precise-proposed; urgency=low

  * debian/upstart.logrotate: don't create empty files after rotation;
    upstart will automatically create new log files for jobs as needed.
  * init/main.c: restore the fix for bug #540256; we know the console setup
    is taken care of by plymouth in Ubuntu, so upstart changing the console
    settings just makes trouble (such as turning echo back on when it
    shouldn't be). LP: #876626.

  [ James Hunt ]
  * debian/upstart-job: Only attempt to handle disabled jobs if the running
    version of Upstart supports such a query (LP: #985755, #984474).
  * debian/manpages/upstart-events.7: Fixed typo and corrected reference to
    'kill signal' stanza.
 -- Steve Langasek <email address hidden> Thu, 26 Apr 2012 07:48:17 -0700

Is someone able to test the upstart package in precise-proposed now that a fix is available?

The proposed fix does resolve the issue for me on precise.
Many thanks.

This bug was fixed in the package upstart - 1.5-0ubuntu7

---------------
upstart (1.5-0ubuntu7) precise-proposed; urgency=low

  * Correct a build failure from the previous upload.

upstart (1.5-0ubuntu6) precise-proposed; urgency=low

  * debian/upstart.logrotate: don't create empty files after rotation;
    upstart will automatically create new log files for jobs as needed.
  * init/main.c: restore the fix for bug #540256; we know the console setup
    is taken care of by plymouth in Ubuntu, so upstart changing the console
    settings just makes trouble (such as turning echo back on when it
    shouldn't be). LP: #876626.

  [ James Hunt ]
  * debian/upstart-job: Only attempt to handle disabled jobs if the running
    version of Upstart supports such a query (LP: #985755, #984474).
  * debian/manpages/upstart-events.7: Fixed typo and corrected reference to
    'kill signal' stanza.
 -- Steve Langasek <email address hidden> Thu, 26 Apr 2012 07:48:17 -0700

This bug was fixed in the package upstart - 1.3-0ubuntu12

---------------
upstart (1.3-0ubuntu12) oneiric-proposed; urgency=low

  * init/main.c: restore the fix for bug #540256; we know the console setup
    is taken care of by plymouth in Ubuntu, so upstart changing the console
    settings just makes trouble (such as turning echo back on when it
    shouldn't be). LP: #876626.
 -- Steve Langasek <email address hidden> Tue, 24 Apr 2012 13:16:43 -0700

quantal includes plymouth 0.8.4, which has the upstream changes to support locking the terminal.