[OSSA-2018-001] Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)

I think the priority here is that the user loses their data. The DoS potential is real, but relatively inefficient. I think we could use a CVE to communicate the problem to users, but I don't think this particular issue is important enough to jump through secrecy hoops while we work on the data-loss bug.

I would personally be in favour of early disclosure, handling this openly, and getting it done quickly. We could possibly leave disclosure til early January, though, as we're unlikely to even start work on the fix until then. I'll obviously defer to the VMT and the reporter, just my 2c.

Agreed, the data loss bug is of much higher importance than the potential DoS, just wanted to ensure VMT members were at least aware of the possibility for this to be used maliciously. If there are no further concerns or objections then I'm fine with early disclosure so we can fix this as a priority, in the open.

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

This sounds similar to OSSA-2016-012, thus I this this is a class A according to VMT taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

If this doesn't warrant an embargoed disclosure, or if a fix isn't proposed in timely manner, we should loop ossg-coresec before making this public, as explained in the embargo exception process here: https://security.openstack.org/vmt-process.html#embargo-exceptions

Given Matthew and Lee are both suggesting we switch to our public workflow in hope of getting the data loss problem solved in a more timely manner (at the expense of disclosing a somewhat inefficient DoS risk), I've gone ahead and subscribed ossg-coresec reviewers to get their input first.

Have these problems have existed since the introduction of LUKS support? Or did the ability to perform a volume-update/swap_volume come later? Are all currently supported stable branches (let's say back to stable/ocata anyway since stable/newton is likely to finally get EOL'd by the time this is fixed) at risk from this?

Yes this has been an issue since the introduction of the LUKS encryptors and is present all the way back to stable/ocata (and stable/newton FWIW).

Any additional thoughts on this? I'd like to open this up and get the bug fix in sooner rather than later if possible.

I know it's been holiday time for a lot of people, but given it's been two weeks at this point and we've got some consensus from the reporter and another core reviewer as well as some of the VMT, I'm going ahead and switching this one to our public workflow ending the embargo.

Fix proposed to branch: master
Review: https://review.openstack.org/531233

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/531233
Reason: ca323b87fa85a454fca9d46ada3677f18fe50022

Lee: Is https://review.openstack.org/460243 aimed at solving this instead now?

Yes, the refactor in https://review.openstack.org/460243 will resolve this.

data loss bugs are critical.

Reviewed: https://review.openstack.org/460243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cd3eb60c2c00bcccfa9ccd4bf9d1a96ae7a5cd88
Submitter: Zuul
Branch: master

commit cd3eb60c2c00bcccfa9ccd4bf9d1a96ae7a5cd88
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 24 19:23:44 2017 +0100

    libvirt: Collocate encryptor and volume driver calls

    This change introduces new utility methods for attaching and detaching
    frontend volume encryptors. These methods centralise the optional
    fetching of encryption metadata associated with a volume, fetching of the
    required encryptor and calls to detach or attach the encryptor.

    These new utility methods are called either after initially connecting
    to or before disconnecting from a volume. This ensures encryptors are
    correctly connected when swapping volumes for example, where previously
    no attempt was made to attach an encryptor to the target volume.

    The request context is provided to swap_volume and various other config
    generation related methods to allow for the lookup of the relevant
    encryption metadata if it is not provided.

    Closes-bug: #1739593
    Change-Id: Ica323b87fa85a454fca9d46ada3677f18fe50022

Related BZ where swap volume won't work with native LUKS encrypted volumes if you're using new enough libvirt/qemu:

https://bugzilla.redhat.com/show_bug.cgi?id=760547

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

Backport to stable branches doesn't seems trivial, is this planed to be fixed in stable/pike and stable/ocata?

I wanted to give the master change sometime to bed in and while there are conflicts there don't appear to be any show stoppers AFAICT. I'll try to post something for review in the coming week so other stable cores can review.

Related fix proposed to branch: master
Review: https://review.openstack.org/539893

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/540861

Rather than backport these libvirt driver changes to stable branches, which are non-trivial refactors of the code, why can't we just put something in the stable branches that explicitly fails a swap volume of encrypted volumes? If swapping encrypted volumes doesn't work until queens, and only then if you're not using native luks encryption, it seems OK to just deny the operation outright in stable branches as a resolution to this bug for pike and ocata.

If swapping encrypted volumes is already unusable in stable, then I don't see any significant concerns with making it safely unusable. Sounds far less likely to introduce regressions in stable too.

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/540861

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/543569

So at this point I think we're just waiting for review of the stable/pike fix and then a similar fix backported to stable/ocata? Looks like we can probably get moving on CVE assignment from the VMT end in that case. Thanks!

I'm +1 now on the stable/pike backport https://review.openstack.org/#/c/543569/ - my +2 is pending discussion about a 'security' release note in the patch that mentions the CVE / OSSA which isn't yet published, and sounds like we might have a chicken-and-egg with that until the stable/ocata backport is proposed?

CVE-2017-18191 is already reserved for this and can, I suppose, be included in that release note already. You're right that OSSA numbers have historically been assigned at the time of publication so I would consider either amending the release note later and not blocking the patch on it, or we can attempt to work out some sort of coordination mechanism for earlier OSSA assignment.

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561604

Reviewed: https://review.openstack.org/543569
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5b64a1936122eeb35f37a09f9d38159e1a224c58
Submitter: Zuul
Branch: stable/pike

commit 5b64a1936122eeb35f37a09f9d38159e1a224c58
Author: Lee Yarwood <email address hidden>
Date: Mon Feb 12 18:07:14 2018 +0000

    libvirt: Block swap volume attempts with encrypted volumes prior to Queens

    Prior to Queens any attempt to swap between encrypted volumes would
    result in unencrypted data being written to the new volume. This
    unencrypted data would then be overwritten the next time the volume was
    attached to an instance as Nova no longer identified the volume as
    encrypted, resulting in the volume being reformatted.

    This stable only change uses limited parts of the following changes to
    block all swap_volume attempts with encrypted volumes prior to Queens
    where this was resolved by Ica323b87fa85a454fca9d46ada3677f18 and also
    blocked when using QEMU to decrypt LUKS volumes by
    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1.

    Ica323b87fa85a454fca9d46ada3677f18fe50022

    The request context is provided to swap_volume in order to look up the
    encryption metadata of a volume.

    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1355

    Attempts to swap from an encrypted volume are blocked with a
    NotImplementedError exception raised.

    I258127fdcd011ccec721d5ff62eb7f128f130336

    Attempts to swap from an unencrypted volume to an encrypted volume are
    also blocked with a NotImplementedError exception raised.

    Ie02d298cd92d5b5ebcbbcd2b0e8be01f197bfafb

    The serial of a volume is used as the id if connection_info for the
    volume doesn't contain the volume_id key. Required to avoid bug #1746609.

    Closes-bug: #1739593
    Change-Id: If12e7860baad2899380f06144a0270784a5466b8

Reviewed: https://review.openstack.org/561604
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0225a61fc4557c1257383a654f0741f7ef2ddeac
Submitter: Zuul
Branch: stable/ocata

commit 0225a61fc4557c1257383a654f0741f7ef2ddeac
Author: Lee Yarwood <email address hidden>
Date: Mon Feb 12 18:07:14 2018 +0000

    libvirt: Block swap volume attempts with encrypted volumes prior to Queens

    Prior to Queens any attempt to swap between encrypted volumes would
    result in unencrypted data being written to the new volume. This
    unencrypted data would then be overwritten the next time the volume was
    attached to an instance as Nova no longer identified the volume as
    encrypted, resulting in the volume being reformatted.

    This stable only change uses limited parts of the following changes to
    block all swap_volume attempts with encrypted volumes prior to Queens
    where this was resolved by Ica323b87fa85a454fca9d46ada3677f18 and also
    blocked when using QEMU to decrypt LUKS volumes by
    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1.

    Ica323b87fa85a454fca9d46ada3677f18fe50022

    The request context is provided to swap_volume in order to look up the
    encryption metadata of a volume.

    Ibfa64f18bbd2fb70db7791330ed1a64fe61c1355

    Attempts to swap from an encrypted volume are blocked with a
    NotImplementedError exception raised.

    I258127fdcd011ccec721d5ff62eb7f128f130336

    Attempts to swap from an unencrypted volume to an encrypted volume are
    also blocked with a NotImplementedError exception raised.

    Ie02d298cd92d5b5ebcbbcd2b0e8be01f197bfafb

    The serial of a volume is used as the id if connection_info for the
    volume doesn't contain the volume_id key. Required to avoid bug #1746609.

    Conflicts:
            nova/tests/unit/compute/test_compute_mgr.py
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): Conflict due to cinderv3 support for swap_volume not
    being present in stable/ocata via
    I4b8bd01f1ffe2640fe7313213bf853d2e1bef9dd.

    Closes-bug: #1739593
    Change-Id: If12e7860baad2899380f06144a0270784a5466b8
    (cherry picked from commit 5b64a1936122eeb35f37a09f9d38159e1a224c58)

Reviewed: https://review.openstack.org/539893
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=c5f504bf1d2769191467b6a9749a1bb440750991
Submitter: Zuul
Branch: master

commit c5f504bf1d2769191467b6a9749a1bb440750991
Author: Tristan Cacqueray <email address hidden>
Date: Thu Feb 1 10:24:36 2018 +0000

    Adds OSSA-2018-001 (CVE-2017-18191)

    Change-Id: I43abe5ca3e14010b578a450bf2fa7bc3839b24b1
    Related-Bug: #1739593

This issue was fixed in the openstack/nova 16.1.2 release.

This issue was fixed in the openstack/nova 15.1.1 release.