OpenStack Compute (nova)

nova-manage vm list produces monster query

Bug #961317 reported by Jay Pipes
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Medium
Sumanth Nagadavalli

Bug Description

Marking this as a security vulnerability because this can be used to essentially flood the database server and cause an entire Nova MySQL database to slow to a crawl.

This issue was discovered by HP database admins investigating slow performance on the Nova database nodes. The DBA was seeing the following PROCESSLIST in MySQL:

http://pastebin.ca/2130819

With this being the query predominantly running, with a status of "Sending data":

http://pastebin.ca/2130818

Code in trunk:

https://github.com/openstack/nova/blob/master/bin/nova-manage#L902

AFAICT, no limit/marker offset is ever passed to the query, and so the command by default lists every VM, with joins to all related tables.

The short-term solution would be to put a pagination mechanism into nova-manage.

Tags: db nova-manage
Revision history for this message
Thierry Carrez (ttx) wrote :

Discussed with Jay, and there is no valid threat model for this unless your deployment let attackers indirectly run nova-manage commands... in which case the issue could be considered a vulnerability in the way you deployed rather than in Nova.

Those affected deployers should definitely patch this before we open the bug to the public... and fix the bug in Nova publicly.
Hopefully that will happen before the next RC round.

Setting to Incomplete until Jay can confirm this is OK to open.

Changed in nova:
status: New → Incomplete
Revision history for this message
Thierry Carrez (ttx) wrote :

Adding PTL, though I wouldn't consider this a real security issue in Nova.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

also agree that this is not a security issue.

Revision history for this message
Russell Bryant (russellb) wrote :

agreed here too FWIW

Revision history for this message
Thierry Carrez (ttx) wrote :

Removing security bit. Anyone allowing non-admins to run nova-manage should probably apply a number of improvements to make it safer to use by potential attackers.

@jay: feel free to remove the private bit whenever this is workarounded on your side and the issue can be made public.

Changed in nova:
status: Incomplete → Confirmed
security vulnerability: yes → no
Revision history for this message
Thierry Carrez (ttx) wrote :

@Jay: any progress here ? Should we just close the bug as EIMPROPERUSEOFNOVAMANAGE ?

Revision history for this message
Robert Clark (robert-clark) wrote :

I think the sticking issue is that it's not getting any intention as it's still private.

What's the impact of making it public?

Revision history for this message
Jay Pipes (jaypipes) wrote :

Marked public.

visibility: private → public
aeva black (tenbrae)
tags: added: db
Thierry Carrez (ttx)
tags: added: nova-manage
Changed in nova:
importance: Undecided → Medium
Sumanth Nagadavalli (sumanth-nagadavalli)
Changed in nova:
assignee: nobody → Sumanth Nagadavalli (sumanth-nagadavalli)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/42866

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

nova-manage is more or less deprecated in place of using the nova apis when possible except for things you want to do directly against the database like managing services and doing db migrations.

In Icehouse Steve Kaufer added paging support for nova api so servers list should having that working, so marking this as invalid in light of that.

Plus the patch was abandoned.

Changed in nova:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.