euca-describe-instances shows all instances in all states (even terminated)

Bug #824008 reported by Kevin Bringard
54
This bug affects 9 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Brian Waldon

Bug Description

Running 2011.3~d4~20110809.1411-0ubuntu0ppa1~natty1 and it doesn't look to filter out terminated instances properly. When I run euca-describe-instances it shows everything, they're all just "terminating":

INSTANCE i-00000009 ami-00000004 terminating None (sandbox, enc2b11) 5 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
INSTANCE i-0000000a ami-00000004 terminating None (sandbox, enc2b14) 6 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
INSTANCE i-0000000b ami-00000004 terminating None (sandbox, enc2b14) 7 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
INSTANCE i-0000000c ami-00000004 shutdown None (sandbox, enc2b11) 8 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
INSTANCE i-0000000d ami-00000004 terminating None (sandbox, enc2b12) 9 m1.small 2011-06-23T21:33:02Z plano-trunk0 aki-00000003 ami-00000000

However, if I try to euca-terminate the instances, I'm told its an invalid instance:

i-0000000b
InstanceNotFound: Instance %(instance_id)s could not be found.
i-0000000c
InstanceNotFound: Instance %(instance_id)s could not be found.
i-0000000d
InstanceNotFound: Instance %(instance_id)s could not be found.

Related branches

Revision history for this message
Kevin Bringard (kbringard) wrote :

I should also note that this happens regardless of if the user is_admin or not, but (of course) admin users show all instances for all projects, while non admins only show the ones they have access to.

Revision history for this message
George Reese (george-reese) wrote :

I don't believe this is a bug. IIRC, the EC2 API allows that instances may be returned from DescribeInstances that are terminated.

Revision history for this message
Kevin Bringard (kbringard) wrote :

Fair enough... I guess I'm not sure how to proceed then. At the very least I don't think this is desired behavior. When I run euca-terminate-instances, it doesn't seem like it should show up in euca-describe-instances anymore once it's shutdown and gone.

The reason I filed this as a bug is because it was working as desired (perhaps not as expected or to ec2 spec?). I would terminate an instance and it wouldn't show up in describe instances anymore. I upgraded the code and suddenly instances I'd terminated months ago were showing up again.

So, again, I guess I'm not really sure how to proceed because I don't think seeing every instance you've ever run every time you describe-instances is desired most of the time.

Revision history for this message
Vish Ishaya (vishvananda) wrote : Re: [Bug 824008] Re: euca-describe-instances shows all instances in all states (even terminated)

There is definitely a bug here. The instance should be marked deleted by compute when it is done terminating and no longer show up in the list. This has always worked correctly so something is going wrong.

Revision history for this message
Vladimir Popovski (vladimir.p) wrote :

It seems like this bug was introduced when get_all code was changed to use search_opts.

Function instance_get_all_by_filters() doesn't apply filer for:
deleted=can_read_deleted(context)

It seems like this was done intentionally due to requirement to support "changes-since" in server APIs.

Probably it will make more sense in sqlalchemy's instance_get_all_by_filters() return back :
    .filter_by(deleted=can_read_deleted(context))

and in _get_servers (servers.py) to play with context.read_deleted when 'changes-since' is set.

John Tran (jtran)
Changed in nova:
assignee: nobody → John Tran (jtran)
Thierry Carrez (ttx)
Changed in nova:
status: New → In Progress
Revision history for this message
Vladimir Popovski (vladimir.p) wrote :

John,

Seems like fix was delivered together with rev. 1445 and this issue might be already fixed.

http://bazaar.launchpad.net/~hudson-openstack/nova/trunk/revision/1445

However, looking at "changes-since" functionality of Openstack API it seems like we may have another issue there... without setting proper context it may not retrieve recently deleted instances.

Revision history for this message
John Tran (jtran) wrote : RE: [Bug 824008] Re: euca-describe-instances shows all instances in all states (even terminated)

Should I delete the merge proposal?
________________________________________
From: <email address hidden> [<email address hidden>] On Behalf Of Vladimir Popovski [<email address hidden>]
Sent: Tuesday, August 16, 2011 3:10 PM
To: John Tran
Subject: [Bug 824008] Re: euca-describe-instances shows all instances in all states (even terminated)

John,

Seems like fix was delivered together with rev. 1445 and this issue
might be already fixed.

http://bazaar.launchpad.net/~hudson-openstack/nova/trunk/revision/1445

However, looking at "changes-since" functionality of Openstack API it
seems like we may have another issue there... without setting proper
context it may not retrieve recently deleted instances.

--
You received this bug notification because you are a bug assignee.
Matching subscriptions: bugs
https://bugs.launchpad.net/bugs/824008

Title:
  euca-describe-instances shows all instances in all states (even
  terminated)

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Running 2011.3~d4~20110809.1411-0ubuntu0ppa1~natty1 and it doesn't
  look to filter out terminated instances properly. When I run euca-
  describe-instances it shows everything, they're all just
  "terminating":

  INSTANCE i-00000009 ami-00000004 terminating None (sandbox, enc2b11) 5 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
  INSTANCE i-0000000a ami-00000004 terminating None (sandbox, enc2b14) 6 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
  INSTANCE i-0000000b ami-00000004 terminating None (sandbox, enc2b14) 7 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
  INSTANCE i-0000000c ami-00000004 shutdown None (sandbox, enc2b11) 8 m1.small 2011-06-23T21:33:01Z plano-trunk0 aki-00000003 ami-00000000
  INSTANCE i-0000000d ami-00000004 terminating None (sandbox, enc2b12) 9 m1.small 2011-06-23T21:33:02Z plano-trunk0 aki-00000003 ami-00000000

  However, if I try to euca-terminate the instances, I'm told its an
  invalid instance:

  i-0000000b
  InstanceNotFound: Instance %(instance_id)s could not be found.
  i-0000000c
  InstanceNotFound: Instance %(instance_id)s could not be found.
  i-0000000d
  InstanceNotFound: Instance %(instance_id)s could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/824008/+subscriptions

Revision history for this message
Thierry Carrez (ttx) wrote :

@John: if the bug is fixed by the other commit, yes... Also please set this bug to FixCommitted.

Thierry Carrez (ttx)
Changed in nova:
status: In Progress → Incomplete
Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → diablo-4
Changed in nova:
status: Fix Committed → Incomplete
Brian Waldon (bcwaldon)
Changed in nova:
assignee: John Tran (jtran) → Brian Waldon (bcwaldon)
status: Incomplete → In Progress
importance: Undecided → High
milestone: diablo-4 → diablo-rbp
Revision history for this message
Vladimir Popovski (vladimir.p) wrote :

this issue re-appeared as a result of merge 1521. read_deleted was removed from there...

Thierry Carrez (ttx)
Changed in nova:
milestone: diablo-rbp → 2011.3
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.