"Too much data for sort() with no index." While Getting Resources

Bug #1433924 reported by Eren
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
Ilya Tyaptin
ceilometer (Ubuntu)
Fix Released
Medium
Unassigned
Trusty
Triaged
Medium
Unassigned
Vivid
Triaged
Medium
Unassigned
Wily
Fix Released
Medium
Unassigned

Bug Description

We have 47GB of mongo data. When I run "ceilometer resource-list", I get the error in the subject. I isolated the problem down to storage/impl_mongodb.py:746 _get_floating_resources(). It builds an empty query with sort options "{user_id: -1, project_id: -1, last_sample_timestamp: -1}" and it fails.

I ran the query in mongodb to see if it was generating the same error and confirmed it. After a bit digging, I learned that mongodb needs compound index for that kind of sort. I added the following index via mongocl:

db.resource.createIndex({user_id: -1, project_id: -1, last_sample_timestamp: -1}, {background: true})

After this index, the problem went away. I can run "ceilometer resource-list" and "ceilometer meter-list". However, those commands take ~1minute to complete with the index. The performance may be another issue that needs investigation.

I need to mention that this error is different than https://bugs.launchpad.net/ceilometer/+bug/1193906. The timestamp index is in place but we need compound indexes as I explained above.

gordon chung (chungg)
Changed in ceilometer:
status: New → Triaged
Changed in ceilometer:
assignee: nobody → Aleksey Ripinen (aripinen)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/207409

Changed in ceilometer:
assignee: Aleksey Ripinen (aripinen) → Ilya Tyaptin (ityaptin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/207409
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=4b8929206acdd28fecb14652d78f9ca675a4fdd4
Submitter: Jenkins
Branch: master

commit 4b8929206acdd28fecb14652d78f9ca675a4fdd4
Author: Ilya Tyaptin <email address hidden>
Date: Thu Jul 30 14:18:42 2015 +0300

    [MongoDB] Refactor indexes for meter and resources

    Currently Ceilometer uses redundant indexes in MongoDB and in same
    time doesn't cover get_resources sorting by indexes.
    In this CR source field indexing is deleted and resource.project_id,
    resource.user_id indexes are combined to one.
    New compound index covers the get_resource sorting and its parts
    are used in queries by timestamp, project_id, user_id.

    Change-Id: I077317533fd30865eea8a3f32e3c3a7f53fa6343
    Closes: bug 1433924

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Revision history for this message
James Page (james-page) wrote :

b3 has been uploaded to wily - setting to Fix Released.

Changed in ceilometer (Ubuntu Wily):
importance: Undecided → Medium
status: New → Fix Released
Changed in ceilometer (Ubuntu Vivid):
status: New → Triaged
Changed in ceilometer (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → Medium
Changed in ceilometer (Ubuntu Vivid):
importance: Undecided → Medium
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: liberty-3 → 5.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (master)

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/196645
Reason: cleanup

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.