OpenStack Compute (nova)

[VMware] Launching an instance with large image size crashes nova-compute

Bug #1495834 reported by Giridhar Jayavelu
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Radoslav Gerganov
Liberty
Fix Released
High
Dongcan Ye

Bug Description

Created an image of size ~6.5GB.
Launching a new instance from this image crashes nova-compute.

I'm observing nova-compute node running out of memory.

This could probably be due to reading entire file stream in memory without using proper chunk size.

This is from git source and not any distribution.

git log -1
commit 1cf97bd096112b8d2e0eb95fd2a636a53cbf0bcc
Merge: a54c0d6 17fe88a
Author: Jenkins <email address hidden>
Date: Mon Sep 14 02:34:18 2015 +0000

    Merge "Fix typo in lib/keystone"

nova image-show fa376c74-4058-492b-9081-f31522f640f6
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size | 6997009920 |
| created | 2015-09-14T10:53:09Z |
| id | fa376c74-4058-492b-9081-f31522f640f6 |
| minDisk | 0 |
| minRam | 0 |
| name | win2k12-01 |
| progress | 100 |
| status | ACTIVE |
| updated | 2015-09-14T10:59:33Z |
+----------------------+--------------------------------------+

Attached n-cpu.log

See original description
Tags: vmware spawn
Revision history for this message
Giridhar Jayavelu (gjayavelu) wrote :
Revision history for this message
Giridhar Jayavelu (gjayavelu) wrote :
description: updated
Markus Zoeller (markus_z) (mzoeller)
tags: added: spawn vmware
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/231468

Changed in nova:
assignee: nobody → Radoslav Gerganov (rgerganov)
status: New → In Progress
Radoslav Gerganov (rgerganov)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/231468
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=71d2d31b9b86df4071c3728666842eec6cd1c727
Submitter: Jenkins
Branch: master

commit 71d2d31b9b86df4071c3728666842eec6cd1c727
Author: Radoslav Gerganov <email address hidden>
Date: Tue Oct 6 16:01:45 2015 +0300

    VMware: specify chunk size when reading image data

    When reading OVA image data from Glance without specifying chunk size,
    nova-compute tries to read the entire tar stream in memory because this
    is how the tar fileobject is implemented. This patch fixes this by
    specifying the chunk size to be 64KB.

    Change-Id: I953dfa1c28c25886306e1c9afe281cf2faca4db5
    Closes-Bug: #1495834

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/266096

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/266096
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=51bfbbd82916f08daa97ae3ab7400ec581571105
Submitter: Jenkins
Branch: stable/liberty

commit 51bfbbd82916f08daa97ae3ab7400ec581571105
Author: Radoslav Gerganov <email address hidden>
Date: Tue Oct 6 16:01:45 2015 +0300

    VMware: specify chunk size when reading image data

    When reading OVA image data from Glance without specifying chunk size,
    nova-compute tries to read the entire tar stream in memory because this
    is how the tar fileobject is implemented. This patch fixes this by
    specifying the chunk size to be 64KB.

    Change-Id: I953dfa1c28c25886306e1c9afe281cf2faca4db5
    Closes-Bug: #1495834
    (cherry picked from commit 71d2d31b9b86df4071c3728666842eec6cd1c727)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 12.0.1

This issue was fixed in the openstack/nova 12.0.1 release.

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b2

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.