tripleo

CI: cinder fails to allocate memory while creating volume for ping test tenant

Bug #1639970 reported by Gabriele Cerami
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned
tripleo ocata-3 "ocata-3"

Bug Description

HA master jobs are failing very often with this error

ERROR cinder.api.middleware.fault [req-05e49ca5-df00-4359-a6fe-b7890b95dc93 dc6bddee7bbe4405ad837ae3394b74d5 824a1cc0ec2a4723be811cbd8a0080d5 - default default] Caught error: <type 'exceptions.OSError'> [Errno [Errno 12] Cannot allocate memory

full log at http://logs.openstack.org/62/336562/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/e68fefe/logs/overcloud-controller-1/var/log/cinder/cinder-api.txt.gz#_2016-11-07_17_27_53_459

We found some general performance issues during a debug session, but most of them impact on CPU, not memory.

Thomas Herve was suggesting to tweak down httpd config to reduce limits for some services, and free some memory.

Tags: ci
Revision history for this message
Gabriele Cerami (gcerami) wrote :

A shallow memory utilization analysis showd no particular memory eaters. In each controller, mysqld takes ~6%, http ~3%, and there's a long list of processes at 0-2%. Not easy to understand if any of the process is taking more memory than it should.

Revision history for this message
Gabriele Cerami (gcerami) wrote :

This is still happening, not very often, but still happening. I think it happens when test env is under heavy load. Downgrading to High since it's not blocking.

Changed in tripleo:
importance: Critical → High
Revision history for this message
Gabriele Cerami (gcerami) wrote :

Noticed one more thing, the error appears in all three controllers, sequentially.

at ~1:14
http://logs.openstack.org/79/395379/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5cdde75/logs/overcloud-controller-0/var/log/cinder/cinder-api.txt.gz#_2016-11-10_01_14_09_047

at ~1:15
http://logs.openstack.org/79/395379/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5cdde75/logs/overcloud-controller-1/var/log/cinder/cinder-api.txt.gz#_2016-11-10_01_15_01_741

at ~1:16
http://logs.openstack.org/79/395379/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5cdde75/logs/overcloud-controller-2/var/log/cinder/cinder-api.txt.gz#_2016-11-10_01_16_00_111

Looks like all three controllers are asked to create the volume, and all are unable to allocate memory.

This is another point in favor of test env memory shortage under heavy load.

Steven Hardy (shardy)
Changed in tripleo:
milestone: ocata-1 → ocata-2
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: ocata-2 → ocata-3
Emilien Macchi (emilienm)
Changed in tripleo:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.