Cinder

Cinder dies a horrible death with "Insufficient free space for volume creation on host"

Bug #1376941 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Confirmed
High
Unassigned

Bug Description

This job had 27 failed volume tests, looks like some volumes aren't cleaning up and it's eating up the disk causing scheduling errors:

http://logs.openstack.org/73/92073/6/check/check-tempest-dsvm-postgres-full/eba2f01/logs/screen-c-sch.txt.gz?#_2014-10-02_17_57_48_901

http://goo.gl/TeHs6V

message:"Insufficient free space for volume creation on host" AND tags:"screen-c-sch.txt"

346 hits in 7 days, check and gate, all failures.

Tags: gate-failure
Matt Riedemann (mriedem)
no longer affects: barbican
Changed in cinder:
importance: Undecided → High
Revision history for this message
John Griffith (john-griffith) wrote :

So it turns out the issue here is actually that the c-vol service "died". Cleanup probably isn't the issue in this case, but the fact that the service dies the scheduler will report the "No valid host"

Revision history for this message
Matt Riedemann (mriedem) wrote :

e-r query: https://review.openstack.org/#/c/125809/

Changed in cinder:
status: New → Confirmed
Revision history for this message
Matt Riedemann (mriedem) wrote :

http://logs.openstack.org/07/125407/2/check/check-tempest-dsvm-postgres-full/aa0bf40/logs/syslog.txt.gz#_Oct__2_16_26_25

Notice blockdev is hanging:

INFO: task lvm:4403 blocked for more than 120 seconds.

do_blockdev_direct_IO+0x1ce4/0x2910

That sounds similar to what we were seeing with bug 1371677.

Revision history for this message
Matt Riedemann (mriedem) wrote :

message:"INFO: task lvm" AND message:"blocked for more than 120 seconds." AND tags:"syslog"

That's not 100% failure:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSU5GTzogdGFzayBsdm1cIiBBTkQgbWVzc2FnZTpcImJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjAgc2Vjb25kcy5cIiBBTkQgdGFnczpcInN5c2xvZ1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDEyMjg5NDY4MTQyLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

When it does hit, the failure rates are 96% though:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSU5GTzogdGFzayBsdm1cIiBBTkQgbWVzc2FnZTpcImJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjAgc2Vjb25kcy5cIiBBTkQgdGFnczpcInN5c2xvZ1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDEyMjg5NDY4MTQyLCJtb2RlIjoidHJlbmQiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfc3RhdHVzIn0=

Revision history for this message
Joe Gordon (jogo) wrote :

http://logs.openstack.org/97/123597/3/check/check-tempest-dsvm-full/9fb47c8/logs/screen-dstat.txt.gz

At 02-10 17:34:30 the CPU is at 97%

Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

Same issue causing https://bugs.launchpad.net/tempest/+bug/1310896

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.