"Too many open files" are opened by nova-compute

Bug #1090268 reported by YangLei
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Medium
Lance Bragstad

Bug Description

1. nova-comp open too many files about 1112. (but the output of "ulimit -n" just is 1024)

2. nova-comp open too many files inode 3641 /lib/modules/2.6.32-279.el6.x86_64/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko

nova-comp 20916 28u REG 0,9 0 3641 anon_inode
nova-comp 20916 29u REG 0,9 0 3641 anon_inode
...

nova-comp 20975 22u REG 0,9 0 3641 anon_inode
nova-comp 20975 23u REG 0,9 0 3641 anon_inode
...

---------------------------
[test@Openstack_Grizzly_ControlNode ~]$ ps -ef | grep nova
test 10374 10357 0 04:54 pts/1 00:00:00 grep nova
test 11292 11272 0 Dec04 pts/7 00:00:01 python /home/.../nova/bin/nova-api
test 11300 11292 0 Dec04 pts/7 00:00:01 python /home/.../nova/bin/nova-api
test 11303 11292 0 Dec04 pts/7 00:00:50 python /home/.../nova/bin/nova-api
test 11304 11292 0 Dec04 pts/7 00:00:01 python /home/.../nova/bin/nova-api
test 11479 11373 0 Dec04 pts/9 01:21:03 python /home/.../nova/bin/nova-network
test 11650 11485 0 Dec04 pts/10 00:39:28 python /home/.../nova/bin/nova-scheduler
nobody 13152 1 0 Dec04 ? 00:02:29 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/home/.../data/nova/networks/nova-br100.pid --listen-address=10.0.1.1 --except-interface=lo --dhcp-range=set:'private',10.0.1.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/home/.../data/nova/networks/nova-br100.conf --dhcp-script=/home/.../nova/bin/nova-dhcpbridge --leasefile-ro
root 13153 13152 0 Dec04 ? 00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/home/.../data/nova/networks/nova-br100.pid --listen-address=10.0.1.1 --except-interface=lo --dhcp-range=set:'private',10.0.1.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/home/.../data/nova/networks/nova-br100.conf --dhcp-script=/home/.../nova/bin/nova-dhcpbridge --leasefile-ro
test 20916 1 0 Dec05 ? 01:25:35 /usr/bin/python /usr/bin/nova-compute
test 20975 1 0 Dec05 ? 01:24:55 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova2.conf
[test@Openstack_Grizzly_ControlNode ~]$
[test@Openstack_Grizzly_ControlNode ~]$ ulimit -n
1024
[test@Openstack_Grizzly_ControlNode ~]$ lsof -p 20916 | wc -l
1112
[test@Openstack_Grizzly_ControlNode ~]$ lsof -p 20975 | wc -l
1112
[test@Openstack_Grizzly_ControlNode ~]$ lsof -p 20975 | grep 3641 | wc -l
1009
[test@Openstack_Grizzly_ControlNode ~]$ lsof -p 20916 | grep 3641 | wc -l
1009
[test@Openstack_Grizzly_ControlNode ~]$
[test@Openstack_Grizzly_ControlNode ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14874
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[test@Openstack_Grizzly_ControlNode ~]$
[test@Openstack_Grizzly_ControlNode ~]$ ls -li /lib/modules/2.6.32-279.el6.x86_64/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko
3641 -rwxr--r--. 1 root root 49576 Jun 14 2012 /lib/modules/2.6.32-279.el6.x86_64/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko
[test@Openstack_Grizzly_ControlNode ~]$

nova-compute.log:
2012-12-13 22:31:21 20916 ERROR nova.virt.powervm.common [-] Error while trying to connect: Error reading SSH protocol banner[Errno 24] Too many open files
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common Traceback (most recent call last):
2012-12-13 22:31:21 20916 TRACE nova.virt.powervmt.common File "/home/.../nova/nova/virt/powervm/common.py", line 60, in ssh_connect
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common port=port)
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common File "/usr/lib/python2.6/site-packages/paramiko/client.py", line 295, in connect
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common t.start_client()
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common File "/usr/lib/python2.6/site-packages/paramiko/transport.py", line 451, in start_client
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common raise e
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common SSHException: Error reading SSH protocol banner[Errno 24] Too many open files
2012-12-13 22:31:21 20916 TRACE nova.virt.powervm.common

Tags: powervm
Dan Smith (danms)
Changed in nova:
assignee: nobody → Lance Bragstad (ldbragst)
Changed in nova:
status: New → Confirmed
Changed in nova:
importance: Undecided → Medium
YangLei (yanglyy)
description: updated
description: updated
Revision history for this message
Kun Huang (academicgareth) wrote :

Why is this bug "comfirmed" ?

Michael Still (mikal)
tags: added: powervm
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Trying to get in contact with the opener of the bug, he says he can't reproduce and doesn't have any steps. I also can't reproduce. I have spoke with a few others and they haven't seen it either. Still reaching out to the opener to find a way to recreate.

Matt Riedemann (mriedem)
Changed in nova:
status: Confirmed → Incomplete
Revision history for this message
Sean Dague (sdague) wrote :

powervm has been removed from tree, marking won't fix.

Changed in nova:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.