Nova: Permission Denied, Ubuntu 12.04 B1

Bug #956876 reported by Kevin Jackson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I'm getting multiple failures related to permission problems with nova on Ubuntu 12.04 B1

It started off with it being unable to create /var/lib/nova/networks/nova-br100.conf but it turns out there are other issues.

e.g. grep through the logs for Permission denied:
/var/log/nova/nova-compute.log:(nova.compute.manager): TRACE: Stderr: "mkdir: cannot create directory `/var/lib/nova/instances/instance-00000003/': Permission denied\n"
/var/log/nova/nova-compute.log:(nova.rpc.amqp): TRACE: Stderr: "mkdir: cannot create directory `/var/lib/nova/instances/instance-00000003/': Permission denied\n"
/var/log/nova/nova-network.log:(nova.rpc.amqp): TRACE: IOError: [Errno 13] Permission denied: '/var/lib/nova/networks/nova-br100.conf'

Directory /var/lib/nova
drwxr-xr-x 9 nova nova 4096 Mar 14 15:15 .
drwxr-xr-x 41 root root 4096 Mar 14 15:15 ..
drwxr-xr-x 2 root root 4096 Mar 9 18:22 buckets
drwxr-xr-x 8 nova nova 4096 Mar 14 15:15 CA
drwxr-xr-x 2 root root 4096 Mar 9 18:22 images
drwxr-xr-x 2 root root 4096 Mar 9 18:22 instances
drwxr-xr-x 2 root root 4096 Mar 9 18:22 keys
drwxr-xr-x 2 root root 4096 Mar 16 11:25 networks
-rw-r--r-- 1 nova nova 104448 Mar 12 10:54 nova.sqlite
-rw------- 1 nova nova 1024 Mar 12 10:52 .rnd
drwxr-xr-x 2 root root 4096 Mar 9 18:22 tmp

in /etc/nova/nova.conf I set:

--state_dir=/var/lib/nova
I've removed this, but turns out its the default anyway.

I've chown -R this area to be owned by nova and its working now.

Revision history for this message
Yaguang Tang (heut2008) wrote :

add the following to /etc/sudoers.d/nova_sudoers

Cmnd_Alias NOVACMDS = /bin/chmod /var/lib/nova/tmp/*/root/.ssh, \
                      /bin/chown /var/lib/nova/tmp/*/root/.ssh, \
                      /bin/chown, \
                      /bin/chmod, \
                      /bin/dd, \
                      /sbin/ip, \
                      /sbin/route, \
                      /sbin/iptables-save, \
                      /sbin/iptables-restore, \
                      /sbin/ip6tables-save, \
                      /sbin/ip6tables-restore, \
                      /sbin/kpartx, \
                      /sbin/losetup, \
                      /sbin/lvcreate, \
                      /sbin/lvdisplay, \
                      /sbin/lvremove, \
                      /bin/mkdir, \
                      /bin/mount, \
                      /sbin/pvcreate, \
                      /usr/bin/tee, \
                      /sbin/tune2fs, \
                      /bin/umount, \
                      /sbin/vgcreate, \
                      /usr/bin/qemu-nbd, \
                      /usr/sbin/brctl, \
                      /sbin/brctl, \
                      /usr/sbin/radvd, \
                      /usr/sbin/vblade-persist, \
                      /bin/kill, \
                      /usr/sbin/ietadm, \
                      /sbin/vgs, \
                      /sbin/iscsiadm, \
                      /usr/bin/socat, \
                      /sbin/parted, \
                      /usr/sbin/dnsmasq, \
                      /usr/bin/arping

nova ALL = (root) NOPASSWD: SETENV: NOVACMDS

Changed in nova:
status: New → Invalid
Revision history for this message
Thierry Carrez (ttx) wrote :

Clearly not the right fix

Changed in nova:
status: Invalid → New
Revision history for this message
Thierry Carrez (ttx) wrote :

Might be a packaging bug, have you any idea why those were created root-owned in the first place ? I bet you always ran nova as "nova" user, right ?

Changed in nova:
status: New → Incomplete
Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

I've never had to alter or change permissions before. Nova services has always ran as 'nova'.
This is on a fresh install too.
Certainly wouldn't expect a bug to be closed because I can add a number of entries to sudoers.

Incidentally my sudo config is:
nova.conf:
--root_helper=sudo nova-rootwrap

sudoers:
#includedir /etc/sudoers.d

/etc/sudoers.d/nova_sudoers
Defaults:nova !requiretty

nova ALL = (root) NOPASSWD: /usr/bin/nova-rootwrap

Feels to me specifically related to packaging, setting the /var/lib/nova to root:root perms.

Revision history for this message
Thierry Carrez (ttx) wrote :

Packaging makes new installs do chmod /var/lib/nova nova:nova properly.
And looking at your description /var/lib/nova is owned by nova...

Looks like buckets, images, instances, keys and networks were created while running as root, which means nova was running as root when it created them. Could that be the result of some test ? Can you reproduce it ?

Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

I'm repeating the install now.

For clarification: I've not deviated from any startup/boot scripts. When launching services I am the root user, but I'm using the upstart scripts. I've never ran any service directly as the root user (or without the upstart/init script) which implies its an upstart/init script that has done something as root that shouldn't have. All services when ran as the root user change to run as the appropriate unprivileged user.

e.g. (as root): $ start nova-compute

Just doing another 12.04 install now to ensure nothing has crept in that does this, or at least helps identify this.

Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

Happy to mark as invalid.
Install of nova-* on Ubuntu 12.04 - permissions seem good now.

# ls -al /var/lib/nova/
total 148
drwxr-xr-x 9 nova nova 4096 Mar 19 12:21 .
drwxr-xr-x 41 root root 4096 Mar 19 12:20 ..
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 buckets
drwxr-xr-x 8 nova nova 4096 Mar 19 12:19 CA
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 images
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 instances
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 keys
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 networks
-rw-r--r-- 1 nova nova 104448 Mar 19 12:21 nova.sqlite
-rw------- 1 nova nova 1024 Mar 19 12:19 .rnd
drwxr-xr-x 2 nova nova 4096 Mar 16 20:20 tmp

Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.