lxc instance console output spewed to stdout

Not sure I understand correctly... What would be the desired behavior ? Devstack should ignore stuff ? Nova should filter the libvirt console ?

devstack is not relevant here. The issue is that from inside the guest, you can write to what is /dev/pts/0 (or some pts) that you should not have access to.

The console that I typed 'run-instances' on was just a console, sitting there at a prompt. It wasn't the console that nova-compute was run on (or any other remotely relevant console). Its *probably* pts/0 but i'd have to check that.

Got it. Looks like some LXC isolation issue though, more than a Nova thing ?

I haven't yet looked back at your image, but I think this has to do with devtmpfs being mounted.

I do believe this affects both lxc and libvirt (+lxc), but your reproduction of it happens with libvirt, right?

The tool creating containers (lxc or libvirt) has to mount a private instance of devpts on /dev/pts, after unmounting the host's version. After that, every subsequent mount of devpts will be the private instance.

So what I need to figure out is where does the /dev/pts/0 from the host's devpts instance come from?

No, i can't reproduce this with lxc.

Hm, right now I'm not seeing it with libvirt either.

Scott, what release is this on? Can you tell me exactly how you triggered it?

devstack fails to install for me:

2011-12-22 18:32:12,674 INFO migrate.versioning.api [-] 63 -> 64...
2011-12-22 18:32:12,684 ERROR nova [-] foreign key constraint couldn't be removed
2011-12-22 18:32:12,684 DEBUG migrate.versioning.util [-] Disposing SQLAlchemy engine Engine(mysql://root:none@localhost/nova) from (pid=32095) with_engine /usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py:162
2011-12-22 18:32:12,685 CRITICAL nova [-] list index out of range
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/opt/stack/nova/bin/nova-manage", line 2336, in <module>
(nova): TRACE: main()
(nova): TRACE: File "/opt/stack/nova/bin/nova-manage", line 2324, in main
(nova): TRACE: fn(*fn_args, **fn_kwargs)
(nova): TRACE: File "/opt/stack/nova/bin/nova-manage", line 1161, in sync
(nova): TRACE: return migration.db_sync(version)
(nova): TRACE: File "/opt/stack/nova/nova/db/migration.py", line 35, in db_sync
(nova): TRACE: return IMPL.db_sync(version=version)
(nova): TRACE: File "/opt/stack/nova/nova/db/sqlalchemy/migration.py", line 51, in db_sync
(nova): TRACE: return versioning_api.upgrade(FLAGS.sql_connection, repo_path, version)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
(nova): TRACE: return _migrate(url, repository, version, upgrade=True, err=err, **opts)
(nova): TRACE: File "<string>", line 2, in _migrate
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
(nova): TRACE: return f(*a, **kw)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 365, in _migrate
(nova): TRACE: schema.runchange(ver, change, changeset.step)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 91, in runchange
(nova): TRACE: change.run(self.engine, step)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 145, in run
(nova): TRACE: script_func(engine)
(nova): TRACE: File "/opt/stack/nova/nova/db/sqlalchemy/migrate_repo/versions/064_change_instance_id_to_uuid_in_instance_actions.py", line 55, in upgrade
(nova): TRACE: fkey_name = list(instance_actions.c.instance_id.foreign_keys)[0].\
(nova): TRACE: IndexError: list index out of range
(nova): TRACE:
Command failed, please check log for more info
++ failed
++ local r=1
++ set +o xtrace
stack.sh failed: full log in /home/ubuntu/devstack/stack.sh.31435.log
+ for ret in '"${PIPESTATUS[@]}"'
+ '[' 1 -eq 0 ']'
+ exit 1

However I can reproduce this on oneiric (but not precise) with just libvirt.

Adding 'newinstance' to the mount options in /etc/fstab in the brickies rootfs works around this bug. Changing the priority to low to reflect the workaround exists.

Of course this nevertheless shouldn't be happening. libvirt mounts a devpts newinstance and that should be used in the container.

On oneiric, looking at /proc/pid/root/dev/pts for a task in the container shows the host's devpts. On precise it shows a private one. Need to pinpoint the patch which fixed that.

a couple things.
a.) I can't seem to reproduce this on precise at the moment. I've used the attached devstack user-data on a stock ubuntu cloud image and then run an instance insdie the screen with 'euca-run-instances'. I think that is what previously was failing, but I'm not sure.
b.) calling it low because a workaround exists is invalid. There is a bug in the hypervisor (lxc), working around by modifying guests means you can't run certain guests.

"calling it low because a workaround exists is invalid"

Are you saying you consider this an invalid workaround? If so please feel free to change it back to medium. If not, then your opinion disagrees with the community docs (https://wiki.ubuntu.com/Bugs/Importance).

I'm not saying this isn't a bug or shouldn't be fixed. I'm saying that it is a lower priority than a bug which can't be fixed by tweaking the guest image.

On Thu, 22 Dec 2011, Serge Hallyn wrote:

> "calling it low because a workaround exists is invalid"
>
> Are you saying you consider this an invalid workaround? If so please
> feel free to change it back to medium. If not, then your opinion
> disagrees with the community docs
> (https://wiki.ubuntu.com/Bugs/Importance).

Mostly I'm just complaining.
But, from my perspective, saying "guest can be modified" is not a
valid response for lxc if you actually want lxc to be useful, especially
in openstack. There will of course be some limitations for lxc on
openstack, but assuming people are willing to modify guests that work
perfectly well other places explicitly to fit inside lxc is not
reasonable.

So, i'd consider "modify guest to fit your hardware" comment here, very
similar to "install windows to make your hardware work".

(yes, i reallize its completely unreasonable, but if you want "lxc
everywhere", stuff has to "just work").

I see no reason in a debdiff why it would be fixed in precise. Will try precise's libvirt built on oneiric.

Regarding modified guests, note:

for 11.04, the goal was to run unmodified ubuntu images, but with lxc-guest
 installed, in containers.

for 12.04, the goal is to run unmodified ubuntu precise images with no extra
 packages needed.

However, 'running any unmodified image' in a container is not a realistic
goal for 12.04. But, for simple guests like yours, it certainly should be
working.

the kernel's devpts newinstance support is more limited than I remembered. When you do 'mount -t devpts -o newinstance devpts /dev/pts; mount -t devpts devpts2 /mnt', then the second devpts mount under /mnt will be the global kernel instance, not the last instance you mounted with newinstance.

I'm not sure yet why precise isn't exhibiting this behavior. It would seem to be a neat trick, and whatever is stopping this from happening through the fstab entry in precise, in general the guest cannot be stopped from doing this (without LSM or user namespace interference).

I've instrumented libvirt from both oneiric and precise, both on a precise server, to print out all of the mount activity performed during container creation, and print out /proc/mounts and /proc/self/mountinfo right before the container's /sbin/init is exec'd. Following is the result. 'XXX' is mount activity in the oneiric version, 'YYY' is mount activity in the precise version. /proc/mounts and /proc/self/mountinfo are prefixed with 'AAA' for oneiric and 'BBB' for precise.

I don't see any meaningful difference. I'll summarize in a new comment.

=====================================================
00:13:29.994: 1: debug : lxcContainerPivotRoot:323 : XXX made / MS_PRIVATE
00:13:29.994: 1: debug : lxcContainerPivotRoot:345 : XXX mounted tmprootfs onto /home/ubuntu/rootfs/.oldroot type tmpfs
00:13:29.994: 1: debug : lxcContainerPivotRoot:367 : XXX bind-mounted /home/ubuntu/rootfs onto /home/ubuntu/rootfs/.oldroot/new
00:13:29.994: 1: debug : lxcContainerPivotRoot:384 : XXX pivot-rooted . to .oldroot
00:13:29.994: 1: debug : lxcContainerMountBasicFS:435 : XXX mounted devfs on /dev type tmpfs
00:13:29.994: 1: debug : lxcContainerMountBasicFS:435 : XXX mounted /proc on /proc type proc
00:13:29.994: 1: debug : lxcContainerMountBasicFS:435 : XXX mounted /sys on /sys type sysfs
00:13:29.994: 1: debug : lxcContainerMountBasicFS:450 : XXX mounted /.oldroot/home/ubuntu/rootfs/dev/pts on /dev/pts (MS_MOVE)
00:13:29.994: 1: debug : lxcContainerPopulateDevices:502 : XXX bind-mounted /dev/pts/ptmx on /dev/ptmx
00:13:29.994: 1: debug : lxcContainerPopulateDevices:514 : XXX sym-linked /dev/tty1 to /dev/pts/0
00:13:29.994: 1: debug : lxcContainerPopulateDevices:520 : XXX sym-linked /dev/console to /dev/pts/0
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/kernel/security
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/kernel/debug
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/fuse/connections
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/perf_event
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/net_cls
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/memory
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/freezer
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/devices
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/cpuset
00:13:29.994: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/cpuacct
00:13:29.995: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/cpu
00:13:29.995: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup/blkio
00:13:29.995: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys/fs/cgroup
00:13:29.995: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/sys
00:13:29.995: 1: debug : lxcContainerUnmountOldFS:610 : XXX unmounted /.oldroot/ru...

In summary,

there are three robust ways this can be solved.

One would be for the kernel to implement a new (much nicer than the current) semantics, where once a task does 'mount -t devpts -o newinstance', all subsequent devpts mounts (until the next newinstance) mount the new instance. This woudl be ideal.

The second is with user namespaces. The /dev/pts/* will be owned by the user namespace which created it, so a child user namespace won't have access rights. This won't fix the wrong devpts being mounted, but will prevent access to the ptys themselves.

Finally, it can be handled with the LSMs. In apparmor, there will soon the ability to specify allowed mount activity by profile. So we can, after setting up the bulk of the container, switch to a profile not allowing devpts mounts or umounts. (I hope - I don't recall offhand whether umount is covered). So if a container tries to mount devpts, through fstab or otherwise, it will be denied.

Lastly, note that what I previously suggested as a workaround ,namely adding 'newinstance' to the devpts fstab entry, is not entirely correct as it may prevent getty on /dev/pts/0 from talking to libvirt's end of the console.

Marked as affecting the kernel as the ideal solution (solution 1 in comment #18) would be a kernel change.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 900972

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.