xen_emul_unplug=unnecessary on kernel cmdline is required in ec2 hvm

I modified grub to have an additional boot option called 'CUSTOM-TEST' and booted into that.

$ ent="Ubuntu, with Linux $(uname -r)"
$ cust="CUSTOM-TEST"
$ sed -n -e "s/${ent}/${cust}/" \
  -e "/^menuentry '${cust}'/,/}/p" \
  < /boot/grub/grub.cfg | sudo tee /boot/grub/custom.cfg
$ sudo sed -i 's,xen_emul_unplug=unnecessary,,' /boot/grub/custom.cfg
$ sudo grub-set-default "${ent}"
$ sudo grub-reboot "${cust}"
$ sudo update-grub
$ grep -v "^#" /boot/grub/grubenv
saved_entry=CUSTOM-TEST
prev_saved_entry=Ubuntu, with Linux 2.6.37-12-virtual
$ cat /boot/grub/custom.cfg
menuentry 'CUSTOM-TEST' --class ubuntu --class gnu-linux --class gnu --class os {
 recordfail
 set gfxpayload=$linux_gfx_mode
 insmod part_msdos
 insmod ext2
 set root='(hd2,msdos1)'
 search --no-floppy --fs-uuid --set=root f7ac8333-d0de-4c19-b130-794dbd905a44
 linux /boot/vmlinuz-2.6.37-12-virtual root=UUID=f7ac8333-d0de-4c19-b130-794dbd905a44 ro vt.handoff=7 console=ttyS0
 initrd /boot/initrd.img-2.6.37-12-virtual
}
$ sudo reboot

The above does a "boot once" into that custom test kernel, so that subsequent reboots will go back into working kernel.

We have been having ongoing issues with Ubuntu 11.04 HVM instances on AWS EC2 having ATA errors causing long timeouts. AWS engineering is now saying its due / related to this bug. Seems hard to believe, but I thought you might be interested.

See: https://forums.aws.amazon.com/message.jspa?messageID=311995#311995

Is this true? If so is there a Canonical Ubuntu HVM image that doesn't have this problem yet?

Whether using the pv devices or the emulated ones should hardy make a difference with respect of the timeouts and data loss which is reported here. It has a performance benefit for the communication between the instance and the Xen host, though. And we are about to release kernel updates to have the required drivers built-in and not as modules (like they are now). But those are currently in testing (for 11.04 and 11.10).

For the timeouts, I wonder whether that could be an effect of storage on ec2 usually being network attached, together with a high(er) general utilization. Actually moving from emulated to pv devices could make that even worse. Surely there also is a chance that the pv driver handles such delays/contention better than the emulated device driver.

But it should be possible to switch to the pv drivers even with the current images. Just add the pci-platform, xen-netfront and xen-blkfront drivers into /etc/initramfs-tools/modules, then run update-initramfs -u and make sure that the root device in /boot/grub/grub.cfg and /etc/fstab is using uuid or labels (because on reboot with pv drivers enabled device names will switch from hd to xvd for the block devices).

Stefan,

The ec2 kernels already have xen-netfront and xen-blkfront compiled in. If xen-platform-pci was also compiled in, or included in the initramfs, then the HW emulation will be unplugged properly and you'll switch over to the PV drivers. The following results in PV drivers for the root volume on my test instance:

add xen-platform-pci to /etc/initramfs-tools/modules
run update-initramfs -u
remove xen_emu_unplug=unnecessary from the kernel boot command line
reboot

See also bug 804219

If what Matt says is true, shouldn't this be incorporated into the next release of the Ubuntu HVM images? I think people only notice this if they are running applications that get upset when there is a long pause like HBase/Zookeeper. In our case it notices that there is a long pause and considers it a timeout and shuts itself down. But we've also seen spontaneous reboots on some of these.

We've had one other weird issue that I can not be sure is related or not, where even when we do Unix "sync" and then do EBS snapshots and the snapshot is missing large amounts of the data from the server. All the files and directories are there but many files are empty. We've seen this for the HDFS data blocks on the xfs filesystem and ami-f1589598.

The only way we could get a clean EBS snapshot was by detaching the EBS volume.

I would think this would impact at least anyone who is running HBase and it would impact the performance of anyone using the Ubunutu images on Cluster instances. And for some cases, it could lead to data corruption if it caused an HDFS namenode to crash or even just shutdown.

In the mean time we are working on trying out Matt's suggestion on one of our machines. Its non-trivial to do on a running cluster.

Thanks

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.