ec2-init content should only run on first boot

This is fixed now, it will be fixed for the jaunty and karmic images and will be fixed for the hardy and intrepid images soon.

After testing the fixes, I'm starting to realize that some of the processes in the new /etc/init.d/ec2-init should only be run on the first boot (period), and should not be run if the instance is rebundled into a new AMI and an instance of that is run.

For example, if a user customizes the locale, /etc/hosts, and /etc/apt/sources.list and then rebundles the instance into a new AMI, they probably expect those changes to stick when that new AMI is run.

The hostname is a more complex case where it should be set only if it has not been overridden by the user. I.e., if the user has not overridden the hostname, then each new instance should continue to set the hostname to the internal IP DNS name. If the user has overridden the hostname in the AMI, then on first boot the startup code should not change it.

The Ubuntu images I publish on http://alestic.com do not have explicit hostname setting code on startup. Instead, they simply let DHCP set the hostname as provided from EC2 and everything works as it should. Perhaps the best option for the Canonical images would be to simply remove the hostname setting code.

Similarly, /etc/hosts could be a static file created on the original image instead of writing it out on instance startup. The user's changes would then be preserved through rebundling. (This is also how the Ubuntu images on http://alestic.com work.)

I didn't have the locale or apt sources problems on my images because I didn't try to dynamically set them based on the region. The right thing to do here for Canonical is probably to set them only on the first time the startup code is run no matter what AMI is being run (i.e., do not run on rebundled AMI boot).

Running the user-data script is one which should be run on the first boot of each instance, no matter if the instance has been rebundled. Setting up authorized_keys should also be done on the first boot of each instance, though it should probably check to make sure the "ubuntu" user still exists in the off chance the user removed it before rebundling.

Let me know if you'd like me to go through the code with a fine toothed comb and identify the desired behavior for each startup process. The dev AMI I was testing has disappeared, so I'll need access to another one or a pointer to where the latest ec2-init code is. Launchpad doesn't seem to have the code which was fixed.

What about SSH key regeneration? I'm inclined to do that on "first boot for each instance". Sharing host keys is never a good idea.

I've added some primitives to the init script to make this simpler. I'll upload a branch with that in a minute.

Branch pushed. It's completely untested, but I'd rather share it sooner rather than later.

Soren:

ssh host key regeneration gets complicated and there aren't any great solutions for AMI builders or users. Allow me to ramble:

- The ssh host key should be regenerated on the first boot for each instance for all *public* AMIs. Otherwise, every user would know every other user's ssh host key for instances of that image. Unfortunately, it is painful for users to find out the ssh host key for their new instances as the only available method is: wait 3-5 minutes after boot, dump the console output for the EC2 instance, parse it for the output of the ssh host key fingerprint, compare that to the fingerprint returned on the first ssh to the instance. I'm pretty sure that there are very few users who actually take this precaution. (Side note: there may be an opportunity here to create a tool to automate this process, perhaps including waiting for the instance to start running and sshd to be listening.)

- For private AMIs it is convenient and possibly more secure for the ssh host key to remain the same across instances because it is then a known quantity and it is easier to know you are talking to one of your hosts without having to jump through the above hoops. Though you say "sharing host keys is never a good idea" I think EC2 may be a case where if they are not shared, then they are not used, which is a worse idea.

The question we face here is which choice should be the default for new AMIs when users rebundle a running instance.

In the images I publish on http://alestic.com I chose to regenerate the ssh host key on the first boot of my AMIs but let users rebundle the instance into a new AMI where the ssh host key would not be regenerated on boot. If they intend to make that AMI public then in theory they should run
  "chmod +x /etc/init.d/ec2-ssh-host-key-gen"
before bundling. However, I have not done well documenting this and it is difficult to make users aware of it. (I'll admit, I also chose a horrible way of disabling the startup process and am doing penance for it.)

I would like to default to "more secure" but the problem is that one choice is more secure for public AMIs (rare, but important) and the other choice is more secure for private AMIs (common, but most folks don't check the host key fingerprint anyway so it doesn't matter).

Now that I'm forced to think about it again with hindsight, I would recommend regenerating the ssh host key on the first boot of each instance, even rebundled instances by default. People who actually check host keys and who are creating private AMIs can disable this when they rebundle an instance. Making it easy to disable this and documenting the process would be helpful.

So that's a long way of saying "I agree with you" but hopefully with some more background about the use cases.

Updated status

Marking this as fix-released.

karmic/20090811.2/ubuntu-ec2-karmic-amd64.img.gz (with ec2-init 0.4.99-0ubuntu2) has:

        if run_once_per_ami ssh_host_key_regeneration
        then
            regenerate_ssh_host_keys
        fi

I believe this to be in agreement with Eric's comment 5. Please re-open if you disagree or see that this is not functioning properly.