[MIR] ec2-init

Can you please fix these lintian warnings? Should be quite easy.

W: ec2-init: copyright-refers-to-versionless-license-file usr/share/common-licenses/GPL
W: ec2-init: init.d-script-missing-lsb-section /etc/init.d/rightscale-init
E: ec2-init: init.d-script-does-not-implement-required-option /etc/init.d/rightscale-init restart
E: ec2-init: init.d-script-does-not-implement-required-option /etc/init.d/rightscale-init force-reload

Also, some init scripts use echo instead of the lsb_* functions. This really ought to be fixed first.

debian/ec2-init.rightscale-init.init looks questionable and also has some security issues:
 - Uses static file names without further checks in /tmp, allowing symlink attacks
 - Calls apt-get update/install in an init script, which is really not the way things work: please just make those dependencies
 - wgets and installs third-party software (http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz) without any further checks (signatures, checksums, etc.)
 - uses /opt/ which distro packages must not touch
 - calls lsb_release without depending on it
 - changes apt sources without further checks or questions
 - assumes that there is an "ubuntu" user and mucks around in /home. Ubuntu packages must not touch /home.

By and large, this is totally inappropriate as an init script. Setup should be run in postinst, Ubuntu packages should be pulled in as dependencies, and automatically installing third-party packages subverts our trust chain and packaging policy. Please just package the gem and depend on it.

Likewise, ./ec2-init calls regenerate_ssh_host_keys() if ec2-wait-for-meta-data-service() succeeds. Can the latter ever succeed on a non-ec setup? It must not ever, ever, ever change host keys on a normal system.

Packages shuold be installable on a normal system without wreaking havoc. Especially those which have a totally inconspicuous package description like ec2-init. Please at least change the package description to say "THIS IS NOT THE PACKAGE YOU WANT!!!!11!!!", or (preferably) make it inert on a normal system.

Please don't get yourself sucked into the deep black hole that automatix was. Perhaps this isn't appropriate as a package at all, but rather should be an installer-like script on its own, much like ubuntu-vm-builder. Then it can build its own chroot and bang on it as it wants, without endangering normal Ubuntu installations which apt-get install this package?

On Wed, 23 Sep 2009, Martin Pitt wrote:

> Can you please fix these lintian warnings? Should be quite easy.
>
> Also, some init scripts use echo instead of the lsb_* functions. This
> really ought to be fixed first.

I'll look into fixing those warnings. The rightscale-init script was a
source of 3 of them, so we'll figure something out there.

> debian/ec2-init.rightscale-init.init looks questionable and also has some security issues:

I think a little background information on ec2 and rightscale might be
useful. ec2-init is a packaged intended to run only inside of ec2 or an
ec2-compatible "cloud".

A. Amazon's ec2 provides a "cloud" to run instances of virtual machines.
B. Each instance of a virtual machine is a brand new "install" from a
  registered image.
C. Ubuntu builds and registers "official ubuntu images" and makes them
  available for download (uec-images.ubuntu.com). The images are created
  by vm-builder and largely represent a "fresh install from media". These
  images can be used as-is, or can be modified, re-bundled, and
  re-registered with amazon. rebundling is both time consuming and implies
  heavier maintenance compared to using a "off the shelf image".
D. The user is able to pass 64k of "user data" to the instance when they
  start it. This user data gives information to the instance on what to
  do, largely to customize the generic image above. The ec2-init init
  script will execute user-data if it begins with '#!/bin/sh'. Thus, the
  user has the ability to hook into init and install additional packages,
  grab additional software ... all the kinds of things someone would do
  after a fresh install from media.
E. ec2-init only runs user-data "once-per-ami". What this means is that
   it will only run it on first boot.

There is a open bug (bug 434181) to replace the rightscale init script.
It still has issues some of the issues you mention, but the wget
of a ruby gem is removed.

I will agree that the rightscale-init script is somewhat of a wart, and
will look into seeing if we can't find a way for rightscale to feed this
script in via the more generic user-data hook for ec2-init above.

A little background information, though, on the purpose of
rightscale-init.

When users launch new ec2-images from the rightscale interface or tools,
Rightscale utilizes the user-data to pass some rightscale info to script.
This user data is ignored by ec2-init. The user data is seen by the
rightscale-init script, and it acts on it, configuring the system to
do what the rightscale user wants it to do.

The rightscale-init can largely be viewed here as a "post-install" or
"first-boot" script for a full OS install. Rightscale users then (with aid
of rightscale) utilize this hook to turn the generic image into a
functioning LAMP stack, MySQL server or whatever they want. The idea is
that the image comes up, and configures itself and starts doing something
useful all completely automated.

> - Uses static file names without further checks in /tmp, allowing symlink attacks
> - Calls apt-get update/install in an init script, which is really not the way things work: please just make those dependencies
> - wgets and installs th...

Hello Scott,

Scott Moser [2009-09-23 19:13 -0000]:
> I think a little background information on ec2 and rightscale might be
> useful.

Thanks for the heads-up. I roughly know the setup, but my main concern
is that this is not at all a "software package" in the real sense, but
more like an installer or image builder thing.

> ec2-init is a packaged intended to run only inside of ec2 or an
> ec2-compatible "cloud".

Couldn't the package be by and large turned upside down, and instead
of playing evil tricks to change the system that it gets installed on,
rather wrap vmware-builder and change the system that it is building?
That would be so much cleaner and safer IMHO.

> The rightscale-init can largely be viewed here as a "post-install" or
> "first-boot" script for a full OS install.

Why isn't it possible to do most of this setup when building the
image?

> > By and large, this is totally inappropriate as an init script. Setup
> > should be run in postinst, Ubuntu packages should be pulled in as
> > dependencies, and automatically installing third-party packages subverts
> > our trust chain and packaging policy. Please just package the gem and
> > depend on it.
>
> Just to be clear, it is not possible to run these as a post-install
> script, or to depend on any set of packages at image-build (installation
> time).

Hm, I'm curious why that is so?

> We have been discussing a way for the ec2-init to make the ec2-init
> package inert if it believes that it is not running on a ec2 compatible
> cloud.

Right, I'm afraid this needs to be the minimum requirement. But even
with that, it's still quite a hack and circumvents the normal way we
use to do safe package installation.

Thanks!

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

On Wed, 23 Sep 2009, Martin Pitt wrote:

> Thanks for the heads-up. I roughly know the setup, but my main concern
> is that this is not at all a "software package" in the real sense, but
> more like an installer or image builder thing.

Well, its a software package that provides a boot time user hook. That
hook can be used for anything. In my opinion, its not terribly different
installing an empty rc.local and telling the user to do custom things
there.

> Couldn't the package be by and large turned upside down, and instead
> of playing evil tricks to change the system that it gets installed on,
> rather wrap vmware-builder and change the system that it is building?
> That would be so much cleaner and safer IMHO.

We do intend users to 'rebundle' our images, or build ones of their own.
However, either of those options is tremendously heavyweight when compared
to the ability to boot an instance of ubuntu-server on ec2 by typing a
single command. It starts in a minute and costs you $0.10 USD per hour.

You can feed that 10 cent instance a script to run on boot that says "pull
down my php stack and then start apache, register with a dyndns service
and start selling my trinkits"

> > The rightscale-init can largely be viewed here as a "post-install" or
> > "first-boot" script for a full OS install.
>
> Why isn't it possible to do most of this setup when building the
> image?

It is possible to rebundle or create new images with vmbuilder. Its just
expensive in terms of time, storage, network, maintenance when compared to
adding changes to an existing one.

> > We have been discussing a way for the ec2-init to make the ec2-init
> > package inert if it believes that it is not running on a ec2 compatible
> > cloud.
>
> Right, I'm afraid this needs to be the minimum requirement. But even
> with that, it's still quite a hack and circumvents the normal way we
> use to do safe package installation.

Ok. I'll open a bug for this, and we'll see that it gets fixed.

>> I think a little background information on ec2 and rightscale might
>> be useful.
> Thanks for the heads-up. I roughly know the setup, but my main concern
> is that this is not at all a "software package" in the real sense, but
> more like an installer or image builder thing.

I think you really are missing what Scott is saying.

We provide an image for EC2. People will be booting the exact image
we're providing. Sure, we can avoid touching the SSH host keys. I'm just
not sure I would personally be very happy knowing that the private keys
used by my ssh server were well-known.

>> ec2-init is a packaged intended to run only inside of ec2 or an
>> ec2-compatible "cloud".
> Couldn't the package be by and large turned upside down, and instead
> of playing evil tricks to change the system that it gets installed on,
> rather wrap vmware-builder and change the system that it is building?

Err... I don't think I even understand what you mean? ec2-init doesn't..
I mean.. No, wait, what do /you/ mean?

>> The rightscale-init can largely be viewed here as a "post-install" or
>> "first-boot" script for a full OS install.
> Why isn't it possible to do most of this setup when building the
> image?

Because we're building just one image, and people use this same image as
the base for all kinds of different things on EC2. We don't want to
force a web-, database-, or anyting else-server down everone's throats,
we don't want to have the entire world use the same SSH private keys..

>> We have been discussing a way for the ec2-init to make the ec2-init
>> package inert if it believes that it is not running on a ec2
>> compatible cloud.
> Right, I'm afraid this needs to be the minimum requirement. But even
> with that, it's still quite a hack and circumvents the normal way we
> use to do safe package installation.

EC2 is a crazy place. We're just trying to work in it.

Soren Hansen [2009-09-24 13:44 -0000]:
> I think you really are missing what Scott is saying.

Right, it seems we don't understand each other and talk past each
other. That's why I'm trying to understand and untangle this package. :-)

> We provide an image for EC2. People will be booting the exact image
> we're providing. Sure, we can avoid touching the SSH host keys. I'm just
> not sure I would personally be very happy knowing that the private keys
> used by my ssh server were well-known.

That's not what I was saying. Of course you should create the host
keys of an EC2 image on the fly, by the user. However, if I install
ec2-init in my own Karmic desktop system or server, it must not ever
change my existing ssh host keys. The init script does not seem to be
robust against this case, and I asked whether it has any precautions
against this to happen.

> >> ec2-init is a packaged intended to run only inside of ec2 or an
> >> ec2-compatible "cloud".
> > Couldn't the package be by and large turned upside down, and instead
> > of playing evil tricks to change the system that it gets installed on,
> > rather wrap vmware-builder and change the system that it is building?
>
> Err... I don't think I even understand what you mean? ec2-init doesn't..
> I mean.. No, wait, what do /you/ mean?

So, to me it looks like ec2-init takes a stock Ubuntu and installs
some standard stuff into it when you boot it first.

But a package installing more packages, and even fetching them
unauthenticated from the web, and doing all this in an init script,
is just wrong.

I was asking why we can't rather produce a stock ubuntu-ec2 image in
vmbuilder which has all those gems and packages set up already?

> Because we're building just one image, and people use this same image as
> the base for all kinds of different things on EC2. We don't want to
> force a web-, database-, or anyting else-server down everone's throats,

That's not at all what ec2-init is doing, though, and thus doesn't
answer my question at all. The init script I was talking about
doesn't deal with databases and that stuff, it does some static
general things like installing binutils, ruby packages with apt-get,
installing ruby gems, symlinking stuff around, etc. All those seem
to be part of every EC2 image, so I asked why this setup can't happen
when you build the "one" image, instead of running at first boot in a
very questionable way?

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

> > Because we're building just one image, and people use this same image as
> > the base for all kinds of different things on EC2. We don't want to
> > force a web-, database-, or anyting else-server down everone's throats,
>
> That's not at all what ec2-init is doing, though, and thus doesn't
> answer my question at all. The init script I was talking about
> doesn't deal with databases and that stuff, it does some static
> general things like installing binutils, ruby packages with apt-get,
> installing ruby gems, symlinking stuff around, etc. All those seem
> to be part of every EC2 image, so I asked why this setup can't happen
> when you build the "one" image, instead of running at first boot in a
> very questionable way?

I think I've found the problem.
a.) I'm probably not doing a great job of explaining myself. thats not
new.
b.) I really wish that rightscale-init script wasn't in the package that
you reviewed. But I'm glad that Martin took the time to point it out.

Above, when Martin said "The init script I was talking about" he was
referring to 'rightscale-init'. See debian/ec2-init.rightscale-init.init
(http://tinyurl.com/ydxp8c5). It should not be there. And, in bug
434181, I even convinced the rightscale developers of that.

However, at grave risk of confusing things, I want to point out that this
script is an example of one that might be via ec2 user data to ec2-init
(ec2-run-user-data.py)

So, I will get rightscale-init removed from the package, and clean up the
other issues that you pointed out.

> So, I will get rightscale-init removed from the package

Thanks. That'd indeed solve 90% of my woes.

> clean up the other issues that you pointed out.

So, what's left is as far as I can see:

 - ./ec2-init should not overwrite existing host keys, to avoid accidentally changing them if you install the package on a server
 - W: ec2-init: copyright-refers-to-versionless-license-file usr/share/common-licenses/GPL

This should be quite easy to fix.

Thanks for the clarifications!

Could you please check out the changes at

https://code.launchpad.net/~smoser/+junk/ec2-init.karmic , revisions 31 to 35, and let me know if they clear up the issues.

- r31 is unrelated to the concerns here
- r32: OK
- r33: *phew* thanks!
- r34: this seems to address the regeneration of SSH host keys; I thought it would have been easier to just not touch them if they already exists, but if this approach works better for you, it's fine for me.

0.4.999-0ubuntu2 approved once it gets uploaded.

Thanks for the fast fixes!

Sorry, pitti, I completely missed the fact that you were talking about the rightscale init script, and not the other stuff. I completely agree that the majoriy of what that script did should be done a image build time, but since we got rid of it now, this is obviously moot.

This bug was fixed in the package ec2-init - 0.4.999-0ubuntu2

---------------
ec2-init (0.4.999-0ubuntu2) karmic; urgency=low

  * remove rightscale-init from package (see LP: #434181, LP: #434693)
  * fix lintian warning, specify path to GPL-3
  * replace multiple '| logger' in regenerate_ssh_host_keys single one
  * add ec2-is-compat-env, and disable init script by default. it can be
    enabled by setting 'compat=1' in /etc/ec2-init/is-compat-env

 -- Scott Moser <email address hidden> Thu, 24 Sep 2009 16:32:42 -0400

Right, thanks Chuck. I was about to reopen myself, since it's not actually promoted. Thankfully, we just seemed to have a new Launchpad rollout which broke change-override, so I can't actually promote it yet.

I got hold of Celso, I hope it'll be fixed soon.

Promoted. Yay Celso!