LAVA Dispatcher 2012.10

Milestone information

Project:
LAVA Dispatcher
Series:
trunk
Version:
2012.10
Released:
 
Registrant:
Andy Doan
Release registered:
Active:
No. Drivers cannot target bugs and blueprints to this milestone.  

Download RDF metadata

Activities

Assigned to you:
No blueprints or bugs assigned to you.
Assignees:
1 Andy Doan, 2 Dave Pigott
Blueprints:
2 Implemented
Bugs:
1 Fix Released

Download files for this release

After you've downloaded a file, you can verify its authenticity using its MD5 sum or signature. (How do I verify a download?)

File Description Downloads
download icon lava-dispatcher-0.19.3-2012.10.tar.gz (md5, sig) Source release 1,203
last downloaded 124 weeks ago
Total downloads: 1,203

Release notes 

This release does not have release notes.

Changelog 

View the full changelog

2012-10-23 Andy Doan <email address hidden>

 string_to_list(boot_cmds) not working on older python

 The version of python we are running in production hits a problem
 related to our new use of configglue. configglue passes a value
 like "boot_cmds" as a unicode string. Our string_to_list implementation
 on control isn't able to convert this to a list.

2012-10-23 Andy Doan <email address hidden>

 post release bump

2012-10-22 Andy Doan <email address hidden>

 release-0.19.2

 fix openembedded file permission issue

2012-10-22 Andy Doan <email address hidden>

 release-0.19.1

 Two setup.py sdist issues:

 1) I forgot to check in the action lava-test-runner.init.d file
 2) setup.py wasn't pulling in lava_test_shell files

2012-10-22 Andy Doan <email address hidden>

 post release bump

2012-10-22 Andy Doan <email address hidden>

 release-0.19

 2012.10

2012-10-22 Andy Doan <email address hidden>

 fixes for lava-test-shell

2012-10-22 Andy Doan <email address hidden>

 move the workfile before the test starts

 This is a bit smarter than the initial approach of doing it afterwards

2012-10-19 Andy Doan <email address hidden>

 fix weird shutdown issue with lava-test-shell on fastmodel

 I hit a weird issue with lava-test-shell doing Ubuntu based
 FastModels. Even though we call sync, things weren't getting flushed
 to the sd.img file properly.

 This just adds a halt command to the _power_off logic to try and
 make sure things halt as cleanly as possible.

2012-10-18 Andy Doan <email address hidden>

 clean up the CommandRunner logic a bit in target.py

2012-10-18 Andy Doan <email address hidden>

 have lava-test-shell try and clean up after itself

 We want to try and remove our test-runner.conf file so that
 the job won't get executed on the next boot. This is needed in
 case a job file includes a lava_test action after the lava_test_shell
 action.

 In order to do this, the test-runner.conf can't be in /etc for android
 because this partition is mounted read-only. So everything has been
 moved to a read-write partition except for the etc stuff needed to
 launch our test-runner at startup.

2012-10-18 Andy Doan <email address hidden>

 fix dumb mistake from revno 408

2012-10-18 Andy Doan <email address hidden>

 fix dumb mistake from previous commit

2012-10-18 Andy Doan <email address hidden>

 fix health check regression for master images

 Target implementations must now always set self.deployment_data. We
 were missing logic to set this when we find cached tarballs for health
 jobs. This commit does two things:

 1. Add "data" to the tuple that gets cached. This data is flexible, but
 in practice is just "oe" or "ubuntu" to tell us which deployment_data
 is needed.

 2. Make tarballcache more resilient. This changes how tarballcache is
 set up. So current installations not including this new "data" file will
 need to re-generate their tarballs. While I was at it, I also fixed
 logic so that accidental deletion of say "boot.tgz" can be managed
 properly (ie - it will clear the cache and regenerate)

2012-10-17 Andy Doan <email address hidden>

 FastModel: add new support for foundation models

2012-10-17 Andy Doan <email address hidden>

 add oe data for lava_test_shell

 This adds an init.d approach that can work for OE images.

 TODO - this doesn't support a "deps" section in the test defintion.
 Can we do something like this for OE?

2012-10-17 Andy Doan <email address hidden>

 update our _customize_ubuntu logic

 We now have the need to customize OE images and also provide target
 data for them. We could require a flag in our job files to denote if
 its OE or Ubuntu. However, in practice our OE changes are pretty much
 standard linux stuff and our Ubuntu changes take deal with things like
 Dash/Upstart.

 This keys off the existence of /etc/debian_version to help make the
 determination.

2012-10-17 Andy Doan <email address hidden>

 minor change to get ricardos patch to run

 just a minor syntax thing

2012-10-17 Ricardo Salveti de Araujo <email address hidden>

 FastModel: adding support for the foundation model

2012-10-16 Andy Doan <email address hidden>

 all power_off logic requires a sync

 We missed a case where we weren't doing a sync before calling power_off.
 This was causing results for FastModels to not get flushed to disk.

2012-10-16 Andy Doan <email address hidden>

 regression with lava-test for fastmodel

 I'm not sure how this ever worked, but the recent juggling around
 of prompts has indicated an issue with the default timeout set
 for fastmodels. Any command that passes -1 as a timeout uses the
 default which was 90, and way too small to do something like run
 the bigLITTLE suite. This gets it to a sane value used by all
 other device types.

2012-10-16 Andy Doan <email address hidden>

 ensure fastmodel uses lo for network interface

 In the old implementation the client had a function,
 get_android_adb_interface. This doesn't work well with the new
 target approach. And we really already had a way of handling this
 just using the config for the model.

2012-10-16 Andy Doan <email address hidden>

 regression in fastmodel android from latest target refactor

 Somehow I missed the ping alias for fastmodels. I also re-did
 how we write this file. The logging_system way was a bit overkill

2012-10-17 Michael Hudson-Doyle <email address hidden>

 Change the setting of the shell prompts -- instead of setting it after the test
 image boots, just have deployment force the prompt to be what we want.

 Also make the master image code a bit more single-purpose.

2012-10-16 Michael Hudson-Doyle <email address hidden>

 cleanups

2012-10-16 Michael Hudson-Doyle <email address hidden>

 remove tester_hostname config option

2012-10-16 Michael Hudson-Doyle <email address hidden>

 remove another now gratuitous change

2012-10-16 Michael Hudson-Doyle <email address hidden>

 TesterCommandRunner no longer needs to inherit from NetworkCommandRunner

2012-10-16 Michael Hudson-Doyle <email address hidden>

 merge trunk

2012-10-16 Michael Hudson-Doyle <email address hidden>

 leave reliable_session alone for now

2012-10-16 Michael Hudson-Doyle <email address hidden>

 remove dead line

2012-10-16 Michael Hudson-Doyle <email address hidden>

 testing fixes

2012-10-15 Michael Hudson-Doyle <email address hidden>

 oops

2012-10-15 Michael Hudson-Doyle <email address hidden>

 make master_ip clearing work like trunk again

2012-10-15 Michael Hudson-Doyle <email address hidden>

 wait for device to boot in boot_linaro_image; wait for network in reliable_session()

2012-10-15 Michael Hudson-Doyle <email address hidden>

 pass image to _customize_ubuntu

2012-10-15 Michael Hudson-Doyle <email address hidden>

 remove debugging

2012-10-15 Michael Hudson-Doyle <email address hidden>

 remove _power_on/power_on split again

2012-10-15 Michael Hudson-Doyle <email address hidden>

 unrelated bug

2012-10-15 Michael Hudson-Doyle <email address hidden>

 set default shell PS1 when deploying ubuntu for realz this time

2012-10-15 Michael Hudson-Doyle <email address hidden>

 just plain set PS1 for ubuntu deployments too

2012-10-15 Michael Hudson-Doyle <email address hidden>

 push prompt setting into power_on()

2012-10-15 Michael Hudson-Doyle <email address hidden>

 quick 360 to always use deployment_data to determine PS1 to look for

2012-10-15 Michael Hudson-Doyle <email address hidden>

 not sure why i removed that

2012-10-15 Michael Hudson-Doyle <email address hidden>

 bugs found by andy's torture script

2012-10-12 Michael Hudson-Doyle <email address hidden>

 add a comment

2012-10-12 Michael Hudson-Doyle <email address hidden>

 stick INITIAL_TESTER_PS1 in the deployment data

2012-10-12 Michael Hudson-Doyle <email address hidden>

 fixes

2012-10-12 Michael Hudson-Doyle <email address hidden>

 missing some bits

2012-10-12 Michael Hudson-Doyle <email address hidden>

 * for ubuntu deployments, set /etc/hostname to the value of tester_hostname for the device
 * look for root@$tester_hostname on boot, then clobber an explicit PS1 out
 * for android, just clobber PS1 when configuring the image

2012-10-12 Michael Hudson-Doyle <email address hidden>

 trivialities

2012-10-12 Michael Hudson-Doyle <email address hidden>

 in_test_shell is a private method now

2012-10-12 Michael Hudson-Doyle <email address hidden>

 copy CommandRunner.run to MasterCommandRunner, remove some stupid bits

2012-10-12 Michael Hudson-Doyle <email address hidden>

 small drive by things

2012-10-12 Michael Hudson-Doyle <email address hidden>

 remove idiocy

2012-10-12 Michael Hudson-Doyle <email address hidden>

 have the deploy_* methods explicitly boot into the master image

2012-10-12 Michael Hudson-Doyle <email address hidden>

 restore setting of master_ip

2012-10-12 Michael Hudson-Doyle <email address hidden>

 just plain force the PS1 in the master image to be what we want, only look at the config value for the first prompt after boot

2012-10-15 Yongqin Liu <email address hidden>

 update the sed command for mount sdcard of android image with the mount point of /storage/sdcard0

2012-10-15 Yongqin Liu <email address hidden>

 update the sed command for sdcard mount

2012-10-15 Yongqin Liu <email address hidden>

 fix the typo of the calling _hard_reboot

2012-10-15 Yongqin Liu <email address hidden>

 fix type of calling of _hard_reboot

2012-10-10 Andy Doan <email address hidden>

 more documentation updates

2012-10-10 Andy Doan <email address hidden>

 update usage.rst page

2012-10-10 Andy Doan <email address hidden>

 doc fixes

 updated index page
 removed unuseful and hard to maintain pages: code.rst and todo.rst
 fixed configuration page to just show what it was pointing to

2012-10-10 Andy Doan <email address hidden>

 update our documentation

 This was getting pretty out of date. It also includes help on the newest
 lava_test_shell testing format.

2012-10-09 Michael Hudson-Doyle <email address hidden>

 fix MasterImageTarget.file_system to propagate deletes from host to target

2012-10-09 Michael Hudson-Doyle <email address hidden>

 idiot hudson

2012-10-09 Michael Hudson-Doyle <email address hidden>

 in MasterImageTarget.file_system, create the tarball from above the target directory
 and remove the target directory before untarring
 this makes file deletion propagate
 i changed the tarball location so that the permission bits of the target directory are hopefully preserved

2012-10-04 Andy Doan <email address hidden>

 support lava_test_shell action

2012-10-04 Andy Doan <email address hidden>

 more fixes for from code review

 improve some exception handling in the bundling logic

 be explicit about what files to copy to the test results

2012-10-03 Andy Doan <email address hidden>

 allow lava_test_shell to work off filesystem rather than tarfile

 This originally was done via a tarball because I wasn't sure how the
 testshell code would get to interact with the target device. however,
 now that we have a clear API there's no need for this hackery.

 Also make mem/cpu info not required. There's no strong reason to enfore
 this - ie its better to get what bundle results are avaiable than nothing
 at all.

2012-10-03 Andy Doan <email address hidden>

 fix minor comments from review

 * fix subprocess.check_call to not use shell in repo clones

 * iterate repos with less indentation

 * clean up docstrings

2012-10-03 Andy Doan <email address hidden>

 ensure lava-test-shell attachments are really copied

 The old code wasn't actually copying run.sh. We also should copy
 install.sh.

2012-10-03 Andy Doan <email address hidden>

 clean up code for jobs with multiple lava_test_shell actions

 The current code, just sort of worked by piling up the results across
 lava_test_shell actions and parsing them all each time. This puts
 each one in its own unique bundle file that the submit results logic
 can then work with.

2012-10-03 Andy Doan <email address hidden>

 allow action to handle multiple testdef urls

 The lava_test_shell code had this logic already, but the action didn't.
 This changes the action to accept an array of URLs. This implied some
 fixes to badness in the code:

 * making the /lava/tests/<testname> handle repeats by now using an
 index so its /lava/tests/01_testname, etc

 * configure_target decides the name of the test install directory,
 explains how the target's path and the hosts path can differ, and
 then passes this path to its functions.

2012-10-03 Andy Doan <email address hidden>

 fix lava-test-shell interpreter

 This started as /system/bin/mksh for android, then was hacked around
 by calling via "sh" in each runner. This allows the lava_test_shell
 action to determine the proper interpreter and allow us to call it
 properly in our scripts.

2012-10-02 Andy Doan <email address hidden>

 fix some bugs in the MP

2012-10-02 Andy Doan <email address hidden>

 add bundle building logic to lava_test_shell

 This adds the logic to take the test results from the lava-test-shell
 logic and produce a properly formatted lava-dashboard bundle

2012-10-02 Andy Doan <email address hidden>

 add install support to lava_test_shell

 This handles the install part of declarative test in the action.

2012-10-01 Andy Doan <email address hidden>

 add run support to lava_test_shell

 This adds logic to build the run.sh script based on the run section
 of the test definition

2012-10-01 Andy Doan <email address hidden>

 add notion of testdef to lava_test_shell action

 This adds a testdef_url and timeout to the parameters this action
 accepts. The testdef_url should point to a file that supports the
 lp:lava-test's "declarative test" model. The timeout value is how
 long in seconds the test will be given to run.

 This series just gets the testdef in place. The next few patches
 actually do something with this.

 An example declarative test would be:
 {
 "format": "Lava-Test Test Definition 1.0",
 "test_id": "passfail",
 "install": {
 "bzr-repos": ["lp:lava-test"],
 "git-repos": ["git://git.linaro.org/ubuntu/daily-prebuilt-images.git"],
 "steps": ["echo installing!"]
 },
 "run": {
 "steps": ["/bin/echo mem-stress-a7 : PASS"]
 },
 "parse": {
 "pattern": "(?P<test_case_id>.*-*)\\s+:\\s+(?P<result>(PASS|FAIL))",
 "fixupdict": {
 "PASS": "pass",
 "FAIL": "fail"
 }
 }
 }

2012-10-01 Andy Doan <email address hidden>

 include new lava-test-shell scripts

 These are the first stage for the basis of a more black box style approach
 to testing in LAVA. There are two scripts:

 * lava-test-runner(andrdoid/ubuntu): This script is intended to be
 launched near the end of the boot process. Its job is to find tests
 that have been configured to work. It does so by looking at the
 ${WORKFILE} for tests defined to run. Each test directory must
 include a "run.sh" file which is to be run by the lava-test-shell.

 The test runner also saves the information needed later on for the hardware
 and software context information that goes into a LAVA bundle.

 * lava-test-shell: This is super simple at the moment. Its responsible
 for executing the actual command. It streams stdout/stderr to both the
 serial console and to a file. Currently stderr is streamed to stdout.log
 and this should be fixed, but its a start. We also save the exit code
 for the process.

2012-10-01 Andy Doan <email address hidden>

 workaround bug in python's tarfile implementation

 Our version of python has an issue with unicode filenames in a tarball.
 We had old code that just shelled out to "tar -xzf" and that seems like
 the safest bet for the time being.

2012-10-01 Andy Doan <email address hidden>

 convert to using new device.target API for devices

 This converts the dispatcher over to using a more limited, well-defined
 approach for dealing with actual device types. It then keeps the "client"
 part of the implementation down to just what's required by current actions
 in the dispatcher.

2012-10-01 Andy Doan <email address hidden>

 code review updates

 This addresses review notes for the original MP.

 master.py:
 * ubuntu boot_cmds was set wrong
 * use == over "is" in file_system method
 * let SimpleHTTPServer pick its own port
 * fix regex for finding port number (pexpect uses non-greedy matching)

 device-defaults.conf:
 * we already gave soft_boot_cmd a default in config.py

 base.py:
 * use a common empty_buffer function
 * use " over ' for docstring and fix grammar

 target.py:
 * use " over ' for docstring
 * explain partition used for file_system method
 * increase the timeout for the "sync" command in power off to something
 larger, but still arbitrary

2012-09-30 Andy Doan <email address hidden>

 fix deployment data concept in Target

 The original approach in this patchset with a single dictionary
 was pretty clumsy. This approach allows other modules to register
 things in a more flexible way that makes accessing the information
 better.

2012-09-30 Andy Doan <email address hidden>

 create directory if missing on calls to target.file_system

 Create the target directory if the caller is depending on it existing

2012-09-30 Andy Doan <email address hidden>

 convert master client to target based

2012-09-30 Andy Doan <email address hidden>

 support android-deploy in master image device

2012-09-30 Andy Doan <email address hidden>

 add master device support for hwpack+rfs

 This adds in the logic to run l-m-c for Ubuntu images to the master
 image device implementation.

 Tested with snippet:

 from lava_dispatcher.device.master import MasterImageTarget
 from lava_dispatcher.config import get_config, get_device_config
 from lava_dispatcher.context import LavaContext
 config = get_config("lava-dispatcher", "/srv/lava/instances/staging/etc/lava-dispatcher")
 ctx = LavaContext('panda01', config, None, None)
 device_config = get_device_config('panda01', config.config_dir)
 target = MasterImageTarget(ctx, device_config)

 target.deploy_linaro('http://snapshots.linaro.org/precise/hwpacks/panda/304/hwpack_linaro-panda_20120921-304_armhf_supported.tar.gz', 'http://snapshots.linaro.org/precise/images/nano/412/linaro-precise-nano-20120921-412.tar.gz')

2012-09-30 Andy Doan <email address hidden>

 add pre-built flashing logic to master device

 This also introduces a new method (tarballcache) for manage our cached
 tarballs for health jobs. Our caching logic in the client's master.py
 was complex and could have race conditions. This simplifies things quite
 a bit and should actually work better.

 doanac's axiom: Those who devise their own synchronization algorithms
 are bound to lose the 'race'.

 testing done with script:

 from lava_dispatcher.device.master import MasterImageTarget
 from lava_dispatcher.config import get_config, get_device_config
 from lava_dispatcher.context import LavaContext

 config = get_config("lava-dispatcher", "/srv/lava/instances/staging/etc/lava-dispatcher")
 ctx = LavaContext('panda01', config, None, None)
 device_config = get_device_config('panda01', config.config_dir)
 target = MasterImageTarget(ctx, device_config)

 target.deploy_prebuilt('http://snapshots.linaro.org/precise/pre-built/lt-panda/292/lt-panda-precise_nano_20120720-292.img.gz')

2012-09-30 Andy Doan <email address hidden>

 implement file_system support in master image device

2012-09-30 Andy Doan <email address hidden>

 add on/off function to master image device

 This allows us to boot into the linaro image. It might make more
 sense to do the "deploy_image" patches before this, but they are hard
 to verify without boot support. So this was done first.

 These functions were copied from the old master.py client. However,
 the _boot_linaro_image logic was updated to support ubuntu/android in
 one function rather than two functions. This is done by introducing a
 new dictionary to the target device called "deployment_data". This is
 used to set per-deployment options. For example, when the deploy_*
 methods are implemented for master.py they will set the proper boot_cmds.

2012-09-30 Andy Doan <email address hidden>

 create masterimage target device skeleton

 this just gives us the ability to boot into the master image

2012-09-30 Andy Doan <email address hidden>

 make qemu device a generic client implementation

 Make the qemu client be based on the TargetBasedClient

2012-09-30 Andy Doan <email address hidden>

 make fastmodel device a generic client implementation

 The fastmodel client is now basically a generic shell that works with
 a Target object. This makes it totally generic so that new devices built
 completely on the Target API can still work with tradition actions in
 the dispatcher

 We also remove the old fastmodel_deploy action which was never used in
 production.

2012-09-30 Andy Doan <email address hidden>

 move test_data_attachment logic into target device

 This will be needed for target device implementations and is not specific
 to our traditional lava-test oriented clients

2012-09-30 Andy Doan <email address hidden>

 convert fastmodel code over to new target device interface

 This moves everything not needed by the traditional dispatcher client
 code into the device implementation.

2012-09-28 Andy Doan <email address hidden>

 add concept of a target device

 The lava-test-shell work allows us to have a much more narrow view about
 interacting with a target device. This commit defines the minimum interface
 required for a lava-test-shell based action:

 * power_on
 * power_off
 * deploy_* - deploy an image type to the target
 * file_sytem - allows interaction with the files on the target

2012-10-01 Andy Doan <email address hidden>

 configglue bug fixes found while doing fastmodel.py testing

2012-09-27 Andy Doan <email address hidden>

 configglue bug fix for fastmodel.py

2012-10-01 Andy Doan <email address hidden>

 ensure we delete the host_dir

2012-09-27 Andy Doan <email address hidden>

 ensure we delete the host_dir

 While doing the previous commit, I noticed that the temp directory
 created wasn't being deleted. Upon further investigation I noticed
 this also results in littering our /tmp directory with tons of result
 bundles from Android runs. This is a quick fix.

2012-09-27 Antonio Terceiro <email address hidden>

 do not set tester_hostname to empty string in default config

2012-09-27 Michael Hudson-Doyle <email address hidden>

 use configglue to define a schema for our configuration
 buyer beware: this may be a little disruptive, test before upgrading!

2012-09-26 Michael Hudson-Doyle <email address hidden>

 grr

2012-09-26 Michael Hudson-Doyle <email address hidden>

 no, there is always more

2012-09-26 Michael Hudson-Doyle <email address hidden>

 final (?) touches

2012-09-26 Michael Hudson-Doyle <email address hidden>

 even more mechanical progress

2012-09-26 Michael Hudson-Doyle <email address hidden>

 more

2012-09-26 Michael Hudson-Doyle <email address hidden>

 more mechanical progress

2012-09-26 Michael Hudson-Doyle <email address hidden>

 random progress

2012-09-26 Michael Hudson-Doyle <email address hidden>

 more working

2012-09-26 Michael Hudson-Doyle <email address hidden>

 works a little bit now

2012-09-26 Michael Hudson-Doyle <email address hidden>

 kill stupid config accessors

2012-09-26 Michael Hudson-Doyle <email address hidden>

 return config objects that support access by attributes

2012-09-25 Michael Hudson-Doyle <email address hidden>

 sort schema, mark a couple of types

2012-09-25 Michael Hudson-Doyle <email address hidden>

 kill name arg to get_config

2012-09-25 Michael Hudson-Doyle <email address hidden>

 mostly mechanically create schema for devices and the overall config

2012-09-24 Andy Doan <email address hidden>

 post release bump

2012-09-24 Andy Doan <email address hidden>

 release-0.18

 2012.09

2012-09-21 Michael Hudson-Doyle <email address hidden>

 do not explode if /etc/resolv.conf is missing in the testrootfs

2012-09-20 Michael Hudson-Doyle <email address hidden>

 cheap hack to not die if /etc/resolv.conf is not present when setting up chroot partition

2012-09-21 Michael Hudson-Doyle <email address hidden>

 remove never-used features that claimed to support testing ubuntu and android images with custom kernels

2012-09-07 Michael Hudson-Doyle <email address hidden>

 kill equivalent feature for android

2012-09-07 Michael Hudson-Doyle <email address hidden>

 brutally slay code and docs (!) to do with kernel matrix feature

2012-09-19 Antonio Terceiro <email address hidden>

 Use qemu-system-arm by default

 People who are trying architectures other than ARM can still override that in
 their configuration if needed

2 blueprints and 1 bug targeted

Blueprint Priority Assignee Delivery
Black Box Test Actions Black Box Test Actions 5 Essential Andy Doan  11 Implemented
Support v8 Model Device in LAVA Support v8 Model Device in LAVA 5 Essential Dave Pigott  11 Implemented
Bug report Importance Assignee Status
1071266 #1071266 No mkdir for deploying android images 2 Critical Dave Pigott  10 Fix Released
This milestone contains Public information
Everyone can see this information.