Automated Server Testing

Registered by James Page on 2010-10-15

Session for discussion about automated testing of Ubuntu Server; automated ISO testing was implemented for Maverick using libvirt/kvm + Hudson (see http://launchpad.net/ubuntu-server-iso-testing). This approach could be applied in other variants and for other aspects of server testing

Blueprint information

Status:
Complete
Approver:
Robbie Williamson
Priority:
Essential
Drafter:
James Page
Direction:
Approved
Assignee:
James Page
Definition:
Approved
Series goal:
Accepted for natty
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-11.04-beta-2
Started by
James Page on 2010-12-01
Completed by
James Page on 2011-04-26

Related branches

Sprints

Whiteboard

General Update:

* Alpha 2 work items now complete
* Testing packages can be found in ppa:ubuntu-server-iso-testing-dev/testing
* -desktop testing using livecd has been added by jibel.
* Packages have been refactored to provide support for this expanded capability.

Content from ideas pool:

 * Testing Automation (through Hudson)
   * Automated Server ISO Testing (using ubuntu-server-iso-testing)
   * Automated EC2 AMI Testing using existing framework + Hudson
   * Automated package testing using puppet, kvm etc... for UEC, Hadoop and any other moderately complex stack.
   * Continuous performance non-regression testing

Topics for discussion:

* Automated Server ISO Testing (http://launchpad.net/ubuntu-server-iso-testing)
  * Achievements to-date
    * Automation of ISO testing in Maverick.
    * Control through ec2 hosted Hudson. (http://ec2-184-73-96-45.compute-1.amazonaws.com:8080/)
  * Current challenges:
    * Test execution overhead and concurrency.
    * Lack of dedicated hardware.
  * Current state of project
    * Not packaged for Ubuntu so not that accessible
    * Codebase needs tidy + full documentation
  * Next Steps:
    * General tidyup of project and codebase
    * Packaging for Ubuntu to ease adoption
    * Release 1.0, comms to potential adopters to provide physical resources.
* Automated EC2 AMI Testing using existing framework + Hudson
* Automated package testing using puppet, kvm etc... for UEC, Hadoop and any other moderately complex stack.
* Continuous performance non-regression testing

Work items for natty-alpha-2:
[hggdh2] Way forward on production deployment of ISO testing: DONE
[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE
[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE
[james-page] Package ubuntu-server-iso-testing and locate in PPA: DONE

Work items for natty-alpha-3:
[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: DONE
[james-page] Server ISO test - fix concurrency in ISO download: DONE
[cr3] Output plugin for checkbox to write to couchdb: POSTPONED
[cr3] Checkbox plugin to download tests from couchdb to execute: POSTPONED
[james-page] Move Server ISO tests to checkbox and integrate with guest: POSTPONED
[james-page] proof of concept for complex package testing - use case openldap/mysql: POSTPONED

Work items for ubuntu-11.04-beta-1:
[james-page] Test Jenkins automated ec2 tests: DONE
[james-page] Find a long term home for ec2 automated tests: DONE

Work items for ubuntu-11.04-beta-2:
[james-page] Update ubuntu-server-ec2-testing to collate data from inside instances: DONE
[james-page] Move to exectuing tests via SSH rather than cloud-init: DONE
[james-page] Rework Jenkins jobs to use Matrix Projects: DONE

(?)

Work Items