From Mandriva Community Wiki

Jump to: navigation, search

This page attempts to specify the needs of a Mandriva testing framework, partly inspired by Testzilla and EDOS outcomes. Creating a testing framework is part of the tasks incumbing to Mandriva within the Qualipso European research project.

Specifications

Test plan management

The testing framework can manage test plans, which comprise test cases:

TestPlan (contains a set of test cases): name, description
TestCase (reports TestReport to the testing server): name, package-name, test-script, type (manual or automatic)
TestReport: tester (user or machine id), status (running, completed, canceled), result (pass, xfail, fail, untested, unsupported), start-date, end-date, log, log-format
Product (Mandriva 2009.1, etc.): name, url, description
Build (build of a given product: Mandriva 2009.1-RC2, etc.): name, url, description
TestPlanProcess (is an instance of a TestPlan, to be executed on a target machine): test-plan-id, target-host, start-date, end-date

Test cases are automated tests or manual tests. Test plans are launched manually or chronically either in Virtual Machines or on non virtual machines. The machine on which the tests are executed reports to a testing server over HTTP. Test plans could be defined as wiki pages, conforming to test case templates. Test cases consist of test packages or of test shell scripts (possibly written in a wiki for bootstrapping it and creating a community of testers, which may differ from the community of Cooker contributors).

Pre-install tests

pre-packaging tests: unit tests included in upstream projects
rpmlint: checks the package conforms with Mandriva packaging policies
rpmcheck: looks for broken installation dependencies
ISO generation test

Installation tests

The installation tests will be run in a virtual machine such as a Qemu one. See what had been achieved within the EDOS

Testing a package installation

Install each package of the distribution and their dependency in a minimal system and check that:

the installation completes successfully
all executables can be launched successfully

The installation should take place in a virtual machine. The host machine reports the result to a reporting server via HTTP.

Testing distribution installation

Install all packages (or maximum sets) and check that:

the installation completes successfully
all functional tests complete successfully

Update tests

Testing a package update

configure the media
install a package in a minimal system (version n-1)
update the package (to version n)
use a Mancoosi-instrumented version of urpmi for reporting problems to a Mancoosi server

Testing a distribution update

configure the media
install a complete distribution (version n-1)
update the whole distribution (to version n) using a Mancoosi-instrumented version of urpmi (see what had been achieved in EDOS)

Post-installation tests

Functional tests

LTP (see also: LTP article by IBM)
Linux Desktop Testing Project: http://ldtp.freedesktop.org/wiki/
Gnome desktop testing: http://live.gnome.org/DesktopTesting
MySQL test suite: exemple http://dev.mysql.com/doc/mysqltest/en/running-tests.html
OpenOffice
Apache: http://svn.apache.org/viewvc/httpd/test/framework/trunk/
KDE tests: checks which testing framework is available
http://autotest.kernel.org/ "Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other functions such as qualifying new hardware. It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, and many others."
http://ltp.sourceforge.net/ "The LTP testsuite contains a collection of tools for testing the Linux kernel and related features."
dbench http://samba.org/ftp/tridge/dbench/ File-system stress tool. Mandriva package: dbench.

These testing suites need to be packaged and installed on testing machines.

Beside the tests provided by upstream projects, a set of complementary tests need to be written (and can be sent upstream when appropriate) for testing uncovered features and integration aspects. These tests basically consists of scripts (in bash or other languages such as Python). When getting complex, it will make sense to package them and to install the corresponding packages on testing machines:

Testzilla-like tests build upon a shell library providing the following commands (to be completed):

begin: start recording the output of the test.
end: report the status of the test. The first argument contains the success status (0 = success and anything else failure).
install_packages: call urpmi to install the needed packages from the main media.
configure_x11: install and configure the X server.
remove_packages: call urpme to remove packages.
reinstallation_needed: tell the boot manager that after this test the system needs to be reinstalled.
report_bug: report a bug in bugzilla. The function takes 3 parameters: the name of the package, the summary of the problem and the full text of the problem.
throw_error: report a problem calling the end function with a failure code on the given parameter and reboot.
create_user
create_group
set_user_password
...

UI tests

Manual TestZilla-like procedures: manual test cases described on a Web site + collaboration framework for reporting and displaying the results.

Non functional tests

Compliance testing

Documentation testing

Load testing

Localization testing and Internationalization testing

Security testing

Hardware tests

Automated hardware tests

Manual hardware tests

Testzilla like procedures: test cases described on the Web + collaboration framework for reporting the results.

Distributed tests

Use Pulse for provisioning the machines and launching the tests.

Open questions

Create a mandriva-testing mailing-list (preferably to cooker-testing since tests will relate to all Mandriva distributions)?
See how to handle different architectures

Remarks

Tests are to some extent reusable for all Linux distributions. => see how far we can build the framework with others.

Open issues and research aspects

There's the potential for a testing project alleviating issues requiring academic research.

Framework name

Quartz: Qualipso Revamps TestZilla
...

Mandriva testing framework