From Mandriva

Contents 1 Introduction 2 Features 3 Architecture 4 Using the system 5 Writing procedures 6 Automatic testing 6.1 Examples of automatic tests 6.2 XML file (test.xml) 6.3 test file (msec.test) 6.4 Available functions in the tests

Introduction

Testzilla is a powerful new system for validating large-scale systems in a collaborative way over the Internet. It is designed to work in collaboration with Bugzilla and adds hardware coverage capabilities to the testing process.

Features

1. Positive feedback
2. Hardware coverage
3. Efficient testing
4. Statistics

Architecture

Using the system

1. First you need a bugzilla account. To create one, just follow the instruction on http://qa.mandriva.com/createaccount.cgi.
2. Install the hwdb-clients package on your test system.
3. You can now upload the hardware configuration of your test system using the following command with your bugzilla account information. The [system name] will be used in the web interface of TestZilla: hwdb_add_system [bugzilla login] [system name]
4. You are ready to run procedures. Procedures that you can run according to your hardware are available at http://qa.mandriva.com/test_index.cgi?not_up_to_date=1.

Writing procedures

The Guidelines are a good starting point.

You can browse the cvs module containing the procedures at http://cvs.mandriva.com/cgi-bin/viewvc.cgi/testzilla/procedures/

you can get the CVS testzilla module:

export CVSROOT=:ext:[email protected]:/cooker
cvs co testzilla

If you want to contribute a procedure, just send it to testzilla at mandriva.com.

Automatic testing

The automatic testing capability of Testzilla is active on an internal network in the Paris office. A pool of systems runs tests automatically and reports success or failure in Testzilla/Bugzilla.

Here is how a system included in Testzilla receives its tests:

1. PXE boot in a minimal network autoinstall with a special post that installs a special service called test and calls back the install server to signify that the install is completed. This way the install server can change the PXE config of this host to let it reboot locally at next boot.
2. the system reboots and launches the test service that requests a new test from the testzilla server and saves the test in a file. Testzilla knows the system from the hardware database using the MAC address as the name of the system. So before putting a system in Testzilla, you must upload its hardware configuration using =hwdb_add_system <bugzilla account> <MAC address>=.
3. the system runs the test file and lets the test do whatever it wants on the system. At the end of the test, a request must be sent to the testzilla server to report the status of the test (usually using the end function). The test can also request a reinstall to the install server on the next reboot if it has changed too much the system.
4. reboot

Examples of automatic tests

We have started to write some examples of tests in the [http://cvs.mandriva.com/cgi-bin/cvsweb.cgi/testzilla/tests/ testzilla/tests CVS module] but we need more tests that you usually run to test your packages to be able to test the system on real cases.

To add a new test to the system, you just have to create a new directory under testzilla/tests with 2 files inside. The first one must be named test.xml and must list the name of the test, the involved components and the name of the script file to run. The script file must end .test for the moment.

Here is the msec example to validate that python-base still contains all the needed parts to run msec:

XML file (test.xml)

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE procedure SYSTEM "../../dtd/procedure.dtd">

<procedure>
  <summary>msec and python-base</summary>
  <component>python-base</component>
  <component>msec</component>
  <test>msec.test</test>
</procedure>

test file (msec.test)

begin

install_rpms msec python-base
remove_rpms python

msec -o log=stderr 2

end $? python-base

Available functions in the tests

The test file is a shell script that can use the following functions:

• begin: start recording the output of the test.
• end: report the status of the test. The first argument contains the success status (o = success and anything else failure).
• install_rpms: call urpmi to install the needed packages from the main media.
• install_conribs: call urpmi to install the needed packages from the contrib media.
• install_x11_server: install and configure the X server.
• remove_rpms: call urpme to remove packages.
• reinstall: tell the boot manager that after this test the system needs to be reinstalled.
• bug: report a bug in bugzilla. The function takes 3 parameters: the name of the package, the summary of the problem and the full text of the problem.
• error: report a problem calling the end function with a failure code on the given parameter and reboot.