From Mandriva Community Wiki

This article is a work in progress being written, rewritten or being significantly restructured.
You are invited to discuss the changes and to participate to the article's improvement according to what has been decided on the talk page.

Specifications

Entities

• A test management server, called server.
• An host machine which will run the virtual machine client, called host.
• A client testing guest virtual machine, called guest.
• A collection of programs running under the guest, called client.
• A package testing environment, called test_package containing the package, its dependencies, the architecture on which it runs and its version.

Prerequisites

• A Mandriva distribution comprising the packages of the testing suite on a remote server.
• An hardware architecture supported by one of those packages.
• A software architecture supported by one of those packages (mainly the Linux kernel).
• An Internet connection.
• Some CPU and RAM ressources.

Base principles

Introduction

(add here the introduction from the FP Vodka's project article)

The huge number of packages to be tested imposes a very important infrastructure. We should need dozens or hundreds of machines in order to individually install each package, launching each binary, doing a certain amount of tests (ldd, man page presence for each binary and for each configuration file...). We then need to find a simple and safe mean using the community to proceed these tests. Security is a key here: how to ensure users will know those tests won't become trojans wich will spam mails or proposing illegal files to be downloaded threw ftp... At the same time, simplicity is crucial. No way to need several configuration hours in order some can became a tester.

From these constraints, we defined a scheme described in the further paragraphs.

Principle synthesis

Any user will be able to participate in the package testing process. A virtual machine will be used on the real user machine wich will allow to make one or more tests without compromising the real user computer. To do so, the user will have to install the corresponding package and will have to launch a command which will then run the testing suite under a virtual machine and then stop without having altered anything on the real machine.

Global process

Once the package installed on the host real machine and fully configured, the user will be able to use some commands in order to manage the testing process.

The testing process, on the host side, defaults to these treatments, in this order:

• The user launches the initialization script with the start argument.
• The previous used virtual disk, if any, is removed.
• A copy of the original virtual disk is made.
• The virtual machine boots on this virtual disk copy.
• At the end of the boot, a little script checks if the network is available. If it is it downloads the latest version of the Test-Cloud test-suite then launches it.
• The suite then connects to a remote server (the server) and ask for testing a package.
• The server chooses a not-already-tested package and sends back to the guest the name of the package to be tested. The server will also save that a client will start to test this package.
• The guest then receives the name of the package to be tested. It automatically launches the command that allows to make the full test of the package (see below). When the test has finished, the logs are be sent to the server along with the package name. During the test stage, the guest regularly informs the server that is is well performing the tests. In case of a problem (crash of the guest or of the test set or a network problem...) no information is sent to the server. The test done and the logs sent the guest will halt, giving the hand back to the terminal of the host that launched the test.
• The server receives the logs and the name of the package that has been tested and saves this information,
  • OR
• The server did not received any information from the guest since a T time, considered too long. It then decides that the to-be-tested package won't be fully treated by this guest and won't mark it as treated.
• The virtual machine enters the shutdown process.
• The user has called the initialization script with the stop argument so the process stops here.
  • OR
• The daemon sees there aren't virtual machine running anymore, it then loops to the second step above.

Package testing process specifications

This paragraph describes the progress of the test of a package under a guest. In the order, starting from the fact that the guest already received the package name to be tested (test_package):

• Get the time before the test suite starts
• Installation of the package {test_package} and store the STDOUT and STDERR logs, in the order, in a file log/{test_package}/install.log.
• Store the file list and binary file list contained in the package respectively in the files log/{test_package}/files.log and log/{test_package}/binaries.log.
• Test of the library dependencies on each binary and store the results in the files log/{test_package}/{binary_name}.ldd.log.
• Test of each binary and store the results in the files log/{test_package}/{binaryname}.log
• Launching of all the included executable files (binaries and scripts) (without knowing the possible needed arguments neither from the fact that the program will need a graphical server...) and store the resulting outputs in the files log/{test_package}/{binaryname}.log.
• Launching the same files with the --help and -h arguments.
• Testing of the binary and man page matches and store the results in the files log/{test_package}/{binaryname}.man.log.
• Test of each library and store the results in the files log/{test_package}/{libraryname}.log
• Test of the dependencies on all libraries in {test_package} and store the results in the files log/{test_package}/{libraryname}.ldd.log.
• Test of references (files and URLs) in man page. Report a log in log/{test_package}/{manpage_name}.log if a reference is missing.
• Deinstallation of the package and store the results in the file log/{test_package}/uninstall.log.
• Get the disk structure differences (file presence, file sizes, directory presence) before the package has been installed, just after it has been installed and after it has been uninstalled and store the results in the files log/{test_package}/<prein,postin,postin>-<prein,postin,postin>.<file,dir>-<list,size>.log
• Try to build the package {test_package} from its sources if they are available and store the results in log/{test_package}/build.log.
• Get the time just after the test suite has finished
• Compression of the directory log/{test_package} into the file log/{test_package}.tar.gz.
• Sending the file log/{test_package}.tar.gz to the server.

Technical specifications on client side

We define here the term client as the virtual machine guest client running on the final user's host real machine.

The client will mainly contain several shell several scripts doing all steps of the operations listed in the paragraph Package_testing_process_specifications.

• Automated launch of a starter program that will test the network and install the test-suite:
  • Define a script which will automatically launch the test at the end of the boot process (this implies to create a snapshot before this stage for speeding things up)
• Launch of the beginning of the test (and thus pending commands)
• Taking back the standard outputs and the standard output errors respecting the order:
  • Use the script annotate-output.sh as the command launcher to following commands.
• Package installation:
  • Use the urpmi command with the package name.
• Package uninstallation:
  • Use the urpme command with the package name.
• Taking back the file list (along with binary file list):
  • Use the urpmq -l command with the package name.
  • Use the grep command with the above line result with the filter bin/ in order to get the binary list.
• Package binaries launch tries:
  • Use the grep return line of the previous point for each package.
• Package binaries help command line checks:
  • Launch the program with --help and -h.
• Package binary dependency list:
  • Use the command ldd with the binary name.
• Matches between the package binaries and man pages:
  • Use a combination of command with grep man and the binary file name.
• Testing file alteration (file and directory creation, deletion, no-deletion, modification...):
  • Check each file and directory that has been created or altered between the initial state of the machine (before installation) and after the package installation, then between the package has been installed and after it has been installed, and between the initial state and after it has been uninstalled.
• Logs archive:
  • Use the command gzip on the directory containing the package logs.
• Connexion, remote server identification:
  • A key is provided by the server and will sometimes change. There will then need for the client package to be upgraded for the new key to match.
• Download of the package name to be tested:
  • Use the command curl with the appropriate request (to be defined in the server specs)
• Upload of the data to the remote server:
  • Use the curl -F command with the name of the archive, the URL to the remote server and the appropriate request allowing sending of files (to be defined in the server specs)
• Halting:
  • Simply use the halt command

Server side - Web services specifications

The remote testing server provides two Web services:

• Package names queue service
• Test report storage service

Package queue

The queue service is available at the URL http://doc4.mandriva.org/xwiki/bin/view/test-cloud/queue/package/{vendor-name}/{distribution-version}/{architecture}/{kernel-name-version}/. It can be requested over the HTTP GET method. It returns a package full name in plain text, or 0 if there is no package to be tested at the moment it is requested. Its first implementation simply queries the list of referenced packages in a database, and sends the next one.

Log storage

The storage service is available at a URL with the following scheme: http://doc4.mandriva.org/xwiki/bin/view/quartz/test-cloud/log. It receives request sent over the HTTP POST method only, in multipart/form-data format, with the following parameters:

• logfile: name of the logfile
• pkgname: name of the related package
• distribution: version of Mandriva
• arch: architecture of the distribution
• kernel: name of the kernel used

The current implementation of the service simply stores the log received, logs the operation.

File organisation

On host side

The default file organisation is as follow:

• /etc/test-cloud-client/config
  • Configuration file for the client host.
• /usr/bin/testcloud
  • Script to manage (start or stop) the daemon.
• /usr/bin/tc-host-daemon.sh
  • The Test-Cloud host client daemon. When launched this script will loop running guests on the row until it is stopped.
• /usr/bin/tc-host-admin.sh
  • Virtual machine creation script. This script has to be run once after the test-cloud-client rpm has been installed. It will create the first virtual machine settings for the user having the rights to run test-cloud.
• /usr/bin/tc-host-launch-snapshot.sh
  • Script to run the latest snapshot taken. It is mainly intended to quicken the testing process since it avoids the boot process under the virtual machine. This script has been removed from the project for the moment
• /usr/bin/tc-host-launch_vm.sh
  • Script that runs the test virtual machine from the boot. This script is not intended to be used by end-users.
• /usr/share/test-cloud-client/
  • This directory originally contains the compressed archive of the VirtualBox hard disk to be run. It is automatically uncompressed during post-rpm installation process. The name of the contained file can change regarding the version of the test-cloud-client package, the version of the virtualized OS...
• /usr/share/doc/test-cloud-client/ChangeLog
  • File that lists all the change logs.
• /usr/share/doc/test-cloud-client/README
  • Readme file that can be good for first help.
• /usr/share/doc/test-cloud-client/TODO
  • TODO list.

On guest side

This lists all files needed for a client to be able to run within a guest:

• /etc/{prog_name}/config
  • The configuration file defining the server URLs and requests and the time limit of binary executions.
• /etc/{prog_name}/install
  • The install file which contains the directory and the program name (actually only used by the install program).
• /etc/{prog_name}/key
  • The key to authenticate clients to the server.
• {install_dir}/bin/tc-guest-client.sh
  • The init script that launches the testing suite.
• {install_dir}/bin/tc-guest-annotate-output.sh
  • This script allows to redirect STDIN and STDOUT flows in a synchronous way.
• {install_dir}/bin/tc-guest-client-run.sh
  • This script is the main script that runs the tests.
• {install_dir}/bin/tc-guest-timered-sh.sh
  • This script allows to run a command during a maximum amount of time.
• {install_dir}/share/doc/{prog_name}/README
  • The README file.
• {install_dir}/share/doc/{prog_name}/version
  • The installed version of the client.
• /var/log/{prog_name}/
  • The directory where logs will be put.

with {prog_name} defaults to tc-guest (for Test-Cloud Guest-side), {install_dir} defaults to /usr and {daemon_dir} defaults to /etc/rc3.d (the latter could change regarding the chosen runlevel and the Linux distribution on which the guest runs).

The automated run of the test-suite is done by modifying the file /etc/inittab so that the first terminal (tty1), when the boot process has finished, will launch tc-guest-client.sh. This step is done during the RPM post-installation process. This way was chosen to avoid any problems with initscripts (mainly due to the scrictness of the initscripts policies).

On server side

This section is empty or not enough detailed, you can improve it by editing it !

Issues

• How to take the snapshot of the virtual machine just before the test program launch ? This will greatly improve how fast the test will run since it will allow to avoid most of the part of the boot process.

• How the server could surely authenticate that the connected client is really a good testing client but not a bad program ?

Exemple: A user installs the Automated Testing System package. He then owns VirtualBox and the virtual harddrive (vdi file) corresponding to the guest. He then uses this vdi as a second harddrive of another virtual machine and modifies some files in it in order to make the client behaves badly (server data corruption attempts, DOS attack on the server, or make the server works for nothing...).

• What to do if the server receives the data after the T time ? and do the server have to inform the guest that the test has been considered canceled ?

Architecture

On global process

On a single process

TODO

Tip !
Advice from Skiper - use more templates for commands and program lines such as prog, cmd, pkg, etc... See Page Index (in the left panel) > Templates for the list of templates.

TODO for this wiki

• Add the introduction of the vodka's article.
• General specs on server side.
• Technical specs on the server side.
• Final nomenclature (in progress).
• Specs acceptance.

Test-Cloud TODO list

• add relevant information: du -k pre/post install, date begin, date end [work in progress]
• individual build automation [work in progress]
• include/package existing test suites: ldtp, mysql-testsuite, apache-testsuite
• integrate tests of jmeter/wecka (Qualipso)
• ability to manage test priorities
• server side reporting
• integrate functional tests

Annexes

This section is empty or not enough detailed, you can improve it by editing it !

Useful links

This section is empty or not enough detailed, you can improve it by editing it !

Building Test-Cloud

This section is empty or not enough detailed, you can improve it by editing it !

Making

Making an RPM

Create a subdirectory (let's call it tar/) in the directory where the client resides. Create a new subdirectory in tar/ of the name and the version of the project (test-cloud-client-0.1.0 for example). In this directory copy all sources from the client (including TODO, ChangeLog...), but not sources from the vm. Include also the subdirectory cfg/ and its contents. And finally create a directory named vdi in test-cloud-client-0.1.0.

The structure of the directory should look like this:

client/
 _tar/
  _test-cloud-client-0.1.0/
   _cfg/

Now move the tar.gz in ~/rpmbuild/SOURCES and copy test-cloud-client.spec in ~/rpmbuild/SPECS.

Go to ~/rpmbuild/SPECS and launch:

rpmbuild -ba test-cloud-client.spec

The command should return 0. Now check that the rpm contains all needed files:

rpm -ql -p ../RPMS/i586/test-cloud-client-0.1.0-x.y.rpm where x.y is the rpm revision (as defined in ats.spec).

You can now try to install the rpm.

Application

Installation

Installation is quiete easy if you have a Mandriva 2009.1, if you have an i586 compatible architecture, if you run a window manager or a desktop and if you can have a kernel 2.6.29.1-desktop586-4mnb with VirtualBox kernel module (meaning you don't have a specific kernel for specific uses).

[root@computer ~]# urpmi test-cloud-client-0.0.9-0.1.i586.rpm

will install the package (about 130 Mo), so it might take times. After the package has been installed, a post-configuration is launching and can also take time since it has to decompress a 130 Mo archive.

Post install warnings: It is "normal" (if you ran the urpmi command under a terminal) if you see an error message like this one:

ERROR: Could not find a hard disk with location '/usr/share/ats/XXXX.vdi' in the media registry ('/root/.VirtualBox/VirtualBox.xml').

Execution

To execute the program, you first need to reboot to the specific kernel or if you use dkms you'll just need to modprobe vboxdrv. Then you'll need to ensure the module has been well installed to the kernel:

[user@computer ~]$ lsmod | grep vboxdrv

should return a non-empty line.

If all of this is okay, then you can launch the test program with this command:

[user@computer ~]$ /etc/init.d/ats_init.sh start

The virtual machines will run in the row (depending on a laps-time the deamon will make its checks).

In order to stop the daemon simply run:

[user@computer ~]$ /etc/init.d/ats_init.sh stop

If a virtual machine is currently running, it will finish its work before halting. And no other virtual machines will further run until you start the daemon again.

Troubleshoottings

• If you have a VirtualBox version below 2.2 (naimly 2.0), you won't be able to run the test suite. As far as we know, there aren't any solution to this.
• It seems you do not have root permissions to run the test program since anybody seem to be able to have enough permissions to execute the kernel module. However this might change in the future.
• If the network does not work, launch VirtualBox and try to change the network device type to another one, and try the program again. If none of the 4 cards allow networking to run, you can try to contact us.
• If you have the kvm module installed on your kernel, you'll then have to remove it temporarily, run the test, then you'll be able to insert the kvm module again.

Links

Internal links

• Building a virtual machine with VirtualBox
• Automate virtual machines creation

External links

(todo)