Projects/Testziller

From Mandriva Community Wiki

Jump to: navigation, search
Testziller

Testzilla will be more and more used as a mean to provide feedback on problems encountered in the day-to-day use of the distribution. Furthermore, its goal is to provide a more robust distribution since automatic test process will become more and more used.

Contents


[edit] Goals

Testzillers will be involved in testing the distribution in a very formal way, to validate the tests themselves but furthermore to improve the quality feedback. On this web page, you'll find additionnal information about the testzilla process itself but also links to documentations concerning the distribution.

[edit] Overview

Testzilla itself is presented at this page.

To help testing on Cooker, Read this Bugzilla Howto and create yourself A Bugzilla account here if you don't already have one: Create a new Bugzilla account.

Then Read this Cooker Howto and install a Cooker.

Read this General Test Scenario and then read this Perform a test which needs to be done.

[edit] Documentation

Some general pages about where to look, what to do for a better bug reporting is on its way.

Have a look at this page to be complete in your bug report.

[edit] General Test Scenario

Before performing any test, place yourself in the context of an end user. This means trying to react like a person with no specific linux skills, and always use the GUI or frontend we provide to a software.

When doing so, you test at the same time the GUI, the frontend and the backend. Use the command line only if you can't perform the task with a GUI and/or to debug and locate whether the problem comes from the GUI or the backend.

The GUI can be a wizard, a web frontend (such as webmin), and if there is no GUI, we can even have command line tools which are frontends.

i.e.: drakfirewall is a GUI to shorewall which itself is a frontend to iptables. So first use drakfirewall, then see with shorewall, and at the end check with iptables.

Performing tests in this order is the best way to locate where the bug comes from, and as for an example : if all the functionalities provided by drakfirewall work, implicitly those same functionalities are validated for both shorewall and iptables.

Of course not everything can be handled via drakfirewall, so do only the further tests not feasible with drakfirewall with shorewall, and then the deep tests of functionalities not provided by shorewall with iptables.

For all tests, try to perform them in this order.

The above examples were for drakfirewall->shorewall->iptables, but the same reasoning method has to be performed with draksambashare->samba->smbfs.

In one sentence we could sum it up this way : Start scratching from the surface, and dig further deeper and deeper to reach the bottom.

[edit] To Do before testing

[edit] Create a Bugzilla account

To create a Bugzilla account, click here.

Once done, read carefully the Bugzilla Howto.

Take a good look at the

[edit] Usefull Links for testing

[edit] Test Scenario

[edit] Installation

  • Insert the distribution CD/DVD
  • Boot on the CD/DVD.
  • Select your language, country, and keyboard.
    • Try exotic combinations. i.e. you're an American living in France and you have a German keyboard because your computer was bought in Germany : you should be able to have the distribution in English, with the country (i.e. France) settings for the time zone and other, and the German keyboard.
  • Try different types of partitioning: automatic, expert... All that the wizard allows you to do, try resizing, changing file system (ext3, reiserfs, XFS, etc...) try canceling the changes, etc...
  • Try different types of package selection for installation. Because we support both Gnome and KDE, try an installation with only KDE, another one with only Gnome, and another one with both. Try also the individual package selection.
  • Perform the installation. Try to count (with a stop-watch not out loud ;-) ) how long the installation takes.
  • Set the root password and create users.
  • when prompted for the bootloader, try lilo *and* Grub (not at the same time, but on different installations ;-) ).
  • Check the post install summary. Try also doing modifications, i.e. reconfigure your network interface(s) and give a hostname, check locales, time zone (set ntp for example), try a different screen resolution, try installing/removing (a) printer(s). Try changing back and forth from lilo to grub. etc... Don't leave anything untested out of the provided possibilities.
  • Finish the install, and count also how long it takes before the time you clicked "finish" and the reboot (It should be an acceptable waiting time).
  • Reboot on the freshly installed system, count how long you wait before you can get the login prompt.

[edit] Post-Install

  • Boot the distribution, count the boot time before you get the login prompt.
    • If you have a Windows partition, Try booting windows to check the installer didn't break it.
    • Beware that at the first boot you get the "drakfirsttime" wizard, test this wizard.
  • Log into your session, this shouldn't be too long (yes... count again... ;-) ).
  • Check the Desktop, look for anything that could be wrong.
    • Check the Menu entries -> all applications should run (no broken menu entries).
  • Check the Mandriva tools in the MCC (Mandriva Linux Control Center)
    • Test all the available wizards. If a wizard is incomprehensible, it is a designing/documentation problem in the wizard and it ought to be reported.
    • For all the wizard tests, the available functionalities proposed by the wizard should be feasible at 100% with the wizard. If after the wizard usage somethings needs to be finished via the command line to get things working, it is a Bug in the wizard.
    • Always keep this in mind reasoning method.
  • Check the WebSurfing, Mail retrieving and reading, printing, scanning... All common usage functionalities.
  • Check the usage of removable medias (USB keys, CD/DVD).
  • Check all the applications, they should work normally and not segfault.
    • Test the multimedia applications.
    • Test the Office applications and printing from them.
  • Apply the test cases which need to be done and which are listed by Testzilla here
    • Click on the New Test Button
      • Follow the procedure and then:
        • If successful, click on the Test result OK button.
        • If failed, select the application that failed in the list on the lower right of the page and click on the Report a bug on button beside the list.
    • If the list in the above link of applications which need to be tested is empty, it means all tests have been performed by someone and validated as OK. But you can still perform any of the test cases again even if a test has already been done and validated as OK. All test cases of Testzilla are here, and you may (who knows) find a bug which could have been unseen in the previous test.
  • test MandrivaOnline if you have a Valid Silver, VIP or above Mandriva Club membership (ask warly|AT|mandriva.com if you need a valid account)
  • Be inventive... Do what you would wish to do...

[edit] Report Bugs

Report bugs on Bugzilla.

Retrieved from "http://wiki.mandriva.com/en/Projects/Testziller"
Personal tools