Development/Howto/Iurt
From Mandriva Community Wiki
Iurt is a perl script which is used to recompile SRPMs on another architecture
Contents |
Iurt is a recompilation bot which monitors lists of packages of different architectures and recompiles them in a separated clean chroot each time it is needed.
[edit] Current Iurt performance
Here is a brief show off of Iurt current rebuild status. main and contrib are the total number of packages in the x86_64 cooker repository (noarch are not counted) and by iurt is the number of packages rebuilt by Iurt.
| Date | Distro | main | by iurt | contrib | by iurt | total | by iurt |
|---|---|---|---|---|---|---|---|
| 20060607 | cooker x86_64 | 4056 | 1761 (43%) | 5056 | 1715 (34%) | 9112 | 3476 (38%) |
| 20060615 | cooker x86_64 | 4090 | 1888 (46%) | 5093 | 1727 (33%) | 9183 | 3615 (39%) |
| 20060624 | cooker x86_64 | 4104 | 1991 (48%) | 5191 | 1904 (36%) | 9295 | 3895 (41%) |
| 20060718 | cooker x86_64 | 4071 | 1963 (48%) | 5288 | 2053 (38%) | 9359 | 4016 (42%) |
| 20060804 | cooker x86_64 | 4068 | 1943 (47%) | 5307 | 2214 (41%) | 9375 | 4157 (44%) |
| 20060813 | cooker x86_64 | 4089 | 1907 (46%) | 5364 | 2366 (44%) | 9453 | 4273 (45%) |
| 20060915 | cooker x86_64 | 4129 | 2088 (50%) | 5361 | 2337 (43%) | 9490 | 4425 (46%) |
[edit] How to get it
Iurt is not packaged, however you can fetch it from the Mandriva SVN at http://svn.mandriva.com/cgi-bin/viewvc.cgi/soft/build_system/iurt/trunk/
Note: if you are an official Mandriva packager, you can find iurt in the /home/mandrake/bin/iurt2 file on the compilation cluster.
[edit] Usage
Iurt is monitoring the source and binary packages from the various repositories and media, and recompiles them if needed.
[edit] General usage
There are six different usages:
iurt --distro cooker --config_help to display current values from config files, which will be used
iurt --distro cooker --chroot --arch i586 to create or update the tarball for the chroot
iurt --distro cooker --rebuild i586 cooker /path/to/foo.src.rpm to build the package foo in the chroot
iurt --distro cooker --arch i586 --media bar to rebuild all non up-to-date SRPMS for the media bar
iurt --shell create a empty chroot and dump a shell into it
iurt --distro cooker --arch i586 --dkms To recompile all the dkms packages for each available kernel
[edit] Iurt usage on the Mandriva compilation cluster
On the Mandriva compilation cluster, Iurt is available through sudo /usr/local/bin/iurt.
The available modes are
sudo iurt package.src.rpm recompile the package in a cooker chroot.
sudo iurt2006 package.src.rpm recompile the package in a 2006.0 chroot.
sudo iurt --shell create an empty chroot and give a shell in it.
sudo iurt --clean remove the old previously created chroot of the user.
sudo iurt --stop <step> package.src.rpm create a chroot, install the build dependencies and perform a rpm -b<step> package.src.rpm, then give a shell in the chroot.
[edit] Configuration
You must first create a configuration file in the home of the user running iurt. More precisely a configuration file per distro version is needed. To recompile on cooker, you must create a file named .iurt.cooker.conf. This file is a perl script which will be loaded as a configuration variable.
The --distro foo option is here to decide which config file has to be built. Make use of the option --help_config to be sure your config file is ok.
Here is an example:
{
home => "/home/mandrake/",
cache_home => "/home/mandrake/.bugs/",
supported_arch => [ 'i586', 'x86_64' ],
upload => "kenobi:/home/mandrake/uploads/",
local_home => "/export/home/mandrake/",
unwanted_packages => '^monotone-',
repository => '/mnt/BIG/dis/',
rsync_to => "mandrake\@qa:/var/www/html/build/iurt/",
admin => '[email protected]',
sendmail => 1,
packager => 'Iurt the rebuild bot <[email protected]>',
install_chroot_binary => '~warly/files/cvs//mdk/soft/rpm-rebuilder/install-chroot-tar.sh',
distribution => 'Mandriva Linux',
vendor => 'Mandriva',
basesystem_media => ''
}
- home your home dir, defaults to $HOME;
- cache_home where to store the iurt cache (for cooker this cache is about 3 MB), defaults to $HOME/.bugs/;
- upload where to copy resulting packages, defaults to $HOME/uploads/;
- local_home where to build packages (this can be different from you $HOME if you are on NFS and you want to compile locally), defaults to $HOME;
- unwanted_packages if you do not want to compile a certain package, you can enter here a regexp, defaults to nothing;
- repository is the location of your local packages, note that your packages must be in a Mandriva like structure;
- rsync_to can be set if you want to send the reports on another machine, the default is to do nothing;
- admin who to spam with all the warning messages. Default is nothing;
- sendmail if a warning mail will be sent for encountered error. Default is not to send mail;
- packager field used as packager tag and mail origin. Default is 'Iurt';
- install_chroot_binary program to create and manage chroot, it is not yet packaged and needs to be fetched for the CVS. Default is install-chroot-tar.sh;
- vendor default is Mandriva;
- distribution default is Mandriva Linux;
- basesystem_media to select which media will be used to create the initial chroot.
To have a full listing of available options, use iurt --config_help, and, even better, iurt --config_help cooker x6_64 main so that you can debug your default values
Note: at the first run iurt will be quite slow because it will need to scan all the packages to populate its cache.
Note2: It is highly recommended to use the same tree as Mandriva, for storing your repository: /some/path/cooker/$arch/media/main
[edit] Options
[edit] Unionfs (--unionfs)
The default behaviour is to use a basesystem with rpm-build compressed in a tar.gz. For each new packet to build, the old chroot is removed, and a new one is created with the tar.gz, then the build dependencies are installed and the package compiled. If you can have unionfs with your kernel, and enough ram, you can bypass all these untar steps. The chroot is then just merged with a tmpfs filesystem with unionfs, and once the compilation is finished, the unionfs is just umounted. Everything is done in ram and as a consequence really faster. You can bypass the initial chroot check with --nocheckchroot.
[edit] Debug (--debug)
You can call iurt with the --debug flag to debug your configuration. You can run iurt with the debug mode even if another iurt is already running: it will create an alternate chroot and try to compile only one package. It will not update the cache, not send any mail or move any packages.
[edit] Shell (--shell)
If you need to work out or compile several packages in a clean chroot, you can use the --shell option to have a shell with your local user in an empty chroot with basesytem, sudo and urpmi. A sudo will be setup on rpm, urpmi, urpme and urpmi.addmedia.
[edit] DKMS (--dkms)
DKMS packages are kernel external module sources which are recompiled when the package is installed or when a new kernel is installed on the machine. However we can pre compile them for the kernel available in the distribution, so that they can be directly used (to create live CDs for example, or just for user convenience).
[edit] Extra Options
- --clean: this option removes the old chroot remaining in the user's temporary chroot directory
- --clean_all: remove all the chroots for all the users in the temporary chroot directory.
- --chrooted-urpmi: this option adds the urpmi media inside the chroot and does not use the urpmi --root command, which usually leads to an urpmi database locking problem.
[edit] Log files
You can see the current Mandriva Iurt log files at http://qa.mandriva.com/build/iurt/.
Iurt is logging several things during the package's build:
- install_deps_<srpm>.<build_nb>.log: result of the urpmi -s of the srpm in the chroot;
- rpm_qa_<srpm>.<build_nb>.log: list of packages installed before the build in the chroot;
- build_<srpm>.<build_nb>.log: result of the rpm --rebuild of the srpm in the chroot;
- binary_test_<srpm>.<build_nb>.log: result of urpmi of the generated binary packages in the chroot.
[edit] Internals
[edit] Cache
Iurt is caching 6 things:
- rpm_srpm: the correspondence between binaries and source packages;
- failure: all the packages which build has failed, not to recompile them, if you want to force a recompilation, you must remove the package entry in this hash;
- queue: all the binary packages which have been built;
- warning: iurt keeps a list of warning messages sent, to limit the number of mails sent;
- run: the number of iurt runs, this is used in the log file extension, to regroup all the log files of a same run;
- needeed: keep a list of build requires and only try to rebuild a given package when all it's buildrequires are fulfilled;
- no_unionfs: some packages could not be built with unionfs (they need too much space, or they need to access to /proc/self/exe which is not correct with unionfs)
[edit] Algo
Iurt first checks if another iurt is running, then scans the packages in the repository. After that it computes the list of packages which need to be recompiled, creates or verifies the chroot tar which will be used to recompile the packages. It then loops on the packages to compile, reinstalling a new chroot each time, then the build dependencies, compiling the packages, trying to install the resulting binaries, and copying them to their final destination.
[edit] Fault tolerance
Iurt has to be very robust against environment troubles. It must be able to detect and not stop at each problem, namely filesystem available space, concurrent runs, hdlist problems, chroot corruption, temporarily missing resources and such.
[edit] Known bugs
Some packages cause troubles with iurt. The two most common are gconftool-2 and gst-register-0.8. Sometimes they don't exit and an urpmi process is left running in memory. When several gnome packages are recompiled, for each one gconftool-2 and one or two urpmi will stay in memory, eating slowly all your RAM. The current workaround here is to kill these two processes as soon as they appear. watch 'killall -9 gconftool-2 gst-register-0.8 2>/dev/null' in a screen will do the job.
[edit] Needed Packages
For running it, you'll need at least: perl-Filesys-Statvfs_Statfs_Df perl-RPM4 perl-File-NCopy perl-Data-Dump perl-MIME-tools mkcd ( urpmi them).
Non packaged script: install-chroot-tar.sh (http://cvs.mandriva.com/cgi-bin/viewvc.cgi/soft/rpm-rebuilder/ )
[edit] TODO
First feel free to send comments or feedback to [email protected].
Here is a list of important things to look at:
- have it more generic, to be used in several environments ( iurt is quite linked to Mandriva though, with calls to urpmi and other Mandriva tools. However, thanks to RafaelGarciaSuarez, urpmi is now on CPAN);
- need to test also packages updating (installing old version and updating the newly generated packages).
- need to add a compilation set option, to compile all the packages of a group altogether
