Development/Packaging/BuildSystem/Details
From Mandriva Community Wiki
The Build System's gory implementation details
Contents |
[edit] Actual implementation
[edit] The real big picture
A number of details make the real system more complicated than presented in Build System Explained. First, the build user has a work area (called local home) in each cluster node besides the home directory that can be NFS-mounted, and Ulri and Iurt must be correctly configured to use these two directories to perform all disk-intensive operations in the local home and get log files in the correct place. Additionally, complexity also exists in package submission and collection: both operations involve Youri copies to another location in the local host. Finally,
[edit] Local spool
A work area in each nodes local filesystem is called local home or local spool and used by Ulri and Iurt to build packages. In the build cluster each node has the regular /home/mandrake directory mounted from kenobi, and the local work area is /export/home/mandrake. Ulri must invoke Iurt with --config local_spool <dir> (actual value in cluster is --config local_spool /export/home/mandrake/iurt/__DIR__). The packages and log fields in bot configuration in Ulri must also point to the local spool location.
[edit] Scheduling
[edit] The scheduling cycle
Ulri is executed periodically from crontab, so retrieving results of a build is not performed until a later execution of the scheduler. Builds on different architectures are totally asynchronous. Each scheduler run starts with a verification if previously scheduled builds finished, and then it schedules new jobs.
The scheduler cycle (View large | SVG)
[edit] Lock files
When a compilation job is submitted to a remote node, a lock file in the format <time>.<user>.<host>.<pid>_<arch>.iurt.<node>.<time>.<pid>.lock is created in the same directory where the source is in the todo list. A real example of a package/lock pair is:
/home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_i586.iurt.frohike.20061102014649.18860.lock /home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_@17117:byacc-1.9-mdk.src.rpm
The lock file is removed only in a subsequent execution of Ulri.
[edit] Retrieving binaries
Binary packages are retrieved after checking the lock file. Ulri checks for a lock file; if it exists, it verifies the status.log file in the remote node and, if the compilation succeeded, it copies the binary files to the local uploads directory. An example is:
/home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_@17117:byacc-1.9-mdk.src.rpm
Note that the packages are not actually moved, copies of the source and binary packages remain in the build node and must be eventually cleaned up.
[edit] Example
This is a real log snippet showing collection of binaries from a previous job and scheduling of a new job:
ulri: checking rebuild bot results ulri: checking for status for i586 on frohike done by iurt (22264) in /export/home/mandrake/iurt///cooker/contrib/release/20061028122600.goetz.kenobi.10207/\ log/status.log ulri: checking for result for i586 on frohike done by iurt (22264) in /export/home/mandrake/iurt///cooker/contrib/release/20061028122600.goetz.kenobi.10207/ ulri: @73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.src.rpm: ok ulri: compilation succeeded for @73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.src.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/ @73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv200 100% 464KB 464.0KB/s 00:00 ulri: compilation succeeded for gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.i586.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/ gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.i58 100% 105KB 104.9KB/s 00:00 ulri: compilation succeeded for gaim-encrypt-debug-2.0.0_3.0-0.beta6.1mdv2007.1.i586.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/ gaim-encrypt-debug-2.0.0_3.0-0.beta6.1mdv2007 100% 160KB 159.9KB/s 00:00 ulri: launching new compilations ulri: searching a bot to compile @73247:seahorse-0.9.6-2mdv2007.0.src.rpm ulri: searching a free bot for arch i586 ulri: compiling @73247:seahorse-0.9.6-2mdv2007.0.src.rpm by iurt on frohike for i586 ulri: copying /home/mandrake/uploads//todo///cooker/contrib/release/20061026195350.goetz.kenobi.12742_@73247:seahorse-0.9.6-2mdv2007.0.src.rpm to \ mandrake@frohike:/export/home/mandrake/iurt///cooker/contrib/release/20061026195350.goetz.kenobi.12742/@73247:seahorse-0.9.6-2mdv2007.0.src.rpm 20061026195350.goetz.kenobi.12742_@73247:seah 100% 1390KB 1.4MB/s 00:00 ulri: ssh -o ConnectTimeout<tt>20 -o BatchMode</tt>yes -x mandrake@frohike sudo -u mandrake -H /usr/local/bin/iurt2.sh --copy_srpm --group -v 1 \ --config local_spool /export/home/mandrake/iurt//cooker/contrib/release/20061026195350.goetz.kenobi.12742 --no_rsync --chrooted-urpmi http://monstro/dis/ \ -r cooker i586 /export/home/mandrake/iurt///cooker/contrib/release/20061026195350.goetz.kenobi.12742/@73247:seahorse-0.9.6-2mdv2007.0.src.rpm \ (/tmp//ulri.tmp.20061026195350.goetz.kenobi.12742.QYhtb) ulri: remote pid 32602 ulri: creating lock /home/mandrake/uploads//todo///cooker/contrib/release/20061026195350.goetz.kenobi.12742_i586.iurt.frohike.20061102115550.32602.lock
[edit] Uploads
[edit] Submission and input queue
The actual package submission process is more complex than what is presented in The Build System Explained the files checked out from the Subversion repository don´t land directly on the input queue, they ...TODO?
[edit] Collection and the output queue
[edit] Configuration files
[edit] Ulri configuration file
{
admin => 'root@monstro',
bot => {
i586 => {
frohike => {
iurt => { user => 'mandrake' ,
command => 'sudo -u mandrake -H /usr/local/bin/iurt2.sh --copy_srpm --group -v 1 --config local_spool \
/home/mandrake/iurt/__DIR__ --no_rsync --chrooted-urpmi http://monstro/dis/ -r __TARGET__ __ARCH__',
packages => '/home/mandrake/iurt/',
log => '/home/mandrake/iurt/',
} ,
},
},
x86_64 => {
....
},
},
default_mail_domain => 'default_mail_domain',
faildelay => 36000,
http_queue => 'http://monstro/queue/',
queue => "/home/mandrake/uploads/",
cache_home => "$HOME/.bugs",
repsys_conf => '/etc/repsys.conf',
tmp => "$HOME/tmp/",
ssh_options => "-o ConnectTimeout=20 -o BatchMode=yes",
}
[edit] Iurt configuration file
[edit] Filesystem structure
[edit] Upload tree
[edit] uploads/done
[edit] uploads/todo
- First level: target (cooker, 2007.0, etc)
- Second level: media (restricted, main, contrib, non-free, debug_main, debug_contrib, debug_non-free)
- Third level: release, testing, updates, backports
Example:
todo/cooker todo/cooker/restricted todo/cooker/restricted/release todo/cooker/debug_contrib todo/cooker/debug_contrib/updates todo/cooker/debug_contrib/testing todo/cooker/debug_contrib/release todo/cooker/debug_main todo/cooker/debug_main/updates todo/cooker/debug_main/testing todo/cooker/debug_main/release todo/cooker/main todo/cooker/main/updates todo/cooker/main/testing todo/cooker/main/release todo/cooker/main/backports todo/cooker/contrib todo/cooker/contrib/updates todo/cooker/contrib/testing todo/cooker/contrib/release todo/cooker/contrib/backports todo/cooker/debug_non-free todo/cooker/debug_non-free/updates todo/cooker/debug_non-free/testing todo/cooker/debug_non-free/release todo/cooker/non-free todo/cooker/non-free/updates todo/cooker/non-free/testing todo/cooker/non-free/release
