Build System Details

From Mandriva Community Wiki

The content of this article is obsolete or outdated
The following information can be obsolete or outdated and needs to be revised. You can also check the other pages that require revision.

Actual implementation

The real big picture

A number of details make the real system more complicated than presented in Build System Explained. First, the build user has a work area (called local home) in each cluster node besides the home directory that can be NFS-mounted, and Ulri and Iurt must be correctly configured to use these two directories to perform all disk-intensive operations in the local home and get log files in the correct place. Additionally, complexity also exists in package submission and collection: both operations involve Youri copies to another location in the local host. Finally,

Local spool

A work area in each nodes local filesystem is called local home or local spool and used by Ulri and Iurt to build packages. In the build cluster each node has the regular /home/mandrake directory mounted from kenobi, and the local work area is /export/home/mandrake. Ulri must invoke Iurt with --config local_spool <dir> (actual value in cluster is --config local_spool /export/home/mandrake/iurt/__DIR__). The packages and log fields in bot configuration in Ulri must also point to the local spool location.

Scheduling

The scheduling cycle

Ulri is executed periodically from crontab, so retrieving results of a build is not performed until a later execution of the scheduler. Builds on different architectures are totally asynchronous. Each scheduler run starts with a verification if previously scheduled builds finished, and then it schedules new jobs.

Scheduler figure

Lock files

When a compilation job is submitted to a remote node, a lock file in the format <time>.<user>.<host>.<pid>_<arch>.iurt.<node>.<time>.<pid>.lock is created in the same directory where the source is in the todo list. A real example of a package/lock pair is:

/home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_i586.iurt.frohike.20061102014649.18860.lock
/home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_@17117:byacc-1.9-mdk.src.rpm

The lock file is removed only in a subsequent execution of Ulri.

Retrieving binaries

Binary packages are retrieved after checking the lock file. Ulri checks for a lock file; if it exists, it verifies the status.log file in the remote node and, if the compilation succeeded, it copies the binary files to the local uploads directory. An example is:

/home/mandrake/uploads/todo/cooker/main/release/20061101142126.claudio.monstro.26213_@17117:byacc-1.9-mdk.src.rpm

Note that the packages are not actually moved, copies of the source and binary packages remain in the build node and must be eventually cleaned up.

Example

This is a real log snippet showing collection of binaries from a previous job and scheduling of a new job:

ulri: checking rebuild bot results
ulri: checking for status for i586 on frohike done by iurt (22264) in /export/home/mandrake/iurt///cooker/contrib/release/20061028122600.goetz.kenobi.10207/\
log/status.log
ulri: checking for result for i586 on frohike done by iurt (22264) in /export/home/mandrake/iurt///cooker/contrib/release/20061028122600.goetz.kenobi.10207/
ulri: @73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.src.rpm: ok
ulri: compilation succeeded for @73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.src.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/
@73601:gaim-encrypt-2.0.0_3.0-0.beta6.1mdv200 100%  464KB 464.0KB/s   00:00
ulri: compilation succeeded for gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.i586.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/
gaim-encrypt-2.0.0_3.0-0.beta6.1mdv2007.1.i58 100%  105KB 104.9KB/s   00:00
ulri: compilation succeeded for gaim-encrypt-debug-2.0.0_3.0-0.beta6.1mdv2007.1.i586.rpm, copying files in /home/mandrake/uploads//done///cooker/contrib/release/
gaim-encrypt-debug-2.0.0_3.0-0.beta6.1mdv2007 100%  160KB 159.9KB/s   00:00
ulri: launching new compilations
ulri: searching a bot to compile @73247:seahorse-0.9.6-2mdv2007.0.src.rpm
ulri: searching a free bot for arch i586
ulri: compiling @73247:seahorse-0.9.6-2mdv2007.0.src.rpm by iurt on frohike for i586
ulri: copying /home/mandrake/uploads//todo///cooker/contrib/release/20061026195350.goetz.kenobi.12742_@73247:seahorse-0.9.6-2mdv2007.0.src.rpm to \
mandrake@frohike:/export/home/mandrake/iurt///cooker/contrib/release/20061026195350.goetz.kenobi.12742/@73247:seahorse-0.9.6-2mdv2007.0.src.rpm
20061026195350.goetz.kenobi.12742_@73247:seah 100% 1390KB   1.4MB/s   00:00
ulri: ssh -o ConnectTimeout<tt>20 -o BatchMode</tt>yes -x mandrake@frohike sudo -u mandrake -H /usr/local/bin/iurt2.sh --copy_srpm --group -v 1 \
--config local_spool /export/home/mandrake/iurt//cooker/contrib/release/20061026195350.goetz.kenobi.12742 --no_rsync --chrooted-urpmi http://monstro/dis/ \
-r cooker i586  /export/home/mandrake/iurt///cooker/contrib/release/20061026195350.goetz.kenobi.12742/@73247:seahorse-0.9.6-2mdv2007.0.src.rpm \
(/tmp//ulri.tmp.20061026195350.goetz.kenobi.12742.QYhtb)
ulri: remote pid 32602
ulri: creating lock /home/mandrake/uploads//todo///cooker/contrib/release/20061026195350.goetz.kenobi.12742_i586.iurt.frohike.20061102115550.32602.lock

Uploads

Submission and input queue

The actual package submission process is more complex than what is presented in The Build System Explained the files checked out from the Subversion repository don´t land directly on the input queue, they ...TODO?

Collection and the output queue

This section is empty or not enough detailed, you can improve it by editing it !

Configuration files

Ulri configuration file

{
    admin               => 'root@monstro',
    bot                 => {
            i586 => {
                frohike => {
                    iurt => { user => 'mandrake' ,
                        command => 'sudo -u mandrake -H /usr/local/bin/iurt2.sh --copy_srpm --group -v 1 --config local_spool \
/home/mandrake/iurt/__DIR__ --no_rsync --chrooted-urpmi http://monstro/dis/ -r __TARGET__ __ARCH__',
                        packages => '/home/mandrake/iurt/',
                        log => '/home/mandrake/iurt/',
                    } ,
                },
            },
            x86_64 => {
                ....
            },
      },
   default_mail_domain  => 'default_mail_domain',
   faildelay            => 36000,
   http_queue           => 'http://monstro/queue/',
   queue                => "/home/mandrake/uploads/",
   cache_home           => "$HOME/.bugs",
   repsys_conf          => '/etc/repsys.conf',
   tmp                  => "$HOME/tmp/",
   ssh_options          => "-o ConnectTimeout=20 -o BatchMode=yes",
}

Iurt configuration file

This section is empty or not enough detailed, you can improve it by editing it !

Filesystem structure

This section is empty or not enough detailed, you can improve it by editing it !

Upload tree

uploads/done

uploads/todo

• First level: target (cooker, 2007.0, etc)
• Second level: media (restricted, main, contrib, non-free, debug_main, debug_contrib, debug_non-free)
• Third level: release, testing, updates, backports

Example:

todo/cooker
todo/cooker/restricted
todo/cooker/restricted/release
todo/cooker/debug_contrib
todo/cooker/debug_contrib/updates
todo/cooker/debug_contrib/testing
todo/cooker/debug_contrib/release
todo/cooker/debug_main
todo/cooker/debug_main/updates
todo/cooker/debug_main/testing
todo/cooker/debug_main/release
todo/cooker/main
todo/cooker/main/updates
todo/cooker/main/testing
todo/cooker/main/release
todo/cooker/main/backports
todo/cooker/contrib
todo/cooker/contrib/updates
todo/cooker/contrib/testing
todo/cooker/contrib/release
todo/cooker/contrib/backports
todo/cooker/debug_non-free
todo/cooker/debug_non-free/updates
todo/cooker/debug_non-free/testing
todo/cooker/debug_non-free/release
todo/cooker/non-free
todo/cooker/non-free/updates
todo/cooker/non-free/testing
todo/cooker/non-free/release

Appendix

All the scripts

Most of these scripts are available in the CVS modules soft/bugs/bin or soft/robot/

compilation cluster scripts (kenobi)

• uplftp: script to upload packages (used as ftpcooker, ftpcontrib, ftpcookerforce, ftp2006.0...)
• Upload: script to compile and upload packages
• iurt: recompilation bot
• iurt2: recompilation bot using only chroots
• topbastards: script which lists disk usage per user on the compilation cluster

Distribution server scripts (ken)

• admin: scripts which go through the upload queue and uploads (or rejects) the given packages into the different media
• admin_old_structure: upload scripts for packages prior to repository strucure change (community 10.0)
• break_hardlink: remove existing hard links from a directory (create a real copy)
• build-hdlist: build hdlist files for non standard dirs (not handled by gendistrib)
• cdcom-push-kenobi: deprecated, copy 8.2 cdcom to kenobi
• ch: give the last changelog entry of an rpm package
• changelog: create a changelog message for the changelog mailing list
• changelog_old_structure: create a changelog message for old structure based version
• check_bad_cvs_spec.sh: reintroduce package spec file into the CVS for all CVS based package
• check_changelog.sh: check if a new package contains the old package changelog entry
• check_corpo.sh: check that various packages are or are not on the corporate CDs
• check-distrib: check coherency between SRPM and binaries.
• check-multiarch.sh: check for multiarch files into the packages of a directory
• check_old.pl: Check if a rpm file is newer than one in a directory
• check_owner_of_rpm.pl: Check if a rpm package contains local user file owner
• check_updates.sh: check coherency of the updates file into the updates tree
• clic-get-192.168.100.118: rsync script
• community_pending: Send a report mail listing all the packages in the community versions upload queue.
• community-push-kenobi: mirror community tree to kenobi
• community-push-rsync: mirror community tree to rsync
• contrib2rss.sh: create a rss feed for contrib packages
• cooclean: remove the doble of cooker
• cooker2rss.sh: create a rss feed for cooker main packages
• cooker-get-amd64: rsync script
• cooker-get-kenobi: get data from kenobi (updates and internal CVS)
• cooker-push-kenobi: push cooker to kenobi
• cooker-push-rsync: push cooker to rsync
• corpo_copy: rsync corporate to another machine
• corpo_push_n1: rsync signed packages to n1 build machine (corpo/10.0 chroot)
• corpo_resign: use resing_dir to resign the corporate directories
• cvcp: commit the SRPM package into the SRPM CVS
• diffrpm.sh: create a rpm diff for a list of packages
• diffsrcrpm: create a diff between source package
• diffsrpm.sh: One more diff script
• doble: remove doble in a list of rpm
• doble2: remove doble in a list of directory
• doble3: slight variation regarding kernel handling and delta rpm
• doble_no_stat: use only urpm_range_overlap to find old packages
• extracts: extracts menu, initscripts and co from the packages to commit them to the CVS
• fedora-rsync: rsync fedora packages locally
• gen-description.py: generate a description from the changelog for community packages (to be able to use MandrivaUpdate with community sources)
• gen-descriptions: run gen-descryption.py for 10.0
• gen-descriptions2: run gen-descryption.py for old install structures (Mandrake/base)
• gen-descriptions3: run gen-descryption.py for new install structure (media/)
• get-jpackage: mirror jpackages repository
• get-plf: get plf mirror
• incoming: get submitted packages from contributors who do not have an access to the compilation cluster
• make_delta_dir: create all the delta rpm from two directories
• make_delta_rpm: create a delta rpm
• mir: mirror packages to rsync.mandriva.com, main mirror reference machine
• mkcd_ppc.pl: modified mkcd version to create ppc boot ISO files.
• ndoble: remove doble only based on the date.
• old: move package to the old dir and touch them
• other-get-kenobi: get club packages, testing packages, sparc, sparc64, alpha packages and ppc install from kenobi
• other-push-kenobi: push extra packages to kenobi (old packages, rejected packages...)
• other-push-rsync: push extra packages to rsync (testing section)
• process-obsoletes: check the obsoletes tag of an uploaded package and remove them from the repository19181
• process_queue: copy kenobi rpmctl command queue from kenobi
• rebuild_srpms: rebuild all the srpms of a given directory
• rejected: send a error mail about a rejected package to the appropriate person/mailing list with the rejected reason
• resign_contrib_dir: resign a dir with the contrib signature
• resign_cooker_dir: resign a dir with the cooker signature
• resing_dir: check the signature of all the packages of a directory and resign them with the official key
• rpmctl: script to delete or move packages between packages media
• rpmheader: python script to display a given header from a RPM
• rpmmon: script to find information about who maintain a given packages (local modified version to give full mail address).
• rsync-cooker-cvs: rsync the CVS to ken.
• rsync_corpo: rsync the corporate server and desktop 3.0 from the 10.0 chroot of n1.
• rsync_mkcd: rsync cooker and community to guerd.
• rsync_old_ken: rsync cooker, updates, rejected and old to old_ken and raoh.
• rsync_old_ken_other: rsync less changing parts (community, corporate, old updates) to backup machines (old_ken and raoh)
• sync_beta: create a snapshot of cooker for a beta
• sync_beta_ppc: create a snapshot of the cooker ppc for a beta
• sync_community_old_updates: includes the updates for the offcial branch for pre-10.2 release
• sync_community_updates: includes the updates for the official branch into the community branch and resign them with the community devel key
• sync_noarch: synchronize the noarch packages between two architecture
• sync-official: in beta period, create a cooker snapshot to a separated tree, for internal testing purposed (this tree is also the one put into the community tree of the beta version on the mirrors.
• sync-official-rsync: create a hardlinked copy of cooker to an official tree
• tagcvs: tag the initscripts cvs modules
• update-rsync: send the updates packages to the rsync machine
• updates-pro: synchronize professional updates frpm the machine secupdtpro

Crontabs

Distribution server crontab

# upload
20 1,3,5,7,9,11,13,15,17,19,21,23 * * * $HOME/robot/admin && $HOME/robot/cooker-push-kenobi; $HOME/robot/community-push-kenobi; \
$HOME/robot/other-push-kenobi;  $HOME/robot/other-get-kenobi
20 */2 * * * $HOME/robot/admin && $HOME/robot/cooker-push-kenobi;
49 0 * * * $HOME/robot/admin --blind && $HOME/robot/cooker-push-kenobi;
30 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi 
40 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi && $HOME/robo/get-kenobi && /bin/sh $HOME/robot/rsync_old_ken
50 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi 
0 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi && $HOME/robot/get-kenobi && /bin/sh $HOME/robot/rsync_old_ken
10 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi; 
#26 */3 * * * /bin/sh $HOME/robot/rsync_old_ken
56 7 */3 * * /bin/sh $HOME/robot/rsync_mkcd
27 23 * * * /bin/sh $HOME/robot/rsync_old_ken_other
55 * * * * /bin/sh $HOME/robot/process_cvs_queue
# download
1 */4 * * * $HOME/robot/admin_old_structure
3,33 * * * * $HOME/robot/cooker-get-kenobi &> /dev/null
15 0 * * * $HOME/robot/fedora-rsync; $HOME/robot/get-plf; $HOME/robot/get-jpackage
17 * * * * $HOME/robot/rsync-cooker-cvs
5 */6 * * * $HOME/robot/updates-pro
# local
8 0 * * * /usr/bin/find $HOME/uploads/*/rejected $HOME/old/ -ctime +30 -type f| xargs rm -vf
9 1 * * * $HOME/robot/mir -c
24 */7 * * * $HOME/robot/sync_community_old_updates --genhdlist 10.0
25 */7 * * * $HOME/robot/sync_community_updates --genhdlist 10.1
25 */5 * * * $HOME/robot/sync_community_updates --genhdlist 10.2
25 */3 * * * $HOME/robot/sync_community_updates --genhdlist 2006.0
50 *  * * * $HOME/robot/check_updates.sh
18 1 * * * $HOME/robot/save.sh
20 * * * * $HOME/robot/process_queue
21 4 * * * $HOME/robot/rpmctl --update
22 * * * * $HOME/robot/rpmctl -p
16 1 * * * $HOME/robot/sync_noarch
33 3 * * 0 /bin/sh $HOME/robot/community_pending | sendmail -oi -t