Development/Packaging/BuildSystem

From Mandriva Community Wiki

Jump to: navigation, search
Mandriva Build System

General view of the existing machine, environment, scripts and procedure to build the Mandriva distribution

Contents


See also:

Note: This page is aimed at describing the current structure. In red, what is currently being reworked, or should be

[edit] Fetching information

[edit] Checking if new versions are available

Different scripts are run from ken to warn the developer of the existence of new version:

  • rpmwatcho: compare cooker version with software listed at freshmeat (rpmwatcho is currently broken)
  • mdkvsrh: compare cooker version with Fedora version
  • mdkvsdeb: compare cooker version with Debian version

[edit] Getting the sources

Usually via wget, lftp, curl, any ftp/http client. Note: Maybe a local repository can gather automaticaly downloaded sources

[edit] Current packages version

The sources of a packages are available directly from the source media, the SPECS CVS hold all the local modifications (specfile, patches...).

The current SPECS CVS is being migrated to a SVN repository which will hold everything (both the official sources and the local modifications)

[edit] Development

See:

[edit] Build machines

The build machines, namely 'the compilation cluster', is a set of x86 32 bit and x86 64 bit machines. Any official Mandriva maintainer has an access to all these machines. The home directories are spread over the machines.

The machines run a stable version of Mandrivalinux, and include a chroot of the development version, cooker, where the developers are working. The machines also include old stable development chroot for support and maintenance of old distributions.

The machines are accessible via a gateway, kenobi, which is holding also a local mirror of the distributions.

[edit] Development tools

[edit] Building

In the current architecture, the final build should be performed by a rebuild bot after that the source package is validated. In some critical case, the build can happen on the cluster machines, for various reasons (missing buildrequires, short-circuit, lack of time). Only a few admin of the Mandriva Community can force packages upload without automatic rebuild.

[edit] Manual building

[edit] Automatic building

Two bots are used to rebuild the distributions, Iurt to rebuild the packages and sync two architectures, and rpm-rebuilder to recompile the whole distribution.

[edit] Uploading

[edit] Developer submit

Developers can use a collection of scripts to submit their package. The core script is mdvsys, associated with youri-submit.

Upload is a more sophisticated script which is compiling the packages and also performing several checks before submitting the package.

[edit] Upload process

Once the packager has submitted a package, it is copied on a temporary queue on kenobi, then Ulri the compilation scheduler will send packages on the build host so Iurt can rebuild them. Once compiled, the packages are moved in another queue, waiting for the distribution server to fetch them and to remove them from kenobi. Once on the distribution server, the package is checked, and then eventually put into cooker or rejected.

When a source package is uploaded into the distribution, a mail is sent to the changelog mailing list, when a package is rejected, a mail is sent to the maintainers mailing list and the owner of the package. The rejected packages are kept for some time.

The frequency of the upload process is directly linked to the time needed to upload a package. Initially an upload consisted of :

  • checking the package
  • committing the package into the CVS
  • putting the package in the repository
  • rebuilding the hdlist
  • mirroring to the compilation cluster so that the developers can use it
  • mirroring to the mirrors reference machine, where external public mirrors will fetch it

Now committing to the CVS and mirroring to the external mirror reference machine have been made asynchronous, so developers don't have to suffer from the network trouble between Mandriva, CVS server and the mirror reference machine.

[edit] Mirroring

[edit] Local mirroring

The distribution server is mirroring twice per hour cooker to a local machine, after a successful upload, so that the hdlists are correct.

[edit] External mirroring

This local machine then mirrors cooker to the external mirror machine, so that public mirrors can get the packages from it.

[edit] Monitoring

[edit] Contributors

http://qa.mandriva.com/contributors.cgi

pkg script queries the packages database and output some statistics regarding maintainers activity. It gives the number of packages maintained by a developer, the number of builts of these packages, and the number he personnaly built. Ratio is the quotient between the packages he built compared to the total uplods of his packages, the more the better. This script also outputs the total number of packages built by a maintainer (his packages or other ones).

Output of pkg -p 2005-01-01:

|          Maintainer            | Packages | Built since 2005-01-01 | He built | Ratio | Total |
|--------------------------------|----------|----------------------|----------|-------|-------|

[...]

| [email protected]          |       41 |                  171 |66        | 0.386 |    80 |
| [email protected]       |       50 |                  176 |68        | 0.386 |   217 |
| [email protected]      |        4 |                   18 |7         | 0.389 |    13 |
| [email protected]          |      119 |                  290 |117       | 0.403 |   226 |
| [email protected]            |      248 |                  834 |354       | 0.424 |   464 |
| [email protected]                 |       33 |                   78 |34        | 0.436 |    98 |
| [email protected]         |       68 |                  202 |89        | 0.441 |   229 |
| [email protected]           |       15 |                   46 |21        | 0.457 |   105 |
| [email protected]     |      206 |                  695 |328       | 0.472 |   505 |
| [email protected]          |        5 |                   14 |7         | 0.500 |   202 |
| [email protected]           |        3 |                    4 |2         | 0.500 |     6 |
| [email protected]          |      129 |                  507 |274       | 0.540 |  1522 |
| [email protected]           |       18 |                   61 |34        | 0.557 |    34 |
| [email protected]              |      132 |                  344 |213       | 0.619 |   333 |
| [email protected]           |      127 |                  375 |233       | 0.621 |   273 |
| [email protected]            |      356 |                 1421 |883       | 0.621 |  1596 |
| [email protected]         |     1256 |                 5402 |3733      | 0.691 |  4431 |
| [email protected]    |       97 |                  333 |248       | 0.745 |   321 |
| [email protected]             |        6 |                    7 |6         | 0.857 |    29 |
|--------------------------------|----------|----------------------|----------|-------|-------|

pkg is also able to report various problems in the packages dependencies:

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Your packages changelogs

Last week these packages which you are maintainer of were modified 
    by other one that you:

ppracer-0.5-0.alpha1mdk modified by :
- New release 0.5alpha

kde-style-tiblit-1.1-4mdk modified by [email protected]:
- Remove redundant Buildrequires
- use mkrel

kftpgrabber-0.7.0-1.4mdk modified by :
- Rebuild

konverter-0.92-4mdk modified by [email protected]:
- Remove redundant Buildrequires

kmenc15-0.04-4mdk modified by [email protected]:
- Clean Patch 0

kdenetwork-3.5.0-5mdk modified by [email protected]:
+ 2005-12-22 11:50:03 (1332)
- Fix kppp suid (pb with script with extract debug)

knmap-2.0-1.beta1mdk modified by :
- Fix BuildRequires

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Unresolved dependencies

These packages have unresolved dependencies in the distro:

package                                 depends on

ppracer-0.5-0.alpha1mdk.                squirrel (in contrib)
ppracer-0.5-0.alpha1mdk.                libsquirrel-devel (in contrib)

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Your packages changelogs

Last week these packages which you are maintainer of were modified 
    by other one that you:

kradio-0.3.0-snapshot.20040228.2mdk modified by [email protected]:
- fix x86_64 build
- %mkrel

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Packages needing to be rebuilt

These packages have not been rebuild since 200412:

package                                 last built

dlume-0.2.4-2mdk                        2004-09-21 07:09:25
freelords-0.3.4-1mdk                    2004-11-03 12:11:12
multi-gnome-terminal-1.6.2-3mdk         2004-07-19 01:07:38
powershell-0.9-0.pre3.3mdk              2004-02-06 04:02:00
gpgme03-0.3.16-6mdk                     2004-10-21 10:10:50
urlgfe-0.7.1-3mdk                       2004-08-23 09:08:51

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Your packages changelogs

Last week these packages which you are maintainer of were modified 
    by other one that you:

mozilla-thunderbird-1.0.7-4mdk modified by [email protected]:
- fix build on 10.2 (club request)

xtux-20030306-6mdk modified by :
- fix X11LIB for lib64
- fix menu section

kleansweep-0.2.3-3mdk modified by [email protected]:
- Remove redundant buildRequires

To: [email protected]
From: Mandriva Bugzilla <[email protected]>
Reply-to: Mandriva Bugzilla <[email protected]>
Subject: [QA] Packages needing to be rebuilt

These packages have not been rebuild since 200412:

package                                 last built

binutils-isem-2.15.90.0.3-4mdk          2004-10-24 09:10:07
cross-sun4-binutils-2.15.90.0.3-1mdk    2004-10-24 12:10:17
tkisem-4.5.12-6mdk                      2004-10-25 12:10:21

[edit] Packages

[edit] Rpmlint errors on packages

http://qa.mandriva.com/mainstat.cgi

[edit] Installed packages

http://qa.mandriva.com/rpmstats.cgi

[edit] Appendix

[edit] All the scripts

Most of these scripts are available in the CVS modules soft/bugs/bin or soft/robot/

[edit] compilation cluster scripts (kenobi)

  • uplftp: script to upload packages (used as ftpcooker, ftpcontrib, ftpcookerforce, ftp2006.0...)
  • Upload: script to compile and upload packages
  • iurt: recompilation bot
  • iurt2: recompilation bot using only chroots
  • topbastards: script which lists disk usage per user on the compilation cluster

[edit] Distribution server scripts (ken)

  • admin: scripts which go through the upload queue and uploads (or rejects) the given packages into the different media
  • admin_old_structure: upload scripts for packages prior to repository strucure change (community 10.0)
  • break_hardlink: remove existing hard links from a directory (create a real copy)
  • build-hdlist: build hdlist files for non standard dirs (not handled by gendistrib)
  • cdcom-push-kenobi: deprecated, copy 8.2 cdcom to kenobi
  • ch: give the last changelog entry of an rpm package
  • changelog: create a changelog message for the changelog mailing list
  • changelog_old_structure: create a changelog message for old structure based version
  • check_bad_cvs_spec.sh: reintroduce package spec file into the CVS for all CVS based package
  • check_changelog.sh: check if a new package contains the old package changelog entry
  • check_corpo.sh: check that various packages are or are not on the corporate CDs
  • check-distrib: check coherency between SRPM and binaries.
  • check-multiarch.sh: check for multiarch files into the packages of a directory
  • check_old.pl: Check if a rpm file is newer than one in a directory
  • check_owner_of_rpm.pl: Check if a rpm package contains local user file owner
  • check_updates.sh: check coherency of the updates file into the updates tree
  • clic-get-192.168.100.118: rsync script
  • community_pending: Send a report mail listing all the packages in the community versions upload queue.
  • community-push-kenobi: mirror community tree to kenobi
  • community-push-rsync: mirror community tree to rsync
  • contrib2rss.sh: create a rss feed for contrib packages
  • cooclean: remove the doble of cooker
  • cooker2rss.sh: create a rss feed for cooker main packages
  • cooker-get-amd64: rsync script
  • cooker-get-kenobi: get data from kenobi (updates and internal CVS)
  • cooker-push-kenobi: push cooker to kenobi
  • cooker-push-rsync: push cooker to rsync
  • corpo_copy: rsync corporate to another machine
  • corpo_push_n1: rsync signed packages to n1 build machine (corpo/10.0 chroot)
  • corpo_resign: use resing_dir to resign the corporate directories
  • cvcp: commit the SRPM package into the SRPM CVS
  • diffrpm.sh: create a rpm diff for a list of packages
  • diffsrcrpm: create a diff between source package
  • diffsrpm.sh: One more diff script
  • doble: remove doble in a list of rpm
  • doble2: remove doble in a list of directory
  • doble3: slight variation regarding kernel handling and delta rpm
  • doble_no_stat: use only urpm_range_overlap to find old packages
  • extracts: extracts menu, initscripts and co from the packages to commit them to the CVS
  • fedora-rsync: rsync fedora packages locally
  • gen-description.py: generate a description from the changelog for community packages (to be able to use MandrivaUpdate with community sources)
  • gen-descriptions: run gen-descryption.py for 10.0
  • gen-descriptions2: run gen-descryption.py for old install structures (Mandrake/base)
  • gen-descriptions3: run gen-descryption.py for new install structure (media/)
  • get-jpackage: mirror jpackages repository
  • get-plf: get plf mirror
  • incoming: get submitted packages from contributors who do not have an access to the compilation cluster
  • make_delta_dir: create all the delta rpm from two directories
  • make_delta_rpm: create a delta rpm
  • mir: mirror packages to rsync.mandriva.com, main mirror reference machine
  • mkcd_ppc.pl: modified mkcd version to create ppc boot ISO files.
  • ndoble: remove doble only based on the date.
  • old: move package to the old dir and touch them
  • other-get-kenobi: get club packages, testing packages, sparc, sparc64, alpha packages and ppc install from kenobi
  • other-push-kenobi: push extra packages to kenobi (old packages, rejected packages...)
  • other-push-rsync: push extra packages to rsync (testing section)
  • process-obsoletes: check the obsoletes tag of an uploaded package and remove them from the repository19181
  • process_queue: copy kenobi rpmctl command queue from kenobi
  • rebuild_srpms: rebuild all the srpms of a given directory
  • rejected: send a error mail about a rejected package to the appropriate person/mailing list with the rejected reason
  • resign_contrib_dir: resign a dir with the contrib signature
  • resign_cooker_dir: resign a dir with the cooker signature
  • resing_dir: check the signature of all the packages of a directory and resign them with the official key
  • rpmctl: script to delete or move packages between packages media
  • rpmheader: python script to display a given header from a RPM
  • rpmmon: script to find information about who maintain a given packages (local modified version to give full mail address).
  • rsync-cooker-cvs: rsync the CVS to ken.
  • rsync_corpo: rsync the corporate server and desktop 3.0 from the 10.0 chroot of n1.
  • rsync_mkcd: rsync cooker and community to guerd.
  • rsync_old_ken: rsync cooker, updates, rejected and old to old_ken and raoh.
  • rsync_old_ken_other: rsync less changing parts (community, corporate, old updates) to backup machines (old_ken and raoh)
  • sync_beta: create a snapshot of cooker for a beta
  • sync_beta_ppc: create a snapshot of the cooker ppc for a beta
  • sync_community_old_updates: includes the updates for the offcial branch for pre-10.2 release
  • sync_community_updates: includes the updates for the official branch into the community branch and resign them with the community devel key
  • sync_noarch: synchronize the noarch packages between two architecture
  • sync-official: in beta period, create a cooker snapshot to a separated tree, for internal testing purposed (this tree is also the one put into the community tree of the beta version on the mirrors.
  • sync-official-rsync: create a hardlinked copy of cooker to an official tree
  • tagcvs: tag the initscripts cvs modules
  • update-rsync: send the updates packages to the rsync machine
  • updates-pro: synchronize professional updates frpm the machine secupdtpro

[edit] Crontabs

[edit] Distribution server crontab

# upload
20 1,3,5,7,9,11,13,15,17,19,21,23 * * * $HOME/robot/admin && $HOME/robot/cooker-push-kenobi; $HOME/robot/community-push-kenobi; \
$HOME/robot/other-push-kenobi;  $HOME/robot/other-get-kenobi
20 */2 * * * $HOME/robot/admin && $HOME/robot/cooker-push-kenobi;
49 0 * * * $HOME/robot/admin --blind && $HOME/robot/cooker-push-kenobi;
30 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi 
40 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi && $HOME/robo/get-kenobi && /bin/sh $HOME/robot/rsync_old_ken
50 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi 
0 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi && $HOME/robot/get-kenobi && /bin/sh $HOME/robot/rsync_old_ken
10 * * * * $HOME/robot/admin --fast && $HOME/robot/cooker-push-kenobi; 
#26 */3 * * * /bin/sh $HOME/robot/rsync_old_ken
56 7 */3 * * /bin/sh $HOME/robot/rsync_mkcd
27 23 * * * /bin/sh $HOME/robot/rsync_old_ken_other
55 * * * * /bin/sh $HOME/robot/process_cvs_queue
# download
1 */4 * * * $HOME/robot/admin_old_structure
3,33 * * * * $HOME/robot/cooker-get-kenobi &> /dev/null
15 0 * * * $HOME/robot/fedora-rsync; $HOME/robot/get-plf; $HOME/robot/get-jpackage
17 * * * * $HOME/robot/rsync-cooker-cvs
5 */6 * * * $HOME/robot/updates-pro
# local
8 0 * * * /usr/bin/find $HOME/uploads/*/rejected $HOME/old/ -ctime +30 -type f| xargs rm -vf
9 1 * * * $HOME/robot/mir -c
24 */7 * * * $HOME/robot/sync_community_old_updates --genhdlist 10.0
25 */7 * * * $HOME/robot/sync_community_updates --genhdlist 10.1
25 */5 * * * $HOME/robot/sync_community_updates --genhdlist 10.2
25 */3 * * * $HOME/robot/sync_community_updates --genhdlist 2006.0
50 *  * * * $HOME/robot/check_updates.sh
18 1 * * * $HOME/robot/save.sh
20 * * * * $HOME/robot/process_queue
21 4 * * * $HOME/robot/rpmctl --update
22 * * * * $HOME/robot/rpmctl -p
16 1 * * * $HOME/robot/sync_noarch
33 3 * * 0 /bin/sh $HOME/robot/community_pending | sendmail -oi -t

[edit] Other Documentation

Retrieved from "http://wiki.mandriva.com/en/Development/Packaging/BuildSystem"
Personal tools