Development/Tasks/Packaging/Tools/repsys

From Mandriva Community Wiki

Jump to: navigation, search


WARNING: This document is deprecated and relates only to buildsystem administration.

Repository System Tool

This page will explain how to setup the repsys tool. Basic usage examples are shown elsewhere.

Contents


Introduction

The repository system tool (repsys for short) was developed to help manage RPM repositories stored in SVN. It allows one to submit packages, obtain an SRPM, convert commit messages to %changelog format and much more. Below we will see in details what this tool can do in server and client mode.

The repsys package is available in the contrib repository.

Repsys as a server

Repsys is used in server mode in the XML RPC part of the buildsystem:

Image:repsys.png

In this mode, repsys generates a .src.rpm file from the SVN repository and sends it to the build daemon/machine. This means it has to:

  • checkout the SOURCES and SPECS directories of the given package at the revision the client requested;
  • create a %changelog section and append it to the spec file;
  • optionally run an external rebrand program which can, for example, choose a release tag (like %mkrel) or change the spec file in any other way;
  • run rpm -bs to construct a .src.rpm file;
  • place this .src.rpm file in a directory specified in /etc/repsys.conf for this target.

Installation steps

  1. Read this documentation fully
  2. Install the repsys rpm package and its dependencies;
  3. Install apache;
  4. If using a rebrand script, create it
  5. Configure /etc/repsys.conf
  6. Configure apache authentication
  7. Configure apache to execute the XML RPC CGI script

Detailed explanations are below.

The /etc/repsys.conf configuration file

This file is used both in server and in client mode. Here is a sample:

[global]
verbose = no
default_parent = svn+ssh://svn.mandriva.com/svn/packages/cooker/

[log]
oldurl = svn+ssh://svn.mandriva.com/svn/packages/misc/

[users]
andreas = Andreas Hasenack <andreas@mandriva.com>
oeriksson = Oden Eriksson <oeriksson@mandriva.com>

[submit]
name = cooker
target = /buildsystem/cooker
allowed = svn+ssh://svn.mandriva.com/svn/packages/cooker/
scripts = /svnroot/cooker.admin/rebrand-cooker.sh

Below is an explanation of these options:

  • default_parent: URL to prepend to the package name when no url is provided. This is mostly used in client mode. For example, with the above configuration file, both commands below would end up accessing the same URL:
   repsys getsrpm -l kolab
   repsys getsrpm -l svn+ssh://svn.mandriva.com/svn/packages/cooker/kolab
  • oldurl: this URL has the old %changelogs from the pre-SVN "era". When packages are imported into this SVN scheme, the %changelog tag has to be removed because we start using the SVN commit messages instead. In order to not loose these old %changelog entries, they are stored at this URL, one for each package. When repsys is building a .src.rpm, it appends this old changelog to the spec file;
  • [users] section: contains the svn login to email mapping used to create %changelog entries. Since SVN only stores the login name of the person who made the commit, this mapping has to be used to write out a compliant %changelog of the form Name <email>. Currently this list has to be maintained manually at the server, but I'm sure this could be easily expanded to use some sort of LDAP lookup, for example. If the SVN commit logs contains a username which is not listed here, then only the username will be used in the %changelog (i.e., no email part);
  • [submit] section: contains the definition of all targets supported by this installation. Each target has the following parameters:
    • name: the target name. Used as the -t argument to the repsys submit client command;
    • target: whitespace-separated list of directories indicating where the .src.rpm file has to be placed. For example, to submit packages simultaneously to ia32 and x86_64 buildsystems one could use /buildsystem/cooker-ia32 /buildsystem/cooker-x86_64 here;
    • allowed: URLs pointing to which SVN repositories can have packages submitted through this target;
    • scripts: if used, contains a whitespace-separated list of scripts that will be called with three parameters, in this order:
      • topdir: rpm's topdir, i.e., where SOURCES and SPECS are;
      • specfile: the spec filename of the package being processed;
      • svn-revision: the SVN revision that was extracted

Sample rebrand script

The rebrand script (parameter scripts in the [submit] section) can be used to modify the SPEC file after it was extracted from SVN and already had a %changelog section appended. This modification is only local and will only appear in packages which went through the buildsystem: nothing is commited back to SVN.

One use we found for such a script in the Conectiva case was to automatically use the SVN revision number of the last commit (obtained from the already added %changelog section, see below for an example) as an RPM Release tag:

#!/bin/bash

TOPDIR=$1
SPEC=$2
REV=$3

# Use svn revision as release
tac $SPEC | perl -pe '/^\+ [\d-]+ [\d:]+ \((\d+)\)/ && ($r = $1); s/^(Release:).*/$1 ${r}mdk/;' | tac > ${SPEC}.tmp;
mv -f $SPEC{.tmp,}

To better see what this sample does, consider the following %changelog section which was automatically created by repsys and appended to the SPEC file:

%changelog
* Fri Aug 26 2005 Andreas Hasenack <andreas@mandriva.com>
+ 2005-08-26 09:36:08 (749)
- rebuild and recover changelog which was mistakenly erased in
  the previous package

* Fri Aug 26 2005 Andreas Hasenack <andreas@mandriva.com>
+ 2005-08-25 19:13:47 (743)
- Compile with -fPIC to fix python binding, see #17246
  (fix by Couriousous <couriousous@mandriva.org>)
(...)

With this %changelog, the shown rebrand script would change the Release tag in the SPEC file to be Release: 749mdk.

This was just an example of the kind of transformations that can be done at this stage before the .src.rpm package is created and sent to the build daemon/machine.

The XML RPC CGI script

The XML RPC CGI script is the actual repsys "server". It's the part that listens to the repsys submit client command and creates the .src.rpm file that is to be sent to the build system/daemon.

This is the actual CGI script:

/svnroot/mdv.scripts/xmlrpc:

#!/usr/bin/python
from RepSys.cgi import xmlrpcserver
import tempfile
tempfile.tempdir="/var/tmp"
xmlrpcserver.main()

This script is then called as a CGI from Apache when the repsys command line tool is used to submit a package. The following Apache configuration can be used to enable script execution and to authenticate and authorize the user:

/etc/httpd/conf/webapps.d/repsys.conf:

ScriptAlias /scripts/mdv "/svnroot/mdv.scripts/"
<Directory "/svnroot/mdv.scripts">
    AllowOverride All
    Options ExecCGI
    <LimitExcept GET>
        AuthType Basic
        AuthName "Authorization Realm"
        AuthUserFile /svnroot/mdv.admin/svn.passwd
        Require valid-user
    </LimitExcept>
    <IfModule mod_access.c>
        Order allow,deny
        Allow from all
    </IfModule>
</Directory>

This means read-only repsys client commands can access the repository without authentication, but a package submission and other write operations must be authenticated.

The web submission form

Packages can also be submitted via a web interface if so desired. The repsys package contains such a script. Sample screenshot:

File:Submit-cgi-sm.png

The CGI script which allows that is:

/svnroot/mdv.scripts/submit:

#!/usr/bin/python
from RepSys.cgi import submit
import tempfile
tempfile.tempdir="/var/tmp"
submit.main()

The same Apache configuration we used for the xmlrpc server is used again, which means only authenticated users can submit packages.

Repsys as a client

Repsys has many uses at the client side, the most important one being of course the package submission. This section will show the most used client side commands.

Most commands take as an argument a repository URL. Many users find it convenient to use a shell variable which points to the repository root and then just append the package name. For example:

~/.bashrc:

export REPOS="svn+ssh://svn.mandriva.com/svn/packages/cooker"

This can be used in a checkout, for example:

svn co $REPOS/bm/current bm

Package submission: submit

Please see Repository System Quickstart -- Submitting for an example.

 %changelog output: rpmlog

The rpmlog repsys command converts the SVN commit log messages to the RPM %changelog format. This is how the .src.rpm packages from the buildsystem end up with a proper %changelog section even though the SPEC file itself in SVN doesn't have one.

If you just want to get a proper .src.rpm with a %changelog entry, however, then you may prefer to use the getsrpm command shown further below.

Example

[andreas@pandora smart]$ repsys rpmlog svn+ssh://svn.mandriva.com/svn/packages/cooker/smart > changelog.txt
[andreas@pandora smart]$ head changelog.txt
* Wed Oct 05 2005 Andreas Hasenack <andreas@mandriva.com>
+ 2005-10-05 20:57:36 (990)
- updated to version 0.40
- releasing 0.40-1mdk

* Tue Sep 20 2005 Helio Chissini de Castro <helio@mandriva.com>
+ 2005-09-20 10:31:05 (922)
- Reenabled gpg signatire check ( Thanks Andreas )

* Mon Sep 12 2005 Andreas Hasenack <andreas@mandriva.com>

SRPM package extraction: getsrpm

Syntax: repsys getsrpm [-l] <package-url>

Example: repsys getsrpm -l svn+ssh://svn.mandriva.com/svn/packages/cooker/smart

This command is used to obtain an .src.rpm file with data stored in SVN. This file is then stored in the current working directory.

A parameter which is used very often is -l. With this parameter, the %changelog section is added to the spec file prior building the .src.rpm package.

There are many other parameters used to checkout specific versions of a package, please see the output of repsys getsrpm --help.

Example

[andreas@pandora ~]$ mkdir smart
[andreas@pandora ~]$ cd smart
[andreas@pandora smart]$ repsys getsrpm -l svn+ssh://svn.mandriva.com/svn/packages/cooker/smart
[andreas@pandora smart]$ l smart-0.40-1mdk.src.rpm
-rw-r--r--  1 andreas andreas 544K Out  6 11:37 smart-0.40-1mdk.src.rpm
[andreas@pandora smart]$ rpm -qp --changelog smart-0.40-1mdk.src.rpm | head
* Qua Out 05 2005 Andreas Hasenack <andreas@mandriva.com>
+ 2005-10-05 20:57:36 (990)
- updated to version 0.40
- releasing 0.40-1mdk

* Ter Set 20 2005 Helio Chissini de Castro <helio@mandriva.com>
+ 2005-09-20 10:31:05 (922)
- Reenabled gpg signatire check ( Thanks Andreas )

* Seg Set 12 2005 Andreas Hasenack <andreas@mandriva.com>

Checking out cooker-testing packages

To checkout cooker-testing packages, don't use "repsys checkout" but instead do an "svn checkout" of the appropriate directory from the repository under packages/cooker/$PACKAGE//branches. E.g:

 svn co svn+ssh://svn.mandriva.com/svn/packages/cooker/rpm/branches/5.3/current
Personal tools