Development/Packaging/RepositorySystem
From Mandriva
Subversion based package repository management
The new repository system is based on the original Conectiva's package building and testing system. Current status of the Mandriva repository is TRANSITION. Many packages are already using the repository and the build tools are being adapted.
Contents
|
Repository software
The repository system is based on Subversion. After many different ideas, Subversion has solved many problems we'd like to fix in our previous implementation. This document will explain how the repository tree is organized.
Quickstart
Please see Development/Packaging/RepositorySystem/Quickstart for a hands-on example of how to work with packages stored in SVN. The example shows how to checkout a package, change it, revise the changes and commit them back to the repository.
Subversion Concepts
Cheap copies
Subversion uses a method called bubble-up for revisioning file trees. What it means exactly won't be explained here, since it's enough for our purposes to explain that this turns copying of entire file trees into a very cheap operation. So when you see mentioned somewhere that a tree is going to be cloned, don't worry.
Repository structure
This section explains how the repository is structured. It will introduce some of the concepts you'll be involved with when dealing with it.
Repository tree
Here's an example of how a repository tree looks. Reticence was used when the additional information wouldn't help in the explanations.
- cooker/ (1)
-
pkgname/ (2)
- current/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.3.tar.bz2 (3)
- pkgname-foo.patch
- SPECS/
- pristine/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.1.tar.bz2 (4)
- SPECS/
- releases/
- 1:2.0/
- 1cl/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.0.tar.bz2
- pkgname-foo.patch
- SPECS/
- 2cl/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.0.tar.bz2
- pkgname-foo.patch
- SPECS/
- 1cl/
- 1:2.1/
- 1cl/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.1.tar.bz2
- pkgname-foo.patch
- SPECS/
- 1cl/
- 1:2.0/
- branches/
- 1:3.0beta/
- pkgname
- current
- pristine
- releases
- pkgname
- 1:3.0beta/
- current/
-
pkgname/ (2)
- releases/ (5)
- 2005/
- pkgname/
- current/
- ...
- releases/
- ...
- pristine/
- ...
- branches/
- ...
- current/
- pkgname/
- 2005/
- updates/ (6)
- 2005/ (7)
- pkgname/ (8)
- current/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.0.tar.bz2
- pkgname-foo.patch
- SPECS/
- pristine/
- SPECS/
- ...
- SOURCES/
- ...
- SPECS/
- releases/
- 1:2.0/
- 1cl/
- SPECS/
- ...
- SOURCES/
- ...
- SPECS/
- 1cl/
- 1:2.0/
- branches/
- ...
- current/
- pkgname/ (8)
- 2005/ (7)
- branches/ (9)
-
pkgname-threaded/
- pkgname/
- current/
- SPECS/
- pkgname.spec
- SOURCES/
- pkgname-2.0.tar.bz2
- pkgname-foo.patch
- SPECS/
- releases/
- ...
- ...
- current/
- pkgname/
-
pkgname-threaded/
- tags/ (10)
- bootstrap/
- pkgname/
- current/
- ...
- releases/
- ...
- current/
- pkgname/
- pristine/
- ...
- bootstrap/
- misc/ (11)
-
pkgname/
- log (12)
- ...
-
pkgname/
Mutable places
Most of the directories in the Repository System are not meant to be changed. Those places which are meant to be changed are marked with bold letters in the repository tree above. In the rest of this document you'll find how and when they're meant to be changed.
Cooker directory
It's in the cooker/ (1) directory that most of the package development occurs. That's where the packages begin their life in the repository, and where most of the changes will be introduced during their life. Inside this directory, you'll find one directory (2) for each package which is currently part of Mandriva Linux cooker. These directories must always match the current package name. It means that as soon as the package name change is confirmed and just before the spec file is edited, you must rename this directory (that's the only reason why it is mutable).
Package directory structure
Inside each package directory (2) you'll find three directories:
- current/: This directory always holds the most recent changes to the packages. Here you'll find a partial RPM %_topdir structure, with SPECS/ and SOURCES/ subdirectories. The information they hold match with what was expected in a common SRPM package. That's where changes in this package will be introduced.
- releases/: In this directory you'll find cheap copies of the current/ directory. These copies where made in the exact moment that specific package versions were released to the cooker users. Notice that changes are not allowed here. These directories and files are just informative, and are created by an automated system.
- pristine/: This directory holds the current/ directory state when the last package was released to the cooker users. This is useful to know what has been changed in current/ since the last package release. Just like the release/ directory, changes here are not allowed, since this is informative and created by an automated system.
- branches/: Sometimes we want to have two different major versions of a package in the distro. Usually, the higher numbered one is a beta version or an experimental one. Or sometimes we just want to offer two stable major versions (for example, postfix 2.2 and postfix 2.3). In these cases, we can use the per-package branches/ directory. Each package inside it will again have the current, releases and pristine tree (but not branches, of course).
In the example above you may observe, looking at (3) and (4), that the current/ directory holds changes that were not yet released to the public (notice that changes are not always so obvious, but Subversion provides resources to easily detect what has changed).
Releases directory
The releases/ directory (5) holds complete cooker/ (1) images taken whenever a Mandriva Linux was released. It's meant to be a reference, and no changes inside it are allowed.
Updates directory
At first, the updates/ (6) directory looks exactly like the releases/ (5) directory. Indeed, both hold a similar structure, and their first level subdirectories (7) are always created at the same time. The fundamental difference between them is that updates/ is mutable, since it's meant to hold changes to a Mandriva Linux release after it has been released. Sometimes, needed changes are valid for more than one released Mandriva Linux and/or the cooker packages. When this is the case, you must change the package in all places, that is, all updates/ (6) subdirectories correspondent to the affected distributions and/or the cooker/ (1) directory. You may want to use Subversion's merge features to help you during this task. Notice that package directories (8) are not mutable here, because unlike cooker/, they're not meant to be renamed.
Branches directory
Sometimes, you may want to change one or more packages in the cooker without affecting what other packagers do. That's where the branches/ (9) directory is used. Its first level subdirectories are copies of the whole cooker/ directory, even if only a single package is being changed. There are some strong reasons to do that:
- copying the whole cooker/ directory is as cheap as copying a subdirectory of it;
- if you later discover that changes in other packages are necessary, they are already there;
- for free, you get the information of how was the cooker at the moment you forked the development;
- it's more consistent.
Notice that unless you want to make changes to other packages in the style explained above, branches/ is not the place to introduce new packages. They usually should go directly to the cooker/ directory.
The period of life of a directory inside branches/ is uncertain, since it's meant to be removed once the packagers using it are sure that it's not needed anymore.
NOTE: DO NOT simply copy the development branch over the cooker when you're done! Always merge your work back in the cooker, comparing what has changed since you forked to what is currently there. Losing changes introduced by others in the period is unacceptable.
Tags directory
This directory is similar to the releases/ directory (5), but it holds more general tag copies of the cooker/ directory. It's used to mark the cooker state in a given moment which happened somewhere between two Mandriva Linux releases.
Misc directory
The misc/ (11) directory holds miscellaneous information about each package. Right now, the only information being stored in this directory is the old changelog of each named package, inside a file named "log" (12). This file, if present, must contain an rpm changelog without the _%changelog_ tag.
Tips
- Understand these concepts! They are essential if you want to work with the repository system.
- It's not necessary to send a package to the cooker every time you change something in the repository. You can "stage" your change and rest assured that the next user checking out the package will get it.
- Be careful when merging back changes in the branches directory. You may lose your head if you overwrite changes made by others.
Changelog messages
Writing your log message
Before introducing the basic repository operation, it's necessary to establish some conventions for the changelog messages. Remember that these messages are used for RPM %changelog section generation, so the format does matter. First, be sure to describe whatever change you do to a package, and to use capitals and punctuation correctly. Besides being used to generate the %changelog section, the log is one of the best documentation we have for the packages. If you're going to write a single entry, just go ahead and type it as you would normally do:
This is a changelog entry, which could have one or more lines.
If your log has more than one entry, use a dash to separate them, in the RPM %changelog style:
- This is a changelog entry which is very long. - And I also have a second entry.
"SILENT" changelog messages
You may hide Subversion log messages fron the package %changelog by using the SILENT (all caps) keyword. The rules are:
- "SILENT" as the first word of the first line of a Subversion log will suppress all log messages from the generated package %changelog
- otherwise, "SILENT" anywhere on a line will hide only that line from the generated package %changelog
Suppressing whole message:
SILENT This whole message will be suppressed. You can even do ASCII art here.
Suppressing single lines (only the first line in this example will appear in the %changelog):
- upgraded to version x.y.z - SILENT: cleanup whitespace in the spec file - renamed patch to fix typo (SILENT: not relevant for %changelog) - nothing to see here, move along (SILENT)
Fixing wrong log entries
If you have done something wrong in a log entry, you'll be glad to know that you can use the following command:
svn pedit svn:log --revprop -r <revision> svn+ssh://repos
Discarding old %changelog sessions
Whenever you edit a spec file in the repository and find out that it still has a _%changelog_ session, please strip it out (including the %changelog tag itself). Old changelog entries, available before the repository system was implemented, will be safe in old revisions, and can be easily accessed in tags/ and releases/ subdirectories.
Saving old %changelog sessions
If you want to keep the old %changelog session of some package you're including in the repository, you may do so by saving it in a file named /misc/pkgname/log. This file will be appended to the changelog extracted from the repository commit logs. See below for more information on how to do this.
Advice
- Write down your changes! Nobody will ever complain if you explain what you do, and people (including yourself) will find out problems quickly.
Repository operation
Here is described the procedure to do some of the usual operations on the repository system.
Repository location
After you've understood the concepts above, you're ready to check the real repository. It is located in the following URL:
For a good visualization and compares on web use: https://svn.mandriva.com/cgi-bin/viewcvs.cgi/?root=svn
Subversion provides basic access through your browser, so that you can navigate in this URL. The choice of access on Mandriva subversion is svn+ssh.
NOTE: In the examples below, for the benefit of brevity, the URL svn+ssh://<repos> will be used instead of the real one presented above.
Changing a package
When you want to change a package, follow these steps:
Checkout your working copy to a directory /path/pkgname, for example:
svn checkout svn+ssh://<repos>/cooker/pkgname/current /path/pkgname
This will create the basic structure, with SPECS/ and SOURCES/ directories. Change the package, editing the spec file and/or removing, adding, or moving files in SOURCES/. For example:
# Add new version cp pkgname-1.2.tar.bz2 /path/pkgname/SOURCES svn add /path/pkgname/SOURCES/pkgname-1.2.tar.bz2 # Remove old one svn rm /path/pkgname/SOURCES/pkgname-1.1.tar.bz2 # Move patch svn mv /path/pkgname/SOURCES/pkgname-1.1-foo.patch /path/pkgname/SOURCES/pkgname-1.2-foo.patch
NOTE: Remember to use svn add/rm/cp/mv commands when you're adding, removing, copying, or moving a file inside the repository (not when editing). Now, test the package, using one of the procedures explained below. After you're done, commit your changes with:
svn commit /path/pkgname
NOTE: Please, read the "Changelog messages" section before entering the log message.
NOTE: If you're going to change only the spec file, you'll probably want to checkout only the SPECS/ directory, instead of the whole current/ directory.
Testing a package
Whenever you do changes to a package, you will want to test your changes building the package. It's pretty easy to do that. Here are some different techniques to achieve the same results.
Using BuildManager
The easiest way to test the package is using bm. Besides other features, this tool simplifies working with RPMs which are outside of a predefined %_topdir. Using it is completely straightforward. If you're somewhere inside of the package's %_topdir:
bm
if you're outside of _%_topdir_:
bm /path/pkgname
You may also specify manually which spec file to build:
bm /path/pkgname/SPECS/pkgname.spec
Using just rpm
You may want to use just plain rpm. If so, follow these simple steps. Create missing directories:
mkdir -p /path/pkgname/{RPMS,SRPMS,BUILD}
And build the package, setting the RPM _%_topdir_ variable:
rpm -ba --define "_topdir /path/pkgname" /path/pkgname/SPECS/pkgname.spec
Renaming a package
If you're sure you want to rename a package, follow these steps:
Move the cooker package directory, and the misc/ subdirectory, if existent. Notice that these are operations done completely in the repository, so you have no chance for mistakes:
svn mv svn+ssh://<repos>/cooker/oldname svn+ssh://<repos>/cooker/newname svn mv svn+ssh://<repos>/misc/oldname svn+ssh://<repos>/misc/newname
Checkout your working copy of the directory:
svn checkout svn+ssh://<repos>/cooker/newname/current /path/newname
Edit the spec to rename the package, and move it as well:
svn mv /path/newname/SPECS/oldname.spec /path/newname/SPECS/newname.spec
After you're done, commit your changes:
svn commit /path/newname
NOTE: Please, read the "Changelog messages" section before entering the log message. Notice that packages in updates/ should probably not be renamed.
Creating a package
If you're sure you want to create a package, use one of the methods below.
Easy way
The repsys tool will do all of the work for you. You simply have to execute (assuming that default_parent is set correctly).
NOTE: repsys is currently in beta test on the Mandriva repository. Automatic submission ( no local build necessary ) will be implemented.
Creating a cooker package with repsys is easy:
repsys create svn+ssh://svn.mandriva.com/svn/packages/cooker/newpkgname
svn checkout svn+ssh://<repos>/cooker/newpkgname/current newpkgname Add your package files
You can restore SRPM with full changelog coming from svn, using:
repsys getsrpm -l pkgname
This will generate an SRPM ready to build on the cluster
Hard way
Create the cooker package directory. Notice that this is an operation done completely in the repository:
svn mkdir svn+ssh://<repos>/cooker/newpkgname
Checkout your working copy of the whole package dir:
svn checkout svn+ssh://<repos>/cooker/newpkgname /path/newpkgname
Create the current directory structure (locally!):
svn mkdir /path/newpkgname/current svn mkdir /path/newpkgname/current/SPECS svn mkdir /path/newpkgname/current/SOURCES
Add the package files to the structure:
cp SPECS/newpkgname.spec /path/newpkgname/current/SPECS cp SOURCES/newpkgname-1.0.tar.bz2 /path/newpkgname/current/SOURCES svn add /path/newpkgname/current/SPECS/newpkgname.spec svn add /path/newpkgname/current/SOURCES/newpkgname-1.0.tar.bz2
After you're done, commit your changes:
svn commit /path/newpkgname
NOTE: Please, read the "Changelog messages" section before entering the log message.
Creating a package using another as base
Many times a package is created using another package as the base. For example, Mandriva Linux is currently using a scheme where SRPM packages which hold libraries have their name changed to include the soname of the library, allowing more than one version to exist at a given time. This can be gracefuly handled by the repository system. Suppose we have a package named libgal5, and we're going to introduce libgal7. Since we don't want to lose libgal5's history, we just take a copy of it, and call it libgal7:
svn cp svn+ssh://<repos>/cooker/gal5 svn+ssh://<repos>/cooker/gal7
NOTE: Please, read the "Changelog messages" section before entering the log message. Now you can safely checkout libgal7 and work on it.
Detecting changes in a package
To detect what has changed since the last package release:
svn diff svn+ssh://<repos>/cooker/pkgname/pristine svn+ssh://<repos>/cooker/pkgname/current
To detect what has changed since the given Mandriva Linux release:
svn diff svn+ssh://<repos>/releases/2005/pkgname/pristine svn+ssh://<repos>/cooker/pkgname/current
You can also use the RepSys tool to check if a package has changed, or which packages have changed. Check below, in the RepSys ( not available yet ) "changed" subcommand, for more information.
Saving the original logs of a package
If you want to maintain the original log of a package, you may do so by creating a file named /misc/pkgname/log, with the original rpm %changelog session content, without the %changelog tag itself. The content of this file will be appended to the changelog extracted from the repository commit logs, whenever this package is submitted to the test system. Here is an example:
# Create and checkout misc subdirectory svn mkdir svn+ssh://<repos>/misc/pkgname svn checkout svn+ssh://<repos>/misc/pkgname misc_pkgname # Paste the content from the original log, WITHOUT the %changelog tag vi misc_pkgname/log # Add to the repository, and commit it svn add misc_pkgname/log svn commit misc_pkgname
Using Branches
Here is explained a complete branch cycle.
Preparing
In our example, we'll change the package openssh to include kerberos support. Since we're not sure about whether it is going to become the official package one day, and we also don't want to mess with the usual openssh package which is in the cooker (others may change it freely), we'll use a branch for it.
First thing to do is to create the branch. That operation is done completely in the repository, so you have no chance for mistakes:
svn cp svn+ssh://<repos>/cooker svn+ssh://<repos>/branches/openssh-krb
NOTE: Please, read the "Changelog messages" section before entering the log message.
Using it
Now that we have the branch, let's checkout our working copy of the openssh package from the branch:
svn checkout svn+ssh://<repos>/branches/openssh-krb/openssh/current /path/openssh-krb
Now you can change it at will. In our example, we're going to introduce a new patch, called openssh-krb.patch:
# Add the file cp openssh-krb.patch /path/openssh-krb/SOURCES svn add /path/openssh-krb/SOURCES/openssh-krb.patch # Introduce needed changes for the patch vi /path/openssh-krb/SPECS/openssh.spec # Commit your changes svn commit /path/openssh-krb
NOTE: Please, read the "Changelog messages" section before entering the log message.
You probably want to release test packages from your changed openssh. To do that, use one of the techniques described above, in the "Testing a package" section.
Merging it back
After some weeks of changes and testing procedures, we decided to use these changes in our main cooker package. Thus, it's time to merge our work.
First, checkout a working copy of the openssh package from cooker:
svn co svn+ssh://<repos>/cooker/openssh/current /path/openssh
Then, we need to know the revision in which we created our branch, so that we can merge changes introduced in this period. This command will help us in the task:
svn log svn+ssh://<repos>/branches/openssh-krb | head -n 2
Now that we know the revision (let's say it was 18003), let's try to merge the work done in the spec file of branch openssh-krb.
svn merge -r 18003:HEAD svn+ssh://<repos>/branches/openssh-krb/openssh/current/SPECS/openssh.spec /path/openssh/SPECS/openssh.spec
NOTE: DO NOT simply copy the file from the branch over the cooker file, since it may contain changes you don't know about.
If the command above results in any conflicts, you must resolve them before going ahead.
We must also check whatever files were introduced or removed in the branch and merge the work in the cooker package. Again, DO NOT simply copy the files from there over the cooker. Check out which changes were introduced by you in the branch, and what changes were introduced by others in the cooker. You may easily do that with the following commands:
svn log -v -r 18003:HEAD svn+ssh://<repos>/branches/openssh-krb/openssh/current svn log -v -r 18003:HEAD svn+ssh://<repos>/cooker/openssh/current
Supposing that no changes were introduced in the cooker, with the commands above you'll find out our introduced file, openssh-krb.patch. Instead of simply coping it manually to the SOURCES/ directory of the cooker openssh package and adding it to subversion, you'll copy it using subversion, so that subversion will use internally the same data to represent both. Doing that is easier than you might expect:
svn copy svn+ssh://<repos>/branches/openssh-krb/openssh/current/SOURCES/openssh-krb.patch /path/openssh/SOURCES
Notice that we've added it to our working copy, so that we will commit everything in a single operation:
svn commit /path/openssh
NOTE: Please, read the "Changelog messages" section before entering the log message.
Removing the branch
Ok. We're ready! The openssh-krb branch has been integrated in the cooker. Now we can safely remove our branch:
svn rm svn+ssh://<repos>/branches/openssh-krb
NOTE: Please, read the "Changelog messages" section before entering the log message.
Tips
- Change files in your working copy in place. Do not overwrite a file in your working copy directory with an external file.
- If you really have to overwrite a file in your working copy, do a svn diff operation to make sure that you're changing really what you were expecting to;
- Use branches when you want to fork a package development for a long time.
- Use branches when testing a new version of software which should not yet get into the cooker.
- Be careful when doing a full repository branch as in the example above. It may take a little longer (I did it right now and it finished after 31 minutes);
Contact
For information on this system, contact:
