Development/Howto/GenLibs
From Mandriva Community Wiki
Howto use RPM features for easier package development
For some time now Mandriva have split libraries in small packages to reduce and make easier the package manager's life. The problem is that still there are no formal policies for standalone libraries beyond the regular %mklibname macro, meaning that a lot of mistakes can be made making difficult the lives of both packager and package manager, leading to a nasty case of huge dependency problems. The idea is to propose an easy addon on the spec side to fix 99% of the cases, reducing spec size, taking from the packager the burden of adding a lot of macros and leaving the dependency intelligence on the package manager's side, in Mandriva's case: RPM.
Contents |
[edit] The whole process idea explained
This idea is based on work done in Conectiva times, by Arnaldo Carvalho, for easy control of i18n packaging.
The whole concept was quite simple. Usually a package has a set of i18n translation files that comes with the package. Not all languages might be available, but usually they fall down in relatively large language packages attached to a main package or a subpackage called doc package. The first step was to decide to split packages by language, and in the worst cases where we have more than 3 or 4 languages we have:
- A huge spec
- A lot of repetitive work for the maintainer
- A boring process highly prone to error
Doing the process in such a way would get us stuck in an endless process and probably will be the worst job ever in packaging. So, the question was:
How can we do some nifty "trick" in rpm to make our life easier?
The point was the set of installed files in the %install spec stage for i18n having a uniform layout. We already know where to look, and how to get packages. A small script could do the job. What we would need is a way to create subpackages based on some template, with no interference on the regular package, and adding more constraints, like keeping references from the original srpm from which package is built originally, amongst others. And surprise, surprise, there's a way. And the adapted process for libraries, which have some more restrictions will be explained now.
[edit] Adding "intelligence" in the spec
Our whole idea consists of a new special rpm macro, a script ( which can be in any language, i choose shell ), and add a line as a last step on the %install stage.
During the package install process, we put it on hold right before the last package stage. The new macro will receive the spec data, will pass the script, which generate a package on the fly with all data, removing the required files in BUILDROOT and builds a subpackage with all these data.
After all subpackages are generated, the control returns back to the main build process, not with buildroot without files packaged by the macro.
All the mklibname, and subpackages in the main spec can just be removed, and we never will have to deal with things like discovering the right soname, or adding a lib suffix because we are in a different arch...
[edit] A real result example
The current libqt3 package represents a real sample with lots of details. Today libqt3 has a set of libraries inside, which leads people to install even unnecessary libraries to run a program.
Contents of libqt3 package:
/usr/lib/qt3 /usr/lib/qt3/lib /usr/lib/qt3/lib/libdesignercore.prl /usr/lib/qt3/lib/libeditor.prl /usr/lib/qt3/lib/libqassistantclient.prl /usr/lib/qt3/lib/libqt-mt.la /usr/lib/qt3/lib/libqt-mt.prl /usr/lib/qt3/lib/libqt-mt.so.3 /usr/lib/qt3/lib/libqt-mt.so.3.3 /usr/lib/qt3/lib/libqt-mt.so.3.3.5 /usr/lib/qt3/lib/libqui.prl /usr/lib/qt3/lib/libqui.so.1 /usr/lib/qt3/lib/libqui.so.1.0 /usr/lib/qt3/lib/libqui.so.1.0.0 /usr/lib/qt3/plugins/lib/inputmethods /usr/lib/qt3/plugins/lib/inputmethods/libqimsw-multi.so /usr/lib/qt3/plugins/lib/inputmethods/libqimsw-none.so /usr/lib/qt3/plugins/lib/inputmethods/libqsimple.so /usr/lib/qt3/plugins/lib/inputmethods/libqxim.so /usr/lib/qt3/plugins/lib/styles /usr/lib/qt3/plugins/lib/styles/libqcdestyle.so /usr/lib/qt3/plugins/lib/styles/libqcompactstyle.so /usr/lib/qt3/plugins/lib/styles/libqmotifplusstyle.so /usr/lib/qt3/plugins/lib/styles/libqmotifstyle.so /usr/lib/qt3/plugins/lib/styles/libqplatinumstyle.so /usr/lib/qt3/plugins/lib/styles/libqsgistyle.so
The trick part is that all .so.* packages represent a library package. In qt we have modules and .la, .so, .prl files that are strictly -devel files.
What I did is add the new macro ( will be shown later ) as last step of the %install stage:
%genlibs -O %libqt3name@<@3.3.5-12mdk -R %name-common@=@%{version}-%{release}
And, of course, I removed from the spec the original lib subpackage. The -O and -R was meant to be Obsoletes and Requires
And now we have this subpackages generated:
libdesignercore1-3.3.5-15mdk.i586.rpm libeditor1-3.3.5-15mdk.i586.rpm libqassistantclient1-3.3.5-15mdk.i586.rpm libqt-mt3-3.3.5-15mdk.i586.rpm libqui1-3.3.5-15mdk.i586.rpm
And the common and devel packages received the non-library packages
Common: /usr/lib/qt3/plugins/lib64 /usr/lib/qt3/plugins/lib64/inputmethods /usr/lib/qt3/plugins/lib64/inputmethods/libqimsw-multi.so /usr/lib/qt3/plugins/lib64/inputmethods/libqimsw-none.so /usr/lib/qt3/plugins/lib64/inputmethods/libqsimple.so /usr/lib/qt3/plugins/lib64/inputmethods/libqxim.so /usr/lib/qt3/plugins/lib64/sqldrivers /usr/lib/qt3/plugins/lib64/styles /usr/lib/qt3/plugins/lib64/styles/libqcdestyle.so /usr/lib/qt3/plugins/lib64/styles/libqcompactstyle.so /usr/lib/qt3/plugins/lib64/styles/libqmotifplusstyle.so /usr/lib/qt3/plugins/lib64/styles/libqmotifstyle.so /usr/lib/qt3/plugins/lib64/styles/libqplatinumstyle.so /usr/lib/qt3/plugins/lib64/styles/libqsgistyle.so
[edit] The macro and the RPM parser
The macro is quite common and the same for many cases, just like the i18n as explained and new ones from the libraries:
%genlibs(n:e:v:r:l:R:O:) /usr/lib/rpm/library-packages.sh \\\
--name "%{-n*}%{!-n:%{name}}" \\\
--epoch "%{-e*}%{!-e:%{?epoch:%{epoch}}}" \\\
--version "%{-v*}%{!-v:%{version}}" \\\
--release "%{release}" \\\
--license "%{license}" \\\
--buildarch "%{_arch}" \\\
--buildroot "%{buildroot}" \\\
--define "_topdir %{_topdir}" \\\
--define "_tmppath %{_tmppath}" \\\
%{-R:--requires "%{-R*}"} \\\
%{-O:--obsoletes "%{-O*}"} \\\
%{*}
It is easy to see from the script called above, and all the spec data that we received. A point to be observed is that RPM parser is one of the most difficult things to deal with, so the glob in the end can bring a lot of surprises. Taking the example above of qt3, I have for obsoletes:
- -O %libqt3name@<@3.3.5-12mdk
Note that my obsoletes should be %libqt3name < 3.3.5-12mdk, but the rpm parser doesn't like too many spaces, and of course, being a character parse, it just doesn't accept anything between commas either. So, I decided to use some special charcter, @ to pass parameters and substitute them on the script later.
This is all for macro side.. Nothing more to see here.. move along.. :-)
[edit] Choosing proper libraries
A library isn't an i18n translation file. It is far more complicated and choosing what is a candidate to be a split library and what is not is a delicate process. First of all some mandatory rules:
- Library subpackages should not have any explicit requires !!! RPM handles everything
- Library subpackages should not have any more than the real library and symlinks
- A library should not be a run time module at the same time ( i.e. subversion )
Ok, explaining the three cases.
The first one will bring a lot of discussion around, in situations like libfontconfig requiring the client fontconfig package, and libldap requiring ldap just because of a config file. WRONG. Both situations suffer from dependency assumption. The library is a library and doesn't depend on any of the client programs. In both examples, users install some unnecessary things, like installing ldap server because a config file that fits not on any install, because the user should always make changes.
The second one is more obvious. Libraries usually provide functions, and requires functions from other libraries, but act not as a client program. This can be easily seen on the libao2 package, which requires the libxorg-X11 package, just because they need libXau.so.0. So, to install the libao2 package you need xml2 fontconfig and the whole X setup, installing more than 15 mb for a 97 k package ! The dep graph for libao with 2 dep levels.
The third one is the drawback, the 1% prone to fail cases. The subversion example is a good naughty sample. Libraries on subversion show the behaviour of both library and module, which invalidates the case. subversion packages need have the "whole set" of libraries to satisfy at least command line clients.
So the trick is to find the correct libraries. We can ( safely ) assume that real libraries have a full generated symlink. So we have at least libbla.so.0 and the libbla.so.0.0.0 symlink. All .so and .la files of this library go into the main devel package. This is what we are looking for. Cases like qt3, kde, where some .so files are modules will not be caught by the script.
The library naming, including soname and all Mandriva mandatory macros, like %mklibname are dealt with by the template script package.
[edit] The "SCRIPT"
Here's the rough initial shell script already tested and working for some huge packages like kdelibs and qt3, but there's plenty of room to improve it.
For log reasons, i decided to use the bm utility to compile subpackages, and to see the results, but it can be easily replaced by a direct rpmbuild
[edit] Drawbacks...
As said in the beginning of this document, this solution can be accomplished in 99% of the cases. But even for this 99%, one thing becomes important now: the maintainer!
The maintainer of the package ( or maintainers, or the team which is a better word now ) will need to know at least part of the software install and work process. And know very well in case of delicate packages like gnome or kde core ones.
This experience will be needed to see if a package IS NOT a candidate for this change.
If there's no good reason to not split libraries, or if it is a very specific case ( rare, not just because "we think that it should" ), the package should benefit from this.
[edit] How to test now ( Mandriva cooker or 2006 )
Easy steps
- Copy the script provided to /usr/lib{64}/rpm/
- Add the %genlibs% macro definition to your ~/.rpmmacros
- Add the %genlibs% macro as a last line in the %install stage of your spec
You can add Obsoletes and Requires, just remember that you should use @ instead of spaces after the option:
- -O libbla@<[email protected] ( for Obsoletes: libbla <= 3.4.5-12mdk )
