complicated build, subversion client

["rupert.thurner" <ru...@gmail.com>]

for some time i participate in a small group of people packaging
subversion for solaris within the opencsw project. while we love to
use subversion a lot because it easily scales to terabytes of data
managed, we continue to have two problems building, for years now:

1. client build
personally i find it particularly difficult to separate out a "common
build", and "client build" and "server build" which both may depend on
it.

2. comlicated build
the build itself is so complicated that, since i can remember, we
continually fail to package a current version of subversion.

what would be a good way to address this in your opinion? would it be
possible to switch the build system to something easier to handle and
introduce proper dependencies?

kr, rupert.

Re: complicated build, subversion client

[Stefan Sperling <st...@elego.de>]

On Sun, May 29, 2011 at 11:19:05PM +0200, rupert THURNER wrote:
> and we separate out the apache module, but imo a client should not contain
> svnserve, svnsync, svndumpfilter, etc., see
> http://www.opencsw.org/search/subversion/ for list of files we have
> currently in subversion. currently we have sasl, ldap and berkely db
> dependencies in this package

You could separate out svnserve and other repository administration
*binaries* (svnadmin, svnlook, svndumpfilter) into a separate package.

With the svn libraries it's a different story. You will want all
of them on both the client and server. This means you'll need most
(and probably all) dependencies on both sides.

SASL is a dependency of svnserve (server-side implementation of
the svn:// protocol) and of libsvn_ra_svn (client-side implementation
of same). You'll need it on both servers and client if you want
to support SASL authentication.

You'll need berkeley DB on the server, and on the client for the
libsvn_ra_local (file:// repository access protocol) to support
Subversion repositories using BDB accessed via file://.

I suppose the LDAP dependecy comes from berkeley DB?
Subversion has no direct dependency on LDAP.

> ... which seems bloat.

Well, Subversion has a large set of features, some of which are optional.
And with all those features come a lot of external dependencies.

You could choose not to compile some optional features in to keep 
package dependencies down but I think that would be to the detriment
of your users more than it helps them. You usually won't know in advance
which features users of your packages will need.

You could also take a look at how the Debian packages for Subversion
split things up. They probably have things split up as much as
reasonably possible.

> > > 2. comlicated build
> > > the build itself is so complicated that, since i can remember, we
> > > continually fail to package a current version of subversion.
> >
> > Can you point out more specifically what problems you are running into?
> >
> > If there is something concrete we could do in to improve the build system
> > we will consider it. Suggestions (and of course patches) are always
> > welcome.
> >
> >
> some files compile with the standard sun compiler, some don't, see:
> http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/Makefile
> 

The files that don't compile with SUN's cc are generated by SWIG.
Can you figure out if it's possible to make SWIG generate code that
compiles with gcc as well as Sun's cc? If so we might be able to
fix your problem by changing the way we generate these files.

> we patch certain files, see:
> *
> http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/0001-make-subversion-sysconfigdir-as-it-should-be-for-csw.patch

That patch looks fine. If there is a pre-processor macro that identifies
a CWS build we could also include this change upstream wrapped in the
right #ifdef.

It's a very small patch though and doesn't really cause maintenance
effort at your end, does it?

> *
> http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/fixme.sh

This I don't really understand.

Editing Makefiles and libtool scripts like that is going to get you into
trouble sooner or later, no doubt. This seems very hackish and hard to
maintain. You are fixing up *generated* files. They going to change more
often than not from release to release. It's probably easier to fix up
the source files these were generated from, and regenerate.
Preferably in a way that we can integrate upstream.

Does CSW have a generic libtool replacement to deal with special
requirements that you impose on shared libraries?
OpenBSD does have one:
http://www.openbsd.org/cgi-bin/cvsweb/ports/infrastructure/bin/libtool
(It's only used for package builds -- my dev builds don't use this but
they work fine on OpenBSD regardless.)

> * (not sure if this is still active:
> http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/javahl_headers_for_nested_classes.diff
> )

Note sure what the above is doing.

> *
> http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/subversion161.diff

This is OK. We do similar things in OpenBSD. Unfortunately everyone has
their own ideas about where stuff like python libraries should go.

SVN should probably add a new configure script parameter to take away the
need to patch this.

> > > what would be a good way to address this in your opinion? would it be
> > > possible to switch the build system to something easier to handle and
> > > introduce proper dependencies?
> >
> > I don't think that smart handling of dependencies belongs into the
> > Subversion core build system because there are so many different ways
> > people manage dependencies on different systems.
> > It would be very hard to come up with something that works for everyone.
> >
> 
> autotools / libtool seems to be responsible for 80% of the problems. beside
> making the build very slow we always need to tinker (see above), and still

I know that libtool is hard to deal with sometimes (I pulled some of
my own hair out over it as well). But it serves a great purpose for
Subversion because it is the most widespread way of dealing with
shared libraries on many UNIX-like platforms. I don't think we can
afford not to use it. If libtool is broken for you please try to
get libtool fixed instead. This would also help you port other software
that uses libtool more easily.

Note that as of Subversion 1.7 the build system will always use the libtool
program that was used to compile APR. This was done so we could remove
the configure script code for finding the right libtool and let APR sort
this out for us instead.

> get errors like just now when trying to build 1.6.16:
> 
> cd subversion/bindings/swig/python ; /bin/bash
> /home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/build-isa-sparcv8/subversion-1.6.16/libtool
> --mode=install /opt/csw/bin/ginstall -c
> _core.la/home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/install-isa-sparcv8/opt/csw/lion2.3/libsvn/_
> core.la
> libtool: install: error: cannot install `_core.la' to a directory not ending
> in /opt/csw/lib/python/site-packages/libsvn
> gmake[2]: *** [install-swig-py] Error 1
> gmake[2]: Leaving directory
> `/home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/build-isa-sparcv8/subversion-1.6.16'
> gmake[1]: *** [svn-python] Error 2
> gmake[1]: Leaving directory `/home/rupert/mgar/pkg/subversion/trunk'
> gmake: *** [merge-isa-sparcv8] Error 2

I suppose the above is something with the fixme.sh gone wrong?
I've never seen an error like this.

> but the subversion community is very nice and helpful which makes it still a
> pleasure :)

Well, I hope my hints and suggestions will help. I'm sorry I don't have
any conclusive answers to your problems.

Re: complicated build, subversion client

[rupert THURNER <ru...@gmail.com>]

On Sun, May 29, 2011 at 20:16, Stefan Sperling <st...@elego.de> wrote:

> On Sun, May 29, 2011 at 06:56:13AM -0700, rupert.thurner wrote:
> > for some time i participate in a small group of people packaging
> > subversion for solaris within the opencsw project. while we love to
> > use subversion a lot because it easily scales to terabytes of data
> > managed, we continue to have two problems building, for years now:
> >
> > 1. client build
> > personally i find it particularly difficult to separate out a "common
> > build", and "client build" and "server build" which both may depend on
> > it.
>
> Hi Rupert,
>
> You could build all svn binaries and then package them separately into
> client and server packages (with the mod_dav_svn Apache module depending
> on the HTTPD package). This is what Linux distributions and BSD systems do.


do you have an easy reference what goes into client, and what into server?
we use gar and are able to state which files go where, and have also
dependencies for every package:
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/Makefile

and we separate out the apache module, but imo a client should not contain
svnserve, svnsync, svndumpfilter, etc., see
http://www.opencsw.org/search/subversion/ for list of files we have
currently in subversion. currently we have sasl, ldap and berkely db
dependencies in this package ... which seems bloat.


> > 2. comlicated build
> > the build itself is so complicated that, since i can remember, we
> > continually fail to package a current version of subversion.
>
> Can you point out more specifically what problems you are running into?
>
> If there is something concrete we could do in to improve the build system
> we will consider it. Suggestions (and of course patches) are always
> welcome.
>
>
some files compile with the standard sun compiler, some don't, see:
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/Makefile

we patch certain files, see:
*
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/0001-make-subversion-sysconfigdir-as-it-should-be-for-csw.patch
*
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/fixme.sh
* (not sure if this is still active:
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/javahl_headers_for_nested_classes.diff
)
*
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/subversion/trunk/files/subversion161.diff



> > what would be a good way to address this in your opinion? would it be
> > possible to switch the build system to something easier to handle and
> > introduce proper dependencies?
>
> I don't think that smart handling of dependencies belongs into the
> Subversion core build system because there are so many different ways
> people manage dependencies on different systems.
> It would be very hard to come up with something that works for everyone.
>

autotools / libtool seems to be responsible for 80% of the problems. beside
making the build very slow we always need to tinker (see above), and still
get errors like just now when trying to build 1.6.16:

cd subversion/bindings/swig/python ; /bin/bash
/home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/build-isa-sparcv8/subversion-1.6.16/libtool
--mode=install /opt/csw/bin/ginstall -c
_core.la/home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/install-isa-sparcv8/opt/csw/lion2.3/libsvn/_
core.la
libtool: install: error: cannot install `_core.la' to a directory not ending
in /opt/csw/lib/python/site-packages/libsvn
gmake[2]: *** [install-swig-py] Error 1
gmake[2]: Leaving directory
`/home/rupert/mgar/pkg/subversion/trunk/work/solaris9-sparc/build-isa-sparcv8/subversion-1.6.16'
gmake[1]: *** [svn-python] Error 2
gmake[1]: Leaving directory `/home/rupert/mgar/pkg/subversion/trunk'
gmake: *** [merge-isa-sparcv8] Error 2

but the subversion community is very nice and helpful which makes it still a
pleasure :)


rupert.

Re: complicated build, subversion client

[Stefan Sperling <st...@elego.de>]

On Sun, May 29, 2011 at 06:56:13AM -0700, rupert.thurner wrote:
> for some time i participate in a small group of people packaging
> subversion for solaris within the opencsw project. while we love to
> use subversion a lot because it easily scales to terabytes of data
> managed, we continue to have two problems building, for years now:
> 
> 1. client build
> personally i find it particularly difficult to separate out a "common
> build", and "client build" and "server build" which both may depend on
> it.

Hi Rupert,

You could build all svn binaries and then package them separately into
client and server packages (with the mod_dav_svn Apache module depending
on the HTTPD package). This is what Linux distributions and BSD systems do.

> 2. comlicated build
> the build itself is so complicated that, since i can remember, we
> continually fail to package a current version of subversion.

Can you point out more specifically what problems you are running into?

If there is something concrete we could do in to improve the build system
we will consider it. Suggestions (and of course patches) are always welcome.

> what would be a good way to address this in your opinion? would it be
> possible to switch the build system to something easier to handle and
> introduce proper dependencies?

I don't think that smart handling of dependencies belongs into the
Subversion core build system because there are so many different ways 
people manage dependencies on different systems.
It would be very hard to come up with something that works for everyone.

I am an svn upstream developer and also maintain the OpenBSD port of
Subversion as well as packages for OpenSUSE. I've never had problems
packaging Subversion for these systems.

For my svn development builds I've written a custom GNU Makefile that
wraps Subversion's build system, downloads required dependencies,
compiles them with debug symbols and installs everything in the user's
home directory. The requirements for this build are very different from
those of downstream packagers, of course. But it should work on any
UNIX-like system and is available in the Subversion repository for reference:
https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build

For the OpenBSD port I rely on dependency management of the
ports/package infrastructure, which works quite well:
http://www.openbsd.org/cgi-bin/cvsweb/ports/devel/subversion

For OpenSUSE there is an RPM build that declares required dependencies
which are installed automatically before the package is built (see
the subversion.spec file):
https://build.opensuse.org/package/files?package=subversion&project=devel%3Atools%3Ascm%3Asvn

Note that both the above packages provide at least the mod_dav_svn
server module in a separate package. OpenSUSE even has svnserve in a
separate package. Both use the upstream build system to get the binaries
compiled and installed into a temporary directory, and then package
everything up into nice bundles in a separate step.

I don't know much about the packaging infrastructure that is available
on Solaris. Maybe your problems are rooted there rather than in the
upstream build system?