You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by tsteven four <ts...@gmail.com> on 2011/07/29 17:12:57 UTC
1.7 check-swig-py failure
I get the following error when trying to make check-swig-py:
/usr/bin/python
> /apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/bindings/swig/python/tests/run_all.py
> Traceback (most recent call last):
> File
> "/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/bindings/swig/python/tests/run_all.py",
> line 22, in ?
> import mergeinfo, core, client, delta, pool, ra, wc, repository, auth,
> \
> File
> "/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/bindings/swig/python/tests/mergeinfo.py",
> line 22, in ?
> from svn import core, repos, fs
> File
> "/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/bindings/swig/python/svn/core.py",
> line 26, in ?
> from libsvn.core import *
> File
> "/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/bindings/swig/python/libsvn/core.py",
> line 29, in ?
> import _core
>
ImportError:
> /apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/libsvn_ra_serf/.libs/libsvn_ra_serf-1.so.0:
> undefined symbol: apr_file_buffer_set
> gmake: *** [check-swig-py] Error 1
>
I have built apr, apr-util and subversion with a non-standard prefix.
However, when I check the library mentioned in the error message above, it
is finding my apr, apr-util and svn libraries, not the system ones.
ldd libsvn_ra_serf-1.so.0
> linux-vdso.so.1 => (0x00007fff33dfc000)
> libsvn_delta-1.so.0 =>
> /apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/libsvn_delta/.libs/libsvn_delta-1.so.0
> (0x00002b618e828000)
> libsvn_subr-1.so.0 =>
> /apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/libsvn_subr/.libs/libsvn_subr-1.so.0
> (0x00002b618ea35000)
> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00002b618ed0a000)
> libserf-0.so.0 =>
> /apps/install/subversion-1.7.0-beta2-Linux_x86_64/serf/.libs/libserf-0.so.0
> (0x00002b618ef19000)
> libaprutil-1.so.0 =>
> /apps/Linux_x86_64/subversion-1.7.0-beta2/lib/libaprutil-1.so.0
> (0x00002b618f12e000)
> libexpat.so.0 =>
> /apps/Linux_x86_64/subversion-1.7.0-beta2/lib/libexpat.so.0
> (0x00002b618f34e000)
> libapr-1.so.0 =>
> /apps/Linux_x86_64/subversion-1.7.0-beta2/lib/libapr-1.so.0
> (0x00002b618f570000)
> libuuid.so.1 => /lib64/libuuid.so.1 (0x00002b618f7bc000)
> librt.so.1 => /lib64/librt.so.1 (0x00002b618f9c0000)
> libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b618fbc9000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b618fe02000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00002b619001d000)
> libm.so.6 => /lib64/libm.so.6 (0x00002b6190221000)
> libz.so.1 => /usr/lib64/libz.so.1 (0x00002b61904a5000)
> libssl.so.6 => /lib64/libssl.so.6 (0x00002b61906b9000)
> libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002b6190905000)
> libc.so.6 => /lib64/libc.so.6 (0x00002b6190c57000)
> /lib64/ld-linux-x86-64.so.2 (0x000000326ec00000)
> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
> (0x00002b6190faf000)
> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002b61911de000)
> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002b6191473000)
> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3
> (0x00002b6191675000)
> libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
> (0x00002b619189b000)
> libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002b6191aa3000)
> libresolv.so.2 => /lib64/libresolv.so.2 (0x00002b6191ca5000)
> libselinux.so.1 => /lib64/libselinux.so.1 (0x00002b6191ebb000)
> libsepol.so.1 => /lib64/libsepol.so.1 (0x00002b61920d3000)
>
readelf -d libsvn_ra_serf-1.so.0 | grep RPATH
> 0x000000000000000f (RPATH) Library rpath:
> [/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/libsvn_delta/.libs:/apps/install/subversion-1.7.0-beta2-Linux_x86_64/subversion/libsvn_subr/.libs:/apps/install/subversion-1.7.0-beta2-Linux_x86_64/serf/.libs:/apps/Linux_x86_64/subversion-1.7.0-beta2/lib:/usr/lib64]
>
apr_file_buffer_set is not defined in the system libapr-1.so.0, but it is
defined in mine
(/apps/Linux_x86_64/subversion-1.7.0-beta2/lib/libapr-1.so.0). I have
verified
this with readelf -s. Somehow it appears that when check-swig-py is run the
system library /usr/lib64/libapr-1.so.0 is used instead of the one I built,
despite the results of
ldd and readelf shown above. Is there some method to get python use the
RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or is
the python check overriding RPATH with
some method like LD_LIBRARY_PATH?
Re: 1.7 check-swig-py failure
Posted by tsteven four <ts...@gmail.com>.
>
> Is libpython.*.so in /usr/lib64, by chance?
>
yes
In my build I get this (quote from swig/python/_wc.la:)
Not sure what the is about, but in my _wc.la file the only reference to
/usr/lib64 is
in dependency_libs, which includes /usr/lib64/libexpat.la.
My apr-util is built with the internal expat to avoid a dependency on
/usr/lib64/libexpat in other libs such as libaprutil,
libsvn_subr, etc. because that was bringing in /usr/lib64 which caused the
wrong libapr to be found. The internal
expat solved that, but the same issue has cropped up again now in the
binding libraries.
Interestingly all installed libs are correct, it is just an issue before
installation. All the installed libs refer to the correct
libexpat, libapr, libaprutil.
Re: 1.7 check-swig-py failure
Posted by Stefan Sperling <st...@elego.de>.
On Fri, Jul 29, 2011 at 10:48:47AM -0600, tsteven four wrote:
> Thanks, I will look at your workarounds.
>
> On my system the troubles seem to be because /usr/lib64 gets put into RPATH
> in all the .so libs in subversion/bindings/swig/python/* (including the
> subdirectories). It is put in RPATH before the path element for the apr,
> apr-util that I have built. If I use patchelf to reset the RPATH removing
> /usr/lib64 entirely in subversion/bindings/swig/python/.libs/_core.so then
> all the checks pass. Note that I removed /usr/lib64, I didn't move it to
> the end of the RPATH. So now I ask why does /usr/lib64 get put into the
> RPATH at all? It seems to cause a lot of problems, and not only with the
> bindings.
Is libpython.*.so in /usr/lib64, by chance?
I suppose libtool puts the RPATH in there to make sure the .so files
in the bindings can find libpython.*.so.
This is not necessary since python is already running when these
.so files get loaded, and python has loaded libpython.*.so.
So without the RPATH you can run the tests, since no incompatible
apr libraries are loaded from /usr/lib64.
But how can libtool, which is unaware of this and just treats the
bindings .so files as general shared libraries, know about that?
In my build I get this (quote from swig/python/_wc.la:)
-Wl,-rpath=/home/stsp/svn/prefix/python/lib
-L/home/stsp/svn/prefix/bz2/lib -L/home/stsp/svn/prefix/python/lib
-lpython2.7
It boils down two choices options:
1) use only system dependencies
2) use only dependencies you have compiled yourself and installed into a
custom prefix, and which do not pull in incompatible system deps.
If any dependency pulls in incompatible deps, compile those deps
yourself, too, until everything is smooth
The middle ground between those options is littered with countless
problems and spurious test failures.
Re: 1.7 check-swig-py failure
Posted by tsteven four <ts...@gmail.com>.
Thanks, I will look at your workarounds.
On my system the troubles seem to be because /usr/lib64 gets put into RPATH
in all the .so libs in subversion/bindings/swig/python/* (including the
subdirectories). It is put in RPATH before the path element for the apr,
apr-util that I have built. If I use patchelf to reset the RPATH removing
/usr/lib64 entirely in subversion/bindings/swig/python/.libs/_core.so then
all the checks pass. Note that I removed /usr/lib64, I didn't move it to
the end of the RPATH. So now I ask why does /usr/lib64 get put into the
RPATH at all? It seems to cause a lot of problems, and not only with the
bindings.
On Fri, Jul 29, 2011 at 10:32 AM, Stefan Sperling <st...@elego.de> wrote:
> On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> > ldd and readelf shown above. Is there some method to get python use the
> > RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or is
> > the python check overriding RPATH with
> > some method like LD_LIBRARY_PATH?
>
> Yes, you need to set LD_LIBRARY_PATH.
> You also need to make sure that none of the libraries that get loaded
> indirectly load the incompatible system libraries... it can get quite
> hairy.
> See this for various workarounds that I use in my build:
> https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/
>
Re: 1.7 check-swig-py failure
Posted by tsteven four <ts...@gmail.com>.
Thanks for your consideration of this issue. Unfortunately I have to
apologize, this is not an issue with r878293.
The issue arises when you are building subversion against apr/apu that has
already
been installed in a special location, was built with the internal expat,
and there is another incompatible apr/apu in the system location that
contains libexpat. Then many libraries will load the undesired (system)
apr/apu which leads to the test
failure.
The problem starts when the subversion make tries to configure serf.
It uses the results of "apu-1-config --link-libtool --libs" as a portion of
SERF_LIBS. Because
the apu is already installed this can include something like
"/specialpath/lib/libaprutil-1.la -lexpat".
When libtool links libserf the -lexpat option can turn into a dependency_lib
entry corresponding to the
system libexpat.la. This dependency then is inherited by libsvn_ra_serf,
and then by
libsvn_ra. In resolving these dependencies when linking downstream libs the
RPATH of the downstream libs
may contain a reference to the system location of libexpat (which contains
the downlevel system apr/apu).
As a result of the order of the generated RPATH entries libsvn_ra_serf
correctly loads the special apr/apu,
but libsvan_ra does not. Further downstream libraries also incorrectly load
the system apr/apu and
cause the test failure.
This can be avoided if serf is built outside of the subversion make by
configuring serf with
LDFLAGS=-L/specialpath/lib. Then the dependency_lib entry of libserf for
expat will refer to the
specially built libexpat.la instead of the system one. It can also be
avoided by building against the
special apr/apu build directories instead of the installed version. In this
case apu-1-config will
return something like
"/apr-util-buildpath/libaprtuil-1.la/apr-util-buildpath/xml/expat/
libexpat.la" for use
in SERF_LIBS, and the dependancy_lib entry in libserf-0.la will not refer to
the system libexpat.la,
which can prevent the path of the system libexpat from being added to RPATH
of the downstream libs.
My test system based on a RHEL 5 derivative.
It would be nice if it was easier to build subversion on this system,
but I am not sure this is a subversion problem, it seems more like an issue
with apr-util and serf.
But it certainly made it hard for me to build subversion!
On Mon, Aug 1, 2011 at 6:06 PM, Daniel Shahaf <d....@daniel.shahaf.name>wrote:
> I can't keep track of the chain of reversions there. Can you post this
> to dev@ please with a description of the regression --- i.e., how
> specifically the results you get from invocation X in 1.6.x differ from
> the results of X in 1.7.0-beta2, and where do you think
> configure.ac/Makefile.in are wrong (it sounds like you have that
> pinpointed).
>
> Thanks.
>
> tsteven four wrote on Mon, Aug 01, 2011 at 08:21:52 -0600:
> > I believe I have chased this down to r878293, a change to apr.m4 and
> > aprutil.m4. SVN_APR_LIBS and SVN_APRUTIL_LIBS used to be set based on
> > --link-libtool, now they are set on --link-ld. But when building the
> > swig-py target the libraries are linked with libtool and thus according
> to
> > the apr-1-config/apu-1-config help the --link-libtool option is correct.
> > When I look at my 1.6.17 build the subversion/bindings/swig/python/.libs
> > correctly include the local apr and aprutil, but in 1.7.0-beta2 they are
> > picking up the system versions.
> >
> > On Fri, Jul 29, 2011 at 12:21 PM, Daniel Shahaf <d.s@daniel.shahaf.name
> >wrote:
> >
> > > Stefan Sperling wrote on Fri, Jul 29, 2011 at 18:32:40 +0200:
> > > > On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> > > > > ldd and readelf shown above. Is there some method to get python
> use
> > > the
> > > > > RPATH in libsvn_ra_serf as ldd does so the correct libapr is found,
> or
> > > is
> > > > > the python check overriding RPATH with
> > > > > some method like LD_LIBRARY_PATH?
> > > >
> > > > Yes, you need to set LD_LIBRARY_PATH.
> > > > You also need to make sure that none of the libraries that get loaded
> > > > indirectly load the incompatible system libraries... it can get quite
> > > hairy.
> > >
> > > I'm not sure I recommend your build --- it rebuilds its own python and
> > > bzip2.
> > >
> > > Personally I'm trying to see if just installing the system svn to an
> > > out-of-${*PATH} location works. (ie, I used this technique on one box
> > > and now I'm trying to see if I can survive that way)
> > >
> > > > See this for various workarounds that I use in my build:
> > > >
> https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/
> > >
>
Re: 1.7 check-swig-py failure
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
I can't keep track of the chain of reversions there. Can you post this
to dev@ please with a description of the regression --- i.e., how
specifically the results you get from invocation X in 1.6.x differ from
the results of X in 1.7.0-beta2, and where do you think
configure.ac/Makefile.in are wrong (it sounds like you have that
pinpointed).
Thanks.
tsteven four wrote on Mon, Aug 01, 2011 at 08:21:52 -0600:
> I believe I have chased this down to r878293, a change to apr.m4 and
> aprutil.m4. SVN_APR_LIBS and SVN_APRUTIL_LIBS used to be set based on
> --link-libtool, now they are set on --link-ld. But when building the
> swig-py target the libraries are linked with libtool and thus according to
> the apr-1-config/apu-1-config help the --link-libtool option is correct.
> When I look at my 1.6.17 build the subversion/bindings/swig/python/.libs
> correctly include the local apr and aprutil, but in 1.7.0-beta2 they are
> picking up the system versions.
>
> On Fri, Jul 29, 2011 at 12:21 PM, Daniel Shahaf <d....@daniel.shahaf.name>wrote:
>
> > Stefan Sperling wrote on Fri, Jul 29, 2011 at 18:32:40 +0200:
> > > On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> > > > ldd and readelf shown above. Is there some method to get python use
> > the
> > > > RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or
> > is
> > > > the python check overriding RPATH with
> > > > some method like LD_LIBRARY_PATH?
> > >
> > > Yes, you need to set LD_LIBRARY_PATH.
> > > You also need to make sure that none of the libraries that get loaded
> > > indirectly load the incompatible system libraries... it can get quite
> > hairy.
> >
> > I'm not sure I recommend your build --- it rebuilds its own python and
> > bzip2.
> >
> > Personally I'm trying to see if just installing the system svn to an
> > out-of-${*PATH} location works. (ie, I used this technique on one box
> > and now I'm trying to see if I can survive that way)
> >
> > > See this for various workarounds that I use in my build:
> > > https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/
> >
Re: 1.7 check-swig-py failure
Posted by tsteven four <ts...@gmail.com>.
I believe I have chased this down to r878293, a change to apr.m4 and
aprutil.m4. SVN_APR_LIBS and SVN_APRUTIL_LIBS used to be set based on
--link-libtool, now they are set on --link-ld. But when building the
swig-py target the libraries are linked with libtool and thus according to
the apr-1-config/apu-1-config help the --link-libtool option is correct.
When I look at my 1.6.17 build the subversion/bindings/swig/python/.libs
correctly include the local apr and aprutil, but in 1.7.0-beta2 they are
picking up the system versions.
On Fri, Jul 29, 2011 at 12:21 PM, Daniel Shahaf <d....@daniel.shahaf.name>wrote:
> Stefan Sperling wrote on Fri, Jul 29, 2011 at 18:32:40 +0200:
> > On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> > > ldd and readelf shown above. Is there some method to get python use
> the
> > > RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or
> is
> > > the python check overriding RPATH with
> > > some method like LD_LIBRARY_PATH?
> >
> > Yes, you need to set LD_LIBRARY_PATH.
> > You also need to make sure that none of the libraries that get loaded
> > indirectly load the incompatible system libraries... it can get quite
> hairy.
>
> I'm not sure I recommend your build --- it rebuilds its own python and
> bzip2.
>
> Personally I'm trying to see if just installing the system svn to an
> out-of-${*PATH} location works. (ie, I used this technique on one box
> and now I'm trying to see if I can survive that way)
>
> > See this for various workarounds that I use in my build:
> > https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/
>
Re: 1.7 check-swig-py failure
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Stefan Sperling wrote on Fri, Jul 29, 2011 at 18:32:40 +0200:
> On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> > ldd and readelf shown above. Is there some method to get python use the
> > RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or is
> > the python check overriding RPATH with
> > some method like LD_LIBRARY_PATH?
>
> Yes, you need to set LD_LIBRARY_PATH.
> You also need to make sure that none of the libraries that get loaded
> indirectly load the incompatible system libraries... it can get quite hairy.
I'm not sure I recommend your build --- it rebuilds its own python and bzip2.
Personally I'm trying to see if just installing the system svn to an
out-of-${*PATH} location works. (ie, I used this technique on one box
and now I'm trying to see if I can survive that way)
> See this for various workarounds that I use in my build:
> https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/
Re: 1.7 check-swig-py failure
Posted by Stefan Sperling <st...@elego.de>.
On Fri, Jul 29, 2011 at 09:12:57AM -0600, tsteven four wrote:
> ldd and readelf shown above. Is there some method to get python use the
> RPATH in libsvn_ra_serf as ldd does so the correct libapr is found, or is
> the python check overriding RPATH with
> some method like LD_LIBRARY_PATH?
Yes, you need to set LD_LIBRARY_PATH.
You also need to make sure that none of the libraries that get loaded
indirectly load the incompatible system libraries... it can get quite hairy.
See this for various workarounds that I use in my build:
https://svn.apache.org/repos/asf/subversion/trunk/tools/dev/unix-build/