You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ant.apache.org by Juan Jimenez <vi...@gmail.com> on 2006/06/14 11:09:22 UTC
UnsatisfiedLinkError when invoking native library from junit task
Back in June 2004 there was a thread briefly discussing the problem of
native libraries having dependencies on further native libraries and the
apparent lack of support in ANT to modify Path or LD_LIBRARY_PATH as
appropriate. I am facing the problem (which I explain in more detail below)
and I am posting this in the hope that things have moved on or someone had a
bright idea on how to tackle this.
##### The Problem #####
I use Cruisecontrol to build several branches of several different
projects. The build itself is done through ANT scripts which I have
refactored extensively to avoid duplication. As such, all projects have a
standard location where they keep third party dependencies. That includes
both jar files and natibe libraries (DLLs for Windows and SOs for RHEL-3).
Within the <junit> task I have included the following inner elements:
<sysproperty key="java.library.path" path="${junit.lib.path}"/>
<sysproperty key="LD_LIBRARY_PATH" path="${ junit.lib.path}"/>
<sysproperty key="PATH" path="${junit.lib.path}"/>
where junit.lib.path is set to a path structure that includes all project
specific dependencies (including DLLs and SOs).
Our Java code can find any native library it needs (be it DLL or SO).
However, if that native library tries to load further native libraries we
get an UnsatisfiedLinkError
Below is a sample stack trace showing that libIRTKJava.so cannot find
libIRTKApi.so even though they are both in the same directory:
java.lang.UnsatisfiedLinkError:
/vob/gcib/etrading/projects/PricingImpl/lib/package/libIRTKJava.so:
libIRTKApi.so: cannot open shared object file: No such file or directory at
java.lang.ClassLoader$NativeLibrary.load(Native Method) at
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1751) at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1676) at
java.lang.Runtime.loadLibrary0(Runtime.java:822) at
java.lang.System.loadLibrary(System.java:992) at
irtkjava.LibLoader.loadNativeLibrary(LibLoader.java:46) at
com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.initialise(
IRTKCurveBuilderTest.java:21) at
com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.setUp(
IRTKCurveBuilderTest.java:34)
Am I going to be forced to look into a nominated directory and preload
anything that matches *.so before invoking a non-forked JVM for the unit
tests?
Re: UnsatisfiedLinkError when invoking native library from junit task
Posted by Juan Jimenez <vi...@gmail.com>.
For completeness, I am using:
Apache Ant version 1.6.5 compiled on June 2 2005
java version "1.5.0_05"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_05-b05)
Java HotSpot(TM) Server VM (build 1.5.0_05-b05, mixed mode)
On 14/06/06, Juan Jimenez <vi...@gmail.com> wrote:
>
> Back in June 2004 there was a thread briefly discussing the problem of
> native libraries having dependencies on further native libraries and the
> apparent lack of support in ANT to modify Path or LD_LIBRARY_PATH as
> appropriate. I am facing the problem (which I explain in more detail below)
> and I am posting this in the hope that things have moved on or someone had a
> bright idea on how to tackle this.
>
> ##### The Problem #####
> I use Cruisecontrol to build several branches of several different
> projects. The build itself is done through ANT scripts which I have
> refactored extensively to avoid duplication. As such, all projects have a
> standard location where they keep third party dependencies. That includes
> both jar files and natibe libraries (DLLs for Windows and SOs for RHEL-3).
>
> Within the <junit> task I have included the following inner elements:
>
> <sysproperty key="java.library.path" path="${junit.lib.path
> }"/>
> <sysproperty key="LD_LIBRARY_PATH" path="${ junit.lib.path}"/>
> <sysproperty key="PATH" path="${junit.lib.path}"/>
>
> where junit.lib.path is set to a path structure that includes all project
> specific dependencies (including DLLs and SOs).
>
> Our Java code can find any native library it needs (be it DLL or SO).
> However, if that native library tries to load further native libraries we
> get an UnsatisfiedLinkError
>
> Below is a sample stack trace showing that libIRTKJava.so cannot find
> libIRTKApi.so even though they are both in the same directory:
>
> java.lang.UnsatisfiedLinkError:
> /vob/gcib/etrading/projects/PricingImpl/lib/package/libIRTKJava.so:
> libIRTKApi.so: cannot open shared object file: No such file or directory
> at java.lang.ClassLoader$NativeLibrary.load(Native Method) at
> java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1751) at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1676) at
> java.lang.Runtime.loadLibrary0(Runtime.java:822) at
> java.lang.System.loadLibrary(System.java:992) at
> irtkjava.LibLoader.loadNativeLibrary(LibLoader.java:46) at
> com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.initialise(
> IRTKCurveBuilderTest.java:21) at
> com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.setUp(
> IRTKCurveBuilderTest.java:34)
>
> Am I going to be forced to look into a nominated directory and preload
> anything that matches *.so before invoking a non-forked JVM for the unit
> tests?
>
Re: UnsatisfiedLinkError when invoking native library from junit task
Posted by Dominique Devienne <dd...@gmail.com>.
> Our tests have always been forked and I do want to avoid hacks, which is why
> I am coming to this forum.
Cool.
> <!-- we now reset java.library.path to something specific to this project -->
> <sysproperty key="java.library.path" path="${junit.lib.path}"/>
This is enough for the libraries directly loaded by System.loadLibrary,
but not for their dependencies.
> <!--
> However, it doesn't seem to work under Linux. Native libraries
> needed
> by Java are loaded correctly but other native libraries needed by
> native libraries loaded by Java cannot be found
>
> None of the following worked (aside from not being platform
> independent)
>
> <sysproperty key="LD_LIBRARY_PATH" path="${junit.lib.path}"/>
Needs to be an <env>.
> <env key="LD_LIBRARY_PATH" path="${junit.lib.path}" />
> <env key="LD_LIBRARY_PATH" path="${env.LD_LIBRARY_PATH}${path.separator
> }${lib.package}" />
> <env key="LD_LIBRARY_PATH" path="${env.LD_LIBRARY_PATH}${path.separator
> }${lib.package}" />
I don't see why this wouldn't work. On some platform you need a 32 or
64 suffix, or SHLIB something instead of LD_LIBRARY_PATH. Dynamic
loading is very platform-specific.
> the following also works on Windows if you also remove the sysproperty
> above, however this
> is not platform independent and should be avoided
> <env key="PATH" path="${junit.lib.path}" />
> -->
PATH or Path has to be used on Windows. That's the equivalent of LD_...
I had a custom task that would give me the name and the value of the
binary and library paths in a platform independent manner, i.e. the
task looked at the current OS, and lookup which var to use. Supporting
a new platform required updating the task. Basically it was a switch.
--DD
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: UnsatisfiedLinkError when invoking native library from junit task
Posted by Juan Jimenez <vi...@gmail.com>.
I found some interesting reading here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4280189
http://forum.java.sun.com/thread.jspa?threadID=707176
However, since my script will not work without the sysproperty, one of the
following must be true:
a) the described behaviour of java.library.path has changed in Tiger
b) the implementation of sysproperty uses the hack described in 707176
c) it was never true that you couldn't change java.library.pathprogrammatically
Tonight I will be downloading ANT sources from my unrestricted network
access ...
Sadly, I still do not have an elegant solution to my problem
On 14/06/06, Juan Jimenez <vi...@gmail.com> wrote:
>
> Thank you for taking an interest.
>
> Our tests have always been forked and I do want to avoid hacks, which is
> why I am coming to this forum.
>
> This is what I have so far ( lib.package contains all shared objects ):
>
> <property name="junit.lib.path" value="${java.library.path}${
> path.separator}${lib.package}" />
> <junit failureproperty=" junit.failed"
> fork="true"
> includeAntRuntime="true"
> dir="${basedir}"
> showoutput="true">
>
> <classpath>
> <path refid="test.classpath"/>
> </classpath>
>
> <!-- we now reset java.library.path to something specific to this
> project -->
>
> <sysproperty key="java.library.path" path="${junit.lib.path}"/>
>
> <!--
> However, it doesn't seem to work under Linux. Native libraries
> needed
> by Java are loaded correctly but other native libraries needed by
>
> native libraries loaded by Java cannot be found
>
> None of the following worked (aside from not being platform
> independent)
>
>
> <sysproperty key="LD_LIBRARY_PATH" path="${junit.lib.path}"/>
> <env key="LD_LIBRARY_PATH" path="${ junit.lib.path}" />
> <env key="LD_LIBRARY_PATH" path="${env.LD_LIBRARY_PATH }${
> path.separator}${lib.package}" />
> <env key="LD_LIBRARY_PATH" path="${ env.LD_LIBRARY_PATH}${
> path.separator}${lib.package}" />
>
> the following also works on Windows if you also remove the sysproperty
> above, however this
> is not platform independent and should be avoided
> <env key="PATH" path="${ junit.lib.path}" />
> -->
>
> <formatter type="xml"/>
> <formatter type="plain"/>
>
> <batchtest todir="${ testresults.dir}">
> <fileset dir="${src.test}">
> <include name="**/*Test*.java"/>
> </fileset>
> </batchtest>
>
> </junit>
>
>
>
>
>
> On 14/06/06, Dominique Devienne <dd...@gmail.com> wrote:
> >
> > > Presumably you are running <junit> forked? I think you also need to
> > make
> > > sure that your path and library search path includes the existing env
> > > variables.
> >
> > Mostly likely not. The only "right" solution is to <junit fork="true">.
> >
> > > > Am I going to be forced to look into a nominated directory and
> > preload
> > > > anything that matches *.so before invoking a non-forked JVM for the
> > unit
> > > > tests?
> >
> > That would be a hack, and even then you'd have to load them in the
> > proper dependency order. I had a too-clever colleague use such an
> > approach to work around a similar issue in WebStart, but it's very
> > brittle, and very hacky.
> >
> > It's not a matter of using LoadLibraryEx or not, since *nix in general
> > does not allow you to alter the library loading path of an existing
> > process.
> >
> > So just fork your tests ;-) --DD
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> > For additional commands, e-mail: user-help@ant.apache.org
> >
> >
> >
>
Re: UnsatisfiedLinkError when invoking native library from junit task
Posted by Juan Jimenez <vi...@gmail.com>.
Thank you for taking an interest.
Our tests have always been forked and I do want to avoid hacks, which is why
I am coming to this forum.
This is what I have so far ( lib.package contains all shared objects ):
<property name="junit.lib.path" value="${java.library.path}${path.separator
}${lib.package}" />
<junit failureproperty="junit.failed"
fork="true"
includeAntRuntime="true"
dir="${basedir}"
showoutput="true">
<classpath>
<path refid="test.classpath"/>
</classpath>
<!-- we now reset java.library.path to something specific to this
project -->
<sysproperty key="java.library.path" path="${junit.lib.path}"/>
<!--
However, it doesn't seem to work under Linux. Native libraries
needed
by Java are loaded correctly but other native libraries needed by
native libraries loaded by Java cannot be found
None of the following worked (aside from not being platform
independent)
<sysproperty key="LD_LIBRARY_PATH" path="${junit.lib.path}"/>
<env key="LD_LIBRARY_PATH" path="${junit.lib.path}" />
<env key="LD_LIBRARY_PATH" path="${env.LD_LIBRARY_PATH}${path.separator
}${lib.package}" />
<env key="LD_LIBRARY_PATH" path="${env.LD_LIBRARY_PATH}${path.separator
}${lib.package}" />
the following also works on Windows if you also remove the sysproperty
above, however this
is not platform independent and should be avoided
<env key="PATH" path="${junit.lib.path}" />
-->
<formatter type="xml"/>
<formatter type="plain"/>
<batchtest todir="${testresults.dir}">
<fileset dir="${src.test}">
<include name="**/*Test*.java"/>
</fileset>
</batchtest>
</junit>
On 14/06/06, Dominique Devienne <dd...@gmail.com> wrote:
>
> > Presumably you are running <junit> forked? I think you also need to make
> > sure that your path and library search path includes the existing env
> > variables.
>
> Mostly likely not. The only "right" solution is to <junit fork="true">.
>
> > > Am I going to be forced to look into a nominated directory and preload
> > > anything that matches *.so before invoking a non-forked JVM for the
> unit
> > > tests?
>
> That would be a hack, and even then you'd have to load them in the
> proper dependency order. I had a too-clever colleague use such an
> approach to work around a similar issue in WebStart, but it's very
> brittle, and very hacky.
>
> It's not a matter of using LoadLibraryEx or not, since *nix in general
> does not allow you to alter the library loading path of an existing
> process.
>
> So just fork your tests ;-) --DD
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
>
>
>
Re: UnsatisfiedLinkError when invoking native library from junit task
Posted by Dominique Devienne <dd...@gmail.com>.
> Presumably you are running <junit> forked? I think you also need to make
> sure that your path and library search path includes the existing env
> variables.
Mostly likely not. The only "right" solution is to <junit fork="true">.
> > Am I going to be forced to look into a nominated directory and preload
> > anything that matches *.so before invoking a non-forked JVM for the unit
> > tests?
That would be a hack, and even then you'd have to load them in the
proper dependency order. I had a too-clever colleague use such an
approach to work around a similar issue in WebStart, but it's very
brittle, and very hacky.
It's not a matter of using LoadLibraryEx or not, since *nix in general
does not allow you to alter the library loading path of an existing
process.
So just fork your tests ;-) --DD
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: UnsatisfiedLinkError when invoking native library from junit
task
Posted by Steve Loughran <st...@apache.org>.
Juan Jimenez wrote:
> Back in June 2004 there was a thread briefly discussing the problem of
> native libraries having dependencies on further native libraries and the
> apparent lack of support in ANT to modify Path or LD_LIBRARY_PATH as
> appropriate. I am facing the problem (which I explain in more detail
> below)
> and I am posting this in the hope that things have moved on or someone
> had a
> bright idea on how to tackle this.
>
> ##### The Problem #####
> I use Cruisecontrol to build several branches of several different
> projects. The build itself is done through ANT scripts which I have
> refactored extensively to avoid duplication. As such, all projects have a
> standard location where they keep third party dependencies. That includes
> both jar files and natibe libraries (DLLs for Windows and SOs for RHEL-3).
>
> Within the <junit> task I have included the following inner elements:
>
> <sysproperty key="java.library.path" path="${junit.lib.path}"/>
> <sysproperty key="LD_LIBRARY_PATH" path="${ junit.lib.path}"/>
> <sysproperty key="PATH" path="${junit.lib.path}"/>
>
> where junit.lib.path is set to a path structure that includes all project
> specific dependencies (including DLLs and SOs).
>
> Our Java code can find any native library it needs (be it DLL or SO).
> However, if that native library tries to load further native libraries we
> get an UnsatisfiedLinkError
>
> Below is a sample stack trace showing that libIRTKJava.so cannot find
> libIRTKApi.so even though they are both in the same directory:
>
> java.lang.UnsatisfiedLinkError:
> /vob/gcib/etrading/projects/PricingImpl/lib/package/libIRTKJava.so:
> libIRTKApi.so: cannot open shared object file: No such file or directory at
> java.lang.ClassLoader$NativeLibrary.load(Native Method) at
> java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1751) at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1676) at
> java.lang.Runtime.loadLibrary0(Runtime.java:822) at
> java.lang.System.loadLibrary(System.java:992) at
> irtkjava.LibLoader.loadNativeLibrary(LibLoader.java:46) at
> com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.initialise(
> IRTKCurveBuilderTest.java:21) at
> com.bofa.ib.pricer.impl.irtk.IRTKCurveBuilderTest.setUp(
> IRTKCurveBuilderTest.java:34)
>
> Am I going to be forced to look into a nominated directory and preload
> anything that matches *.so before invoking a non-forked JVM for the unit
> tests?
>
This looks a bit like the problem that on windows, Java loads files
using ::LoadLibrary, not
::LoadLibraryEx( LOAD_WITH_ALTERED_SEARCH_PATH)
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/loadlibrary.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/loadlibraryex.asp
if they did that, then when you loaded a library from a dir, it would
add that dir to the search path for the next libraries to load.
This is the full search order
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dynamic-link_library_search_order.asp
Presumably you are running <junit> forked? I think you also need to make
sure that your path and library search path includes the existing env
variables.
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org