You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Stuart Maclean <st...@apl.washington.edu> on 2013/01/04 20:54:46 UTC
Re: JNI jars dependencies
Hi from a newbie.
I too have been fighting with how best to develop maven artifacts
containing jni parts. Though I found several useful posts on jni and
maven , e.g.
http://www.tricoder.net/blog/?p=197
http://docs.codehaus.org/display/MAVENUSER/Projects+With+JNI
http://www.humboldt.co.uk/2009/02/wrapping-a-native-library-with-maven.html
in the end I rolled my own solution, something I wanted to avoid. My
goal was to produce usable artifacts that worked across platforms, ie
linux 32 and 64 bit and perhaps windows. Here's some notes on my
experiences over the past 3-4 days. My particular jni effort was to
'Java-ify' an existing C library.
I isolated the Java classes which had native methods into a single Maven
module, so producing a single artifact. I placed the Java sources under
the regular src/main/java. In the pom, I used maven-native-plugin to
invoke javah to produce the headers (jn the default location
target/native/javah). In the same pom, I then used the exec plugin to
invoke make in ./native/${os.name}. Both of these are bound to the
compile phase, so a simple
mvn
will compile Java, run javah, then locate correct Makefile and run it.
I then added makefiles to ./native/Linux and ./native/Windows. In
the Linux makefile, I used VPATH to locate the C sources in
${basedir}/src/main/c. The final .so file (which has no platform/arch
encoded into its name, ie is just libfoo.so) is copied by the make into
${basedir}/target/classes/META-INF/lib, so the package phase will locate
the .so at /META-INF/lib/libfoo.so, from which the nifty
com.wapmx.native loader can unpack it at runtime.
The whole artifact is then named
NAME-${os.arch}-${os.name} and installed and/or deployed. this is sort
of a poor man's AOL (from the nar plugin, which I ended up NOT using).
Then a project/module which depends on this artifact simply uses the
name above. It's a bit crude, but for 2 or 3 platforms maximum, I think
it will be manageable.
Points:
the javah invocation could have been done either from Maven or make. I
chose to to it in Maven, mostly to avoid duplication across many
platform Makefiles.
I use the wapmx artifact to extract the native lib from the jar, better
than requiring LD_LIBRARY_PATH solutions.
My particular jni scenario required that the Java side maintain C
pointers (in both directions). Knowing that these would be 64 bits wide
on 64 bit platforms, I had to use Java longs and jlong in the C code.
Compiling on 32 bit Linux, I then had to add
ifeq ($(shell uname -m),i686)
# want any warning to raise error, but silence any 64/32 bit conversions
CFLAGS += -Wno-int-to-pointer-cast -Wno-pointer-to-int-cast
endif
to the Makefile to silence the compiler's warnings. I always compile
with -Werror (any warning is an error, fail to build)
In the C link phase in my makefiles, I statically linked the C library I
am wrapping, call it W. To do this, I used an explicit file name, eg.
cc -o libfoo.so /path/to/libW.a my1.o my2.o
I tried fancy -Wl,-static and -Wl,-shared linker options but it always
failed to link. I should point out that I had sources to libW, so could
configure/make it as desired.
Then my end user (who is handed just a single jar) does not need any
libW.so at runtime.
Paths I investigated but did not finally use:
having the Java classes containing the native methods (call it A) and
the actual .so (call it B) as separate Maven modules, as explained in
the first url above. Then B depends upon A being built so that javah can
locate .class files, but then how do you run test cases in A, since you
need native code in B.
I did not use maven-nar-plugin, since as suggested, it is really a Maven
solution to wider native development. I had a localised (or so I think)
jni issue.
I used maven-native-plugin over the antrun-plugin, which apparently
would also give access to javah (would this require an extra Ant
build.xml or can this be 'embedded' in the pom???)
Comments, suggestions welcomed
Stuart
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: JNI jars dependencies
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi Stuart,
> I didn't use NAR as I was worried it was unsupported,fragmented and
> overly optimistic on what it was trying to achieve.
FYI, there is now an attempt at unification of the maven-nar-plugin at:
https://github.com/maven-nar/maven-nar-plugin
As well as a maven-nar-plugin mailing list:
https://groups.google.com/forum/?fromgroups#!forum/maven-nar
But since all the involved folks have other full-time projects, you are
probably right about the project goals being overly optimistic. :-) More
help on the project is very welcome.
Regards,
Curtis
On Fri, Jan 4, 2013 at 2:45 PM, Stuart Maclean <st...@apl.washington.edu>wrote:
> Sorry Martin, I missed some of your points:
>
>
> Hi from a newbie.MG>a newbie with extensive experience with maven and
>>> make files!
>>>
>>>
> Ah yes, I meant new to mailing list ;)
>
>
> com.wapmx.native loader can unpack it at runtime. MG>is this NativeLoader
> what you're alluding toMG>http://dev.loci.wisc.edu/**
> trac/software/browser/**branches/maven/projects/**
> native-library-util/src/main/**java/com/wapmx/nativeutils/**
> jniloader/NativeLoader.java?**rev=7574<http://dev.loci.wisc.edu/trac/software/browser/branches/maven/projects/native-library-util/src/main/java/com/wapmx/nativeutils/jniloader/NativeLoader.java?rev=7574>
>
> Essentially yes, that is it. I did try contacting the original author, to
> no avail. I can't recall now if I was able to get that artifact from a
> repo or had to install it manually.
>
>
> of a poor man's AOL (from the nar plugin, which I ended up NOT using).
> MG>did'nt catch the reasons why you do'nt want to use nar..explain please
>
> I didn't use NAR as I was worried it was unsupported,fragmented and overly
> optimistic on what it was trying to achieve. I could live with the exec
> plugin firing make, I am happy in make myself.tion across many
>
> platform Makefiles. MG>smart ..OS and compiler specific variables should
>>> be aggregated in OS and compiler specific profiles
>>>
>>>
> Yes. I cannot fathom why there is no easier way to run javah from Maven.
> It certainly isn't 'native' the way gcc is 'native'. It's a standard Java
> tool running on .class files. OK, it produces C headers but that doesn't
> make it 'native'. I may switch to the antrun-plugin for the javah step,
> since the native-maven-plugin looks a bit alpha itself, and I don't use it
> for any 'heavy lifting', I use make.
>
> MG>app code?
>> MG>JNI ?MG>How is the app PATHed to native binaries?
>>
>
> See last post, you don't need any LD_LIBRARY_PATH nor -Djava.library.path.
>
> I should qualify this by saying that in PRINCIPLE this should all work on
> Windows, I have NOT actually tested it, I have just tested Linux 32 and 64
> bit.
>
>
>
> Stuart
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@maven.**apache.org<us...@maven.apache.org>
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: JNI jars dependencies
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi all,
>
http://dev.loci.wisc.edu/trac/software/browser/branches/maven/projects/native-library-util/src/main/java/com/wapmx/nativeutils/jniloader/NativeLoader.java?rev=7574
Some notes about this code:
1) The above URL is to an SVN repository that has since been migrated to
Git. The current URL for the project is:
https://github.com/scijava/native-lib-loader
2) The SciJava native-lib-loader project is a fork [1] of Richard van der
Hoff's mx-native-loader project, which is also available from its own Maven
repository [2], and has several other forks floating around on GitHub, too.
Since we forked (at version 1.7), the author has released a newer version
(1.8), which we have not reconciled with the SciJava code.
Regards,
Curtis
[1] https://groups.google.com/d/topic/scijava/D4SVBhGhb1g/discussion
[2]
http://opensource.mxtelecom.com/maven/repo/com/wapmx/native/mx-native-loader/
On Fri, Jan 4, 2013 at 2:45 PM, Stuart Maclean <st...@apl.washington.edu>wrote:
> Sorry Martin, I missed some of your points:
>
>
> Hi from a newbie.MG>a newbie with extensive experience with maven and
>>> make files!
>>>
>>>
> Ah yes, I meant new to mailing list ;)
>
>
> com.wapmx.native loader can unpack it at runtime. MG>is this NativeLoader
> what you're alluding toMG>http://dev.loci.wisc.edu/**
> trac/software/browser/**branches/maven/projects/**
> native-library-util/src/main/**java/com/wapmx/nativeutils/**
> jniloader/NativeLoader.java?**rev=7574<http://dev.loci.wisc.edu/trac/software/browser/branches/maven/projects/native-library-util/src/main/java/com/wapmx/nativeutils/jniloader/NativeLoader.java?rev=7574>
>
> Essentially yes, that is it. I did try contacting the original author, to
> no avail. I can't recall now if I was able to get that artifact from a
> repo or had to install it manually.
>
>
> of a poor man's AOL (from the nar plugin, which I ended up NOT using).
> MG>did'nt catch the reasons why you do'nt want to use nar..explain please
>
> I didn't use NAR as I was worried it was unsupported,fragmented and overly
> optimistic on what it was trying to achieve. I could live with the exec
> plugin firing make, I am happy in make myself.tion across many
>
> platform Makefiles. MG>smart ..OS and compiler specific variables should
>>> be aggregated in OS and compiler specific profiles
>>>
>>>
> Yes. I cannot fathom why there is no easier way to run javah from Maven.
> It certainly isn't 'native' the way gcc is 'native'. It's a standard Java
> tool running on .class files. OK, it produces C headers but that doesn't
> make it 'native'. I may switch to the antrun-plugin for the javah step,
> since the native-maven-plugin looks a bit alpha itself, and I don't use it
> for any 'heavy lifting', I use make.
>
> MG>app code?
>> MG>JNI ?MG>How is the app PATHed to native binaries?
>>
>
> See last post, you don't need any LD_LIBRARY_PATH nor -Djava.library.path.
>
> I should qualify this by saying that in PRINCIPLE this should all work on
> Windows, I have NOT actually tested it, I have just tested Linux 32 and 64
> bit.
>
>
>
> Stuart
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@maven.**apache.org<us...@maven.apache.org>
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: JNI jars dependencies
Posted by Stuart Maclean <st...@apl.washington.edu>.
Sorry Martin, I missed some of your points:
>> Hi from a newbie.MG>a newbie with extensive experience with maven and make files!
>>
Ah yes, I meant new to mailing list ;)
com.wapmx.native loader can unpack it at runtime. MG>is this
NativeLoader what you're alluding
toMG>http://dev.loci.wisc.edu/trac/software/browser/branches/maven/projects/native-library-util/src/main/java/com/wapmx/nativeutils/jniloader/NativeLoader.java?rev=7574
Essentially yes, that is it. I did try contacting the original author,
to no avail. I can't recall now if I was able to get that artifact from
a repo or had to install it manually.
of a poor man's AOL (from the nar plugin, which I ended up NOT using).
MG>did'nt catch the reasons why you do'nt want to use nar..explain please
I didn't use NAR as I was worried it was unsupported,fragmented and
overly optimistic on what it was trying to achieve. I could live with
the exec plugin firing make, I am happy in make myself.tion across many
>> platform Makefiles. MG>smart ..OS and compiler specific variables should be aggregated in OS and compiler specific profiles
>>
Yes. I cannot fathom why there is no easier way to run javah from
Maven. It certainly isn't 'native' the way gcc is 'native'. It's a
standard Java tool running on .class files. OK, it produces C headers
but that doesn't make it 'native'. I may switch to the antrun-plugin
for the javah step, since the native-maven-plugin looks a bit alpha
itself, and I don't use it for any 'heavy lifting', I use make.
> MG>app code?
> MG>JNI ?MG>How is the app PATHed to native binaries?
See last post, you don't need any LD_LIBRARY_PATH nor -Djava.library.path.
I should qualify this by saying that in PRINCIPLE this should all work
on Windows, I have NOT actually tested it, I have just tested Linux 32
and 64 bit.
Stuart
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: JNI jars dependencies
Posted by Stuart Maclean <st...@apl.washington.edu>.
Hi Martin, all,
the native code, either a .so or .dll, lives in the uber.jar at
/META-INF/lib/. We then link this in via wapmx's native loader:
import com.wapmx.nativeutils.jniloader.NativeLoader;
class LoadJNI {
static {
String libName = "tsk4j";
try {
System.err.println( "Loading: " + libName );
NativeLoader.loadLibrary( libName );
} catch( IOException ioe ) {
throw new ExceptionInInitializerError( ioe );
}
}
}
The NativeLoader will create ./tmplib and dump the .so in there and then
call Java's regular loader. So the only side-effect is the ./tmplib
creation. And I imagine you could register a deleteOnExit hook to even
clean that up, I don't bother.
My LoadJNI class is essentially a dummy class purely to do the native
load. In classes where I actually HAVE native methods which are to be
found in the .so, I force a class dependency:
class HasNatives {
void native foo();
static {
LoadJNI dummy = new LoadJNI();
}
Short of multi-class-loader issues, this seems to work. All of my
classes which depend on the .so for their native are 'regular' classes
with no fancy class loader things going on.
Any better ideas on how to ensure the .so loaded before any use would be
welcomed.
Stuart
Stuart
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
RE: JNI jars dependencies
Posted by Martin Gainty <mg...@hotmail.com>.
> Date: Fri, 4 Jan 2013 11:54:46 -0800
> From: stuart@apl.washington.edu
> To: users@maven.apache.org
> Subject: Re: JNI jars dependencies
>
> Hi from a newbie.MG>a newbie with extensive experience with maven and make files!
>
> I too have been fighting with how best to develop maven artifacts
> containing jni parts. Though I found several useful posts on jni and
> maven , e.g.
>
> http://www.tricoder.net/blog/?p=197
>
> http://docs.codehaus.org/display/MAVENUSER/Projects+With+JNI
>
> http://www.humboldt.co.uk/2009/02/wrapping-a-native-library-with-maven.html
>
> in the end I rolled my own solution, something I wanted to avoid. My
> goal was to produce usable artifacts that worked across platforms, ie
> linux 32 and 64 bit and perhaps windows. Here's some notes on my
> experiences over the past 3-4 days. My particular jni effort was to
> 'Java-ify' an existing C library.
>
> I isolated the Java classes which had native methods into a single Maven
> module, so producing a single artifact. I placed the Java sources under
> the regular src/main/java. In the pom, I used maven-native-plugin to
> invoke javah to produce the headers (jn the default location
> target/native/javah). In the same pom, I then used the exec plugin to
> invoke make in ./native/${os.name}. Both of these are bound to the
> compile phase, so a simple
>
> mvn
>
> will compile Java, run javah, then locate correct Makefile and run it.
>
> I then added makefiles to ./native/Linux and ./native/Windows. In
> the Linux makefile, I used VPATH to locate the C sources in
> ${basedir}/src/main/c. The final .so file (which has no platform/arch
> encoded into its name, ie is just libfoo.so) is copied by the make into
> ${basedir}/target/classes/META-INF/lib, so the package phase will locate
> the .so at /META-INF/lib/libfoo.so, from which the nifty
> com.wapmx.native loader can unpack it at runtime. MG>is this NativeLoader what you're alluding toMG>http://dev.loci.wisc.edu/trac/software/browser/branches/maven/projects/native-library-util/src/main/java/com/wapmx/nativeutils/jniloader/NativeLoader.java?rev=7574
>
> The whole artifact is then named
>
> NAME-${os.arch}-${os.name} and installed and/or deployed. this is sort
> of a poor man's AOL (from the nar plugin, which I ended up NOT using). MG>did'nt catch the reasons why you do'nt want to use nar..explain please
>
> Then a project/module which depends on this artifact simply uses the
> name above. It's a bit crude, but for 2 or 3 platforms maximum, I think
> it will be manageable.
>
> Points:
>
> the javah invocation could have been done either from Maven or make. I
> chose to to it in Maven, mostly to avoid duplication across many
> platform Makefiles. MG>smart ..OS and compiler specific variables should be aggregated in OS and compiler specific profiles
>
> I use the wapmx artifact to extract the native lib from the jar, better
> than requiring LD_LIBRARY_PATH solutions.
>
> My particular jni scenario required that the Java side maintain C
> pointers (in both directions). Knowing that these would be 64 bits wide
> on 64 bit platforms, I had to use Java longs and jlong in the C code.
> Compiling on 32 bit Linux, I then had to add
>
> ifeq ($(shell uname -m),i686)
> # want any warning to raise error, but silence any 64/32 bit conversions
> CFLAGS += -Wno-int-to-pointer-cast -Wno-pointer-to-int-cast
> endif
>
> to the Makefile to silence the compiler's warnings. I always compile
> with -Werror (any warning is an error, fail to build)
>
> In the C link phase in my makefiles, I statically linked the C library I
> am wrapping, call it W. To do this, I used an explicit file name, eg.
>
> cc -o libfoo.so /path/to/libW.a my1.o my2.o
>
> I tried fancy -Wl,-static and -Wl,-shared linker options but it always
> failed to link. I should point out that I had sources to libW, so could
> configure/make it as desired.
>
>
> Then my end user (who is handed just a single jar) does not need any
> libW.so at runtime.MG>whats inside the uber.jar?
MG>app code?
MG>JNI ?MG>How is the app PATHed to native binaries?
>
>
> Paths I investigated but did not finally use:
>
> having the Java classes containing the native methods (call it A) and
> the actual .so (call it B) as separate Maven modules, as explained in
> the first url above. Then B depends upon A being built so that javah can
> locate .class files, but then how do you run test cases in A, since you
> need native code in B.MG>perhaps aggregate the plugins for B into a child module?
>
> I did not use maven-nar-plugin, since as suggested, it is really a Maven
> solution to wider native development. I had a localised (or so I think)
> jni issue.
>
> I used maven-native-plugin over the antrun-plugin, which apparently
> would also give access to javah (would this require an extra Ant
> build.xml or can this be 'embedded' in the pom???) MG>forked ant targets cause classloader issues, resource-limitation issues and passing maven-properties to ant build.xml and return MG>back...unless there is a political reason to do so..stay with maven
>
> Comments, suggestions welcomed
>
> Stuart
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>