You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Mike Accola <ma...@us.ibm.com> on 2017/07/17 19:30:09 UTC
Using native library in Flink
I am new Flink user just trying to learn a little bit. I am trying to
incorporate an existing C++ library into a new Flink application. I am
seeing some strange behavior when trying to link in the native (C++)
library using java via JNI.
I am running this on Linux (RHEL6)
I can run my application once without error. Sometimes it will run
successfully a 2nd or 3rd time. However, eventually on a subsequent run,
I get an exception about the the native library not being found:
java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:10)
For debugging purposes for now, my native library does not have any
external references. It really contains 1 method that essentially does
nothing.
The behavior seems to indicate that there is some kind of cleanup being
done that "unloads" the native library. I suspect this is somehow related
to Flink's implementation of its library cache manager, but I have not
been able to prove this yet.
A few more details:
- I have a c++ library libdummy2native.so that contains a method that can
be invoked via JNI.
- I have a jar containing a class, called Dummy2. The Dummy2 constructor
will invoke the JNI method.
- The libdummy2native.so library is invoked with System.loadLibrary() like
this:
static {System.loadLibrary("dummy2native"); }
- In my simple Flink application, I have extended the ProcessFunction
class. Within this class, I have overriden processElement method that
declares a Dummy2 object.
- The Dummy2 class can be called and invoked without error when used in a
standalone java program.
Any thoughts or ideas on what to try next would be appreciated. Initially,
I'd be happy to be able to just explain this behavior. I will worry about
fixing it afterwards.
Thanks.
Re: Using native library in Flink
Posted by Mike Accola <ma...@us.ibm.com>.
Timo/Eron -
Thank you for the responses. To answer a few of your questions:
- For now, I am just running in a simple, local environment
(start-local.sh)
- I have this entry in the flink-conf.yaml file: env.java.opts :
"-Djava.library.path=/myPathWithTheLibrary". From looking at logs, it
looks like the JVM is picking up this setting. (plus if I remove the
setting, things don't work at all).
- I am loading the library within the processElement() method of my
ProcessFunction class.
I applied Eron's example of adding the library to my jar file and then
extracting/loading. This seems to work for me. So at least I have a
workaround for now (thank you!). However, it really seems like this is a
hack that I should not have to do. I am running on a single system and
the java.library.path is an absolute path on this system. I'd love to
figure out why this is happening and a better way to get around it.
One things I've noted: In the job manager logs, it appears
java.library.path is getting set as expected. But if I do
System.getProperty("java.library.path") within my processElement method to
check the property, the results are erratic: Sometimes I see the value
from my flink-conf.yaml. Other times I see something totally different
that appears to be the jvm default. More confusing is that seeing these
different values for java.library.path do NOT seem to correlate to whether
the library loads successfully or not. If I run this same application
twice in succession, am I running in different processes or JVMs?
Please reply if anyone has suggestions on other things to try.
--Mike
From: Eron Wright <er...@gmail.com>
To: dev@flink.apache.org
Date: 07/18/2017 04:40 PM
Subject: Re: Using native library in Flink
The solution mentioned by Timo works well with a standalone Flink cluster
but might not work with a YARN or Mesos cluster. An alternative is to
have
your Java library contain the native library within itself, and to extract
it to a temporary directory before calling `System.loadLibrary(...)`.
Note that you lose the advantages of using the native OS's packaging
system
(e.g. security patches, dependency management). The TensorFlow Java
library demonstrates the technique:
https://github.com/tensorflow/tensorflow/blob/v1.2.1/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java
-Eron
On Tue, Jul 18, 2017 at 8:02 AM, Timo Walther <tw...@apache.org> wrote:
> Hi Mike,
>
> do you run Flink locally or in a cluster? You have to make sure that VM
> argument -Djava.library.path is set for all Flink JVMs. Job Manager and
> Task Managers might run in separate JVMs. Make also sure that the
library
> is accessible from all node. I don't know what happens if the file is
> accessed by multiple processes/threads at the same time. It might also
> important where you put the static { ... } loading. It should be in the
> Function, because these classes get deserialized on the TaskManager.
>
> I hope this helps.
>
> Timo
>
>
> Am 17.07.17 um 21:30 schrieb Mike Accola:
>
> I am new Flink user just trying to learn a little bit. I am trying to
>> incorporate an existing C++ library into a new Flink application. I am
>> seeing some strange behavior when trying to link in the native (C++)
>> library using java via JNI.
>> I am running this on Linux (RHEL6)
>> I can run my application once without error. Sometimes it will run
>> successfully a 2nd or 3rd time. However, eventually on a subsequent
run,
>> I get an exception about the the native library not being found:
>> java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
>> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>> at java.lang.System.loadLibrary(System.java:1122)
>> at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:
>> 10)
>> For debugging purposes for now, my native library does not have any
>> external references. It really contains 1 method that essentially does
>> nothing.
>> The behavior seems to indicate that there is some kind of cleanup
being
>> done that "unloads" the native library. I suspect this is somehow
related
>> to Flink's implementation of its library cache manager, but I have not
>> been able to prove this yet.
>> A few more details:
>> - I have a c++ library libdummy2native.so that contains a method that
>> can
>> be invoked via JNI.
>> - I have a jar containing a class, called Dummy2. The Dummy2
constructor
>> will invoke the JNI method.
>> - The libdummy2native.so library is invoked with System.loadLibrary()
like
>> this:
>> static {System.loadLibrary("dummy2native"); }
>> - In my simple Flink application, I have extended the ProcessFunction
>> class. Within this class, I have overriden processElement method that
>> declares a Dummy2 object.
>> - The Dummy2 class can be called and invoked without error when used in
a
>> standalone java program.
>> Any thoughts or ideas on what to try next would be appreciated.
>> Initially,
>> I'd be happy to be able to just explain this behavior. I will worry
about
>> fixing it afterwards.
>> Thanks.
>>
>>
>>
>>
>>
>
Re: Using native library in Flink
Posted by Eron Wright <er...@gmail.com>.
The solution mentioned by Timo works well with a standalone Flink cluster
but might not work with a YARN or Mesos cluster. An alternative is to have
your Java library contain the native library within itself, and to extract
it to a temporary directory before calling `System.loadLibrary(...)`.
Note that you lose the advantages of using the native OS's packaging system
(e.g. security patches, dependency management). The TensorFlow Java
library demonstrates the technique:
https://github.com/tensorflow/tensorflow/blob/v1.2.1/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java
-Eron
On Tue, Jul 18, 2017 at 8:02 AM, Timo Walther <tw...@apache.org> wrote:
> Hi Mike,
>
> do you run Flink locally or in a cluster? You have to make sure that VM
> argument -Djava.library.path is set for all Flink JVMs. Job Manager and
> Task Managers might run in separate JVMs. Make also sure that the library
> is accessible from all node. I don't know what happens if the file is
> accessed by multiple processes/threads at the same time. It might also
> important where you put the static { ... } loading. It should be in the
> Function, because these classes get deserialized on the TaskManager.
>
> I hope this helps.
>
> Timo
>
>
> Am 17.07.17 um 21:30 schrieb Mike Accola:
>
> I am new Flink user just trying to learn a little bit. I am trying to
>> incorporate an existing C++ library into a new Flink application. I am
>> seeing some strange behavior when trying to link in the native (C++)
>> library using java via JNI.
>> I am running this on Linux (RHEL6)
>> I can run my application once without error. Sometimes it will run
>> successfully a 2nd or 3rd time. However, eventually on a subsequent run,
>> I get an exception about the the native library not being found:
>> java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
>> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>> at java.lang.System.loadLibrary(System.java:1122)
>> at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:
>> 10)
>> For debugging purposes for now, my native library does not have any
>> external references. It really contains 1 method that essentially does
>> nothing.
>> The behavior seems to indicate that there is some kind of cleanup being
>> done that "unloads" the native library. I suspect this is somehow related
>> to Flink's implementation of its library cache manager, but I have not
>> been able to prove this yet.
>> A few more details:
>> - I have a c++ library libdummy2native.so that contains a method that
>> can
>> be invoked via JNI.
>> - I have a jar containing a class, called Dummy2. The Dummy2 constructor
>> will invoke the JNI method.
>> - The libdummy2native.so library is invoked with System.loadLibrary() like
>> this:
>> static {System.loadLibrary("dummy2native"); }
>> - In my simple Flink application, I have extended the ProcessFunction
>> class. Within this class, I have overriden processElement method that
>> declares a Dummy2 object.
>> - The Dummy2 class can be called and invoked without error when used in a
>> standalone java program.
>> Any thoughts or ideas on what to try next would be appreciated.
>> Initially,
>> I'd be happy to be able to just explain this behavior. I will worry about
>> fixing it afterwards.
>> Thanks.
>>
>>
>>
>>
>>
>
Re: Using native library in Flink
Posted by Timo Walther <tw...@apache.org>.
Hi Mike,
do you run Flink locally or in a cluster? You have to make sure that VM
argument -Djava.library.path is set for all Flink JVMs. Job Manager and
Task Managers might run in separate JVMs. Make also sure that the
library is accessible from all node. I don't know what happens if the
file is accessed by multiple processes/threads at the same time. It
might also important where you put the static { ... } loading. It should
be in the Function, because these classes get deserialized on the
TaskManager.
I hope this helps.
Timo
Am 17.07.17 um 21:30 schrieb Mike Accola:
> I am new Flink user just trying to learn a little bit. I am trying to
> incorporate an existing C++ library into a new Flink application. I am
> seeing some strange behavior when trying to link in the native (C++)
> library using java via JNI.
>
> I am running this on Linux (RHEL6)
>
> I can run my application once without error. Sometimes it will run
> successfully a 2nd or 3rd time. However, eventually on a subsequent run,
> I get an exception about the the native library not being found:
>
> java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:10)
>
> For debugging purposes for now, my native library does not have any
> external references. It really contains 1 method that essentially does
> nothing.
>
> The behavior seems to indicate that there is some kind of cleanup being
> done that "unloads" the native library. I suspect this is somehow related
> to Flink's implementation of its library cache manager, but I have not
> been able to prove this yet.
>
> A few more details:
>
> - I have a c++ library libdummy2native.so that contains a method that can
> be invoked via JNI.
> - I have a jar containing a class, called Dummy2. The Dummy2 constructor
> will invoke the JNI method.
> - The libdummy2native.so library is invoked with System.loadLibrary() like
> this:
> static {System.loadLibrary("dummy2native"); }
> - In my simple Flink application, I have extended the ProcessFunction
> class. Within this class, I have overriden processElement method that
> declares a Dummy2 object.
> - The Dummy2 class can be called and invoked without error when used in a
> standalone java program.
>
> Any thoughts or ideas on what to try next would be appreciated. Initially,
> I'd be happy to be able to just explain this behavior. I will worry about
> fixing it afterwards.
>
> Thanks.
>
>
>
>