You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by charlie quillard <ch...@epitech.eu> on 2015/07/10 16:04:09 UTC
Problem to submit a topology to my storm cluster
Hello,
I want to execute a C++ lib on Storm and to implement this native lib in a bolt thought Swig. It doesn't work : I have a problem to submit a storm topology to my storm cluster.
First, I generated a c++ library. I want to use it in Java with Swig (an evolved wrapper based on JNI). The swig wrapper generate a java library (.jar) and a shared library (.so).
Then, I create the jar of the Storm application with Maven. This jar file contain : my java classes, my swig jar and my shared library (.so).
I include the Swig jar library in the pom.xml with theses following lines :
<dependency>
<groupId>com.example</groupId>
<artifactId>JavaEncoder</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>${project}/src/main/resources/JavaNativeEncoder.jar</systemPath>
</dependency>
I enable the jar generation plug-in in the pom.xml with these following lines :
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.example.JavaEncoder.PerfTopology</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
When I submit my topology to my cluster, I have the following Error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/example/JavaEncoder/NativeEncoder
at com.example.JavaEncoder.PerfTopology.main(PerfTopology.java:59)
Caused by: java.lang.ClassNotFoundException: com.example.JavaEncoder.NativeEncoder
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
I checked that if I unzipped the generated file, the swig class and the .so file is well contained in the Storm jar file.
To conclude, I don't understand why that does not works if someone have an idea, i would appreciate it.
Thanks in advance,
Charlie QUILLARD
RE: Problem to submit a topology to my storm cluster
Posted by charlie quillard <ch...@epitech.eu>.
I resolved my problem with the add of java library path in "storm.yaml", I didn't find any solution to avoid this.
________________________________
De : cutoutsy Feng <cu...@gmail.com>
Envoyé : jeudi 16 juillet 2015 06:58
À : user@storm.apache.org
Objet : Re: Problem to submit a topology to my storm cluster
Exception:Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar;
you can read the source code storm-core/jvm/utils/Utils.java getConfigFileInputStream()
2015-07-15 22:12 GMT+08:00 charlie quillard <ch...@epitech.eu>>:
Yes thank you, i put my storm-core dependency to provided and now I running my topology into my cluster.
Normally, the add of maven-shade-plugin had to resolve the NoClassDefFoundError but i have them in runtime , i would like to know if someone solved this problem with JNI library in the past.
Thanks in advance
Re: Problem to submit a topology to my storm cluster
Posted by cutoutsy Feng <cu...@gmail.com>.
Exception:Found multiple defaults.yaml resources. You're probably bundling
the Storm jars with your topology jar;
you can read the source code storm-core/jvm/utils/Utils.java
getConfigFileInputStream()
2015-07-15 22:12 GMT+08:00 charlie quillard <ch...@epitech.eu>:
> Yes thank you, i put my storm-core dependency to provided and now I
> running my topology into my cluster.
> Normally, the add of maven-shade-plugin had to resolve the
> NoClassDefFoundError but i have them in runtime , i would like to know if
> someone solved this problem with JNI library in the past.
>
> Thanks in advance,
> Charlie
>
> ________________________________________
> De : Matthias J. Sax <mj...@informatik.hu-berlin.de>
> Envoyé : mercredi 15 juillet 2015 15:20
> À : user@storm.apache.org
> Objet : Re: Problem to submit a topology to my storm cluster
>
> Your jar file contains two copies of "defaults.yaml". You need to make
> sure that there is at max one.
>
> Do you include "storm-core.jar" in your own jar? For this case, exclude
> "defaults.yaml" that is contained in "storm-core.jar"
>
> -Matthias
>
>
> On 07/15/2015 02:08 PM, charlie quillard wrote:
> > Hi,
> >
> >
> > Just following up on my previous email, I found a similar problem on
> > "stack overflow" (
> >
> http://stackoverflow.com/questions/29917158/apache-storm-java-lang-noclassdeffounderror-com-google-gson-gson
> > ), this one gives a solution with the *maven-shade-plugin .*
> >
> > I tried this solution and that partially works but I had the next error :
> >
> > /java.lang.RuntimeException: Found multiple defaults.yaml resources.
> > You're probably bundling the Storm jars with your topology jar/
> >
> >
> > //
> >
> > I someone can help me, i would appreciate it.
> >
> > Charlie
> > **
> >
> >
> >
> > ------------------------------------------------------------------------
> > *De :* charlie quillard
> > *Envoyé :* vendredi 10 juillet 2015 16:04
> > *À :* user@storm.apache.org
> > *Objet :* Problem to submit a topology to my storm cluster
> >
> >
> > Hello,
> >
> >
> >
> > I want to execute a C++ lib on Storm and to implement this native lib in
> > a bolt thought Swig. It doesn’t work : I have a problem to submit a
> > storm topology to my storm cluster.
> >
> >
> >
> > First, I generated a c++ library. I want to use it in Java with Swig (an
> > evolved wrapper based on JNI). The swig wrapper generate a java library
> > (.jar) and a shared library (.so).
> >
> >
> >
> > Then, I create the jar of the Storm application with Maven. This jar
> > file contain : my java classes, my swig jar and my shared library (.so).
> >
> >
> >
> > I include the Swig jar library in the pom.xml with theses following
> lines :
> >
> >
> > </dependency>
> > <groupId>com.example</groupId>
> > <artifactId>JavaEncoder</artifactId>
> > <scope>system</scope>
> > <version>1.0</version>
> >
> >
> <systemPath>${project}/src/main/resources/JavaNativeEncoder.jar</systemPath>
> > </dependency>/
> >
> > / /
> >
> > I enable the jar generation plug-in in the pom.xml with these following
> > lines :
> >
> >
> > /<build>
> > <plugins>
> > <plugin>
> > <artifactId>maven-assembly-plugin</artifactId>
> > <configuration>
> > <descriptorRefs>
> > <descriptorRef>jar-with-dependencies</descriptorRef>
> > </descriptorRefs>
> > <archive>
> > <manifest>
> > <mainClass>com.example.JavaEncoder.PerfTopology</mainClass>
> > </manifest>
> > </archive>
> > </configuration>
> > </plugin>/
> >
> > / //</plugins>/
> >
> > /</build>/
> >
> >
> >
> >
> >
> > When I submit my topology to my cluster, I have the following Error :
> >
> >
> >
> > /Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/example/JavaEncoder/NativeEncoder
> > at com.example.JavaEncoder.PerfTopology.main(PerfTopology.java:59)
> > Caused by: java.lang.ClassNotFoundException:
> > com.example.JavaEncoder.NativeEncoder
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358/)
> >
> >
> >
> > I checked that if I unzipped the generated file, the swig class and the
> > .so file is well contained in the Storm jar file.
> >
> >
> >
> > To conclude, I don't understand why that does not works if someone have
> > an idea, i would appreciate it.
> >
> >
> > Thanks in advance,
> >
> >
> > Charlie QUILLARD
> >
>
>
RE: Problem to submit a topology to my storm cluster
Posted by charlie quillard <ch...@epitech.eu>.
Yes thank you, i put my storm-core dependency to provided and now I running my topology into my cluster.
Normally, the add of maven-shade-plugin had to resolve the NoClassDefFoundError but i have them in runtime , i would like to know if someone solved this problem with JNI library in the past.
Thanks in advance,
Charlie
________________________________________
De : Matthias J. Sax <mj...@informatik.hu-berlin.de>
Envoyé : mercredi 15 juillet 2015 15:20
À : user@storm.apache.org
Objet : Re: Problem to submit a topology to my storm cluster
Your jar file contains two copies of "defaults.yaml". You need to make
sure that there is at max one.
Do you include "storm-core.jar" in your own jar? For this case, exclude
"defaults.yaml" that is contained in "storm-core.jar"
-Matthias
On 07/15/2015 02:08 PM, charlie quillard wrote:
> Hi,
>
>
> Just following up on my previous email, I found a similar problem on
> "stack overflow" (
> http://stackoverflow.com/questions/29917158/apache-storm-java-lang-noclassdeffounderror-com-google-gson-gson
> ), this one gives a solution with the *maven-shade-plugin .*
>
> I tried this solution and that partially works but I had the next error :
>
> /java.lang.RuntimeException: Found multiple defaults.yaml resources.
> You're probably bundling the Storm jars with your topology jar/
>
>
> //
>
> I someone can help me, i would appreciate it.
>
> Charlie
> **
>
>
>
> ------------------------------------------------------------------------
> *De :* charlie quillard
> *Envoyé :* vendredi 10 juillet 2015 16:04
> *À :* user@storm.apache.org
> *Objet :* Problem to submit a topology to my storm cluster
>
>
> Hello,
>
>
>
> I want to execute a C++ lib on Storm and to implement this native lib in
> a bolt thought Swig. It doesn’t work : I have a problem to submit a
> storm topology to my storm cluster.
>
>
>
> First, I generated a c++ library. I want to use it in Java with Swig (an
> evolved wrapper based on JNI). The swig wrapper generate a java library
> (.jar) and a shared library (.so).
>
>
>
> Then, I create the jar of the Storm application with Maven. This jar
> file contain : my java classes, my swig jar and my shared library (.so).
>
>
>
> I include the Swig jar library in the pom.xml with theses following lines :
>
>
> </dependency>
> <groupId>com.example</groupId>
> <artifactId>JavaEncoder</artifactId>
> <scope>system</scope>
> <version>1.0</version>
>
> <systemPath>${project}/src/main/resources/JavaNativeEncoder.jar</systemPath>
> </dependency>/
>
> / /
>
> I enable the jar generation plug-in in the pom.xml with these following
> lines :
>
>
> /<build>
> <plugins>
> <plugin>
> <artifactId>maven-assembly-plugin</artifactId>
> <configuration>
> <descriptorRefs>
> <descriptorRef>jar-with-dependencies</descriptorRef>
> </descriptorRefs>
> <archive>
> <manifest>
> <mainClass>com.example.JavaEncoder.PerfTopology</mainClass>
> </manifest>
> </archive>
> </configuration>
> </plugin>/
>
> / //</plugins>/
>
> /</build>/
>
>
>
>
>
> When I submit my topology to my cluster, I have the following Error :
>
>
>
> /Exception in thread "main" java.lang.NoClassDefFoundError:
> com/example/JavaEncoder/NativeEncoder
> at com.example.JavaEncoder.PerfTopology.main(PerfTopology.java:59)
> Caused by: java.lang.ClassNotFoundException:
> com.example.JavaEncoder.NativeEncoder
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358/)
>
>
>
> I checked that if I unzipped the generated file, the swig class and the
> .so file is well contained in the Storm jar file.
>
>
>
> To conclude, I don't understand why that does not works if someone have
> an idea, i would appreciate it.
>
>
> Thanks in advance,
>
>
> Charlie QUILLARD
>
Re: Problem to submit a topology to my storm cluster
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
Your jar file contains two copies of "defaults.yaml". You need to make
sure that there is at max one.
Do you include "storm-core.jar" in your own jar? For this case, exclude
"defaults.yaml" that is contained in "storm-core.jar"
-Matthias
On 07/15/2015 02:08 PM, charlie quillard wrote:
> Hi,
>
>
> Just following up on my previous email, I found a similar problem on
> "stack overflow" (
> http://stackoverflow.com/questions/29917158/apache-storm-java-lang-noclassdeffounderror-com-google-gson-gson
> ), this one gives a solution with the *maven-shade-plugin .*
>
> I tried this solution and that partially works but I had the next error :
>
> /java.lang.RuntimeException: Found multiple defaults.yaml resources.
> You're probably bundling the Storm jars with your topology jar/
>
>
> //
>
> I someone can help me, i would appreciate it.
>
> Charlie
> **
>
>
>
> ------------------------------------------------------------------------
> *De :* charlie quillard
> *Envoyé :* vendredi 10 juillet 2015 16:04
> *À :* user@storm.apache.org
> *Objet :* Problem to submit a topology to my storm cluster
>
>
> Hello,
>
>
>
> I want to execute a C++ lib on Storm and to implement this native lib in
> a bolt thought Swig. It doesn’t work : I have a problem to submit a
> storm topology to my storm cluster.
>
>
>
> First, I generated a c++ library. I want to use it in Java with Swig (an
> evolved wrapper based on JNI). The swig wrapper generate a java library
> (.jar) and a shared library (.so).
>
>
>
> Then, I create the jar of the Storm application with Maven. This jar
> file contain : my java classes, my swig jar and my shared library (.so).
>
>
>
> I include the Swig jar library in the pom.xml with theses following lines :
>
>
> </dependency>
> <groupId>com.example</groupId>
> <artifactId>JavaEncoder</artifactId>
> <scope>system</scope>
> <version>1.0</version>
>
> <systemPath>${project}/src/main/resources/JavaNativeEncoder.jar</systemPath>
> </dependency>/
>
> / /
>
> I enable the jar generation plug-in in the pom.xml with these following
> lines :
>
>
> /<build>
> <plugins>
> <plugin>
> <artifactId>maven-assembly-plugin</artifactId>
> <configuration>
> <descriptorRefs>
> <descriptorRef>jar-with-dependencies</descriptorRef>
> </descriptorRefs>
> <archive>
> <manifest>
> <mainClass>com.example.JavaEncoder.PerfTopology</mainClass>
> </manifest>
> </archive>
> </configuration>
> </plugin>/
>
> / //</plugins>/
>
> /</build>/
>
>
>
>
>
> When I submit my topology to my cluster, I have the following Error :
>
>
>
> /Exception in thread "main" java.lang.NoClassDefFoundError:
> com/example/JavaEncoder/NativeEncoder
> at com.example.JavaEncoder.PerfTopology.main(PerfTopology.java:59)
> Caused by: java.lang.ClassNotFoundException:
> com.example.JavaEncoder.NativeEncoder
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358/)
>
>
>
> I checked that if I unzipped the generated file, the swig class and the
> .so file is well contained in the Storm jar file.
>
>
>
> To conclude, I don't understand why that does not works if someone have
> an idea, i would appreciate it.
>
>
> Thanks in advance,
>
>
> Charlie QUILLARD
>
RE: Problem to submit a topology to my storm cluster
Posted by charlie quillard <ch...@epitech.eu>.
Hi,
Just following up on my previous email, I found a similar problem on "stack overflow" ( http://stackoverflow.com/questions/29917158/apache-storm-java-lang-noclassdeffounderror-com-google-gson-gson ), this one gives a solution with the maven-shade-plugin .
I tried this solution and that partially works but I had the next error :
java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar
I someone can help me, i would appreciate it.
Charlie
________________________________
De : charlie quillard
Envoyé : vendredi 10 juillet 2015 16:04
À : user@storm.apache.org
Objet : Problem to submit a topology to my storm cluster
Hello,
I want to execute a C++ lib on Storm and to implement this native lib in a bolt thought Swig. It doesn't work : I have a problem to submit a storm topology to my storm cluster.
First, I generated a c++ library. I want to use it in Java with Swig (an evolved wrapper based on JNI). The swig wrapper generate a java library (.jar) and a shared library (.so).
Then, I create the jar of the Storm application with Maven. This jar file contain : my java classes, my swig jar and my shared library (.so).
I include the Swig jar library in the pom.xml with theses following lines :
<dependency>
<groupId>com.example</groupId>
<artifactId>JavaEncoder</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>${project}/src/main/resources/JavaNativeEncoder.jar</systemPath>
</dependency>
I enable the jar generation plug-in in the pom.xml with these following lines :
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.example.JavaEncoder.PerfTopology</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
When I submit my topology to my cluster, I have the following Error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/example/JavaEncoder/NativeEncoder
at com.example.JavaEncoder.PerfTopology.main(PerfTopology.java:59)
Caused by: java.lang.ClassNotFoundException: com.example.JavaEncoder.NativeEncoder
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
I checked that if I unzipped the generated file, the swig class and the .so file is well contained in the Storm jar file.
To conclude, I don't understand why that does not works if someone have an idea, i would appreciate it.
Thanks in advance,
Charlie QUILLARD