You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by monstereo <me...@gmail.com> on 2018/09/04 13:36:30 UTC

Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

I want to integrate ignite with storm via ignite-storm dependency
Because of Storm works with Threads, I had to use:

*Ignite node =Ignition.getOrStart(...),*

and this method only takes IgniteConfiguration instance. Here is the method
declaration:
   
*public static Ignite getOrStart(IgniteConfiguration cfg) throws
IgniteException *

If I do not use this method, I will get Ignite already started exception.

Now, I want to set ignite bolt via:

 igniteStormStreamer = new StormStreamer<>();
...
*builder.setBolt("igniteBolt",
igniteStormStreamer).shuffleGrouping("OtherBolt");*

and this igniteStormStreamer needs
*igniteStormStreamer.setIgniteConfigFile(getProperties().getProperty("ignite.config"));*

however, when i put class path to this ignite.config property, gives me
error:

*Caused by: org.apache.ignite.IgniteException: Spring XML configuration path
is invalid: apacheIgnite.CreateIgniteConfiguration. Note that this path
should be either absolute or a relative local file system path, relative to
META-INF in classpath or valid URL to IGNITE_HOME.*

when i put xml file to this ignite.config property, gives me error:
*8137 [Thread-26-igniteBolt-executor[5 5]] ERROR o.a.s.d.executor - 
java.lang.IncompatibleClassChangeError: Implementing class*


how can i should this issue?










--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by monstereo <me...@gmail.com>.
thank you for your response
you comments helped me alot

But there is point about IgniteStorm, one way or another I can run the code
via ide.(using ignite-storm). However when I convert to there was error like
you said backward compability. That's why I gave up to do with ignite-storm,
just do it simply cache.put(...)

may i ask how to get instance of specific ignite node: (assume that in
terminal mode ignite run on port 47501 and when i write Ignite ignite =
Ignition.ignite(), I just want to connect that node) (not creating new one)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by Saikat Maitra <sa...@gmail.com>.
Hi,

Yes, you can connect to a remote(local running in some 475XX port or
another host in network ) ignite node using IgniteClient. Please take a
look into some examples available here

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/client/ClientPutGetExample.java

Regards
Saikat

On Thu, Sep 6, 2018 at 1:27 AM, monstereo <me...@gmail.com> wrote:

> I thought that I could connect specific ignite node via setting
> userAttributes in the xml configuration.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by monstereo <me...@gmail.com>.
I thought that I could connect specific ignite node via setting
userAttributes in the xml configuration.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by Saikat Maitra <sa...@gmail.com>.
Hello,

1. I think the issue with Incompatible class error is due to backward
incompatible changes between slf4j-api 1.7.25 and ignite-log4j. I agree for
most cases we need not worry on the dependency compatibility but incase of
incompatible changes excluding dependencies helps.

2. The link I shared is to check if ignite instance is already present in
same jvm. When you are starting one node in terminal and another using
storm project then they are in separate jvm. You can try the same by
starting 2 nodes in terminal. The key point to note here is you are
using TcpDiscoveryMulticastIpFinder
in the xml config and you have also provided 127.0.0.1:47500..47530 as your
ip address and port range so each node as you start will discover each
other and create a cluster.

3. This issue can be solved using following options:

a) One way to solve the issue you described would be to use a IgniteClient
in your storm project and read the data from the remote node (prod or local
running in port assume 47500), process the data and stream the data using
StormStreamer in another node (assume for example node running in 47501).

b) You can also choose to use TcpDiscoveryVmIpFinder and create you own
cluster and read data from one node and write to another.

HTH

Regards,
Saikat



On Wed, Sep 5, 2018 at 1:09 AM, monstereo <me...@gmail.com> wrote:

> I have many questions
>
> *1)* This issue really annoying me, here is the error:(occurs in storm
> project)
> *java.lang.IncompatibleClassChangeError: Implementing class     at
> java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_181]        at
> java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_181]  at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> ~[?:1.8.0_181]  at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> ~[?:1.8.0_181]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> ~[?:1.8.0_181]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_181]
>       at
> java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_181]   at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> ~[?:1.8.0_181]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_181]
>        at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> ~[?:1.8.0_181]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
>        at
> java.lang.Class.forName0(Native Method) ~[?:1.8.0_181]  at
> java.lang.Class.forName(Class.java:264) ~[?:1.8.0_181]  at
> org.apache.ignite.internal.util.IgniteUtils.addLog4jNoOpLogger(
> IgniteUtils.java:8366)
> ~[ignite-core-2.5.0.jar:2.5.0]  at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:930)
> ~[ignite-core-2.5.0.jar:2.5.0]  at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
> ~[ignite-core-2.5.0.jar:2.5.0]  at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
> ~[ignite-core-2.5.0.jar:2.5.0]  at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
> ~[ignite-core-2.5.0.jar:2.5.0]*
>
> now, i have solved but i need proper solution, because i know that when i
> convert to jar, it gives me error(always it gives).
> Here is the my solution: according to the this line
> "org.apache.ignite.internal.util.IgniteUtils.addLog4jNoOpLogger",
> I have realized that ignite-storm uses storm and log4j as a dependency
> Then I excluded these dependency from pom.xml, and it works.
> However, this is really stupid issue, do i have to care all dependency all
> the time?
>
> *2)* Link that you provided says that  // if an ignite instance is already
> started in same JVM then use it.
> can you explain this. Right now i am working on local mode.
> I already open one ignite node on the terminal.
> then I open one ignite node inside the storm project
> Now, I am opening 2 different JVM or one. (Same question for production
> mode, think of it terminal and storm project are inside different machines)
>
> *3)* As far as I know, StormStreamer<String, Integer> igniteStormStreamer =
> new StormSt.. () this, it creates new ignite node.
> For my project, I have one ignite node on a machine,
> Then to get cache data from that machine, I will open another ignite node
> in
> storm (actually in spout inside the open method)
> Then I process the datas via spouts bolts etc..
> Then I will write the last processed data to new ignite cache (uses
> StormStreamer)
> However, because of I have created ignite node once, when I use
> stormstreamer, i am getting IgniteAlreadyStarted exceptions
> How can i solved that? (Note that: I have to be sure that one ignite node
> will be created in the storm)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by monstereo <me...@gmail.com>.
I have many questions

*1)* This issue really annoying me, here is the error:(occurs in storm
project)
*java.lang.IncompatibleClassChangeError: Implementing class 	at
java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_181] 	at
java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_181] 	at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
~[?:1.8.0_181] 	at
java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_181] 
at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_181] 
at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_181] 	at
java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_181] 	at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181] 
at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_181] 
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_181] 	at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_181] 
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181] 	at
java.lang.Class.forName0(Native Method) ~[?:1.8.0_181] 	at
java.lang.Class.forName(Class.java:264) ~[?:1.8.0_181] 	at
org.apache.ignite.internal.util.IgniteUtils.addLog4jNoOpLogger(IgniteUtils.java:8366)
~[ignite-core-2.5.0.jar:2.5.0] 	at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:930)
~[ignite-core-2.5.0.jar:2.5.0] 	at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
~[ignite-core-2.5.0.jar:2.5.0] 	at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
~[ignite-core-2.5.0.jar:2.5.0] 	at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
~[ignite-core-2.5.0.jar:2.5.0]*

now, i have solved but i need proper solution, because i know that when i
convert to jar, it gives me error(always it gives).
Here is the my solution: according to the this line
"org.apache.ignite.internal.util.IgniteUtils.addLog4jNoOpLogger", 
I have realized that ignite-storm uses storm and log4j as a dependency
Then I excluded these dependency from pom.xml, and it works. 
However, this is really stupid issue, do i have to care all dependency all
the time?

*2)* Link that you provided says that  // if an ignite instance is already
started in same JVM then use it.
can you explain this. Right now i am working on local mode.
I already open one ignite node on the terminal.
then I open one ignite node inside the storm project
Now, I am opening 2 different JVM or one. (Same question for production
mode, think of it terminal and storm project are inside different machines)

*3)* As far as I know, StormStreamer<String, Integer> igniteStormStreamer =
new StormSt.. () this, it creates new ignite node.
For my project, I have one ignite node on a machine,
Then to get cache data from that machine, I will open another ignite node in
storm (actually in spout inside the open method)
Then I process the datas via spouts bolts etc..  
Then I will write the last processed data to new ignite cache (uses
StormStreamer)
However, because of I have created ignite node once, when I use
stormstreamer, i am getting IgniteAlreadyStarted exceptions
How can i solved that? (Note that: I have to be sure that one ignite node
will be created in the storm)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by monstereo <me...@gmail.com>.
Here is the pom.xml ->   pom2.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1901/pom2.xml>  
Here is the node configuration ->   node.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1901/node.xml>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Need IgniteConfiguration and XML at the same time? IgniteStormStreamer?

Posted by Saikat Maitra <sa...@gmail.com>.
Hi,

StormStreamer already creates Ignite instance as part of start() method. In
your example since multiple thread is trying to create ignite instances you
are getting the exception of "Ignite already started".

Please take a look into the below sample test implementation of
StormStreamer.

https://github.com/apache/ignite/blob/master/modules/storm/src/test/java/org/apache/ignite/stream/storm/StormIgniteStreamerSelfTest.java

Also can you please share your Ignite configuration xml file.

Regards,
Saikat



On Tue, Sep 4, 2018 at 8:36 AM, monstereo <me...@gmail.com> wrote:

> I want to integrate ignite with storm via ignite-storm dependency
> Because of Storm works with Threads, I had to use:
>
> *Ignite node =Ignition.getOrStart(...),*
>
> and this method only takes IgniteConfiguration instance. Here is the method
> declaration:
>
> *public static Ignite getOrStart(IgniteConfiguration cfg) throws
> IgniteException *
>
> If I do not use this method, I will get Ignite already started exception.
>
> Now, I want to set ignite bolt via:
>
>  igniteStormStreamer = new StormStreamer<>();
> ...
> *builder.setBolt("igniteBolt",
> igniteStormStreamer).shuffleGrouping("OtherBolt");*
>
> and this igniteStormStreamer needs
> *igniteStormStreamer.setIgniteConfigFile(getProperties().getProperty("
> ignite.config"));*
>
> however, when i put class path to this ignite.config property, gives me
> error:
>
> *Caused by: org.apache.ignite.IgniteException: Spring XML configuration
> path
> is invalid: apacheIgnite.CreateIgniteConfiguration. Note that this path
> should be either absolute or a relative local file system path, relative to
> META-INF in classpath or valid URL to IGNITE_HOME.*
>
> when i put xml file to this ignite.config property, gives me error:
> *8137 [Thread-26-igniteBolt-executor[5 5]] ERROR o.a.s.d.executor -
> java.lang.IncompatibleClassChangeError: Implementing class*
>
>
> how can i should this issue?
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>