You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Jessie Lin <je...@gmail.com> on 2017/08/09 00:09:34 UTC
Re: Streaming test
Val, I replace atomicSequence with atomicLong and used the method below.
I need to specify an initial value, start, according to requirement.
return ignite.atomicLong(
"id", // name.
start, // Initial value
true // Create if it does not exist.
);
But the process got stuck as well.
Shall I do something differently? Appreciate your advice.
Best,
Jessie
On Tue, Jul 25, 2017 at 7:55 PM, Jessie Lin <je...@gmail.com>
wrote:
> Thank you Val. I'll give it a try.
>
> On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Jessie,
>>
>> atomicLong() method will do this for you automatically. It either creates
>> new instance or returns existing one. There is no need for separate
>> initialization step.
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Streaming-test-tp14039p15646.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
Jessie,
If it's a preprocessing done on client side, then I think it should happen
on application level. Do required preprocessing, convert to Ignite's data
model and then do cache operations.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16349.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Hi Val, thank you getting back. The source type will be different, and the
client will deal with the difference in a preprocess that's unique to that
data source. Once that's done, it'll be ready for a standard
transformation. So we can encapsulate it in a single service or callable.
Thanks for explaining the difference between client and servers. It
clarifies things and make sense.
We wanted to build different type of clients to accommodate the differences
in the data sources.
Does this make sense? what kind of solutions would you suggest?
Jessie
On Fri, Aug 18, 2017 at 3:34 PM, vkulichenko <va...@gmail.com>
wrote:
> If data sources are different, then transformations are different as well,
> no? If so, how are you going to encapsulate them in a single service? I
> would really appreciate a concrete example, it might make this much easier.
> I'm probably missing something now.
>
> Difference between clients and servers is the same is in other systems.
> Servers store data, run services and computations, etc. while clients are
> only used to interact with the cluster.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16310.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
If data sources are different, then transformations are different as well,
no? If so, how are you going to encapsulate them in a single service? I
would really appreciate a concrete example, it might make this much easier.
I'm probably missing something now.
Difference between clients and servers is the same is in other systems.
Servers store data, run services and computations, etc. while clients are
only used to interact with the cluster.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16310.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
We'd like to take either input from streaming or a cache to do some
transformation and insert data into another cache.
Because the input could be different types, I thought I could encapsulate
the transformation and just let a client to call it.
The difference between client and server in Ignite isn't very clear to me,
to be honest :)
One thing I can think of is client doesn't hold data. But I don't know the
difference in terms of computation.
Jessie
On Thu, Aug 17, 2017 at 3:56 PM, vkulichenko <va...@gmail.com>
wrote:
> What does business logic do exactly? Is it supposed to be executed on
> server
> or client side?
>
> Can you give an example?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16270.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
What does business logic do exactly? Is it supposed to be executed on server
or client side?
Can you give an example?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16270.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Val, I'd like to build a piece of logic ( could be service, callable, etc)
and encapsulate business logic there. And build different clients to for
different type of sources, for example file, streaming, jdbc, etc, the
client will invoke the business logic by calling the service or callables.
What architecture/components would you recommend to use in this type of
situation?
Thank you very much!
Jessie
On Tue, Aug 15, 2017 at 5:28 PM, vkulichenko <va...@gmail.com>
wrote:
> Can you describe the use case and what you want to achieve, step by step?
> I'll try to suggest a solution.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16213.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
Can you describe the use case and what you want to achieve, step by step?
I'll try to suggest a solution.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16213.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Btw, I tried two ways to get service proxy on a client node, the first one
worked fine, but the one with Stream Visitor failed.
1) I stream the file and put them into a cache, and then loop through the
cache entries and call Service
2) I stream the file and define a stream visitor, in which I get the
key(String) and value(a defined class) from the Map<K,V> and call the
service. The service interface is defined as String, the defined class.
Kinda interesting... Thank you for helping out!
On Tue, Aug 15, 2017 at 10:07 AM, Jessie Lin <je...@gmail.com>
wrote:
> The class gets a service proxy and invokes the service.
> I thought service could be invoked on client as well...Looks like it's not
> designed to be used that way.
>
> If I wrap the Service in a Callable, can I call it from a client node?
> The client node would stream from a local file and call the Callable, I
> think?
> Does that sound right?
>
> Jessie
>
> On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> What is this class used for? I'm a bit confused - service is usually
>> invoked
>> on the server, so it will obviously need all classes it works with. Am I
>> missing something?
>>
>> Can you show the whole trace?
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Streaming-test-tp14039p16189.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
The class gets a service proxy and invokes the service.
I thought service could be invoked on client as well...Looks like it's not
designed to be used that way.
If I wrap the Service in a Callable, can I call it from a client node?
The client node would stream from a local file and call the Callable, I
think?
Does that sound right?
Jessie
On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko <va...@gmail.com>
wrote:
> What is this class used for? I'm a bit confused - service is usually
> invoked
> on the server, so it will obviously need all classes it works with. Am I
> missing something?
>
> Can you show the whole trace?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16189.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
What is this class used for? I'm a bit confused - service is usually invoked
on the server, so it will obviously need all classes it works with. Am I
missing something?
Can you show the whole trace?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16189.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by waterg <je...@gmail.com>.
Hello Val,
I now put the stream visitor code in a different model and packaged it in a
different jar with dependencies.
Interestingly, when I ran the program again, the server node has weird error
messages.
The FileConnector is a class in the client jar. Why would the service needs
the caller's signature?
Btw, I turned enablePeerClassLoader on, but I still have the same error.
Jessie
Caused by: java.lang.ClassNotFoundException: com.sample.client.FileConnector
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[?:1.8.0_
91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
~[?:1.8.0_91]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
~[?:1.8
.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
~[?:1.8.0_91]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_91]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_91]
at
org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:
8478) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar:?]
at
org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerC
ontextImpl.java:340) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar:
?]
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUt
ils.classDescriptor(OptimizedMarshallerUtils.java:268) ~[sample-client:?]
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16188.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Val, thanks for pointing it out. Now I call AtomicLong Function from
service#execute() and it's working. Thank you very much!
Jessie
On Thu, Aug 10, 2017 at 3:08 PM, vkulichenko <va...@gmail.com>
wrote:
> Jessie,
>
> You still call atomicLong() method from Service#init(). As I already
> mentioned, this is causing the startup hang. You should move
> IgniteAtomicLong creation out of init() method to avoid it.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16113.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
Jessie,
You still call atomicLong() method from Service#init(). As I already
mentioned, this is causing the startup hang. You should move
IgniteAtomicLong creation out of init() method to avoid it.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16113.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Val, please see thread print attached.
This is take after a server is run by "bin\ignite.bat
config\ignite-writebehind.xml" and the service initialization didn't
complete.
Thank you very much for helping out!
"srvc-deploy-#33%null%" #59 prio=5 os_prio=0 tid=0x00000000577b8000
nid=0x1ef8 waiting on condition [0x00000000608fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c09fbdc8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7419)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.awaitInitialization(DataStructuresProcessor.java:1112)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:517)
at
org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3436)
at com.sample.SampleServiceImpl.init(SampleServiceImpl.java:63)
Jessie
On Wed, Aug 9, 2017 at 2:29 PM, vkulichenko <va...@gmail.com>
wrote:
> I can't reproduce it, your project works fine for me. Can you attach thread
> dumps?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16088.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
I can't reproduce it, your project works fine for me. Can you attach thread
dumps?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16088.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Posted by Jessie Lin <je...@gmail.com>.
Val, I think the ignite.atomicLong() function never returned and just hang
there. If you run the test project I attached it should be reproducable.
I don't expect to have memory issue here. As the service init doesn't do
much else. And I only have one server node and one client node.
Jessie
On Tue, Aug 8, 2017 at 5:36 PM, vkulichenko <va...@gmail.com>
wrote:
> Jessie,
>
> What do you mean by "stuck"? Did you check thread dumps? Is it possible you
> have memory/GC issues?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Streaming-test-tp14039p16066.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Streaming test
Posted by vkulichenko <va...@gmail.com>.
Jessie,
What do you mean by "stuck"? Did you check thread dumps? Is it possible you
have memory/GC issues?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.