You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by alpha centauri <he...@hotmail.com> on 2015/12/06 01:51:27 UTC
RE: From one cache to another
Hello,
Thanks for your answers, guys.
It works now but I struggled a lot with the serialization process.
Maybe somewhere strongly underline the fact in the documentation that ONLY certain type of classes (I think it's closures and some other
inner tasks) are part of the zero deployment mechanisms. For like two hours, I didn't understood why any object inside my tasks would not be automagically deployed on the cluster nodes.
It is still a bit annoying. Once I create a new processor, I have to restart all my nodes to have it properly deployed everywhere. Is there something I'm missing ?
Anyway thanks, Ignite is a really powerfull thing!
Date: Fri, 27 Nov 2015 13:55:08 +0300
Subject: Re: From one cache to another
From: agura@gridgain.com
To: user@ignite.apache.org
Hi,
You should not serialize cache instances. Actually your cache is already created on all nodes of your cluster and you can obtain it using ignite.cache(<name>) or ignite.getOrCreateCache(<name>) methods.
I assume that you try to start Ignite instance with default name from your ComplexObjectProcessor but Ignite is started already. You can declare Ignite reference in ComplexObjectProcessor and annotate it with @IgniteInstanceResource. When your processor will be deserialized on remote node Ignite instance will be injected automatically. It looks like this:
public class ComplexObjectProcessor {
@IgniteInstanceResource private Ignite ignite
...
}
Make sure that your ComplexObjectProcessor is regular class or inner static class. Non-static inner class instances have reference to enclosing class that will be serialized with inner class.
On Thu, Nov 26, 2015 at 9:08 PM, alpha centauri <he...@hotmail.com> wrote:
Hello,
I'm a new Apache Ignite User. I'm trying to prototype a simple use case :
1. create two caches
2. read strings from a file and put them in the first cache
3. compute strings into complex objects put the complex objects into second cache
My code looks like this :
// Part 1
final IgniteCache<Integer, String> cacheString = ignite.getOrCreateCache("somestrings");
final IgniteCache<String, SomeObjects> cache = ignite.getOrCreateCache("someobjects");
// Part 2
try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer(cacheString.getName())) {
stmr.allowOverwrite(true);
try (BufferedReader br = Files.newReader(file, StandardCharsets.UTF_8)) {
for (String line; (line = br.readLine()) != null; ) {
if (line !=null && line.length() > 0) {
stmr.addData(index, line);
}
}
}
}
// Part 3 is where I struggle and need help
IgniteCompute compute = ignite.compute();
for (int i = 0; i < cacheString.size(); i++) {
final int key = i;
compute.affinityRun(cacheString.getName(), key, new ComplexObjectProcessor(cache, cacheString.get(key)));
}
The issue is that the ComplexObjectProcessor is serialized to my grid nodes and if I put the destination cache, it will crash because the cache cannot be serialized. If I initialize the cache from my ComplexObjectProcessor, it will complain about the fact that Ignite has already been instanciated outside. ComplexObjectProcessor has a lot of code so an inside/inline class is not an option.
The cacheString contains 100Millions lines so I really want to split the complex object processor with compute tasks.
Any solution? Or code sample would greatly help.
Thks ahead,
AC
--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com
Re: From one cache to another
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
Actually if a processor is being deployed on a remote node all other
classes, the processor refers to from its code, are deployed as well.
In my understanding you should take into account the deployment mode
that is used on your side.
By default SHARED mode is activated which means for you that if you had
several nodes that were sending the processor via compute.affinityRun
(or other method) then either all of them has to be restarted once the
processor is modified (no need to restart the nodes that executed the
task and deployed the processor class remotely) or restart only a single
node that modified the processor but provide updated class version in
META-INF/ignite.xml. Please refer to DeploymentMode.SHARED documentation
for more info.
--
Denis
On 12/6/2015 3:51 AM, alpha centauri wrote:
>
> Hello,
>
> Thanks for your answers, guys.
>
> It works now but I struggled a lot with the serialization process.
>
> Maybe somewhere strongly underline the fact in the documentation that
> ONLY certain type of classes (I think it's closures and some other
> inner tasks) are part of the zero deployment mechanisms. For like two
> hours, I didn't understood why any object inside my tasks would not be
> automagically deployed on the cluster nodes.
>
> It is still a bit annoying. Once I create a new processor, I have to
> restart all my nodes to have it properly deployed everywhere. Is there
> something I'm missing ?
>
> Anyway thanks, Ignite is a really powerfull thing!
>
>
>
> ------------------------------------------------------------------------
> Date: Fri, 27 Nov 2015 13:55:08 +0300
> Subject: Re: From one cache to another
> From: agura@gridgain.com
> To: user@ignite.apache.org
>
> Hi,
>
> You should not serialize cache instances. Actually your cache is
> already created on all nodes of your cluster and you can obtain it
> using ignite.cache(<name>) or ignite.getOrCreateCache(<name>) methods.
>
> I assume that you try to start Ignite instance with default name from
> your ComplexObjectProcessor but Ignite is started already. You can
> declare Ignite reference in ComplexObjectProcessor and annotate it
> with @IgniteInstanceResource. When your processor will be deserialized
> on remote node Ignite instance will be injected automatically. It
> looks like this:
>
> public class ComplexObjectProcessor {
>
> @IgniteInstanceResource
> private Ignite ignite
>
> ...
> }
>
> Make sure that your ComplexObjectProcessor is regular class or inner
> static class. Non-static inner class instances have reference to
> enclosing class that will be serialized with inner class.
>
>
> On Thu, Nov 26, 2015 at 9:08 PM, alpha centauri <helium667@hotmail.com
> <ma...@hotmail.com>> wrote:
>
> Hello,
>
> I'm a new Apache Ignite User. I'm trying to prototype a simple use
> case :
> 1. create two caches
> 2. read strings from a file and put them in the first cache
> 3. compute strings into complex objects put the complex objects
> into second cache
>
> My code looks like this :
>
> // Part 1
> final IgniteCache<Integer, String> cacheString =
> ignite.getOrCreateCache("somestrings");
> final IgniteCache<String, SomeObjects> cache =
> ignite.getOrCreateCache("someobjects");
>
> // Part 2
> try (IgniteDataStreamer<Integer, String> stmr =
> ignite.dataStreamer(cacheString.getName())) {
> stmr.allowOverwrite(true);
> try (BufferedReader br = Files.newReader(file,
> StandardCharsets.UTF_8)) {
> for (String line; (line = br.readLine()) !=
> null; ) {
> if (line !=null && line.length() > 0) {
> stmr.addData(index, line);
> }
> }
>
> }
> }
>
> // Part 3 is where I struggle and need help
> IgniteCompute compute = ignite.compute();
>
> for (int i = 0; i < cacheString.size(); i++) {
> final int key = i;
> compute.affinityRun(cacheString.getName(), key, new
> ComplexObjectProcessor(cache, cacheString.get(key)));
> }
>
> The issue is that the ComplexObjectProcessor is serialized to my
> grid nodes and if I put the destination cache, it will crash
> because the cache cannot be serialized. If I initialize the cache
> from my ComplexObjectProcessor, it will complain about the fact
> that Ignite has already been instanciated outside.
> ComplexObjectProcessor has a lot of code so an inside/inline class
> is not an option.
> The cacheString contains 100Millions lines so I really want to
> split the complex object processor with compute tasks.
>
> Any solution? Or code sample would greatly help.
>
> Thks ahead,
> AC
>
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com <http://www.gridgain.com/>