Continuous reduce in Ignite

[Raymond Wilson <ra...@trimble.com>]

I’m looking at the very nice MapReduce implementation in Ignite and have a
question about the reduce stage.



It appears that the reduce stage only occurs once all mapped results have
been accumulated. I have a use case where the size of that data set is very
large and problematic to hold in memory until all mapped results are
returned.



Is it possible to do a continuous Reduce in Ignite, similar to the way it
is possible to do a continuous Map?



I can see one possible way of doing this by slurping off the responses as
they are provided to OnResult, but I’m wary of mutating the list provided
to that method to pull mapped results out of the list being accumulated.



Thanks,

Raymond.

Re: Continuous reduce in Ignite

[Raymond Wilson <ra...@trimble.com>]

Hi Val,

This would be from .Net. Is there an option for extending the .Net client to support this?

Thanks,
Raymond. 

Sent from my iPhone

> On 11/02/2017, at 11:19 AM, vkulichenko <va...@gmail.com> wrote:
> 
> Hi Raymond,
> 
> Are you implementing tasks in Java or .NET? For Java there is a special
> @ComputeTaskNoResultCache annotation that prevents from caching all job
> results in a single collection. Not sure there is such an option for .NET
> though.
> 
> -Val
> 
> 
> 
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Continuous-reduce-in-Ignite-tp10552p10553.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: Continuous reduce in Ignite

[Raymond Wilson <ra...@trimble.com>]

It turns out this annotation is also present in .Net and works the same
way as the Java annotation.

Eg:
    [Serializable]
    [ComputeTaskNoResultCache]
    class MyComputeTask : IComputeTask<String, String, String>
{
}

Sweet! :)

-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Saturday, February 11, 2017 11:19 AM
To: user@ignite.apache.org
Subject: Re: Continuous reduce in Ignite

Hi Raymond,

Are you implementing tasks in Java or .NET? For Java there is a special
@ComputeTaskNoResultCache annotation that prevents from caching all job
results in a single collection. Not sure there is such an option for .NET
though.

-Val



--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Continuous-reduce-in-Ignite
-tp10552p10553.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Continuous reduce in Ignite

[vkulichenko <va...@gmail.com>]

Hi Raymond,

Are you implementing tasks in Java or .NET? For Java there is a special
@ComputeTaskNoResultCache annotation that prevents from caching all job
results in a single collection. Not sure there is such an option for .NET
though.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Continuous-reduce-in-Ignite-tp10552p10553.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.