You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by gsaxena888 <gs...@gmail.com> on 2018/10/30 19:02:27 UTC
Comparison between java 8 streams functionality and Apache Ignite
I'm new to Apache Ignite, but a long-time user of jdk8 streams. (And I've
used Google Cloud Dataflow.) I'm trying to understand the example described
in latest doc:
https://apacheignite.readme.io/docs/distributed-closures#apply-methods
<https://apacheignite.readme.io/docs/distributed-closures#apply-methods>
and comparing it to jdk8 stream functionality.
Here's the code snippet from the Ignite doc:
```
IgniteCompute compute = ignite.compute();
// Execute closure on all cluster nodes.
Collection<Integer> res = compute.apply(
String::length,
Arrays.asList("How many characters".split(" "))
);
// Add all the word lengths received from cluster nodes.
int total = res.stream().mapToInt(Integer::intValue).sum();
```
My question is: does the "res.stream.mapToInt()" *also* supposed to run on
*all* nodes in the cluster or does it only run only on the local node? (The
doc clealy specifies that the "compute.apply" runs on all nodes, but is
silent on the res.stream() portion.) I'm assuming that Ignite works
similarily to Infinispan, which I *think* will run any streaming logic (even
ones with custom collectors) on *all* nodes (see
https://blog.infinispan.org/2018/01/improving-collect-for-distributed-java.html
<https://blog.infinispan.org/2018/01/improving-collect-for-distributed-java.html>
)
(Also, assuming the res.stream.mapToInt... logic runs on all nodes, is it
reasonable to assume that *within* each node it will run in parallel, and
that the number of threads *within* each node is equal to the number of
cores for that node?)
Also, if nodes differ in number of cores (or in current utilization), does
Ingnite take that into when distributing the work? If so, I'm assuming in
the above example that it makes that decision *not* in the res.stream() part
but instead in the *initial* "compute.apply" section, and that it then tries
to maintain the data partitioning?
Finally, assuming that res.stream.mapToInt() does run on all nodes, would my
existing jdk8 custom collectors (that follow the jdk8 streaming api for
collectors) also *automatically* work on all nodes or would I need to do
some minor refactoring as implied by the Infinispan doc (by using
Infinispan's "SerializableSupplier<Collector<T, ?, R>>." instead of just
"Supplier<...>"? (In particular, is Ignite smart enough to first do the
collection *within* a node, and then start collecting across nodes? And,
although this is not that important for my needs, is Ignite even smart
enough to do the collection in a "logical" order between nodes, ie collect
first within a node, then collect between "close" nodes, then do the final
collection between the furthest nodes (eg on different racks and then next
even from different data centers)? (This way, we minimize data transfer
etc?))
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Comparison between java 8 streams functionality and Apache Ignite
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I would believe that reduce may run multiple times as you have suggested.
Regards,
--
Ilya Kasnacheev
чт, 1 нояб. 2018 г. в 5:48, gsaxena888 <gs...@gmail.com>:
> I've been thinking about this some more: I think the ignite solution is
> nearly perfect, *if* the reduce operation runs within every node (so that,
> for example, the results of ~96 threads on one google compute enginer were
> reduced/summarized to a single value) and then either a single final
> reduction occurrs on one node (eg a leader node?) or, if we want to get
> fancy, the last reduction can occur in parallel accross the nodes (but it's
> not clear whether doing a parallel reduction accross al the nodes is a
> performant in the majority of real-life cases given the overhead of doing
> so). So the question is: does the "reduce" operation run first within each
> node? If not, wouldn't it make sense to do so (to minimize transferring
> data
> to a single node to do a giant reduction and to also make good use of all
> the nodes' cores?) And if not, is there an alternative way of achieving
> this
> in Ignite (ie what do developers do?)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Comparison between java 8 streams functionality and Apache
Ignite
Posted by gsaxena888 <gs...@gmail.com>.
I've been thinking about this some more: I think the ignite solution is
nearly perfect, *if* the reduce operation runs within every node (so that,
for example, the results of ~96 threads on one google compute enginer were
reduced/summarized to a single value) and then either a single final
reduction occurrs on one node (eg a leader node?) or, if we want to get
fancy, the last reduction can occur in parallel accross the nodes (but it's
not clear whether doing a parallel reduction accross al the nodes is a
performant in the majority of real-life cases given the overhead of doing
so). So the question is: does the "reduce" operation run first within each
node? If not, wouldn't it make sense to do so (to minimize transferring data
to a single node to do a giant reduction and to also make good use of all
the nodes' cores?) And if not, is there an alternative way of achieving this
in Ignite (ie what do developers do?)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Comparison between java 8 streams functionality and Apache
Ignite
Posted by gsaxena888 <gs...@gmail.com>.
Ahh, I see. But still, even with "map reduce" stratedgy, I *think* that there
is only a SINGLE node which will do the reduction, right? As in, the
reduction will NOT occur in parallel across nodes, right? (In fact, it
sounds like the reduction won't even occur in parallel *within* a node,
right?) If correct, are there any plans to change this to be more like
Infinispan (see this article from 2014:
https://blog.infinispan.org/2014/02/mapreduce-parallel-execution.html -- I
think they've made improvements since then.) Also, could this single-reducer
explain why one user complained about slow group-by? (see here:
http://apache-ignite-users.70518.x6.nabble.com/Slow-Group-By-td20236.html )
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Comparison between java 8 streams functionality and Apache Ignite
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
String::length will be run on all nodes, but Integer::intValue will be run
locally.
If you want it to be smarter than that, you could use MapReduce & ForkJoin:
https://apacheignite.readme.io/docs/compute-tasks
Regards,
--
Ilya Kasnacheev
вт, 30 окт. 2018 г. в 22:02, gsaxena888 <gs...@gmail.com>:
> I'm new to Apache Ignite, but a long-time user of jdk8 streams. (And I've
> used Google Cloud Dataflow.) I'm trying to understand the example described
> in latest doc:
> https://apacheignite.readme.io/docs/distributed-closures#apply-methods
> <https://apacheignite.readme.io/docs/distributed-closures#apply-methods>
> and comparing it to jdk8 stream functionality.
>
> Here's the code snippet from the Ignite doc:
> ```
> IgniteCompute compute = ignite.compute();
>
> // Execute closure on all cluster nodes.
> Collection<Integer> res = compute.apply(
> String::length,
> Arrays.asList("How many characters".split(" "))
> );
>
> // Add all the word lengths received from cluster nodes.
> int total = res.stream().mapToInt(Integer::intValue).sum();
> ```
>
> My question is: does the "res.stream.mapToInt()" *also* supposed to run on
> *all* nodes in the cluster or does it only run only on the local node? (The
> doc clealy specifies that the "compute.apply" runs on all nodes, but is
> silent on the res.stream() portion.) I'm assuming that Ignite works
> similarily to Infinispan, which I *think* will run any streaming logic
> (even
> ones with custom collectors) on *all* nodes (see
>
> https://blog.infinispan.org/2018/01/improving-collect-for-distributed-java.html
> <
> https://blog.infinispan.org/2018/01/improving-collect-for-distributed-java.html>
>
> )
>
> (Also, assuming the res.stream.mapToInt... logic runs on all nodes, is it
> reasonable to assume that *within* each node it will run in parallel, and
> that the number of threads *within* each node is equal to the number of
> cores for that node?)
>
> Also, if nodes differ in number of cores (or in current utilization), does
> Ingnite take that into when distributing the work? If so, I'm assuming in
> the above example that it makes that decision *not* in the res.stream()
> part
> but instead in the *initial* "compute.apply" section, and that it then
> tries
> to maintain the data partitioning?
>
> Finally, assuming that res.stream.mapToInt() does run on all nodes, would
> my
> existing jdk8 custom collectors (that follow the jdk8 streaming api for
> collectors) also *automatically* work on all nodes or would I need to do
> some minor refactoring as implied by the Infinispan doc (by using
> Infinispan's "SerializableSupplier<Collector<T, ?, R>>." instead of just
> "Supplier<...>"? (In particular, is Ignite smart enough to first do the
> collection *within* a node, and then start collecting across nodes? And,
> although this is not that important for my needs, is Ignite even smart
> enough to do the collection in a "logical" order between nodes, ie collect
> first within a node, then collect between "close" nodes, then do the final
> collection between the furthest nodes (eg on different racks and then next
> even from different data centers)? (This way, we minimize data transfer
> etc?))
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>