Computation best practices

[luqmanahmad <lu...@gmail.com>]

Hi there,

Just trying to clarify few bits in my head around data nodes and compute
nodes.

Let's say we have 10 data nodes which are solely storing the data using
affinity collocation and we have 10 compute nodes as well for computing
different tasks on the cluster.

Now we know that if we want to perform some operation on the data and we
know where it resides we can use the affinity API to perform some operation
on it which is indeed much better as there would be no data movement across
the node which makes sense as well. But then on the other side, we have got
compute nodes as well which are just sitting idle. Although we have got the
luxury of using distributed closures then wouldn't it be an overhead of
carrying all the data to a compute node to perform and then sending the data
back.

Just trying to find a use case where the separate group of the cluster could
be useful. For example data-node, compute-node etc. If anyone can clear this
would be much appreciated.

Thanks,
Luqman




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Computation best practices

[ihorps <ih...@gmail.com>]

hi @luqmanahmad
I was thinking about it as well a little bit in my project... and I'm not
sure if the cluster group is the best direction here. One way to think
(probably) about efficient resource usage is to bring job stealing into your
cluster. In this case you do a "default" setup where data collocation is
used and if there are nodes, which are just idle - then you can start data
transferring and bring calculating in there.
But you would have to do evaluation(leverage) of your real data distribution
about the best setup of job stealing vs collocated job calculations.

Best regards,
Ihor P.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Computation best practices

[luqmanahmad <lu...@gmail.com>]

Hi Christos,

Thanks for getting back to me. 

There is no specific need for it - I am just trying to get my head around
for the best practices in that case data nodes does make sense to me to
perform the computation. To be honest just a bit confused with the Cluster
groups and trying to find a use case for it (apart from collecting metrics
and broadcasting). Cluster API does make sense and already using it but then
stuck with groups where it can be useful or maybe I am thinking too much :)

Thanks,
Luqman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Computation best practices

[Christos Erotocritou <ch...@gridgain.com>]

Hi Luqman,

Is there a specific reason why you want to keep the data nodes separate from the compute nodes?
As you say this beats the point of collocation. You should use the data nodes for compute and ensure you have a way to monitor and kill spurious tasks that may be executed on the grid.

C.

> On 13 Nov 2017, at 11:50, luqmanahmad <lu...@gmail.com> wrote:
> 
> Hi there,
> 
> Just trying to clarify few bits in my head around data nodes and compute
> nodes.
> 
> Let's say we have 10 data nodes which are solely storing the data using
> affinity collocation and we have 10 compute nodes as well for computing
> different tasks on the cluster.
> 
> Now we know that if we want to perform some operation on the data and we
> know where it resides we can use the affinity API to perform some operation
> on it which is indeed much better as there would be no data movement across
> the node which makes sense as well. But then on the other side, we have got
> compute nodes as well which are just sitting idle. Although we have got the
> luxury of using distributed closures then wouldn't it be an overhead of
> carrying all the data to a compute node to perform and then sending the data
> back.
> 
> Just trying to find a use case where the separate group of the cluster could
> be useful. For example data-node, compute-node etc. If anyone can clear this
> would be much appreciated.
> 
> Thanks,
> Luqman
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/