granular configuration of cluster

[Stefano Bortoli <s....@gmail.com>]

Hi all,

is it possible to configure a cluster with different number of task manager
per node?

for example, I have a cluster where I am using a global index, and the
nodes where the global index in running should not be overloaded. However,
the other nodes could run more than 4 task manager per node.

To have such features would be very helpful to fine tune processes on small
clusters.

thanks a lot for your support!

saluti,
Stefano

Re: granular configuration of cluster

[Stephan Ewen <se...@apache.org>]

Yes, each task manager offers a certain number of "task slots" to the
JobManager. That is what you see in the web frontend also. Each TaskManager
can offer a different number of slots, there is no restriction there.

Each slot takes an operator (or a pipeline of operators). So, if a task
manager offers 4 slots, it can execute 4 parallel mappers, reducers, ... of
the same type concurrently. A task manager that offers 8 slots can execute
8 concurrently, ...

On Fri, Nov 28, 2014 at 5:40 PM, Stefano Bortoli <s....@gmail.com>
wrote:

> sweet! so it does not have to be consistent across all nodes! then each
> taskmanager will report to the job manager, that will understand the
> distribution of the work accordingly. I guess.
>
> thanks a lot!
>
> 2014-11-28 17:12 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Sure that's possible. You need different configurations for each node, so
>> instead of having one config directory shared across all nodes, give each
>> node an individual config.
>>
>> The nodes that should do more work need to see a config with a larger JVM
>> heap and more task slots, the nodes that should have fewer load need to
>> have a config with smaller heap and fewer slots.
>>
>> On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <s....@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> is it possible to configure a cluster with different number of task
>>> manager per node?
>>>
>>> for example, I have a cluster where I am using a global index, and the
>>> nodes where the global index in running should not be overloaded. However,
>>> the other nodes could run more than 4 task manager per node.
>>>
>>> To have such features would be very helpful to fine tune processes on
>>> small clusters.
>>>
>>> thanks a lot for your support!
>>>
>>> saluti,
>>> Stefano
>>>
>>
>>
>

Re: granular configuration of cluster

[Stefano Bortoli <s....@gmail.com>]

sweet! so it does not have to be consistent across all nodes! then each
taskmanager will report to the job manager, that will understand the
distribution of the work accordingly. I guess.

thanks a lot!

2014-11-28 17:12 GMT+01:00 Stephan Ewen <se...@apache.org>:

> Sure that's possible. You need different configurations for each node, so
> instead of having one config directory shared across all nodes, give each
> node an individual config.
>
> The nodes that should do more work need to see a config with a larger JVM
> heap and more task slots, the nodes that should have fewer load need to
> have a config with smaller heap and fewer slots.
>
> On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <s....@gmail.com>
> wrote:
>
>> Hi all,
>>
>> is it possible to configure a cluster with different number of task
>> manager per node?
>>
>> for example, I have a cluster where I am using a global index, and the
>> nodes where the global index in running should not be overloaded. However,
>> the other nodes could run more than 4 task manager per node.
>>
>> To have such features would be very helpful to fine tune processes on
>> small clusters.
>>
>> thanks a lot for your support!
>>
>> saluti,
>> Stefano
>>
>
>

Re: granular configuration of cluster

[Stephan Ewen <se...@apache.org>]

Sure that's possible. You need different configurations for each node, so
instead of having one config directory shared across all nodes, give each
node an individual config.

The nodes that should do more work need to see a config with a larger JVM
heap and more task slots, the nodes that should have fewer load need to
have a config with smaller heap and fewer slots.

On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <s....@gmail.com>
wrote:

> Hi all,
>
> is it possible to configure a cluster with different number of task
> manager per node?
>
> for example, I have a cluster where I am using a global index, and the
> nodes where the global index in running should not be overloaded. However,
> the other nodes could run more than 4 task manager per node.
>
> To have such features would be very helpful to fine tune processes on
> small clusters.
>
> thanks a lot for your support!
>
> saluti,
> Stefano
>