Qus: Yarn container in RM

[Aakash Sharma <as...@gmail.com>]

Hello Yarn Gurus,

I am working on a YARN enhancement for optimizing utilization in AWS.
Is it possible to classify a container in resource manager as a Map or
Reduce task? I have looked at the source code of capacity scheduler, and I
dont think it is possible.
If I were to make some placement optimizations in RM to run certain reduce
tasks on a certain type of nodes, without using node labeling, how can I go
about doing this?

Can anyone please give me some suggestions?

-- 
Thanks,
Aakash

Re: Qus: Yarn container in RM

[Hariharan Iyer <hi...@qubole.com>]

It should be possible to do this using placement constraints
<https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html>,
though you may need changes in mapreduce to add allocationTags "map" and
"reduce" to the containers.

Thanks,
Hariharan

On Wed, Dec 18, 2019 at 2:04 PM Aakash Sharma <as...@gmail.com> wrote:

> Hello Yarn Gurus,
>
> I am working on a YARN enhancement for optimizing utilization in AWS.
> Is it possible to classify a container in resource manager as a Map or
> Reduce task? I have looked at the source code of capacity scheduler, and I
> dont think it is possible.
> If I were to make some placement optimizations in RM to run certain reduce
> tasks on a certain type of nodes, without using node labeling, how can I go
> about doing this?
>
> Can anyone please give me some suggestions?
>
> --
> Thanks,
> Aakash
>

Re: Qus: Yarn container in RM

[zhankun tang <ta...@gmail.com>]

Hi Aakash,

This is an interesting topic. Now the scheduler is not aware of the
underlying hardware differences.
In my understanding, the placement constraints or label or node
attributes together with application-level(MR/Spark/Hive to be cloud
instance aware) changes could potentially achieve your goal.

BR,
Zhankun

On Wed, 18 Dec 2019 at 16:34, Aakash Sharma <as...@gmail.com> wrote:

> Hello Yarn Gurus,
>
> I am working on a YARN enhancement for optimizing utilization in AWS.
> Is it possible to classify a container in resource manager as a Map or
> Reduce task? I have looked at the source code of capacity scheduler, and I
> dont think it is possible.
> If I were to make some placement optimizations in RM to run certain reduce
> tasks on a certain type of nodes, without using node labeling, how can I go
> about doing this?
>
> Can anyone please give me some suggestions?
>
> --
> Thanks,
> Aakash
>