Container settings at vertex level

[Madhusudan Ramanna <m....@ymail.com>]

Hello Folks,
Some vertices require more memory than other vertices. These vertices are memory intensive.  The graph, in general, takes a long(ish) time to complete.  Default allocation of a huge chunk of memory to this one DAG/application severely limits concurrent yarn containers that can be run.  How can we influence Tez Runtime to request and execute some vertices in specialized containers ? What is a good solution to this problem?
thanks,Madhu

Re: Container settings at vertex level

[Hitesh Shah <hi...@apache.org>]

Hello Madhu,

If you are using Tez via Hive, then this would need a fix in Hive. I don’t believe Hive supports different settings for each vertex in a given query today.

However, for native jobs, Tez already supports different specs for each vertex:

Vertex::setTaskResource() ( configuring yarn resources i.e. memory/cpu )
Vertex::setTaskLaunchCmdOpts() ( java opts, etc )

Does the above help? Or are you looking for something different? 

thanks
— HItesh


> On Oct 20, 2016, at 10:44 AM, Madhusudan Ramanna <m....@ymail.com> wrote:
> 
> Hello Folks,
> 
> Some vertices require more memory than other vertices. These vertices are memory intensive.  The graph, in general, takes a long(ish) time to complete.  Default allocation of a huge chunk of memory to this one DAG/application severely limits concurrent yarn containers that can be run.  How can we influence Tez Runtime to request and execute some vertices in specialized containers ? What is a good solution to this problem?
> 
> thanks,
> Madhu