container size

[Grandl Robert <rg...@yahoo.com>]

Hi guys,

It seems the size of a container allocated to a task in Tez is static ? Please correct me if I am wrong.


Is there a way such that Tez is dynamically choosing the task container size based on various things ? (like a map or reduce vertex, size of data to process, etc.)

At least for the tpcds/h queries I am running it looks containers are fixed size. 

Thanks,
Robert

Re: container size

[Grandl Robert <rg...@yahoo.com>]

Thank you guys for your answers. Really helpful.

@Bikas: Nice that Tez has these APIs to define container size. 


@Hitesh: Yeah, I think Hive is using a single size for all its vertices, at least this happens for all the queries I am running, unfortunately (or good for my scenario). 


 Thanks,
Robert



On Tuesday, September 9, 2014 9:32 PM, Hitesh Shah <hi...@apache.org> wrote:
 


Hi Robert, 

From a Tez point of view, a user of the Tez APIs can define the container size on a per vertex basis. I believe currently, Hive, when using Tez, uses a single size for all its vertices. 

thanks
— Hitesh


On Sep 9, 2014, at 6:37 PM, Grandl Robert <rg...@yahoo.com> wrote:

> Hi guys,
> 
> It seems the size of a container allocated to a task in Tez is static ? Please correct me if I am wrong.
> 
> Is there a way such that Tez is dynamically choosing the task container size based on various things ? (like a map or reduce vertex, size of data to process, etc.)
> 
> At least for the tpcds/h queries I am running it looks containers are fixed size. 
> 
> Thanks,
> Robert

Re: container size

[Hitesh Shah <hi...@apache.org>]

Hi Robert, 

From a Tez point of view, a user of the Tez APIs can define the container size on a per vertex basis. I believe currently, Hive, when using Tez, uses a single size for all its vertices. 

thanks
— Hitesh

On Sep 9, 2014, at 6:37 PM, Grandl Robert <rg...@yahoo.com> wrote:

> Hi guys,
> 
> It seems the size of a container allocated to a task in Tez is static ? Please correct me if I am wrong.
> 
> Is there a way such that Tez is dynamically choosing the task container size based on various things ? (like a map or reduce vertex, size of data to process, etc.)
> 
> At least for the tpcds/h queries I am running it looks containers are fixed size. 
> 
> Thanks,
> Robert

RE: container size

[Bikas Saha <bi...@hortonworks.com>]

The container size is determined at compile time when creating the DAG.
There is black box user code running inside a task and so Tez cannot
determine a good size since it does not know whats running.



When tasks actually run in a container then Tez has APIs by which it tries
to divide a tasks memory resources among inputs/outputs and processor. At
that point inputs may use data size to make an ask for a % of the memory.



*From:* Grandl Robert [mailto:rgrandl@yahoo.com]
*Sent:* Tuesday, September 09, 2014 6:38 PM
*To:* user@tez.incubator.apache.org
*Subject:* container size



Hi guys,



It seems the size of a container allocated to a task in Tez is static ?
Please correct me if I am wrong.



Is there a way such that Tez is dynamically choosing the task container
size based on various things ? (like a map or reduce vertex, size of data
to process, etc.)



At least for the tpcds/h queries I am running it looks containers are fixed
size.



Thanks,

Robert

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.