What is recommended memory setting for tez.am and tez task?

[Alexander Pivovarov <ap...@gmail.com>]

Hi Everyone

What is recommended value for

tez.am.resource.memory.mb

tez.task.resource.memory.mb


Thank you

Re: What is recommended memory setting for tez.am and tez task?

[Hitesh Shah <hi...@apache.org>]

Hello Alexander,

Are you using Tez natively or via Hive/Pig/Cascading, etc? 

To a large extent, most users I have encountered tend to have tez.am.resource.memory.mb sized to be between 4-8 GB though in some cases, ( until TEZ-776 is addressed ), this might need to increased for DAGs which have very high parallelism and  large scatter-gather edges. ( 4 GB is not a minimum requirement but in general, most YARN clusters usually end up having their minimum allocation configured to 4GB or so in any case ).

As for the task memory, it depends on the kind of workload and there are no standard guidelines from a general Tez perspective. A general rule of thumb on a YARN cluster is that this usually is set atleast the configured minimum size of a YARN container ( minimum-allocation setting ). Hive does not use this value and overrides it directly via its hive.tez.container.size setting. I am not sure if Pig has their own override configuration property or if they treat the tez task memory property as a passthrough. 

For both the above, Tez automatically sets the Xmx value for the JVM to around 0.8 of the container size ( if it has not been set by the user - a general recommendation is to not configure -Xmx in the java opts for this reason ). Furthermore, most of the buffers used by the in-built inputs/outputs usually get auto-scaled down based on the size of the available JVM heap. 

thanks
— Hitesh


On Mar 9, 2015, at 4:04 PM, Alexander Pivovarov <ap...@gmail.com> wrote:

> Hi Everyone
> 
> What is recommended value for
> 
> tez.am.resource.memory.mb
> 
> tez.task.resource.memory.mb
> 
> 
> Thank you
> 
>