You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by "r7raul1984@163.com" <r7...@163.com> on 2015/04/24 10:22:34 UTC
How to Tuning Tez Task Performance
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
r7raul1984@163.com
RE: How to Tuning Tez Task Performance
Posted by Xiaoyong Zhu <xi...@microsoft.com>.
I see, I installed TezUI and saw these values☺
Btw, this setting (tez.task.generate.counters.per.io) only affects the task counter right? i.e. it does not affect the dag/vertex level counter (as indicated by the name itself…) right?
Xiaoyong
From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
Sent: Tuesday, April 28, 2015 8:07 AM
To: user@tez.apache.org
Subject: Re: How to Tuning Tez Task Performance
Not sure if you have Tez-UI which should render this info automatically. Otherwise you can verify from the application logs. Example is given below.
2015-04-27 04:15:12,834 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1429683757595_0452_1][Event:DAG_FINISHED]: dagId=dag_1429683757595_0452_1, startTime=1430133293306, finishTime=1430133312773, timeTaken=19467, status=SUCCEEDED, diagnostics=, counters=Counters: 225, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=43, TOTAL_LAUNCHED_TASKS=43,
.....
.....
TaskCounter_Map_4_OUTPUT_Map_1, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=137200, OUTPUT_BYTES_PHYSICAL=119705, OUTPUT_BYTES_WITH_OVERHEAD=548794, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=27440, SPILLED_RECORDS=0, TaskCounter_Map_5_INPUT_date_dim, INPUT_RECORDS_PROCESSED=10, TaskCounter_Map_5_OUTPUT_Map_1, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=1825, OUTPUT_BYTES_PHYSICAL=1505, OUTPUT_BYTES_WITH_OVERHEAD=7297, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=365, SPILLED_RECORDS=0,
.......
.......
TaskCounter_Map_6_OUTPUT_Map_1, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=909, OUTPUT_BYTES_PHYSICAL=464, OUTPUT_BYTES_WITH_OVERHEAD=2421, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=101, SPILLED_RECORDS=0, TaskCounter_Map_7_INPUT_item, INPUT_RECORDS_PROCESSED=47,
....
....
TaskCounter_Map_7_OUTPUT_Map_1, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=1104000, OUTPUT_BYTES_PHYSICAL=341828, OUTPUT_BYTES_WITH_OVERHEAD=1727999, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=48000, SPILLED_RECORDS=0, TaskCounter_Reducer_2_INPUT_Map_1, ADDITIONAL_SPILLS_BYTES_READ=821473, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, COMBINE_INPUT_RECORDS=0, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=5049, MERGED_MAP_OUTPUTS=36, MERGE_PHASE_TIME=5070, NUM_DISK_TO_DISK_MERGES=0, NUM_FAILED_SHUFFLE_INPUTS=0, NUM_MEM_TO_DISK_MERGES=0, NUM_SHUFFLED_INPUTS=36, NUM_SKIPPED_INPUTS=0, REDUCE_INPUT_GROUPS=47999, REDUCE_INPUT_RECORDS=670353, SHUFFLE_BYTES=16402510, SHUFFLE_BYTES_DECOMPRESSED=52252736, SHUFFLE_BYTES_DISK_DIRECT=821473, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=15581037, SHUFFLE_PHASE_TIME=5056, SPILLED_RECORDS=33317,
....
....
TaskCounter_Reducer_2_OUTPUT_Reducer_3, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=5600, OUTPUT_BYTES_PHYSICAL=0, OUTPUT_BYTES_WITH_OVERHEAD=0, OUTPUT_RECORDS=100, SPILLED_RECORDS=100
On Tue, Apr 28, 2015 at 4:41 AM, Xiaoyong Zhu <xi...@microsoft.com>> wrote:
Btw, Rajesh, I set tez.task.generate.counters.per.io=true in my cluster but did not find the task counter per edge. Could you please give some counter examples when this is enabled so I could verify?
Thanks!
Xiaoyong
From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com<ma...@gmail.com>]
Sent: Friday, April 24, 2015 4:55 PM
To: user@tez.apache.org<ma...@tez.apache.org>
Subject: Re: How to Tuning Tez Task Performance
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io<http://tez.task.generate.counters.per.io>=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.
- In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed>
- Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)
- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)
- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).
- Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)
- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.
- Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)
- Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.
- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)
Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.
On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com<ma...@163.com> <r7...@163.com>> wrote:
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
________________________________
r7raul1984@163.com<ma...@163.com>
--
~Rajesh.B
--
~Rajesh.B
Re: How to Tuning Tez Task Performance
Posted by Rajesh Balamohan <ra...@gmail.com>.
Not sure if you have Tez-UI which should render this info automatically.
Otherwise you can verify from the application logs. Example is given below.
2015-04-27 04:15:12,834 INFO [Dispatcher thread: Central]
history.HistoryEventHandler:
[HISTORY][DAG:dag_1429683757595_0452_1][Event:DAG_FINISHED]:
dagId=dag_1429683757595_0452_1, startTime=1430133293306,
finishTime=1430133312773, timeTaken=19467, status=SUCCEEDED, diagnostics=,
counters=Counters: 225, org.apache.tez.common.counters.DAGCounter,
NUM_SUCCEEDED_TASKS=43, TOTAL_LAUNCHED_TASKS=43,
.....
.....
*TaskCounter_Map_4_OUTPUT_Map_1*, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0,
OUTPUT_BYTES=137200, OUTPUT_BYTES_PHYSICAL=119705,
OUTPUT_BYTES_WITH_OVERHEAD=548794, OUTPUT_LARGE_RECORDS=0,
OUTPUT_RECORDS=27440, SPILLED_RECORDS=0, TaskCounter_Map_5_INPUT_date_dim,
INPUT_RECORDS_PROCESSED=10, TaskCounter_Map_5_OUTPUT_Map_1,
ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0,
ADDITIONAL_SPILL_COUNT=0, OUTPUT_BYTES=1825, OUTPUT_BYTES_PHYSICAL=1505,
OUTPUT_BYTES_WITH_OVERHEAD=7297, OUTPUT_LARGE_RECORDS=0,
OUTPUT_RECORDS=365, SPILLED_RECORDS=0,
.......
.......
*TaskCounter_Map_6_OUTPUT_Map_1*, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0,
OUTPUT_BYTES=909, OUTPUT_BYTES_PHYSICAL=464,
OUTPUT_BYTES_WITH_OVERHEAD=2421, OUTPUT_LARGE_RECORDS=0,
OUTPUT_RECORDS=101, SPILLED_RECORDS=0, TaskCounter_Map_7_INPUT_item,
INPUT_RECORDS_PROCESSED=47,
....
....
*TaskCounter_Map_7_OUTPUT_Map_1*, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0,
OUTPUT_BYTES=1104000, OUTPUT_BYTES_PHYSICAL=341828,
OUTPUT_BYTES_WITH_OVERHEAD=1727999, OUTPUT_LARGE_RECORDS=0,
OUTPUT_RECORDS=48000, SPILLED_RECORDS=0, TaskCounter_Reducer_2_INPUT_Map_1,
ADDITIONAL_SPILLS_BYTES_READ=821473, ADDITIONAL_SPILLS_BYTES_WRITTEN=0,
COMBINE_INPUT_RECORDS=0, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=5049,
MERGED_MAP_OUTPUTS=36, MERGE_PHASE_TIME=5070, NUM_DISK_TO_DISK_MERGES=0,
NUM_FAILED_SHUFFLE_INPUTS=0, NUM_MEM_TO_DISK_MERGES=0,
NUM_SHUFFLED_INPUTS=36, NUM_SKIPPED_INPUTS=0, REDUCE_INPUT_GROUPS=47999,
REDUCE_INPUT_RECORDS=670353, SHUFFLE_BYTES=16402510,
SHUFFLE_BYTES_DECOMPRESSED=52252736, SHUFFLE_BYTES_DISK_DIRECT=821473,
SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=15581037,
SHUFFLE_PHASE_TIME=5056, SPILLED_RECORDS=33317,
....
....
*TaskCounter_Reducer_2_OUTPUT_Reducer_3*, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0,
OUTPUT_BYTES=5600, OUTPUT_BYTES_PHYSICAL=0, OUTPUT_BYTES_WITH_OVERHEAD=0,
OUTPUT_RECORDS=100, SPILLED_RECORDS=100
On Tue, Apr 28, 2015 at 4:41 AM, Xiaoyong Zhu <xi...@microsoft.com>
wrote:
> Btw, Rajesh, I set tez.task.generate.counters.per.io=true in my cluster
> but did not find the task counter per edge. Could you please give some
> counter examples when this is enabled so I could verify?
>
>
>
> Thanks!
>
>
>
> Xiaoyong
>
>
>
> *From:* Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
> *Sent:* Friday, April 24, 2015 4:55 PM
> *To:* user@tez.apache.org
> *Subject:* Re: How to Tuning Tez Task Performance
>
>
>
> Listing some details at very high level,
>
>
>
> - Set "tez.task.generate.counters.per.io=true" to get more details on the
> task counters. Basically this starts printinng the counters per edge, which
> can be a lot more useful for debugging.
>
>
>
> - In case you want to avoid container launches etc when you analyze for
> first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no
> of containers you want in your sesssion to be prewarmed>
>
>
>
> - Container reuse is enabled by default in tez.
> (tez.am.container.idle.release-timeout-min.millis,
> tez.am.container.idle.release-timeout-max.millis controls the amount of
> time a container is held by AM before releasing it)
>
>
>
> - Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check
> task counters in the logs to find out the spills and adjust it accordingly)
>
>
>
> - Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot
> performant than DefaultSorter (this is the default in master branch. But if
> you are using earlier releases, you can turn it on by setting
> tez.runtime.sort.threads=2).
>
>
>
> - Set tez.runtime.compress=true and set tez.runtime.compress.codec
> (SnappyCodec is preferred, but it is upto you to choose)
>
>
>
> - Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle
> heavy workload. This reduces number of connections in shuffle.
>
>
>
> - Adjust memory allocated to different inputs/outputs based on
> tez.task.scale.memory.ratios (but this is more of expert level setting
> which you might want to touch after nailing down any memory pressure)
>
>
>
> - Adjusting shuffle buffers are also possible, but would advise only when
> you nail down an issue related to shuffle/merge codepath.
>
>
>
> - Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when
> data is locally present)
>
>
>
>
>
> Feel free to refer to
> https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for
> any commonly used settings for benchmarks.
>
>
>
> On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com <r7...@163.com>
> wrote:
>
> I want to Tuning Tez Task Performance. This Tez Task is created by
> Hive. How to Tuning Tez Task Performance?
>
> Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
>
>
> ------------------------------
>
> r7raul1984@163.com
>
>
>
>
>
> --
>
> ~Rajesh.B
>
--
~Rajesh.B
RE: How to Tuning Tez Task Performance
Posted by Xiaoyong Zhu <xi...@microsoft.com>.
Btw, Rajesh, I set tez.task.generate.counters.per.io=true in my cluster but did not find the task counter per edge. Could you please give some counter examples when this is enabled so I could verify?
Thanks!
Xiaoyong
From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
Sent: Friday, April 24, 2015 4:55 PM
To: user@tez.apache.org
Subject: Re: How to Tuning Tez Task Performance
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io<http://tez.task.generate.counters.per.io>=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.
- In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed>
- Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)
- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)
- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).
- Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)
- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.
- Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)
- Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.
- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)
Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.
On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com<ma...@163.com> <r7...@163.com>> wrote:
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
________________________________
r7raul1984@163.com<ma...@163.com>
--
~Rajesh.B
Re: How to Tuning Tez Task Performance
Posted by Hitesh Shah <hi...@apache.org>.
@r7raul1984, would you mind filing a documentation jira for your question. The list that Rajesh provided might be good to formalize into a doc and/or wiki.
Also, please take a look at https://issues.apache.org/jira/browse/TEZ-2294 to see all the list of parameters. If you see something off or not clear enough, please add your comments to the jira.
@Rohini,
We recently changed tez.runtime.optimize.local.fetch to true as the default value in master. The feature was introduced and probably kept as false initially as it had not been fully battle tested.
The latter I am assuming depends on how many open connections a cluster’s setup can sustain and needs to be tuned in combination with “tez.runtime.shuffle.keep-alive.max.connections”. Good point on whether we should make this true by default. Will wait for @Rajesh/@Gopal/@Sid to chime in and they can open a new jira if this is generally beneficial in most setups.
thanks
— Hitesh
On Apr 24, 2015, at 9:34 AM, Rohini Palaniswamy <ro...@gmail.com> wrote:
> Rajesh,
> What are the problems with having tez.runtime.shuffle.keep-alive.enabled and tez.runtime.optimize.local.fetch set to true always by default?
>
> Regards,
> Rohini
>
> On Fri, Apr 24, 2015 at 1:54 AM, Rajesh Balamohan <ra...@gmail.com> wrote:
> Listing some details at very high level,
>
> - Set "tez.task.generate.counters.per.io=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.
>
> - In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed>
>
> - Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)
>
> - Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)
>
> - Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).
>
> - Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)
>
> - Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.
>
> - Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)
>
> - Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.
>
> - Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)
>
>
> Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.
>
> On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com <r7...@163.com> wrote:
> I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
> Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
>
> r7raul1984@163.com
>
>
>
> --
> ~Rajesh.B
>
Re: How to Tuning Tez Task Performance
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> What are the problems with having
>tez.runtime.shuffle.keep-alive.enabled and
>tez.runtime.optimize.local.fetch set to true always by default?
Nothing has failed due to these so far - we¹ve gone through one entire
release where we tested both heavily and found that they work very well at
scale.
local.fetch is already enabled by default in 0.7.x (TEZ-2333).
shared.fetch isn¹t getting flipped right now because last release it
didn¹t get enough coverage on customer setups (for my liking) to bake it
in (the broadcast edge didn¹t whitelist that config).
The keep-alive shuffle was tested on 350 nodes, with 10,000 mappers. And
the advantage of these were significant - between those three options a
broadcast JOIN went from about 30 minutes of shuffle time to around 2 1/2
minutes.
You do need a 64 bit OS (not sandbox) with a modern kernel to safely flip
these on - system configs on Centos need to roughly correspond to the
ktune settings for RHEL (other than THP & numad/zone_reclaim).
These configs help shuffle in general - off the top of my head,
tcp_fin_timeout and somaxconn comes to mind immediately as being the
relevant configs to always tune.
There¹s a certain inflection point we hit in shuffle, where it¹s worse to
be faster - fixes like HADOOP-11226 help there, but they need
router/switch configs as well.
Cheers,
Gopal
Re: How to Tuning Tez Task Performance
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Rajesh,
What are the problems with having tez.runtime.shuffle.keep-alive.enabled
and tez.runtime.optimize.local.fetch set to true always by default?
Regards,
Rohini
On Fri, Apr 24, 2015 at 1:54 AM, Rajesh Balamohan <
rajesh.balamohan@gmail.com> wrote:
> Listing some details at very high level,
>
> - Set "tez.task.generate.counters.per.io=true" to get more details on the
> task counters. Basically this starts printinng the counters per edge, which
> can be a lot more useful for debugging.
>
> - In case you want to avoid container launches etc when you analyze for
> first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no
> of containers you want in your sesssion to be prewarmed>
>
> - Container reuse is enabled by default in tez.
> (tez.am.container.idle.release-timeout-min.millis,
> tez.am.container.idle.release-timeout-max.millis controls the amount of
> time a container is held by AM before releasing it)
>
> - Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check
> task counters in the logs to find out the spills and adjust it accordingly)
>
> - Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot
> performant than DefaultSorter (this is the default in master branch. But if
> you are using earlier releases, you can turn it on by setting
> tez.runtime.sort.threads=2).
>
> - Set tez.runtime.compress=true and set tez.runtime.compress.codec
> (SnappyCodec is preferred, but it is upto you to choose)
>
> - Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle
> heavy workload. This reduces number of connections in shuffle.
>
> - Adjust memory allocated to different inputs/outputs based on
> tez.task.scale.memory.ratios (but this is more of expert level setting
> which you might want to touch after nailing down any memory pressure)
>
> - Adjusting shuffle buffers are also possible, but would advise only when
> you nail down an issue related to shuffle/merge codepath.
>
> - Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when
> data is locally present)
>
>
> Feel free to refer to
> https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for
> any commonly used settings for benchmarks.
>
> On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com <r7...@163.com>
> wrote:
>
>> I want to Tuning Tez Task Performance. This Tez Task is created by Hive.
>> How to Tuning Tez Task Performance?
>> Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
>>
>> ------------------------------
>> r7raul1984@163.com
>>
>
>
>
> --
> ~Rajesh.B
>
RE: How to Tuning Tez Task Performance
Posted by Bikas Saha <bi...@hortonworks.com>.
How many of these defaults need to be changed. 0.7 sounds like a good point to make that?
From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
Sent: Friday, April 24, 2015 1:55 AM
To: user@tez.apache.org
Subject: Re: How to Tuning Tez Task Performance
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io<http://tez.task.generate.counters.per.io>=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.
- In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed>
- Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)
- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)
- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).
- Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)
- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.
- Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)
- Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.
- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)
Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.
On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com<ma...@163.com> <r7...@163.com>> wrote:
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
________________________________
r7raul1984@163.com<ma...@163.com>
--
~Rajesh.B
Re: Re: How to Tuning Tez Task Performance
Posted by "r7raul1984@163.com" <r7...@163.com>.
Thank you!
r7raul1984@163.com
From: Rajesh Balamohan
Date: 2015-04-24 17:24
To: user
Subject: Re: How to Tuning Tez Task Performance
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.
- In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed>
- Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)
- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)
- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).
- Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)
- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.
- Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)
- Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.
- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)
Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.
On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com <r7...@163.com> wrote:
I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How to Tuning Tez Task Performance?
Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
r7raul1984@163.com
--
~Rajesh.B
Re: How to Tuning Tez Task Performance
Posted by Rajesh Balamohan <ra...@gmail.com>.
Listing some details at very high level,
- Set "tez.task.generate.counters.per.io=true" to get more details on the
task counters. Basically this starts printinng the counters per edge, which
can be a lot more useful for debugging.
- In case you want to avoid container launches etc when you analyze for
first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no
of containers you want in your sesssion to be prewarmed>
- Container reuse is enabled by default in tez.
(tez.am.container.idle.release-timeout-min.millis,
tez.am.container.idle.release-timeout-max.millis controls the amount of
time a container is held by AM before releasing it)
- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check
task counters in the logs to find out the spills and adjust it accordingly)
- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot
performant than DefaultSorter (this is the default in master branch. But if
you are using earlier releases, you can turn it on by setting
tez.runtime.sort.threads=2).
- Set tez.runtime.compress=true and set tez.runtime.compress.codec
(SnappyCodec is preferred, but it is upto you to choose)
- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle
heavy workload. This reduces number of connections in shuffle.
- Adjust memory allocated to different inputs/outputs based on
tez.task.scale.memory.ratios (but this is more of expert level setting
which you might want to touch after nailing down any memory pressure)
- Adjusting shuffle buffers are also possible, but would advise only when
you nail down an issue related to shuffle/merge codepath.
- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when
data is locally present)
Feel free to refer to
https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for
any commonly used settings for benchmarks.
On Fri, Apr 24, 2015 at 1:52 PM, r7raul1984@163.com <r7...@163.com>
wrote:
> I want to Tuning Tez Task Performance. This Tez Task is created by Hive. How
> to Tuning Tez Task Performance?
> Analyze performance by Tez Task Counts of Tez Log ? Any Suggestion?
>
> ------------------------------
> r7raul1984@163.com
>
--
~Rajesh.B