You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2016/06/06 00:38:59 UTC
[jira] [Comment Edited] (TEZ-3291) Optimize splits grouping when
locality information is not available
[ https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316094#comment-15316094 ]
Rajesh Balamohan edited comment on TEZ-3291 at 6/6/16 12:38 AM:
----------------------------------------------------------------
E.g TPC-DS query_22 @ 200GB scale (Ec2 cluster + S3):
*Without Patch: 291.59 seconds*
{noformat}
Status: Running (Executing on YARN cluster with App id application_1464530613845_5129)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Map 4 .......... container SUCCEEDED 1 1 0 0 0 0
Map 5 .......... container SUCCEEDED 1 1 0 0 0 0
Map 6 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 4 4 0 0 0 0
Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 06/06 [==========================>>] 100% ELAPSED TIME: 291.59 s
----------------------------------------------------------------------------------------------
Status: DAG finished successfully in 291.59 seconds
{noformat}
*With Patch: 38.65 seconds*
{noformat}
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 27 27 0 0 0 0
Map 4 .......... container SUCCEEDED 1 1 0 0 0 0
Map 5 .......... container SUCCEEDED 1 1 0 0 0 0
Map 6 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 4 4 0 0 0 0
Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 06/06 [==========================>>] 100% ELAPSED TIME: 38.65 s
----------------------------------------------------------------------------------------------
Status: DAG finished successfully in 38.65 seconds
{noformat}
Map 1 is the vertex which scans inventory table. With the patch, it ends up creating 27 tasks which helps in improving overall response time.
was (Author: rajesh.balamohan):
E.g TPC-DS query_22 @ 200GB scale:
*Without Patch: 291.59 seconds*
{noformat}
Status: Running (Executing on YARN cluster with App id application_1464530613845_5129)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
Map 4 .......... container SUCCEEDED 1 1 0 0 0 0
Map 5 .......... container SUCCEEDED 1 1 0 0 0 0
Map 6 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 4 4 0 0 0 0
Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 06/06 [==========================>>] 100% ELAPSED TIME: 291.59 s
----------------------------------------------------------------------------------------------
Status: DAG finished successfully in 291.59 seconds
{noformat}
*With Patch: 38.65 seconds*
{noformat}
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 27 27 0 0 0 0
Map 4 .......... container SUCCEEDED 1 1 0 0 0 0
Map 5 .......... container SUCCEEDED 1 1 0 0 0 0
Map 6 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 ...... container SUCCEEDED 4 4 0 0 0 0
Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 06/06 [==========================>>] 100% ELAPSED TIME: 38.65 s
----------------------------------------------------------------------------------------------
Status: DAG finished successfully in 38.65 seconds
{noformat}
Map 1 is the vertex which scans inventory table. With the patch, it ends up creating 27 tasks which helps in improving overall response time.
> Optimize splits grouping when locality information is not available
> -------------------------------------------------------------------
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3 is an example, where all splits would have "localhost" for the location details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small number of splits. Depending on clusters this can end creating long running map jobs.
> Example with hive:
> ==============
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small table.
> 2. With query-22, hive requests with the original splits count as 52 and overall length of splits themselves is around 12061817 bytes. {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+ files be processed in the split. In clusters with split locations, this would have landed up with multiple splits since {{allowSmallGroups}} would have kicked in.
> But in S3, since everything would have "localhost" all splits get added to single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in sequential fashion. Had it been processed by multiple tasks, response time would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0] |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0] |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0] |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817 numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength: 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0] |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1 splitsProcessed: 52
> {noformat}
> Alternate options:
> ==================
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute desiredNumSplits only when number of distinct locations in the splits is > 1. This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)