[jira] [Commented] (TEZ-1152) Optimize broadcast join for scalability

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011041#comment-14011041 ] 

Rajesh Balamohan commented on TEZ-1152:
---------------------------------------

Since #1 and #2 need to be fixed independently, I will create sub tasks.

> Optimize broadcast join for scalability
> ---------------------------------------
>
>                 Key: TEZ-1152
>                 URL: https://issues.apache.org/jira/browse/TEZ-1152
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>              Labels: performance, scalability
>
> Two main issues for large queries using broadcast shuffle
> 1. Lots of tasks communicate to same node for downloading shuffle data. So most of the time, single machine will be overloaded with requests.
> 2. Tasks pertaining to same job (in the same machine) downloads broadcast shuffle data redundantly.  If the data can be copied to temp storage or ramfs, other tasks running in the same machine can refer to the local copy.  Optimizing this would help when running multiple queries in parallel in the cluster.



--
This message was sent by Atlassian JIRA
(v6.2#6252)