[jira] [Commented] (TEZ-3334) Tez Custom Shuffle Handler

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953842#comment-15953842 ] 

Siddharth Seth commented on TEZ-3334:
-------------------------------------

[~jeagles] - there's a lot of comments. I don't think all need to be addressed before merging back.
Some of the important ones from my perspective 1) The deletion setup in the AM along with config parameters, 2) Config propagation when not required, since that opens up possibilities of using configs to pass parameters for future changes, 3) The off by 1 issue / other correctness issues from the list.

> Tez Custom Shuffle Handler
> --------------------------
>
>                 Key: TEZ-3334
>                 URL: https://issues.apache.org/jira/browse/TEZ-3334
>             Project: Apache Tez
>          Issue Type: New Feature
>            Reporter: Jonathan Eagles
>         Attachments: TEZ-3334.1.patch
>
>
> For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom shuffle handler could help reduce the number of fetches and could more efficiently fetch data. In particular if a reducer is fetching 100 pieces serially from the same mapper it could do this in one fetch call. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)