[jira] [Commented] (TEZ-3207) Add support for fetching multiple partitions from the same source task to UnorderedKVInput

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236213#comment-15236213 ] 

Siddharth Seth commented on TEZ-3207:
-------------------------------------

[~mingma] - in case you work on this, could you please model it differently from what is done in the Shuffle case. That ends up clubbing hosts and partitions together to create separate 'host' entities. That has the downside of getting in the way of any host specific logic that exists - e.g. one fetcher per host at any given instance, at a later point fetching multiple partitions with the same call.

> Add support for fetching multiple partitions from the same source task to UnorderedKVInput
> ------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3207
>                 URL: https://issues.apache.org/jira/browse/TEZ-3207
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Ming Ma
>
> The ordered grouped {{ShuffleScheduler}} can support fetching multiple partitions from the same source task. But for the unordered ShuffleManager, it only supports one partition per source task due to the following issue where {{identifier}} doesn't take partition id into account.
> {noformat}
>   public void addKnownInput(String hostName, int port,
>       InputAttemptIdentifier srcAttemptIdentifier, int srcPhysicalIndex) {
>     String identifier = InputHost.createIdentifier(hostName, port);
>     InputHost host = knownSrcHosts.get(identifier);
>     ....
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)