[jira] [Commented] (TEZ-1036) Shuffle deduping should be global and not local to the MapHost

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965981#comment-13965981 ] 

Siddharth Seth commented on TEZ-1036:
-------------------------------------

This can be a temporary fix in the running task - but eventually needs to be fixed in the AM - TEZ-924.

> Shuffle deduping should be global and not local to the MapHost
> --------------------------------------------------------------
>
>                 Key: TEZ-1036
>                 URL: https://issues.apache.org/jira/browse/TEZ-1036
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Bikas Saha
>              Labels: newbie++
>
> The following logic needs to be across all fetchers and not within a maphost.
> {code}
> List<InputAttemptIdentifier> origList = host.getAndClearKnownMaps();
>     Map<Integer, InputAttemptIdentifier> dedupedList = new LinkedHashMap<Integer, InputAttemptIdentifier>();
>     Iterator<InputAttemptIdentifier> listItr = origList.iterator();
>     while (listItr.hasNext()) {
>       // we may want to try all versions of the input but with current retry
>       // behavior older ones are likely to be lost and should be ignored.
>       // This may be removed after TEZ-914
>       InputAttemptIdentifier id = listItr.next();
>       if (inputShouldBeConsumed(id)) {
>         Integer inputNumber = new Integer(id.getInputIdentifier().getInputIndex());
>         InputAttemptIdentifier oldId = dedupedList.get(inputNumber);
>         if (oldId == null || oldId.getAttemptNumber() < id.getAttemptNumber()) {
>           dedupedList.put(inputNumber, id);
>           if (oldId != null) {
>             LOG.warn("Old Src for InputIndex: " + inputNumber + " with attemptNumber: "
>                 + oldId.getAttemptNumber()
>                 + " was not determined to be invalid. Ignoring it for now in favour of "
>                 + id.getAttemptNumber());
>           }
>         }
>       } else {
>         LOG.info("Ignoring finished or obsolete source: " + id);
>       }
>     }{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)