[jira] [Created] (TEZ-1036) Shuffle deduping should be global and not local to the MapHost

["Bikas Saha (JIRA)" <ji...@apache.org>]

Bikas Saha created TEZ-1036:
-------------------------------

             Summary: Shuffle deduping should be global and not local to the MapHost
                 Key: TEZ-1036
                 URL: https://issues.apache.org/jira/browse/TEZ-1036
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Bikas Saha


The following logic needs to be across all fetchers and not within a maphost.

{code}
List<InputAttemptIdentifier> origList = host.getAndClearKnownMaps();

    Map<Integer, InputAttemptIdentifier> dedupedList = new LinkedHashMap<Integer, InputAttemptIdentifier>();
    Iterator<InputAttemptIdentifier> listItr = origList.iterator();
    while (listItr.hasNext()) {
      // we may want to try all versions of the input but with current retry
      // behavior older ones are likely to be lost and should be ignored.
      // This may be removed after TEZ-914
      InputAttemptIdentifier id = listItr.next();
      if (inputShouldBeConsumed(id)) {
        Integer inputNumber = new Integer(id.getInputIdentifier().getInputIndex());
        InputAttemptIdentifier oldId = dedupedList.get(inputNumber);
        if (oldId == null || oldId.getAttemptNumber() < id.getAttemptNumber()) {
          dedupedList.put(inputNumber, id);
          if (oldId != null) {
            LOG.warn("Old Src for InputIndex: " + inputNumber + " with attemptNumber: "
                + oldId.getAttemptNumber()
                + " was not determined to be invalid. Ignoring it for now in favour of "
                + id.getAttemptNumber());
          }
        }
      } else {
        LOG.info("Ignoring finished or obsolete source: " + id);
      }
    }{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)