[jira] [Commented] (TEZ-3607) ShuffleScheduler#allEventsReceived check is too tight

["Kuhu Shukla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328323#comment-16328323 ] 

Kuhu Shukla commented on TEZ-3607:
----------------------------------

In the case of composite event processing, the calculation is off as mentioned in the description and with v1 of the patch, it could still be a problem since only in the composite case do we add the entries to pathToIdentifierMap and also increment skipped counters and it will return early as true with v1 fix in. I am working on the test to be super sure but if we plan to keep the current design choices around pathToIdentifierMap then the check would be a bit more involved AFAICT. 

> ShuffleScheduler#allEventsReceived check is too tight
> -----------------------------------------------------
>
>                 Key: TEZ-3607
>                 URL: https://issues.apache.org/jira/browse/TEZ-3607
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>            Priority: Major
>         Attachments: TEZ-3607.001.patch
>
>
> In case of skipped records being non zero, the allEventsReceived() call can return false even when all the events have in fact been received since pathtoAttemptsMap size plus the skipped counter may be more than the total number of inputs since we don't take out entries from the map if skipped counter is incremented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)