[jira] [Commented] (TEZ-3701) UnorderedPartitionedKVWriter does not wait for submitted tasks to finish before finalMerge.

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987602#comment-15987602 ] 

Jason Lowe commented on TEZ-3701:
---------------------------------

The code doesn't explicitly wait on the executor (e.g.: shutdown() and awaitTermination()) but it does implement its own form of waiting via {{pendingSpillCount}}.  The code in {{close()}} looks like it should properly wait for spills to complete before performing the final merge.  Could you elaborate on how it fails to wait properly?
{code}
  public List<Event> close() throws IOException, InterruptedException {
    List<Event> eventList = Lists.newLinkedList();
    isShutdown.set(true);
    spillLock.lock();
    try {
      LOG.info(destNameTrimmed + ": " + "Waiting for all spills to complete : Pending : " + pendingSpillCount.get());
      while (pendingSpillCount.get() != 0 && spillException == null) {
        spillInProgress.await();
      }
    } finally {
      spillLock.unlock();
    }
{code}


> UnorderedPartitionedKVWriter does not wait for submitted tasks to finish before finalMerge.
> -------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3701
>                 URL: https://issues.apache.org/jira/browse/TEZ-3701
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Harish Jaiprakash
>            Assignee: Harish Jaiprakash
>            Priority: Blocker
>
> UnorderedPartitionedKVWriter add task to the executor, but does not wait for them to finish before starting the final merge. This can cause finalMerge to fail or write incorrect data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)