[jira] [Commented] (TEZ-1914) VertexManager logic should not run on the central dispatcher

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296340#comment-14296340 ] 

Bikas Saha commented on TEZ-1914:
---------------------------------

The patch moves execution of vertex manager code away from the central dispatcher. All invocations of vertexmanager code are now done on a thread pool with callable wrapping around the invocations. A shared threadpool is used to reduce the number of threads being created while allowing multiple vertexmanagers to proceed in parallel. For a given vertexmanager only one operation is active at a given time to prevent any one to monopolize the threadpool. Recovery depended on the serialization of input information events before init event. While this isnt theoretically necessary and something we should look at removing, for now a new transition is allowing to maintain that constraint. This was needed because the return of input information events is no longer sync. Similarly, AMUserCodeException is no longer received inline from the vertexmanager code invocation. [~hitesh] could you please look at the recovery changes [~zjffdu] could you please look at the exception related changes.
This is close to done. So an quick initial review would be good before uploading the next patch.
Existing tests should cover all code paths exercised by this change. TestVertexImpl, TestDAGImpl had to have their supporting code changed to account for the async operations so that tests could continue to pass. Essentially, these tests run the callables on the dispatcher, thus effectively allowing to use dispatcher.await() to continue to guarantee completion of internal operations.

> VertexManager logic should not run on the central dispatcher
> ------------------------------------------------------------
>
>                 Key: TEZ-1914
>                 URL: https://issues.apache.org/jira/browse/TEZ-1914
>             Project: Apache Tez
>          Issue Type: Task
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: TEZ-1914.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)