[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks

["Harish Jaiprakash (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834119#comment-15834119 ] 

Harish Jaiprakash commented on TEZ-1187:
----------------------------------------

Please do tell me if this makes sense:

Define TezSharedExecutors as follows:
{code}
public interface TezSharedExecutors {
  ListeningScheduledExecutorService createScheduledExecutorService();
  void shutdownAll();
}
{code}

The implementation would be roughly like this:
* We have an shared executorService, mostly like a ForkJoinExecutor service with parallelism set to number vcores available for the container.
* createScheduledExecutorService returns an internal implementation of ListeningScheduledExecutorService, delegating the responsibility to the SharedExecutorService for actual execution.
* We track all the Futures returned by shared executor service, to be used in awaitTermination and shutdown calls.
* For shutdownNow we can cancel all these futures.
* shutdown will just prevent more jobs from getting submitted via the current object.
* For the execute method, since it does not return any future and the runnable cannot be removed once submitted. We'll wrap it in a different runnable which will cleanup any references and prevent execution, in case shutdownNow is invoked.

Does this implementation sound fine?

> Share Thread pools between different tasks
> ------------------------------------------
>
>                 Key: TEZ-1187
>                 URL: https://issues.apache.org/jira/browse/TEZ-1187
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Siddharth Seth
>            Assignee: Harish Jaiprakash
>
> Thread pools are used all over for fetchers, sort etc. When running a single task - this is already a problem, and gets worse when running multiple tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)