[jira] [Commented] (HIVE-12376) make hive.compactor.worker.threads use a thread pool, etc

["Eugene Koifman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883468#comment-15883468 ] 

Eugene Koifman commented on HIVE-12376:
---------------------------------------

Together with HIVE-11685 it probably better to simply rely on _hive.compactor.job.queue_ for resource management and have the Worker just submit jobs to it.
Then we can have 1 polling thread to scan COMPACTION_QUEUE  for running jobs and check with Yarn if they are done.
Number of Worker thread could default to 5 or something since generating a compaction job may take time if there are very many deltas, it should not be used to throttle number of jobs.



> make hive.compactor.worker.threads use a thread pool, etc
> ---------------------------------------------------------
>
>                 Key: HIVE-12376
>                 URL: https://issues.apache.org/jira/browse/HIVE-12376
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore, Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>
> # use a thread pool with core/max capacities instead of creating all threads upfront
> # make sure there is a limit (1000 threads?)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)