[jira] [Commented] (HIVE-22081) Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction

["Vipin Vishvkarma (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-22081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973063#comment-16973063 ] 

Vipin Vishvkarma commented on HIVE-22081:
-----------------------------------------

[~Rajkumar Singh] Will, there be any performance improvement with this change, as I don't see changes related to point 2 from the description in the final change and we have used stream() which is sequential in nature.  I may be missing something here, can you please confirm.

> Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction 
> --------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-22081
>                 URL: https://issues.apache.org/jira/browse/HIVE-22081
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 3.1.1
>            Reporter: Rajkumar Singh
>            Assignee: Rajkumar Singh
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-21917.01.patch, HIVE-21917.02.patch, HIVE-21917.03.patch, HIVE-22081.04.patch, HIVE-22081.patch
>
>
> if Automatic Compaction is turned on, Initiator thread check for potential table/partitions which are eligible for compactions and run some checks in for loop before requesting compaction for eligibles. Though initiator thread is configured to run at interval 5 min default, in case of many objects it keeps on running as these checks are IO intensive and hog cpu.
> In the proposed changes, I am planning to do
> 1. passing less object to for loop by filtering out the objects based on the condition which we are checking within the loop.
> 2. Doing Async call using future to determine compaction type(this is where we do FileSystem calls)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)