You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/01/04 11:48:00 UTC
[jira] [Work logged] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-24805?focusedWorklogId=703256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-703256 ]
ASF GitHub Bot logged work on HIVE-24805:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jan/22 11:47
Start Date: 04/Jan/22 11:47
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778021118
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -184,7 +184,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
+ Table t = cacheAndResolveTable(ci);
Review comment:
what do you think if we put `cache` at the end and keep the primary goal at the beginning?
````
resolveTableAndCache
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 703256)
Time Spent: 20m (was: 10m)
> Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
> Reporter: Rajesh Balamohan
> Assignee: Antal Sinkovits
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there are large number of databases/tables, it takes lot of time for Initiator to complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However, it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)