You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/12/22 16:17:42 UTC
[GitHub] [hive] asinkovits opened a new pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
asinkovits opened a new pull request #2906:
URL: https://github.com/apache/hive/pull/2906
…and again for partitioned tables
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Introduce table/partition metadata caching in the initiator and cleaner.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
When there are large number of databases/tables, it takes lot of time for Initiator to complete its initial iteration and load on DB also goes higher.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778029906
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
+ if (CacheAwareCompactor.class.isAssignableFrom(obj.getClass())) {
+ ((CacheAwareCompactor) obj).setCache(cache);
+ }
+ }
+
+ void setCache(CompactorMetadataCache cache);
+
+ class CompactorMetadataCache {
+
+ private final Cache<TableCacheKey, Table> tableCache;
+ private final Cache<PartitionCacheKey, Partition> partitionCache;
+
+ @VisibleForTesting
+ public CompactorMetadataCache(long timeout, TimeUnit unit) {
+ this.tableCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ this.partitionCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ }
+
+ public static CompactorMetadataCache createIfEnabled(Configuration conf) {
+ long timeout = MetastoreConf.getTimeVar(conf,
+ MetastoreConf.ConfVars.COMPACTOR_METADATA_CACHE_TIMEOUT, TimeUnit.SECONDS);
+ if (timeout == 0) {
+ return null;
+ }
+ return new CompactorMetadataCache(timeout, TimeUnit.SECONDS);
+ }
+
+ public Table resolveTable(CompactionInfo ci, Callable<Table> loader) {
+ try {
+ TableCacheKey key = new TableCacheKey(ci);
+ return tableCache.get(key, loader);
+ } catch (ExecutionException e) {
+ throw new UncheckedExecutionException(e);
+ }
+ }
+
+ public Partition resolvePartition(CompactionInfo ci, Callable<Partition> loader) {
+ try {
+ if (ci.partName == null) {
+ return null;
+ }
+ PartitionCacheKey key = new PartitionCacheKey(ci);
Review comment:
same as above:
````
ci.getFullPartitionName
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787845807
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -572,6 +572,9 @@ public static ConfVars getMetaConf(String name) {
"Set this to true on one instance of the Thrift metastore service as part of turning\n" +
"on Hive transactions. For a complete list of parameters required for turning on\n" +
"transactions, see hive.txn.manager."),
+ COMPACTOR_INITIATOR_TABLE_CACHE_ON("metastore.compactor.initiator.table.cache.on",
Review comment:
your call 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r782617149
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -164,7 +164,7 @@ public void run() {
for (CompactionInfo ci : potentials) {
try {
- Table t = resolveTable(ci);
+ Table t = resolveTableAndCache(ci);
Review comment:
Issue is more pronounced in the following code path.
{code}
Set<CompactionInfo> potentials = compactionExecutor.submit(() ->
txnHandler.findPotentialCompactions(abortedThreshold, abortedTimeThreshold, compactionInterval)
.parallelStream()
.filter(ci -> isEligibleForCompaction(ci, currentCompactions, skipDBs, skipTables))
.collect(Collectors.toSet())).get();
{code}
i.e in `isEligibleForCompaction`.
Instead of introducing the cache and invalidating it later, is it possible to create a simple hashmap in the method itself and use it on need basis? (e.g if there are multiple partitions, it will just do one lookup for table and the rest of the partitions will make use of the value in map).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787786833
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -572,6 +572,9 @@ public static ConfVars getMetaConf(String name) {
"Set this to true on one instance of the Thrift metastore service as part of turning\n" +
"on Hive transactions. For a complete list of parameters required for turning on\n" +
"transactions, see hive.txn.manager."),
+ COMPACTOR_INITIATOR_TABLE_CACHE_ON("metastore.compactor.initiator.table.cache.on",
Review comment:
How confident are we in this feature?
Shouldn't we default to false in the beginning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r782918861
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -164,7 +164,7 @@ public void run() {
for (CompactionInfo ci : potentials) {
try {
- Table t = resolveTable(ci);
+ Table t = resolveTableAndCache(ci);
Review comment:
I'm fine changing it to the suggested approach, just want to understand the reasoning behind it.
I do have concerns introducing a hash map, as we have less control over the memory pressure. Table objects are moderately large, and as you mentioned there can be large clusters with numerous tables.
I was using softValues, so the cache can be cleared by the garbage collector in response to memory demand.
As for no 2, do you see some use case where the current approach doesn't address the observed issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778034243
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
##########
@@ -101,6 +103,25 @@ public void init(AtomicBoolean stop) throws Exception {
}
}
+ @Override
+ public void setCache(CompactorMetadataCache metadataCache) {
+ this.metadataCache = metadataCache;
+ }
+
+ protected Table cacheAndResolveTable(CompactionInfo ci) throws MetaException {
+ if (metadataCache != null) {
Review comment:
do you think it's worth using Optional here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787503742
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -589,6 +589,9 @@ public static ConfVars getMetaConf(String name) {
"in which a warning is being raised if multiple worker version are detected.\n" +
"The setting has no effect if the metastore.metrics.enabled is disabled \n" +
"or the metastore.acidmetrics.thread.on is turned off."),
+ COMPACTOR_METADATA_CACHE_TIMEOUT("metastore.compactor.metadata.cache.timeout",
+ "hive.metastore.compactor.metadata.cache.timeout", 60, TimeUnit.SECONDS,
Review comment:
do we still need a timeout config as we currently programmatically evict the cache after each loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778021118
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -184,7 +184,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
+ Table t = cacheAndResolveTable(ci);
Review comment:
what do you think if we put `cache` at the end and keep the primary goal at the beginning?
````
resolveTableAndCache
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778664718
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
+ if (CacheAwareCompactor.class.isAssignableFrom(obj.getClass())) {
+ ((CacheAwareCompactor) obj).setCache(cache);
+ }
+ }
+
+ void setCache(CompactorMetadataCache cache);
+
+ class CompactorMetadataCache {
+
+ private final Cache<TableCacheKey, Table> tableCache;
+ private final Cache<PartitionCacheKey, Partition> partitionCache;
+
+ @VisibleForTesting
+ public CompactorMetadataCache(long timeout, TimeUnit unit) {
+ this.tableCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ this.partitionCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ }
+
+ public static CompactorMetadataCache createIfEnabled(Configuration conf) {
+ long timeout = MetastoreConf.getTimeVar(conf,
+ MetastoreConf.ConfVars.COMPACTOR_METADATA_CACHE_TIMEOUT, TimeUnit.SECONDS);
+ if (timeout == 0) {
+ return null;
+ }
+ return new CompactorMetadataCache(timeout, TimeUnit.SECONDS);
+ }
+
+ public Table resolveTable(CompactionInfo ci, Callable<Table> loader) {
+ try {
+ TableCacheKey key = new TableCacheKey(ci);
Review comment:
Agree, fixed.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
+ if (CacheAwareCompactor.class.isAssignableFrom(obj.getClass())) {
+ ((CacheAwareCompactor) obj).setCache(cache);
+ }
+ }
+
+ void setCache(CompactorMetadataCache cache);
+
+ class CompactorMetadataCache {
+
+ private final Cache<TableCacheKey, Table> tableCache;
+ private final Cache<PartitionCacheKey, Partition> partitionCache;
+
+ @VisibleForTesting
+ public CompactorMetadataCache(long timeout, TimeUnit unit) {
+ this.tableCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ this.partitionCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ }
+
+ public static CompactorMetadataCache createIfEnabled(Configuration conf) {
+ long timeout = MetastoreConf.getTimeVar(conf,
+ MetastoreConf.ConfVars.COMPACTOR_METADATA_CACHE_TIMEOUT, TimeUnit.SECONDS);
+ if (timeout == 0) {
+ return null;
+ }
+ return new CompactorMetadataCache(timeout, TimeUnit.SECONDS);
+ }
+
+ public Table resolveTable(CompactionInfo ci, Callable<Table> loader) {
+ try {
+ TableCacheKey key = new TableCacheKey(ci);
+ return tableCache.get(key, loader);
+ } catch (ExecutionException e) {
+ throw new UncheckedExecutionException(e);
+ }
+ }
+
+ public Partition resolvePartition(CompactionInfo ci, Callable<Partition> loader) {
+ try {
+ if (ci.partName == null) {
+ return null;
+ }
+ PartitionCacheKey key = new PartitionCacheKey(ci);
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778663909
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -201,7 +201,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
Partition p = null;
if (ci.partName != null) {
- p = resolvePartition(ci);
+ p = cacheAndResolvePartition(ci);
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787524083
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -142,6 +145,7 @@ public void run() {
final ShowCompactResponse currentCompactions = txnHandler.showCompact(new ShowCompactRequest());
+ metadataCache.ifPresent(c -> c.invalidateAll());
Review comment:
Could you please add a comment on why do we need to invalidate the cache on every iteration and that's needs to be revisited later?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -142,6 +145,7 @@ public void run() {
final ShowCompactResponse currentCompactions = txnHandler.showCompact(new ShowCompactRequest());
+ metadataCache.ifPresent(c -> c.invalidateAll());
Review comment:
Could you please add a comment on why do we need to invalidate the cache on every iteration and that needs to be revisited later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778667658
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
##########
@@ -101,6 +103,25 @@ public void init(AtomicBoolean stop) throws Exception {
}
}
+ @Override
+ public void setCache(CompactorMetadataCache metadataCache) {
+ this.metadataCache = metadataCache;
+ }
+
+ protected Table cacheAndResolveTable(CompactionInfo ci) throws MetaException {
+ if (metadataCache != null) {
Review comment:
Well I was thinking this previously and because there was no default value for the cache, I decided to allow it as null if absent. But you are right in a sense, that using optional explicitly says that the cache can be null.
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r786957161
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -589,6 +589,9 @@ public static ConfVars getMetaConf(String name) {
"in which a warning is being raised if multiple worker version are detected.\n" +
"The setting has no effect if the metastore.metrics.enabled is disabled \n" +
"or the metastore.acidmetrics.thread.on is turned off."),
+ COMPACTOR_METADATA_CACHE_TIMEOUT("metastore.compactor.metadata.cache.timeout",
+ "hive.metastore.compactor.metadata.cache.timeout", 60, TimeUnit.SECONDS,
Review comment:
The idea was to keep it at a low value so that we don't need extra eviction logic. Now that the cache works for a single cycle of the initiator, the sole purpose of this config is to be able to control the memory consumption of the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits merged pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits merged pull request #2906:
URL: https://github.com/apache/hive/pull/2906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787744784
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -572,6 +572,9 @@ public static ConfVars getMetaConf(String name) {
"Set this to true on one instance of the Thrift metastore service as part of turning\n" +
"on Hive transactions. For a complete list of parameters required for turning on\n" +
"transactions, see hive.txn.manager."),
+ COMPACTOR_INITIATOR_TABLE_CACHE_ON("metastore.compactor.initiator.table.cache.on",
Review comment:
Could you please change the config value to:
````
COMPACTOR_INITIATOR_TABLE_CACHE_ON(
“metastore.compactor.initiator.tablecache.on”, “hive.compactor.initiator.tablecache.on”, true ..
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r782804631
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -164,7 +164,7 @@ public void run() {
for (CompactionInfo ci : potentials) {
try {
- Table t = resolveTable(ci);
+ Table t = resolveTableAndCache(ci);
Review comment:
The idea was, that this way we can have a shared cache between the initiator and the cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778021444
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -201,7 +201,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
Partition p = null;
if (ci.partName != null) {
- p = resolvePartition(ci);
+ p = cacheAndResolvePartition(ci);
Review comment:
same as above
````
resolvePartitionAndCache
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r782824084
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##########
@@ -164,7 +164,7 @@ public void run() {
for (CompactionInfo ci : potentials) {
try {
- Table t = resolveTable(ci);
+ Table t = resolveTableAndCache(ci);
Review comment:
1. In large clusters, there can be 10000s of tables and having additional cache could add up to mem pressure.
2. As mentioned earlier, issue was observed more in isEligibleForCompaction. It would be helpful if the patch can address it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r786519996
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -589,6 +589,9 @@ public static ConfVars getMetaConf(String name) {
"in which a warning is being raised if multiple worker version are detected.\n" +
"The setting has no effect if the metastore.metrics.enabled is disabled \n" +
"or the metastore.acidmetrics.thread.on is turned off."),
+ COMPACTOR_METADATA_CACHE_TIMEOUT("metastore.compactor.metadata.cache.timeout",
+ "hive.metastore.compactor.metadata.cache.timeout", 60, TimeUnit.SECONDS,
+ "Number of seconds the table/partition metadata are cached by the compactor. Setting it to zero disables the feature."),
Review comment:
This is only table level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787508629
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
Review comment:
is it used somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778663702
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -184,7 +184,7 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB, boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
+ Table t = cacheAndResolveTable(ci);
Review comment:
Sounds reasonable, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r778029687
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CacheAwareCompactor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+public interface CacheAwareCompactor {
+
+ static void trySetCache(Object obj, CompactorMetadataCache cache) {
+ if (CacheAwareCompactor.class.isAssignableFrom(obj.getClass())) {
+ ((CacheAwareCompactor) obj).setCache(cache);
+ }
+ }
+
+ void setCache(CompactorMetadataCache cache);
+
+ class CompactorMetadataCache {
+
+ private final Cache<TableCacheKey, Table> tableCache;
+ private final Cache<PartitionCacheKey, Partition> partitionCache;
+
+ @VisibleForTesting
+ public CompactorMetadataCache(long timeout, TimeUnit unit) {
+ this.tableCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ this.partitionCache = CacheBuilder.newBuilder().expireAfterAccess(timeout, unit).softValues().build();
+ }
+
+ public static CompactorMetadataCache createIfEnabled(Configuration conf) {
+ long timeout = MetastoreConf.getTimeVar(conf,
+ MetastoreConf.ConfVars.COMPACTOR_METADATA_CACHE_TIMEOUT, TimeUnit.SECONDS);
+ if (timeout == 0) {
+ return null;
+ }
+ return new CompactorMetadataCache(timeout, TimeUnit.SECONDS);
+ }
+
+ public Table resolveTable(CompactionInfo ci, Callable<Table> loader) {
+ try {
+ TableCacheKey key = new TableCacheKey(ci);
Review comment:
I think you could simplify here to String:
````
ci.getFullTableName()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on pull request #2906:
URL: https://github.com/apache/hive/pull/2906#issuecomment-1010144652
@asinkovits, please remove partition cache from here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r786521394
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -589,6 +589,9 @@ public static ConfVars getMetaConf(String name) {
"in which a warning is being raised if multiple worker version are detected.\n" +
"The setting has no effect if the metastore.metrics.enabled is disabled \n" +
"or the metastore.acidmetrics.thread.on is turned off."),
+ COMPACTOR_METADATA_CACHE_TIMEOUT("metastore.compactor.metadata.cache.timeout",
+ "hive.metastore.compactor.metadata.cache.timeout", 60, TimeUnit.SECONDS,
Review comment:
Might be too small. If we have a query running more than a min, then we will not cross-cache between Initiator and Cleaner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asinkovits commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
asinkovits commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r786948668
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -589,6 +589,9 @@ public static ConfVars getMetaConf(String name) {
"in which a warning is being raised if multiple worker version are detected.\n" +
"The setting has no effect if the metastore.metrics.enabled is disabled \n" +
"or the metastore.acidmetrics.thread.on is turned off."),
+ COMPACTOR_METADATA_CACHE_TIMEOUT("metastore.compactor.metadata.cache.timeout",
+ "hive.metastore.compactor.metadata.cache.timeout", 60, TimeUnit.SECONDS,
+ "Number of seconds the table/partition metadata are cached by the compactor. Setting it to zero disables the feature."),
Review comment:
Missed this, when removed partition caching. Fixed, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2906: HIVE-24805: Compactor: Initiator shouldn't fetch table details again …
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2906:
URL: https://github.com/apache/hive/pull/2906#discussion_r787818367
##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -572,6 +572,9 @@ public static ConfVars getMetaConf(String name) {
"Set this to true on one instance of the Thrift metastore service as part of turning\n" +
"on Hive transactions. For a complete list of parameters required for turning on\n" +
"transactions, see hive.txn.manager."),
+ COMPACTOR_INITIATOR_TABLE_CACHE_ON("metastore.compactor.initiator.table.cache.on",
Review comment:
I think it should be "on" by default, as it's a known limitation. Currently, it's an isolated, simple cache with soft-ref, so shouldn't be risky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org