You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/12/01 05:55:27 UTC

[GitHub] [hive] kokila-19 opened a new pull request, #3822: HIVE-26793: Create a new configuration to override 'no compaction' fo…

kokila-19 opened a new pull request, #3822:
URL: https://github.com/apache/hive/pull/3822

   ### What changes were proposed in this pull request?
   no_auto_compaction configuration should be allowed to overridden on a database level: adding no_auto_compaction=false should override the table level setting forcing the initiator to schedule compaction for all tables.
   
   ### Why are the changes needed?
   Currently a simple user can create a table with no_auto_compaction=true table property and create an aborted write transaction writing to this table. This way a malicious user can prevent cleaning up data for the aborted transaction, creating performance degradation.
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   Unit tests 
   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator2.java
   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1085219185


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -98,29 +100,39 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   @Override
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-      Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (isNoAutoCompactSet(allocWriteIdEvent)) {
+        int numOfWritesToDisabledCompactionTable = Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
+        if (numOfWritesToDisabledCompactionTable >= MetastoreConf.getIntVar(getConf(), MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
+          LOGGER.warn(
+              "There has been a write to table " + allocWriteIdEvent.getDbName() + "." + allocWriteIdEvent.getTableName() + " where auto-compaction is disabled \"no_auto_compact\"=\"true\".");
         }
       }
     }
   }
 
-  private Table getTable(AllocWriteIdEvent allocWriteIdEvent) throws MetaException {
+  private Boolean isNoAutoCompactSet(AllocWriteIdEvent allocWriteIdEvent) throws MetaException {
     String catalog = MetaStoreUtils.getDefaultCatalog(getConf());
     String dbName = allocWriteIdEvent.getDbName();
     String tableName = allocWriteIdEvent.getTableName();
-    if (allocWriteIdEvent.getIHMSHandler() != null) {
-      return allocWriteIdEvent.getIHMSHandler().getMS().getTable(catalog, dbName, tableName);
-    } else {
-      return HMSHandler.getMSForConf(getConf()).getTable(catalog, dbName, tableName);
+    Map<String, String> dbParameters;
+    Table table;
+    try {

Review Comment:
   try scope is unclear, maybe:
   ````
     private Boolean isNoAutoCompactSet(AllocWriteIdEvent allocWriteIdEvent) throws MetaException {
       String catalog = MetaStoreUtils.getDefaultCatalog(getConf());
       String dbName = allocWriteIdEvent.getDbName();
       String tableName = allocWriteIdEvent.getTableName();
       
       RawStore rawStore;
       if (allocWriteIdEvent.getIHMSHandler() != null) {
         rawStore = allocWriteIdEvent.getIHMSHandler().getMS();
       } else {
         rawStore = HMSHandler.getMSForConf(getConf());
       }
       Map<String, String> dbParameters;
       try {
         dbParameters = rawStore.getDatabase(catalog, dbName).getParameters();
       } catch (NoSuchObjectException e) {
         LOGGER.error("Unable to find database " + dbName + ", " + e.getMessage());
         throw new MetaException(String.valueOf(e));
       }
       Table table = rawStore.getTable(catalog, dbName, tableName);
       // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
       if (table != null) {
         return MetaStoreUtils.isNoAutoCompactSet(dbParameters, table.getParameters());
       }
       return false;
     }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3822:
URL: https://github.com/apache/hive/pull/3822#issuecomment-1398153482

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3822)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [14 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3822:
URL: https://github.com/apache/hive/pull/3822#issuecomment-1373747271

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3822)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1063224968


##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,26 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> tblParameters) {
+    String dbNoAutoCompact = getNoAutoCompact(dbParameters);
+    if (dbNoAutoCompact == null) {
+      LOG.debug("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+      String noAutoCompact = getNoAutoCompact(tblParameters);
+      return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+    }
+    LOG.debug("Using database configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+    return dbNoAutoCompact.equalsIgnoreCase("true");
+  }
+
+  // Get no_auto_compact property by checking in both lower and upper cases
+  private static String getNoAutoCompact(Map<String, String> parameters) {

Review Comment:
   Actually no ., when db level NO_AUTO_COMPACT is not set which is null(so i need null to check), we consider the table level ., this method is common to get config for both db and table level ., 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037914520


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {

Review Comment:
   But I see that there is a file in hive-it-util -> TestCleanerWithReplication extending hive-exec -> CompactorTest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1066729412


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,8 +613,9 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();

Review Comment:
   `isEligibleForCompaction` accepts the skipDBs SET, we should populate and reuse it when checking the next candidates 
   ````
   if (isNoAutoCompactSet(dbParams, t.getParameters())) {
       skipDbs.add(ci.dbname)
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3822:
URL: https://github.com/apache/hive/pull/3822#issuecomment-1338988016

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3822)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060426315


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,21 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (table != null) {

Review Comment:
   why is this change? it looks unreleated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060424116


##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,26 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> tblParameters) {
+    String dbNoAutoCompact = getNoAutoCompact(dbParameters);
+    if (dbNoAutoCompact == null) {
+      LOG.debug("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+      String noAutoCompact = getNoAutoCompact(tblParameters);
+      return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+    }
+    LOG.debug("Using database configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+    return dbNoAutoCompact.equalsIgnoreCase("true");

Review Comment:
   why not 
   ````
   return Boolean.parseBoolean(dbNoAutoCompact)
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1072005883


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,8 +613,9 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();

Review Comment:
   ````
   if (replIsCompactionDisabledForTable(t)) {
           skipTables.add(ci.getFullTableName());
           return false;
   }
   ````
   Would be better if we could refactor the method and use a cache of skipTables/skipDBs instead of doing the same evaluation for every Table in skipped Db / every Partition of skipped Table  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060425386


##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,26 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> tblParameters) {
+    String dbNoAutoCompact = getNoAutoCompact(dbParameters);
+    if (dbNoAutoCompact == null) {
+      LOG.debug("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+      String noAutoCompact = getNoAutoCompact(tblParameters);
+      return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+    }
+    LOG.debug("Using database configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+    return dbNoAutoCompact.equalsIgnoreCase("true");
+  }
+
+  // Get no_auto_compact property by checking in both lower and upper cases
+  private static String getNoAutoCompact(Map<String, String> parameters) {

Review Comment:
   can we return boolean right away?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] veghlaci05 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
veghlaci05 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1036973940


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,22 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
+      Database db;
+      try {
+        db = HMSHandler.getMSForConf(getConf()).getDatabase(getDefaultCatalog(getConf()), table.getDbName());

Review Comment:
   You should use table.getCatName() if present, and please make this change in org.apache.hadoop.hive.ql.txn.compactor.Initiator#resolveDatabase too. I just realized this could be an issue if the DB is not in the default catalog.



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorOnTezTest.java:
##########
@@ -259,6 +259,16 @@ void createDb(String dbName, String poolName) throws Exception {
       executeStatementOnDriver("create database " + dbName + " WITH DBPROPERTIES('hive.compactor.worker.pool'='" + poolName + "')", driver);
     }
 
+    void createDb(String dbName, boolean noAutoCompact) throws Exception {
+      executeStatementOnDriver("drop database if exists " + dbName + " cascade", driver);
+      executeStatementOnDriver("create database " + dbName + " WITH DBPROPERTIES('no_auto_compaction'='" + noAutoCompact + "')", driver);
+    }
+
+    void createDbWithLowerCaseProps(String dbName, boolean noAutoCompact) throws Exception {

Review Comment:
   you could add a third param like boolean upperCaseProperty to createDb and remove this function



##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,24 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
-    if (noAutoCompact == null) {
-      noAutoCompact =
-              parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase());
-    }
-    return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> parameters) {
+    String dbNoAutoCompact = dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT) == null ?
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase()) :
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+    if (dbNoAutoCompact == null) {
+      LOG.info("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");

Review Comment:
   This will exeecuted frequently, should be log.DEBUG to avoid spamming the logs



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator2.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.*;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TestInitiator2 extends CompactorOnTezTest {
+  @Test
+  public void dbNoAutoCompactSetTrueUpperCase() throws Exception {

Review Comment:
   The only real difference with the other method is the `testDataProvider.createDbWithLowerCaseProps` call and the expectation whether initiator created a request or not. Consider creating a private method which takes the following arguments: boolean noAutoCompaction, boolean upperCaseProp. This should be called from the test methods instead of duplicating the code.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,22 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
+      Database db;
+      try {
+        db = HMSHandler.getMSForConf(getConf()).getDatabase(getDefaultCatalog(getConf()), table.getDbName());
+      } catch (NoSuchObjectException e) {
+        LOGGER.error("Unable to find database " + table.getDbName() + ", " + e.getMessage());
+        throw new MetaException(String.valueOf(e));
+      }
       // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
+      if (table != null && MetaStoreUtils.isNoAutoCompactSet(db.getParameters(), table.getParameters())) {

Review Comment:
   At this point table is never null as you are already called table.getDbName() a few lines above. This check should be moved up, if table is null an NPE will be thrown even in the catch block.



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {

Review Comment:
   You should not mix up the test classes. The tests should be either in `hive-exec` module and extend `CompactorTest` , or in `hive-it-unit` and extend `CompactorOnTezTest` or `TestCompactorBase`. Altough the preferred place of writing new tests is `hive-it-unit`, if it's easier to write the tests in `hive-exec`, feel free to do it. But you should choose either of the test infrastructure and build your tests only upon that one.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,22 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
+      Database db;
+      try {
+        db = HMSHandler.getMSForConf(getConf()).getDatabase(getDefaultCatalog(getConf()), table.getDbName());
+      } catch (NoSuchObjectException e) {
+        LOGGER.error("Unable to find database " + table.getDbName() + ", " + e.getMessage());
+        throw new MetaException(String.valueOf(e));
+      }
       // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
+      if (table != null && MetaStoreUtils.isNoAutoCompactSet(db.getParameters(), table.getParameters())) {
         int noAutoCompactSet =

Review Comment:
   Please rename this variable to sth like `writeThreshold`, as its current name is really confusing. It holds the threshold, not a boolean value.



##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,24 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
-    if (noAutoCompact == null) {
-      noAutoCompact =
-              parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase());
-    }
-    return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> parameters) {
+    String dbNoAutoCompact = dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT) == null ?
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase()) :
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+    if (dbNoAutoCompact == null) {
+      LOG.info("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+      String noAutoCompact =
+          parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+      if (noAutoCompact == null) {
+        noAutoCompact =
+            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase());
+      }
+      return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+    }
+    LOG.info("Using database configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");

Review Comment:
   This will exeecuted frequently, should be log.DEBUG to avoid spamming the logs



##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,24 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
-    if (noAutoCompact == null) {
-      noAutoCompact =
-              parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase());
-    }
-    return noAutoCompact != null && noAutoCompact.equalsIgnoreCase("true");
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> parameters) {
+    String dbNoAutoCompact = dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT) == null ?
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase()) :
+        dbParameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+    if (dbNoAutoCompact == null) {
+      LOG.info("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");
+      String noAutoCompact =
+          parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+      if (noAutoCompact == null) {
+        noAutoCompact =
+            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT.toUpperCase());
+      }

Review Comment:
   Either use the ternary operator here or use if above for DB parameter. Since you are doing the same logic with only a different parameter you can even create a small private helper method for it.



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactorBase.java:
##########
@@ -180,8 +180,13 @@ private void createTestDataFile(String filename, String[] lines) throws IOExcept
         writer.close();
       }
     }
-
   }
 
-
+  // Sub Class of CompactorTest which is the super class of all compactor test modules

Review Comment:
   Please remove this class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ merged pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #3822:
URL: https://github.com/apache/hive/pull/3822


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037821073


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {

Review Comment:
   But I see that test class TestCleanerWithReplication in hive-it-unit extends CompactorTest from hive-exec



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060434675


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,21 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (table != null) {
+        Database db;
+        try {
+          db = HMSHandler.getMSForConf(getConf()).getDatabase(table.getCatName(), table.getDbName());

Review Comment:
   should be done similarly to getTable to reuse the HMSHandler if available. Maybe even reactor getTable to isNoAutoCompactSet as it's the only place this method is used



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060426315


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,21 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (table != null) {

Review Comment:
   why is this change? it looks unreleated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3822:
URL: https://github.com/apache/hive/pull/3822#issuecomment-1333761117

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3822)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1066729412


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,8 +613,9 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();

Review Comment:
   `isEligibleForCompaction` accepts the skipDBs SET, we should populate and reuse it when checking the next candidates 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1072005883


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,8 +613,9 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();

Review Comment:
   ````
   if (replIsCompactionDisabledForTable(t)) {
           skipTables.add(ci.getFullTableName());
           return false;
   }
   ````
   Would be better if we could refactor the method and use a cache of skipTables/skipDBs instead of doing the same evaluation (isNoAutoCompactSet) for every Table in skipped Db / every Partition of skipped Table  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3822:
URL: https://github.com/apache/hive/pull/3822#issuecomment-1374034809

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3822)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3822&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3822&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3822&metric=duplicated_lines_density&view=list) No Duplication information
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] veghlaci05 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
veghlaci05 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060097452


##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1143,15 +1143,26 @@ public static TableName getTableNameFor(Table table) {
   /**
    * Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved
    * to no_auto_compact, we need to check it in both cases.
+   * Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.
    */
-  public static boolean isNoAutoCompactSet(Map<String, String> parameters) {
-    String noAutoCompact =
-            parameters.get(hive_metastoreConstants.TABLE_NO_AUTO_COMPACT);
+  public static boolean isNoAutoCompactSet(Map<String, String> dbParameters, Map<String, String> tblParameters) {
+    String dbNoAutoCompact = getNoAutoCompact(dbParameters);
+    if (dbNoAutoCompact == null) {
+      LOG.debug("Using table configuration '" + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT + "' for compaction");

Review Comment:
   Since this config now can be both table or DB level, the constant name should be changed to simply NO_AUTO_COMPACT.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1060434675


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -99,15 +103,21 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
       Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (table != null) {
+        Database db;
+        try {
+          db = HMSHandler.getMSForConf(getConf()).getDatabase(table.getCatName(), table.getDbName());

Review Comment:
   should be done similarly to getTable to reuse the HMSHandler if available



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037914520


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {

Review Comment:
   But I see that there is a file in hive-unit-> TestCleanerWithReplication extending hive-exec -> CompactorTest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1037821073


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics2.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.hive.metastore.HMSMetricsListener;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.metrics.Metrics;
+import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TestCompactionMetrics2 extends CompactorOnTezTest {

Review Comment:
   But I see that test class TestCleanerWithReplication in hive-it-unit extends CompactorTest from hive-exec



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1085205621


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -98,29 +100,39 @@ public void onAddPartition(AddPartitionEvent partitionEvent) throws MetaExceptio
   @Override
   public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection dbConn, SQLGenerator sqlGenerator) throws MetaException {
     if (MetastoreConf.getBoolVar(getConf(), MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-      Table table = getTable(allocWriteIdEvent);
-      // In the case of CTAS, the table is created after write ids are allocated, so we'll skip metrics collection.
-      if (table != null && MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
-        int noAutoCompactSet =
-            Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
-        if (noAutoCompactSet >=
-            MetastoreConf.getIntVar(getConf(),
-                MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD)) {
-          LOGGER.warn("There has been a write to table " + table.getDbName() + "." + table.getTableName() +
-              " where auto-compaction is disabled (tblproperties (\"no_auto_compact\"=\"true\")).");
+      if (isNoAutoCompactSet(allocWriteIdEvent)) {

Review Comment:
   please get rid of nesting and combine with above "IF"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1085202881


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,13 +614,22 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
-            "=true so we will not compact it.");
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();
+      if (isNoAutoCompactSet(dbParams, t.getParameters())) {
+        if (MetaStoreUtils.IS_DB_LEVEL_CONFIG) {

Review Comment:
   idea is good , but not the impl. There could be multiple dbs, sharing through static filed in multiple threads is not a good idea. What if you simple use
   ````
   if (getNoAutoCompact(dbParameters)){
   skipDBs.add(ci.dbname);
   } else {
   skipTables.add(ci.getFullTableName());
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] kokila-19 commented on a diff in pull request #3822: HIVE-26793: Create a new configuration to override 'no compaction' for tables

Posted by GitBox <gi...@apache.org>.
kokila-19 commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1067346055


##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
##########
@@ -613,8 +613,9 @@ private boolean isEligibleForCompaction(CompactionInfo ci,
         return false;
       }
 
-      if (isNoAutoCompactSet(t.getParameters())) {
-        LOG.info("Table " + tableName(t) + " marked " + hive_metastoreConstants.TABLE_NO_AUTO_COMPACT +
+      Map<String, String> dbParams = computeIfAbsent(ci.dbname, () -> resolveDatabase(ci)).getParameters();

Review Comment:
   isNoAutoCompactSet considers both db level and table level configs and so if we skip the db when table level config is set, then it will affect other tables. Also isEligibleForCompaction accepts skipTables SET and skipDBs , skipTables both are not used inside  isNoAutoCompactSet., I am not sure if there is any particular reason for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org