You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/04/15 16:04:50 UTC

[GitHub] [hive] klcopp opened a new pull request #2184: HIVE-25022: Metric about incomplete compactions

klcopp opened a new pull request #2184:
URL: https://github.com/apache/hive/pull/2184


   ### What changes were proposed in this pull request?
   See https://issues.apache.org/jira/browse/HIVE-25022
   
   ### How was this patch tested?
   Unit tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] klcopp closed pull request #2184: HIVE-25022: Metric about incomplete compactions

Posted by GitBox <gi...@apache.org>.
klcopp closed pull request #2184:
URL: https://github.com/apache/hive/pull/2184


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] klcopp commented on pull request #2184: HIVE-25022: Metric about incomplete compactions

Posted by GitBox <gi...@apache.org>.
klcopp commented on pull request #2184:
URL: https://github.com/apache/hive/pull/2184#issuecomment-823166632


   > LGTM, however, I don't see real benefit from this metric.
   > 
   > ```
   > major succeeded
   > major failed
   > minor failed
   > minor succeeded
   > ```
   > 
   > This would be reported as incomplete compaction. What action do you expect from the end-users in this case?
   
   In this case end users should re-run major compaction. Major compaction should have run (at 2. major failed) but hasn't since that failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on pull request #2184: HIVE-25022: Metric about incomplete compactions

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on pull request #2184:
URL: https://github.com/apache/hive/pull/2184#issuecomment-823185836


   > > LGTM, however, I don't see real benefit from this metric.
   > > ```
   > > major succeeded
   > > major failed
   > > minor failed
   > > minor succeeded
   > > ```
   > > 
   > > 
   > > This would be reported as incomplete compaction. What action do you expect from the end-users in this case?
   > 
   > In this case end users should re-run major compaction. Major compaction should have run (at 2. major failed) but hasn't since that failure.
   
   How would they know tables/partitions to re-run major compaction? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] deniskuzZ commented on a change in pull request #2184: HIVE-25022: Metric about incomplete compactions

Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2184:
URL: https://github.com/apache/hive/pull/2184#discussion_r616544766



##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java
##########
@@ -108,19 +112,56 @@ private void updateDBMetrics() throws MetaException {
   @VisibleForTesting
   public static void updateMetricsFromShowCompact(ShowCompactResponse showCompactResponse) {
     Map<String, ShowCompactResponseElement> lastElements = new HashMap<>();
+    Map<String, Long> lastUnsuccessfulMajor = new HashMap<>();
+    Map<String, Long> lastUnsuccessfulMinor = new HashMap<>();
     long oldestEnqueueTime = Long.MAX_VALUE;
 
-    // Get the last compaction for each db/table/partition
-    for(ShowCompactResponseElement element : showCompactResponse.getCompacts()) {
+    // sort compactions by ID. This is not done in TxnHandler.
+    List<ShowCompactResponseElement> compactions = showCompactResponse.getCompacts().stream()
+            .sorted((o1, o2) -> (int) (o1.getId() - o2.getId())).collect(Collectors.toList());
+    for (ShowCompactResponseElement element : compactions) {
       String key = element.getDbname() + "/" + element.getTablename() +
           (element.getPartitionname() != null ? "/" + element.getPartitionname() : "");
+
+      // Get the last compaction for each db/table/partition
       // If new key, add the element, if there is an existing one, change to the element if the element.id is greater than old.id
       lastElements.compute(key, (k, old) -> (old == null) ? element : (element.getId() > old.getId() ? element : old));
       if (TxnStore.INITIATED_RESPONSE.equals(element.getState()) && oldestEnqueueTime > element.getEnqueueTime()) {
         oldestEnqueueTime = element.getEnqueueTime();
       }
+
+      // Count incomplete compactions
+      CompactionType type = element.getType();
+      lastUnsuccessfulMajor.compute(key, (k, old) -> {
+        // Add newest unsuccessful compaction to the map
+        if (wasUnsuccessful(element) && type == MAJOR) {

Review comment:
       Could it be simplified ?
   ```
   if (type == MAJOR){
       if (wasUnsuccessful(element)) return element.getId();
       if (wasSuccessful(element)) return null;
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org