[GitHub] [hive] klcopp commented on a change in pull request #2817: HIVE-25740: Avoid compaction heartbeater error logging in race conditions

[GitBox <gi...@apache.org>]

klcopp commented on a change in pull request #2817:
URL: https://github.com/apache/hive/pull/2817#discussion_r759446538



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -233,39 +234,36 @@ void gatherStats() {
     private final CompactionTxn compactionTxn;
     private final String tableName;
     private final HiveConf conf;
-    private final long txnTimeout;
+    private final AtomicBoolean shouldLogError;
 
     public CompactionHeartbeater(CompactionTxn compactionTxn, String tableName, HiveConf conf) {
       this.tableName = Objects.requireNonNull(tableName);
       this.compactionTxn = Objects.requireNonNull(compactionTxn);
       this.conf = Objects.requireNonNull(conf);
-
-      this.txnTimeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+      this.shouldLogError = new AtomicBoolean(true);
 
       setDaemon(true);
       setPriority(MIN_PRIORITY);
       setName("CompactionHeartbeater-" + compactionTxn.getTxnId());
     }
 
+    public void shouldLogError(boolean shouldLogError) {
+      this.shouldLogError.set(shouldLogError);
+    }
+
     @Override
     public void run() {
       LOG.debug("Heartbeating compaction transaction id {} for table: {}", compactionTxn, tableName);
-
       IMetaStoreClient msc = null;
       try {
         // Create a metastore client for each thread since it is not thread safe
         msc = HiveMetaStoreUtils.getHiveMetastoreClient(conf);

Review comment:
       This way we are getting a HMS client for every heartbeat... maybe we only need one for each instance

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -751,11 +748,28 @@ void wasSuccessful() {
       if (status == TxnStatus.UNKNOWN) {
         return;

Review comment:
       We should probably shut down the heartbeater in this case too

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -233,39 +234,36 @@ void gatherStats() {
     private final CompactionTxn compactionTxn;
     private final String tableName;
     private final HiveConf conf;
-    private final long txnTimeout;
+    private final AtomicBoolean shouldLogError;
 
     public CompactionHeartbeater(CompactionTxn compactionTxn, String tableName, HiveConf conf) {
       this.tableName = Objects.requireNonNull(tableName);
       this.compactionTxn = Objects.requireNonNull(compactionTxn);
       this.conf = Objects.requireNonNull(conf);
-
-      this.txnTimeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+      this.shouldLogError = new AtomicBoolean(true);
 
       setDaemon(true);
       setPriority(MIN_PRIORITY);
       setName("CompactionHeartbeater-" + compactionTxn.getTxnId());
     }
 
+    public void shouldLogError(boolean shouldLogError) {
+      this.shouldLogError.set(shouldLogError);
+    }
+
     @Override
     public void run() {
       LOG.debug("Heartbeating compaction transaction id {} for table: {}", compactionTxn, tableName);

Review comment:
       kind of a nit: Is it still necessary to log twice per heartbeat ("Heartbeating compaction transaction..." and "Successfully heartbeated...") Maybe one can be removed?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org